DevSecOps tutorial: What is it, and how can it improve application security?

Dr. David Brumley, a professor at Carnegie Mellon University and CEO of ForAllSecure, explains what DevSecOps is and how companies can use it to improve security.

DevSecOps tutorial: What is it, and how can it improve application security?

If you've been in IT any length of time, you've probably heard the term DevOps--basically a mashup of software development and IT operations. What about DevSecOps? What is it, and what does it bring to the party? In this exclusive TechRepublic cyber security video, Dr. David Brumley explains what DevSecOps is and how companies can use it to improve application security.

Dr. Brumley is a professor at Carnegie Mellon University and CEO of ForAllSecure. He's also been in the application security business for over 20 years, both on the enterprise and the research side.

What is DevSecOps, and how is it different from DevOps?

Bill Detwiler: So let's get right to it. What is DevSecOps?

Dr. David Brumley: Well, in a nutshell, DevSecOps is about building a great application while empowering everyone with the mindset that security is everyone's responsibility. It takes lessons that we've learned about how to build, deploy, and run apps over 40 years of research and practice and builds in this idea that security shouldn't be a step at the end. It's not a checkbox. We've tried that. We've tried that for 40 years. 

It was expensive, error prone, it just broke and instead DevSecOps starts folding in security throughout the process. So when I think about DevSecOps, it's not just a fad. It's not just something I got to choose over a different methodology. It's really the evolution with all of these lessons learned.

SEE: Fuzzing (fuzz testing) tutorial: What it is and how can it improve application security? (TechRepublic)

Bill Detwiler: And that's the way it should be, right? It really should be an evolution.

David Brumley

Dr. David Brumley Ph.D., Professor of Electrical and Computer Engineering at Carnegie Mellon University and CEO of ForAllSecure

Credit: ForAllSecure

Dr. David Brumley: It absolutely should be. So, if we go back a little bit in history and take a bit of a historical tour to understand what DevSecOps is and what problems it fixes, it all starts back in 1976 with a paper and the IEEE conference on software engineering. That paper is the one that really ingrained this idea that security is a checkbox at the end. It's important to think about what that proposed and how we fixed it. Back in the day, we used to think about software security and the Waterfall method. That's really what that 1976 paper led to and it really had five different steps. So we have time right here.

The first thing you do is go out and gather a bunch of requirements. What is the business objective that you're trying to achieve? Then you enter a design phase. This is where you're thinking about what are the different components, how do we break it up, how do we start divvying it up into teams and so on. 

Then you get to an implementation phase. You have all your developers who are building the app and of course, because you want to double check their work, you're going to have a verification step. Really what this meant, originally, was you want to verify the implementation meets the requirements and what happens over time is security evolves. As people started throwing in security, you got to verify it's safe and then you enter in this maintenance phase.

Now, this is a waterfall and what we've learned from this is there's at least three big problems.

Bill Detwiler: At least three, ok? I figured there might be more.

Dr. David Brumley: At least three big problems. So first, it's very linear and that's not the way the world works. We learn things as we start implementing as we start deploying applications out there, so we have to build in a feedback cycle. Now we think of the development component as a cycle and we really have it in four separate steps now. The very first part of the cycle is we're going to plan. 

That's a little bit like the requirements phase. We're going to code, we're then going to build the software. This wasn't in the old phase, we found software is just getting more and more complicated. We have to integrate one development team with another before we can actually come up with the finished application product.This build phase is actually pretty important--we'll get to it more later on how it fits into DevSecOps.
Finally, we have a test phase. This was a realization that verification isn't just verifying the requirements, it's testing to see how well it works. So when we think about development, we're trying to make it a fast moving cycle where you can revisit the planning phase, you can go back through code, build, test, and then of course we're going to have various software releases. So we're going to release it.

The second thing Waterfall didn't really think about is how do you operate the software? If you ask anyone in IT, how you build the software makes a big difference in the security as well as just the speed at which you can build great ops. So DevOps added this idea of there's the operation cycle.

You're talking about IT and there's a couple of different steps here. We have a configuration step--again, it's going to operate in a circle. There's gonna be a deployment step. There's an operations phase. So after you have deployed everything, you're going to sit there and it's going to be doing what it does. Then we've added an observability component to it. You want to be able to, for example, log what's going on, understand how your users are interacting with it. For people who hate trackers, this is where you put in the tracker, right?

So this is what happened with DevOps. What DevSecOps does is it really takes this idea that security is everyone's responsibility. When you talk about DevSecOps and it says security is going to be part of this entire cycle. In a nutshell, we started with this idea of security is last and that wasn't the only problem here--there's really three. The first one was a linear process. We're going to make it a cycle.

Second one is, we have to figure out how we're going to maintain the software if it's actually deployed--that's the operation side. The security side was, it's going to be everyone's responsibility. As I said, this is going to be a tight loop where we're going to take lessons as we have deployed it as we're operating it. That's going to feed back to the planning phase. This is how we've evolved over the last 40 years. 

So DevSecOps is not a fad. It's not something that you choose over some other methodology. It's really taking all those lessons we learned and putting it into one great framework.

SEE: Zero trust security: A cheat sheet (free PDF) (TechRepublic)

How to get started with DevSecOps

Bill Detwiler: So this is a really great explanation. If a company wants to get started integrating DevSecOps into their development process, how do they do that and what are some of the tools to help companies do that effectively?

Dr. David Brumley: I think out there today there's a lot of different tools people can choose from and sometimes people get confused because, 'Well I thought this tool found all the vulnerabilities. Why do I need this other one?' And it's really about putting different tools that are appropriate for different phases. 

I want to take a step back and say DevSecOps is also a mindset. It's about the processes and the people, not just the tools you've got to invest in those first two components, as well. Tools--think of them like an enabler, so as we go around this cycle, you can start thinking about the sort of tools you'd put in. 

The first thing is when we think about the planning stage, people have started building program tracking and issue tracking systems, things like JIRA for example, where you can more quickly come up with this is the set of requirements.

It's also something your development team can look at. It's not a separate, for example, business unit team doing it. We see those coming in here, and this is also a great place to start thinking about the security architecture and the sorts of things that you want to secure. Where is your data? Where is your attack surface? And so on... And the coding step--of course when you're looking at DevSecOps, you want to have revision control and the reason that you want that is so as people are checking in code, as those changes come in, you can start adding in processes like peer review so that you know, if I wrote code, sometimes the author doesn't see the blind spots, someone else--when it's checked in--can review it and make sure it's secure, safe, and appropriately coded. The build step has gotten a lot of attention in DevSecOps.


The reason that we have this, it's gotten that detection is often where we put in hooks into the build step because this is the first place that all the code that's been written gets integrated. So there's a couple things people do here. One of the things that's helped DevSecOps is an idea of reproducible deployments. That doesn't sound anything like security, so let me explain. 

When you look at the old days, back when I was first writing software, you had development and they had come up with maybe a package that you'd go in and install on a completely different system and that this system had one setup and this one had a different setup. Well that's just asking for security problems, right? Tools like Docker become really popular because during development we can test both--a development platform as well as how it's going to be operated and make sure those are consistent and that we've put in all best practices.

The second set of things that people hook into the build system are a set of application security testing tools. In my mind, if you look at Gartner group, they have magic quadrants and so on. I think we want to look at what are the different types that get in here and what they do. At a high level, there's two different sets of tools that people get. 

There is the set of tools that find known vulnerabilities and the set of tools that find unknown vulnerabilities--these are two different sets of tools. A known vulnerability is something like I'm building and I'm using an open source component and there's been a vulnerability found. How do I make sure that I'm following and tracking that latest dependency? The big tool people end up using here is called software component analysis.

Think of it as just checking your library and making sure it's up-to-date. Kind of a funny story is Equifax got hacked and one of the reasons they were vulnerable is they're running a vulnerable version of the patchy struts. Now, Apache had realized that vulnerability and fixed it nine weeks prior to the hack. So if they'd been running software component analysis, they would have had nine weeks lead time to find that issue and fix it.

Bill Detwiler: But they just hadn't done that yet.

Dr. David Brumley: They just hadn't done it and that's one of the lessons is: We need to have these tools automated as part of this cycle. Now, when you're talking about unknown vulnerabilities, these are things that someone hasn't already figured out. Maybe it's code that you just wrote or as part of your own code, not some third-party component and really you can divide this world again into two.

The old school approach to finding security vulnerabilities is a static analysis for SaaS-based solutions. Think of these as like a grammar check for source code. They're looking for insecure patterns. Now, in the old development world, these got extremely popular. What they produce is a report of all the different places. 

There may be a problem, but they also have something called false positives. A false positive is when you have safe code, but the tool flags it, so static analysis is good. It can find a lot of different defects, but when you move into this world, you're going to have to staff someone looking through those static analysis reports, something to think about.

Bill Detwiler: It might slow the process down as well.

Dr. David Brumley: It might slow the process down. It's certainly something you're going to have to staff. Now, there's advantages to static analysis and more mature technology. It also supports more programming languages. On the other side of the fence, we have dynamic analysis and so these are a set of tools that actually check the code as it executes and 10 years ago they weren't very sophisticated. It was preprogrammed attacks, but there's been a little bit of a revolution here and one of the big ones has been the introduction of fuzzing

What fuzzing does is it runs your program and tries to attack it a little bit like an attacker to go and find those vulnerabilities. Now, dynamic analysis, every time it says this is a problem, it can prove it. Zero false positives. When you're looking at what tools to implement, you really have a choice. Do you want static analysis or dynamic analysis for unknown vulnerabilities? The trade off you really have to think about if you're an executive or developer: Am I going to staff someone to go look at those reports or do I want something automatic?


If someone asked me, what do you recommend? Well, if you're not doing something like software component analysis, you're really not doing enough, right? You should make sure all your dependencies are up-to-date. Equifax has taught us that lesson; don't be burned like Equifax. If you're going to choose a tool for the unknown, my rule of thumb is, if there is a fuzzer out there that well-supports your language, this is going to speed up your development and increase your security a lot and this is what you should go for. 

If you're looking at an application that doesn't have a well-supported fuzzer, go with static analysis. I can tell you when you look at people like Google, they almost exclusively do things like software component analysis and fuzzers, and this has been a change as we moved to DevSecOps. Let me tell you about the sort of technologies on the Ops side as well.

So when you look at the Ops side of things, we have the steps of configure, deploy, operations as observed. If you look at various DevSecOps diagrams, they're all gonna have this eight on its side or infinity shape. They're going to look like this and they're all going to have essentially the same sort of characteristics that may differ. 

But, when you look at it, one of the key things is when you configure the app, you want to have it automated and you want to take care of things of, again, making sure you have a reproducible environment so that, God forbid, you're not using an old library here in production while your developers have already updated. At the deployment step, you have a lot of tools that deal with things like secret management. And what I mean by that is if you're going to deploy a web app, you're going to be toiling a web server and that has a cryptographic key.

So you want to come up with mechanisms that automatically deploy your software. Things like Kubernetes are getting extremely popular because of this, because they help manage some of those security functions. Like, 'Hey, how do I go about managing secrets?'

On the operations side, we see a lot of traditional IT tools and they're completely appropriate and DevSecOps. So that's a great thing. It's not like you have to go buy a whole new tool though. You see things like intrusion detection systems. Now, these operate at the network layer, they're really important for monitoring your network and seeing if it's under attack.

We've also seen the rise of something called runtime application self-protection (RASP) and what RASP is doing is making sure that the application layer, that if there's an attack, it's detecting it and preventing that. These are actually complementary technologies--intrusion detection and RASP--because there are two different layers. In fact, we can throw into the bucket something called web application firewall which is at yet another layer. So there's a suite of tools here that you can have to monitor for security.

On the observation side, you get into things like log management. Now, God forbid you get compromised. One of the things you're going to want to do is quickly react, figure out what's compromised, so that you can take action. Logging infrastructures are a big deal here and you may not think of these as security. You may be thinking about these for 'how do I see what pages people are visiting?' But when an incident happens, you'll be very thankful you have them.

Also see

By Bill Detwiler

Bill Detwiler is Editor in Chief of TechRepublic and the host of Cracking Open, CNET and TechRepublic's popular online show. Prior to joining TechRepublic in 2000, Bill was an IT manager, database administrator, and desktop support specialist in the ...