From Foundation to Maturity: How to Build a solid AppSec Program

[1:24]  Welcome everyone. Before we get started, I want to make a quick announcement regarding privacy and data protection. As this webinar may involve the sharing of sensitive information, we kindly ask all participants to respect the confidentiality of information presented. Please do not record, distribute, or utilize any part of this webinar without explicit permission from the organizers and presenters, that is us. So welcome everyone to our webinar today, “From Foundation to Maturity: How to Build a Solid AppSec Program.” My name is Boaz Barzel and I’m the field CTO at OX Security. We are fortunate to have today Dustin Lehr, a co-founder at Katilyst, and David Kosorok, director of information security programs at Toast. They really bring diverse perspectives around building, equipping, and scaling programs, really from the first steps to full maturity, with practical real-world examples and strategies. Whatever you’re hearing today from Dustin and David are their own opinions and do not represent the opinions of their employers or their companies. The webinar today is question friendly. What does it mean? It means that we are waiting for your questions. We want to understand what interests you, what motivates you, what are the items that you would like to get answers for. Since we are recording this session, it will also be shared afterwards, and all the key takeaways and insights that David and Dustin share with us will also be shared with you later on. So let’s start the discussion and face the topic of AppSec that’s not just about vulnerabilities. It’s actually about building different types of relationships between engineering, security, management, operations, and sometimes even finance. And sometimes the journey is a little bit hard. It curves, and to be honest it definitely has its obstacles. It’s not a smooth one. The agenda today will cover three main aspects. The first is how to approach an AppSec program, a project, and the strategy before starting to use the tools. The second, the must-have tools in your AppSec tool belt, what matters and why. And the third, measuring success in AppSec, from metrics to meaningful impact. So just remember, we are going to take a lot of questions from you, so please share those with us. And I’m really excited to give our presenters the ability to introduce themselves. So Dustin, can you start please?

[4:24]  Absolutely. First of all, super excited to be here. I’ve been looking forward to this for a while, so this is going to be a great conversation. My background is in software engineering. I was a software developer for over 13 years before I got into cybersecurity. I worked in the DoD for a little bit, I worked on video games, whole big story behind that one, and I worked in the retail industry as well. And then I got into cybersecurity as an AppSec leader. So I’ve rolled out AppSec programs at large companies, small companies. I also consult with folks on this very topic: the strategy, the right way to approach AppSec, and so forth. So I’ve had a lot of conversations about this. I started a company, Katilyst. We’re focused on security culture and security champion programs. I created a free success guide to help you build your own security champion program, that’s out there on the internet. And then I enjoy interacting with the community, so I do a lot of speaker engagements and webinars like this, as well as podcast appearances, and I even created my own community called Let’s Talk Software Security, where we host open discussions every month about AppSec. So that’s a little bit about me. I’m around, I’m happy to chat if folks want to have a conversation with me. So happy to be here.

[5:46]  Fantastic. That’s amazing. By the way, I really recommend everyone here in the audience, if you’re not part of his community, join it. The article on security champions is amazing, I read through the entire thing. So if you want to get ideas and understand how to build it, that’s a fantastic resource. And now David, over to you.

[6:06]  Awesome. Well, I wanted to share briefly what brought me into security. I focused initially on software testing, and even before that as a technical writer for a computer magazine years and years ago. As I progressed through my career, I had an experience where we released software. We had amazing software quality processes. We released, and within a few hours someone, luckily a benevolent individual, a security researcher, found an issue, reported it to us, and we’re like, what, how did that get through? Turns out we had not given due diligence to a number of application security processes because we were focused on quality, not the security aspect. So I organized that company’s first AppSec team and quit my job as a software tester and focused on AppSec. That was almost a couple decades ago, and I’ve been doing that ever since. I love it. Another part of my inspiration for what I do: I’m a father of nine children, one wife, two grandkids, and that inspires me to be protective. How do we protect our families, our children, our community, our friends? I am involved in the community quite a bit. I love mentoring and working with brilliant students that are looking at getting a career in cybersecurity. And I’d encourage all of you that have a few years of experience in this or more to really look out in your community and say, who needs a little bit of mentoring, and put your arm around them and bring them into the cybersecurity world. But that’s a little bit about me. I love doing this. I’ve been doing it for a number of years and hopefully will do it for a number of years more.

[7:57]  That’s fantastic. And one thing I can tell our audience is, from previous discussions that I had with David, the combination of his experience from both the development side and the security side really clicks and makes sense. And as you’re thinking about your AppSec program, a lot of what he’s telling and how he’s explaining it, and what you said about being protective, it’s really important, because it’s definitely some sort of a mindset that we all have to adjust. And I really want to start from that perspective on how we approach a project. So for example, when you kick off an AppSec initiative, in your opinion, David, where do you start? What are the first conversations that you’re going to have? Developers, leadership, maybe something with yourself?

[8:49]  Yeah. Let’s say I start at a new company and they’re like, “Hey, your job is to start an AppSec team.” The first thing I would do is gather a list of who do I talk to, and understand what the business needs are. These are stakeholders in the software and the products that we’re producing, and really have a conversation with people. And I’m not talking spend a week doing it. It may take several weeks, it may take a couple months, because as you talk to people, you’ll also start to get feedback from them on, hey, these are the things I would love to see happening in your program. And you begin to make your list of initiatives that you want to have as part of your program. That interview process of understanding the business, understanding the stakeholders, and really taking notes on that and developing that strategy is number one for me. It’s not jumping in and saying we’re going to start SAST or SCA or pentesting. You’ve got to first understand what the appetite for risk is. What are they concerned about? If you start doing something which you think is normal AppSec work, you may be focusing on the wrong products, you may be focusing on the wrong people, and if you don’t have buy-in, because security is about developing relationships, not just finding security vulnerabilities, throwing them over the wall and saying fix it or we’re all dead. That philosophy doesn’t work anymore.

[10:20]  Yeah. And the important part is, how do you find the right people? There are a lot of people, specifically in a lot of companies, and sometimes it’s about having those conversations and finding the right people. Is there something that you’re looking for in the conversation to help you understand, okay, that’s the person that I’ll need for that, or can they introduce me to someone else?

[10:44]  Yeah. When you’re hired, the hiring manager, it could be the CTO, it could be the CEO, whoever that is, you ask for that list of who would you suggest I talk to. Then every person that you talk to, you ask the same question: who would you suggest I talk to? And so you build up that list. And you might think, oh, I only have a couple weeks of interviews. By the time you’re done, you probably have three or four months of interviews that you want to conduct, because they keep building on that. But keep in mind that those are your partners as you build a team. Security is a partnership, not a dictatorship.

[11:24]  Exactly. And Dustin, from your perspective, when you’re stepping into a new company or you want to start an AppSec program from zero, there’s no history of AppSec, what are your first moves?

[11:33]  Very similar approach to what David just mentioned, and that’s really learning about the environment by talking to people. I think there’s a very strong relationship aspect in general to AppSec. That starts with listening to the folks that are around you. You’re brand new, you’re trying to create a new program, how do you ensure that it’s going to work and that you’re going to design something that’s actually going to be specific for the culture, specific for the business? The only way to do that is to learn about the culture and about the business. People might have opinions as well about security based on their interactions with security folks in the past. I’ve always made it a mission of mine to maybe break those negative perceptions to some degree, and I think starting to build those relationships is a good way to do that. Beyond that, and maybe we could get into some more details strategically, digging into the details in terms of what the environment looks like technically as well. I like to come in and start with an asset inventory: understanding how things are deployed, what repositories exist out there, how those repos map to the apps and products that we’re trying to build. I think that’s a really good place to start technically, before, David, you mentioned, bringing any sort of scan tools into the conversation. So it’s really just about providing more visibility into the environment, and finding allies. That was another really great thing that David mentioned. I always use that term: follow the trail of conversations until you’re finding people who emerge as friendlies, people who are like, “Hey, I’m glad you’re here, I’ve really been excited about this kind of program, I like what you’re doing.” And those are the people to start working with and bringing into more of the conversation. So that’s where I like to start.

[13:50]  And that’s really fantastic, because as we’re now saying, okay, we understood the relationships, who we needed to talk to, we got better acquainted with some of the people. And I would always say that sometimes the conversations that are not AppSec conversations, but generally sitting over a cup of coffee and talking about the weather, the family, shared hobbies, are really enabling you to create that better relationship. And now that we’re in the realm of starting to scope, and understanding that we can’t really secure everything, Dustin, what do you recommend as we go into that scope? You explained the asset inventory. How do we identify the critical elements that we want to start looking for, for security? Because we can’t just throw a lot of security tools and say, “yeah, we’re secure.”

[14:52]  100%. And you can’t do everything. You can’t make everything as secure as you want it to be. So you always have to prioritize, that’s always part of the conversation. Learning about the environment in terms of the asset inventory, what apps are deployed, and then starting to collect more information about those apps so that you can prioritize them. Things like, are they public facing? There are certainly internal apps that people are using for various reasons, and then there are those public facing apps, that’s important to know. What kind of data do they handle, what’s the data classification of the data that they’re working with? And then what is their interconnectivity? That’s another thing that I’ve always looked into: to what extent do other services and other apps depend on that app or service? So you collect those pieces and then ideally use those to come up with a priority list of assets that tells you here are the most important ones, and you start at the top. Again, you’re not going to do all things all the time. So start with that one critical app that handles sensitive data, that’s public facing, that’s super important to the business, that’s highly interconnected and other services depend on. That’s your first target. And then you can start talking about doing additional things like pentesting, threat modeling, over time really digging into understanding that app more and the risk to the business.

[16:36]  And I think, Dustin, along with your asset gathering and prioritization, a subpart of that, building your team, is understanding and looking at what does a world-class AppSec team do. Because you’re talking about assets, what are you going to do to those assets? Creating that list of things you want to do to those assets is important. And that’s actually a lot of fun to do as a team. You and I could probably create a list of 50 or 60 things that we would expect a world-class AppSec team with unlimited budget to do. But as you hinted, we have a limited scope. We have budget restrictions, headcount restrictions, resource restrictions, lots of restrictions. So gathering that information is a nice fun brainstorming exercise for the team. Then you prioritize on what are the things that we can afford to do now that give us the biggest bang for the buck for reducing risk to the company, and then start building your strategy off of that list. Now you have a target, now you have a strategy to build, and now we start actually implementing that based on what we can afford to do.

[17:54]  Yeah, I think that’s a great point. If I may jump in, Batman. From my perspective, and this is going to come up later when we talk about metrics and ROI, understanding the actual risk in your corporate or production environment is where you build that wish list of world-class tools. Because it’s extremely important to demonstrate that there is risk in our environment in order to justify doing those other things. While a perfect AppSec program should have secure code training and threat modeling and scanning and all that stuff, you have to make a business case for all of those. And the way to make the business case, in my view, is to demonstrate that there is existing risk, or that there have been past issues or incidents or something that happened in the production environment, that justifies you purchasing those tools, or even just spending effort on any of those things, or even building a team that’s going to address those things. If you’re the first hire, I think you again have to demonstrate there’s an issue so that you can justify even building a team in addition to the rest of the tools that we talked about.

[19:28]  Yeah, exactly. And from this point, the feedback that you shared right now, Dustin, is something that I hear from a lot of AppSec teams that I’ve been working with. Sometimes they start with the technology and they realize, after they’ve gone through some evaluations and testing, that they don’t have a business case internally. They don’t understand where they can get the money they need. And I think what you said about building the business case, and also planning that and showing management that you have a road map, where you’re going to take it, and if you want to get from this place to where the dream is, these are the milestones, these are the steps, this is the budget that I’ll have to get, and these are the people and how I build a team. I think this is a really great approach and good feedback.

[20:25]  And while I agree with what you’re saying, Batman, I think a big part of this, and I’m just going to call you Batman, but I want to hear you say the phrase later on whenever you feel inspired to say “I am Batman,” I want to hear it in your own voice sometime. We really must create partnerships with our compliance team, GRC as well, because while we focus on securing the business, the GRC team may be focused on hitting regulatory deadlines and things like that. There’s a vast amount of collaboration we can do. And oftentimes if a company says, “Hey, we want to be PCI compliant,” you can immediately leverage some of those things and say, okay, now I know I can do some scanning, I can do some pentesting, I can justify, as Dustin was saying, some of the expenditures and building the team out. Maybe you start with PCI as the initial budgetary firecracker to get the ball rolling.

[21:21]  I love that. I would add, also understand customer concerns to help justify some of that as well. Look into the sales cycle. Where are deals being won, where are deals not being won, and what are some controls that customers are looking for, to justify your efforts on certain things. You talked about partnering. I’ve gotten a lot of benefit out of partnering with the legal team too, from a legal compliance standpoint, where privacy in a lot of cases is under the legal side, which brings up data classification. There’s a lot of overlap when it comes to that concern as well, that can justify efforts on things.

[22:05]  Yeah, I love that comment. And I think we do need to have partners on the legal team, on the sales team. Talking to the sales team is the next best thing to talking to a customer. They know what customers want and they know what the business needs, so you’re going to get a lot of great feedback. I keep hoping for the day where security becomes a sellable feature. Maybe in some cases it’s there, but the vast majority of products, I saw one of our attendees talk about the fact that we often are the last thing to be thought of, because security happens if there’s an event that is bad. Oh, we should have had security, let’s make sure we do that. So of course we capitalize on events that turn into incidents. My hope is that we can develop a strategy that creates the partnership, so we have something in place before we have an incident that is hair-raising to the company.

[23:10]  Exactly. It’s similar, when we talk about it, it sounds similar to insurance. You buy insurance in case something will happen, but the cost of insurance is usually smaller than the rest of the stuff that you’re putting in. This is really important. Before we get to the tools and the different types of tactics and activities, maybe we can share with the audience some of the common traps or gaps that we might fall into as we build the AppSec program, more from a theory perspective. So this is something that we have to either think about right now, or if we are going to see something, this is a trap we need to start trying to think how to avoid. David, you want to start?

[24:02]  I love that, traps and gaps. Early in my career, of course, you’re like, hey, I’ve heard of these tools, we’ve got to get it into the system. But if your source code, if your technology stack doesn’t meet what the tool does, you’ve just wasted a lot of money, and you’ve spent a lot of credit, meaning the credit that you’re trying to earn with developing partnerships. So you have to understand the technology and then run those POCs. A lot of this comes from that prioritized list, because that’s where you go back to your partners, your stakeholders, and say, hey, this is where we think we want to start, what do you think, and get some opinions. And I would avoid immediately jumping into “hey, this is the top upper-right quadrant tool, we want to get that right now, I think that’s the best investment.” You have no idea. And it might be that for some investments it might be open source for a brief while, to get the ball rolling, to prove the value as Dustin’s talking about. Or it could be that this tool here, it’s one of the top three, it actually fits what we need, you make the investment and then you can start showing that value, and that’s how you gain that credit from your stakeholders. But I would always avoid the bright and shiny object. It’s very tempting. If a company says, hey, we have AI testing now that does X, Y, and Z and your whole problems are all fixed, those are the first things I go, hmm, is it really solving the business risk that I have, or is it introducing new technology for technology’s sake?

[25:43]  Yeah, I think that’s really well said. Buying tools just for the sake of buying tools, or because you want to play with the new technology that’s out there, without really figuring out that it’s appropriate for your environment, for the stage of company you’re in. I think understanding the risk appetite based on the stage and so forth, this is really understanding the business and what the business needs. I always talk about it in terms of, like an MVP type of situation versus a mature and fully established product. They’re going to have different security needs. You don’t want to completely secure that MVP before it’s been tested in the market. It’s like, let’s just get this out the door, see if it even works. There’s no risk if it’s attacked because there’s no users yet. It’s only when you start to get market traction and the product actually becomes important to your business, now you have more risk, now it’s more important to protect it. Also, we talked about this wish list, and ideally we would have a beautiful threat modeling program and SAST tools and all this stuff. But why? Because it’s the right thing to do, because that’s what a mature program should have. There’s a lot of direction out there in terms of the things you should be doing, but it all comes and should come back to, is it appropriate for your business, for the size, for where you are.

[27:26]  And I like that comment. I think also, when we look at the tools, it’s also what the tools generate. So if I’m running an SCA, a software composition analysis, and I’m going to look at libraries and start dumping that information to our developers to update their third-party libraries, am I going to overwhelm them? So how do we reduce the volume to the highest value possible? We may want to start with double-checking and ensuring that there are no false positives in our stuff, so tuning the tools up before we present that to the developers. Ensuring that we only give the top most critical issues that are valid, that are exploitable, to start. Again, it’s gaining that trust. You do not want to shove a bunch of stuff over the wall, because that means that you limit yourself even from expanding your tool set. If you give them small things that are really fixable and remediable, that’s where you start. And then maybe you can say, now let’s expand. We don’t want to prove our value through mass. We don’t want to give you 10,000 things and therefore we’re very valuable as security. That’s the exact opposite. That proves that you have zero value. You have to give them, here’s the five things that matter most to the company. That is going to generate more trust and partnership than 10,000 things that are just blatantly thrown out there without prioritization.

[29:02]  Yeah, I’m glad you brought that up. Can I expand on that a little bit? I think tuning is important. The idea of finding the critical issues that are out there for those critical apps and having those shared with the devs is extremely important. I would also spend a little bit of time on exploitability at first, when you’re trying to build trust, because you’re going to get questions, especially from developers who, this is their baby, they want to know, okay, well, you think there’s some vulnerability with this code, how does that work, how is it realistic that an attacker would focus on this specific thing to actually infiltrate our systems? Show me. And then if you spend a little bit of extra time, especially up front as you’re building that credibility and trust, to create an exploit, or have someone on your team create an exploit, and just demonstrate, look, this is what an attacker would do, that’s why this is an issue that we’re asking you to fix. This is an eye-opening moment that I’ve experienced, or had others experience, and it makes all the difference. Now they know, when you send them another issue down the line, that they can trust it.

[30:34]  Exactly. And building that trust as part of building the AppSec program is really essential. I can see that with a lot of customers that I’ve been working with, as we’re going and helping them with their maturity model, that trust is super critical. And the finding, if it’s exploitable, showing them the exploit, the evidence and the way that you can prove it to them, also positions you as an expert in the field, as someone that you can be confident in. When I’m saying that something is critical for us, we already know, we’ve done the mapping, we’ve done the analysis, we understand the risk about it. And when we’re telling you, listen, now you have to spend time fixing this issue and not coding and not completing other tasks in your sprint, then yeah.

[31:37]  So I have a question here that I want us to address from the chat. Any thoughts on how companies should approach the numerous security-focused compliance and regulatory obligations? Where does a company start? NIST, cloud, ISO, SOC, etc.

[31:53]  Well, I can give you my opinion on that one. That’s where I mentioned earlier, hopefully you have a GRC team already created, because since you’re focused on AppSec, AppSec usually follows. In most businesses they do GRC, they do regulatory stuff, and then they bring on board an AppSec team. You should have a partner in place. That’s where you talk to them and you talk about the business risk. If you’re taking credit cards, obviously you’re going to focus on PCI stuff. So the answer is really, for me, not very easy, because as an AppSec engineer, I want to know that compliance is the minimum bar. It’s not security, it’s the minimum bar where you start. And so we just have to understand where that is. We may want to say, hey, we want to make sure that we’re NIST compliant, and maybe you do a basic review of NIST controls and do some kind of maturity review to see where you are. But the regulatory stuff comes through the partnership with legal and GRC, where they’re like, hey, these are the things we’re obligated to do, and then as an AppSec team we focus on providing evidence to prove that we’re doing those things. I haven’t really been involved in saying, hey, we should be compliant in this regulation, because I’m more focused on the regulations that they have, making sure we have a minimum bar, and then focusing on building that team to ensure the business is secure.

[33:28]  Yeah, and I think this is where understanding the business and the customer needs comes into play as well. A lot of what compliance is trying to do is put in place whatever controls are needed for the customer. So having their input on what’s important to them, and obviously the auditors too, what are the things that they’re looking for when they make their decisions about whether you’ve earned it. I’ve had a lot of close partnerships with GRC, to the point that we’ve walked through the compliance requirements line by line and discussed opportunities for us to do things better, regardless of whether it’s required by the auditor or not. It’s like, look, this is the standard that we said we’re going to hold ourselves to. And there’s a ton of work that just comes out of that, because it’s one thing to say, yeah, we’re doing scanning, it’s something else to say we have a mature VM process in place that we feel good about. So maybe taking, like you said, the compliance as the low bar, but then holding yourself accountable to the spirit of the text beyond just what it says. And I think that creates a lot of opportunities to improve.

[34:50]  And I like what you said about the partnership, Dustin, with compliance, because oftentimes, especially if they’re like, hey, we’re going to be SOC 2 Type 2 compliant, well, we want to talk to them and find out what they mean, because a lot of times we have an opportunity to define what that looks like, and we can influence and drive more towards focused security efforts to ensure that yes, we’re compliant, but we’re compliant in something that’s real, that has real secure value to protect our customer data. And that’s where that partnership is really valuable, to click in early, because often companies, if there’s a budget problem, they’re going to focus on what’s the bottom bar that we can do to survive. So you want to make sure that bottom bar is also pretty solid, that it’s something that we can provide real value to the company. We’re not just saying, hey, we have to scan stuff, okay, what does that mean, oh, it doesn’t matter, just show that you’re scanning. Well, we want to show that we’re fixing stuff. How often do we have to fix stuff? Oh, we don’t care, just show that you’re fixing stuff. If we can define SLAs, be involved in that, and really be part of that process, then we can actually affect and drive that risk down, which is the main purpose of our team: drive the risk down, protect the company and the customer.

[36:02]  That’s fantastic. And from this, we understand that we can use compliance as a jumping board and kind of ride with them, and then use that budget, relationship, partnerships, as you defined, and take the next step in the maturity model. So what would be the must-have tools, the AppSec tool belt, where you’re going to focus and then take the next step? So in your opinions, what are the tools that you would look at? David, you can start.

[36:40]  Well, I like to divide tools up, and you can go into broader categories, but predict, prevent, respond are typically nice categories to have. Assuming that you’ve done some analysis on your business, you’re like, okay, these are some of the tools that we want. We want to do SCA, third-party libraries, we know there are a lot of problems with that. We want to make sure that we have maybe some SAST to understand the business logic of our source code. We want to understand, security champions, want to develop partnership. Now, this depends on the company size. If it’s a five-person team, then maybe that’s a little different. But assuming we’re a mid- to larger-size company, having security champion programs, again establishing and strengthening relationships, and this is a shout out to Dustin’s Katilyst program which is amazing, I love it, it’s a great way to organize a large group of people to help contribute to application security. It’s a wonderful way to involve and develop a community within your company. So that’s another set of tools. But now, Batman, I’m not sure if you’re actually wanting us to recommend actual tools, but I think we can get a lot of that good information from Google and say, top three tools in this area. Understanding the source code, understanding that we need a relationship with developers, and on your AppSec team, as you’re building your team out, assigning emissaries or individuals on your team to represent lines of business, so you create a partnership there, and then they also talk to those security champions that are in those lines of business. That relationship is probably even more important than which tools from your tool belt, because that’s a tool that you have to have before you even expand. But then, looking at your systems, like I said Dustin and I could create a list of 50 or 60 things that you must do if you want to be a world-class team, but that includes typically SAST, DAST, security champions, threat modeling, bug bounty programs, pen testing. Those are some basics that I think, if you have that kind of stuff, you’re probably finding some pretty decent information.

[39:05]  Yeah, I would add, I completely agree, David, with everything you said. I would add pentesting as almost the first one to start with. Not necessarily that there are tools for that, but when we start to talk about finding issues in production environments and compliance, having a regular pentest schedule, a partner that you can work with to make sure that you’re regularly pentesting. I would also add the inventory piece and the prioritization piece. There are tools out there, vendor tools are great, even tools like DefectDojo, very free to use, to help you organize the findings and prioritize and focus on what’s most important. And then when it comes to the inventory side, how do you build a data lake of information, and what are tools out there to help you build that data warehouse, where you can collect information from your cloud environment, from your repos, GitHub, etc., put it into one place so that you can ask some interesting questions about your environment. I think that’s a very important tool to focus on as well. And then culture and champions, like you said, not necessarily tools, there are tools out there that help with such things, but making sure that you focus on that as a component of your program is important too.

[40:37]  Yeah, and those types of tools, especially penetration testing, code security, you usually also find those in a lot of the compliance requirements. So this is also an important part. If you would have to search for some aspect of a tool for yourself, not all SAST or SCA or pentesting are the same. So I would look for a tool that matches the way that my developers work, my environment, and so on. But when I take those tools, now that we’ve founded who to talk to, building relationships, allies, the program, we understand where we are, we’re going to talk with legal, with development, get their buy-in, get some budget, build a team road map, milestones. Now we want to show progress. We’ve got the tools, now what do we do? So how does it actually look like from this point, where I have everything in place and now I want to show development, show management, show different business aspects the progress that we’re making? So where do you start from that, Dustin?

[42:02]  Yeah, I think let’s back up for a second, because part of what you were asking is around how to make sure that the tools that you find are the right tools, and what are the things that you’re looking for when you evaluate tools. One thing I would throw out there is to make sure that you’re working with folks beyond just your security team when you select those tools. Those relationships that you created, the partners that you have, the allies that you found, can you and should you involve them in the evaluation process? Absolutely. And that will help you figure out what are the requirements, what are the things that they’re looking for, probably ease of use, probably things that don’t slow them down, and those are aspects that you should add to your requirements as you’re looking for tools, that you may not know about if you don’t ask them and bring them into the conversation. Now when it comes to how do you start to show value from these tools that you’ve purchased, this is where it goes back to, hopefully you’ve done the due diligence to understand what are the actual risks in your environment, and what requirements you have from customers, compliance, etc. Are you measuring those? Have you created metrics that actually show that you’re affecting the risk based on the tools that you have purchased? There will be some checkboxes for compliance, are you scanning, sure, but at the end of the day, what you want to show is that you’re reducing that risk. There were certain vulnerabilities that you found via pentest in your production environment, maybe even classes of vulnerabilities that emerged as patterns, these are the common issues that our developers insert into the environment, are we effectively reducing those types of issues? Have we purchased a tool that focuses on those issues? Maybe it’s just SQL injection. There’s this prevalent lack of use of prepared statements and the right remediations. Okay, well, does our tool find that, and have we now seen a reduction in that risk because we have the tool in place? And then can we also show maybe shifting left in your SDLC a little bit more, have we through training stopped even getting as far as the scan tool finding those issues, because developers are more aware and they code in a way that prevents those from ever happening? So this is where the details are important and are going to be different in every environment. So it’s important to measure those in your production environment and then set up a way to continuously measure those as you go forward. One of my favorite measurements is an escape rate, vulnerability escape rate, which is defined as, where in your SDLC are you finding certain issues, how far do they get, to what extent do they escape from one part of your process to the next. Meaning, if you have a solid design, training, etc., maybe your escape rate into pre-prod testing is lowered because of those things, whereas it’s high otherwise.

[45:42]  And you’ve implied a lot of cool things here, Dustin. When I look at a tool, the first thing I do is not look at the tool. I look at, I build a requirements list out. Part of that requirements list might be, like this last year, a cool feature in many scanning tools has been, hey, let’s look at contextual risk instead of just focusing on the CVSS score. So looking at contextual risk is a huge thing. So I can start to build up that requirement list, then I can start saying, does that tool meet the requirements? So you not only have your stakeholders that you’re finding out these are the things that they need, you’re looking at, as a researcher, what are the best requirements I should have for a tool that fits my business. One of them I would propose would be, can we move away from measuring straight severity and focus on risk, the criticality of that risk? That way we’re reducing what we give, lower volume to our developers to fix, which increases your credibility, as you talked about, Dustin. But it also gives us the ability to now say, okay, we’re going to measure this, we’re going to show what we’re doing, and we’re going to hold ourselves accountable to fix these things within a particular SLA. The SLA is probably one of the most important things we need to establish early on, even before we start generating tool output, because if I have a tool and I start generating hundreds of issues and they just float out there, if I have a critical issue and it’s going to float out there for two or three months and it’s exploitable, that is really bad. So we have to make sure that we have that SLA defined, and that our CTO and others agree on that and sign off on it, because development says we agree, we’re going to fix this criticality at this level and this speed. Then we can take that tool belt and make sure that we’re going to get results out of it.

[48:07]  Exactly. And this takes us to the next level. I like the escape rate, and the SLA, it’s important to define, as you’ve said David, with the relevant stakeholders, because then they will also be accountable, and they’ll push their people. It’s not your people, it’s their people eventually. So this is also important. Before we get to the audience questions, one final question: what are the bad metrics that you see teams elsewhere using that they shouldn’t, that are counterproductive to their success, and what they should measure instead? David, you want to start?

[48:53]  A bad metric is anything that doesn’t help you lower risk to the company, or that moves you towards bad behavior. So I could create a metric that says, hey, I’m going to pay you by the number of bugs you find, developer. And that’s immediately encouraging every developer to say, I’m going to get paid bonuses for fixing bugs, I’m going to write a few bugs in there and make sure I can get that swimming pool this next year. So we have to be careful that the metrics that we have don’t cause malicious, even if unintentional, behavior from those that we’re asking to fix something. I also think metrics are important to make sure that we focus on the right audience. If you’re the business owner, Batman, and you say I own business line A for product A, and Dustin is business owner for line B, and I am giving you both a big bucket of stuff and say I need you both to fix this, you’re both going to say Batman owns that, or Dustin owns that, so we’re just going to hold off. You have to have complete, distinct, specific ownership. So before you can even give someone something as an AppSec team, you must take those metrics, and I think a simple metric is, am I fixing the things that are found and verified within a certain rate? Am I meeting SLAs? Meeting SLAs for security vulnerabilities is probably one of the most important aspects, to get developers in a good habit. May not be the most important metric, but it’s one that establishes a relationship. So if you’re doing that and you’re ensuring that every one of those is assigned to the correct line of business, that the ownership is there, and you hold them accountable, maybe you send out a weekly newsletter that says, hey CTO, these are the things we’re measuring for and this is where we’re trending. Something as simple as, am I fixing stuff at the right rate, and am I approaching zero, which is where I want to be. That’s one example. Not all businesses are going to be exactly the same, but generally speaking, I would expect, if I find a vulnerability, I want to have it fixed within a certain time once it’s validated.

[51:32]  Exactly. Dustin, what’s, I’ll rapid-fire add a few things there, if I may. Measuring something that’s not effective reminds me, as a dev years and years ago, where they thought measuring lines of code that you developed was a good idea. Sure, I can write a lot of lines of code, there’s no guarantee that it’s going to work or be efficient. So it’s important to think about what you’re measuring, because that’s certainly going to affect the behavior. I think measuring sheer volume of vulns, as if you’ve accomplished something, without really digging into whether they’re quality vulns, is a bit of an issue that I see often. Here’s a bunch of vulns, and then we’ve reduced the vulns, good job, because we threw 90% of them away because they’re false positives. Is that really measuring anything, other than maybe you should get a different tool? And then the other thing I would mention when it comes to metrics is to try to look at the positive, use this to increase behavior and create momentum by showing progress, as opposed to just showing what’s left. Here’s all the vulns that we haven’t fixed, this is a problem, as opposed to, look at all the stuff we have fixed, and look at our velocity of fixing them, that’s increasing. Obviously taking into account any new vulns that are popping up as well, and making sure that you’re getting ahead of them over time. But that’s a human behavior thing. If you’re just showing the negative all the time, I don’t think you’re going to get the outcome that you want, as opposed to building momentum.

[53:26]  Yeah, exactly, trend line. 100% agree. It’s always nice to see where you were, where you are, which way you’re going. And set targets too. I got this advice years ago, instead of here’s all the vulns, we just need to fix all of them, approach it incrementally. Let’s get to this point in the next year when it comes to your backlog. And I think that’s important to show too, did we hit our target, as opposed to just, did we fix all of them, which is sometimes insurmountable.

[54:00]  Exactly. So it’s important not to write and measure how many lines of code and how many bugs and security fixes, because that doesn’t work if I’m correlating. But really, what I want to do before we wrap up, we have a few questions that I’d like us to answer. I’ll start with the first one. How do you guys see AI security coming into AppSec as dev teams are moving into that direction?

[54:30]  Go ahead, David, if you want to go.

[54:33]  Well, let me just quickly say, definitely on the remediation side. I’ve got a few thoughts here. I think using AI as a co-pilot, not an autopilot, to help you work out the code that you want to write, that’s an important way to use AI today. I think AI is a good way to start, it’s almost a solution for writer’s block. Where do I start? Write me the skeleton code for this, I think I want to do this. Oh, okay, that’s a good idea, I can take that, modify it, and run with it. Not just take it for what it is and plop it in my environment, but take it as a starting point, as a co-pilot not an autopilot. And then I do think the remediation side, there’s a lot of potential there as well, to have AI understand more of your codebase, the business logic, and then recommend relevant fixes. Now, we’re not there yet, I’ve seen some tools and vendors popping up that are attempting to focus on this problem. I think we need to reach a stage of maturity where that’s going to be highly useful and relevant, and I don’t think we’re there yet, but I definitely think we can get there.

[56:03]  Yeah. David, you want to add?

[56:05]  You know, just a short while ago, hallucinations were a big part of AI, so there was very low trust, and I think they’ve resolved a lot of hallucination things, but we’re still building trust with AI to find out what is it that we can actually trust and do. But I agree with Dustin that we need to view it as a co-pilot, something that’s going to help us and be something that we can consult. I don’t know that we’re at the point yet where I can push a button and say, okay, analyze all this data for me and tell me the answers and I’m just going to blanket go with it, because most of us would probably be moving towards getting out of a job or looking for retirement at that point. So I think we’re a ways away from truly intelligent consulting on that, but having a co-pilot that can give us advice, point us in the right direction, help us find weak spots, I think that’s a good starting point.

[56:53]  Exactly. And I always remind everyone that before we had AI, we had Stack Overflow. And AI is basically Stack Overflow on steroids, because that’s the data that was used to train the AI. So until we’re able to modify that, we definitely have to understand that, as Dustin and David mentioned, this is a co-pilot. And even if you’re asking, there’s an exercise that I’ve seen a lot of people doing, asking the AI to write secure code, so they get a code, they ask it, okay, now write it securely, and again write it securely, and you go through more and more iterations, but every time the AI fixes one thing, it actually removes the fix from a different thing. It’s still complicated. One final question before we wrap up. I’ll read the question, I am trying to understand what it means: what’s the line that differentiates AppSec and ASPM? That’s an interesting question.

[58:04]  I think ASPM has emerged. I don’t think it’s fully settled on exactly what that term means, but I certainly think that it is an important component of a mature AppSec program, especially on the inventory and prioritization side that I was talking about before. I think ASPM tools, for the most part, and they’re all kind of different, this is why I say it’s not very well defined, but for the most part what they’re doing is aggregating results across multiple tools and then helping you understand, hopefully with contextual information, the landscape, the posture. That’s what ASPM, application security posture management, is, helping you understand the maturity by aggregating those results and then giving you essentially that hit list of where to start first based on the priority.

[59:06]  Yeah, I also think that ASPM is a portion of your application security strategy. Your application security strategy is your end-to-end position of here’s all the things we do, here’s the order we do it in, here’s how long it’s going to take. It’s your team building, it’s all these things, the culture, all of that is the AppSec stuff. ASPM is a small portion of that, that is growing, and maybe it will overlap across prevent, detect, respond, but your strategy is what defines your team and how you’re going to be successful at your company.

[59:37]  Exactly. And this is really important. I think that ASPM is becoming one of those tools in your tool belt. It’s not necessarily all-inclusive at the moment. We’re seeing great strides in the industry, of course, but there’s still ways for us to improve, especially when I’m seeing where the technology can take us and the ideas that people have about what to do, how to do it, how to automate the entire thing. I heard an idea that says, take all of your low-impact vulnerabilities, get an AI and fix those automatically, because who cares, but they’ll be fixed. So, ideas. I don’t know if it’s feasible, I don’t know if it’s going to work or if it’s worth anything to do, but it’s an idea, and people are really amazing in that sense. We are at the time, so I would like to wrap up this session. I want to say big thanks to you, Dustin and David. This was an amazing webinar, very, very insightful. I think that we were able to take everyone through the journey: from I’m starting the journey, understanding what I need to do, who to speak with, building relationships, who are the business owners, my allies, partnerships, how do I think about the program, going through the different tool sets, compliance aspects, metrics, SLAs, how do I prove that success, building that road map. And I want you to remember that AppSec is not a one-size-fits-all playbook. Every company, sometimes even business units, have their own unique needs, their own way of working and workflows, and it’s an evolving practice. We need to understand that every time we take a step forward, we need also to examine that we’re still walking in the right direction. So what I want you all to do: follow Dustin and David, stay in tune on all the amazing AppSec stuff that they’re sharing. Stay tuned to OX Security, follow us on LinkedIn. We’re going to share additional amazing webinars, and the recording of this one with key insights and takeaways. And to everyone, let’s rock AppSec together. I’m Batman.

[1:02:04]  Nice. I heard that. Thanks for having us.

[1:02:09]  This was a pleasure. Bye-bye.