The Science Behind Alert Fatigue in Security Teams, and How to Beat It

[0:24]  Welcome everyone and thank you for joining this webinar. I welcome you to “The Science Behind Alert Fatigue in Security Teams and How to Beat It.” This is really important. My name is Boaz Barzel. I’m the field CTO at OX Security, and I really am fortunate to have Moshe and Matt with me today. They bring really diverse expertise, from deep security knowledge to neuroscience. And we are going to talk about an issue that I constantly hear about, and really it’s about security teams that are beyond drowning. They’re suffocating. And for example, the research that we’ve done showed that an average organization has more than half a million alerts. And really a staggering number is that between 95 and 98% of those alerts are not critical and sometimes not real issues. So really the amount of data that is being bombarded to security teams is enormous, and I’m betting the participants we have now in the webinar are doing something like, “yeah, this is familiar to me.”

[1:49]  So what we’re going to do, we’re going to talk about alert fatigue. We’re talking about more than just automation and shift left and other buzzwords. We’re going to really focus on where you’re going to ensure that you’re not suffering from alert fatigue and really how to beat it. What are the items or actions that you need to do in order to get it? And when you’re hearing this webinar, just ask yourself: when was the last time that you or your team felt rested? And we’ll talk about it. We’ll dive really into that. And what I’d like to start with is an introduction of our speakers. So first, Matt, I would like you to start introducing yourself.

[2:46]  Cool. Yeah. Hey everybody, Matt Johansen. Longtime security practitioner, better part of two decades in the field. I started my career very hands-on keyboard, offensive security. So I was generating the alerts. People would pay me to come hack into the companies and tell them how I did it. I started to really focus on appsec during that time, hacking websites, and SAST and DAST tools, if you’ve ever played in that world, are just filled with false positives, and that was just part of the game back then. Transitioned into more of the blue team role of things. I was protecting a fintech startup for a while; I was the head of security. Then went into banking, and that’s when I really started to cut my teeth on some of the stuff we’re going to talk about today, like incident response and partnering with the SOC and the IR teams. I ran vulnerability management for a while at Bank of America, again generating a lot of findings and vulnerabilities, and my team’s job was to kind of hunt the real ones down and fix them. And then most recently I was the head of software security at Reddit. So any code written at Reddit, my team was responsible for the security of, but we were a pretty small security team, so I did a lot of the incident response there too. Anytime we had a security incident, I was the lead incident responder. So again, still kind of cutting my teeth on a lot of the stuff we’re going to talk about today. And now I run a security publication called Vulnerable U and a YouTube channel and a lot of content about this stuff. And I really like to focus my content on both cybersecurity current events and mental health. So I’m really excited about this topic today for that reason.

[4:28]  Fantastic. Thank you, Matt. I’m also really excited because you are bringing both the red team and the blue team type of experience and can view those items from both sides. And I love what you said in the beginning: I was the one creating all those alerts.

[4:46]  Yeah, and I was creating some noise for sure.

[4:48]  Yeah, the noise. And this is something interesting because I think this is the first time that I’m hosting a neuroscientist and I’m super excited about it. Moshe Bar, please introduce yourself, because we’re going to have a great conversation hopefully.

[5:06]  Yes, so my name is Moshe Bar. I’m a neuroscientist, specifically a cognitive neuroscientist. I did my PhD in Los Angeles and then was a professor at Harvard Medical School for 13 years, then moved to head a big neuroscience center in Israel, and now traveling around the world. So I’ve been hacking the brain. That’s my claim to fame. And I don’t know what’s blue team and what’s red team, but I think the problems kind of transcend different domains, and I think we’ll find a lot of common language between what we do. My expertise is, well, I have a recent book about mind wandering. So I think most of us, or all of us actually, experience this, and this is related also to being overwhelmed at work but also when we’re just bored. So this is a different topic, but the pros and cons, believe it or not, there’s a lot of pros also for mind wandering, for idle time, especially for creativity, for improving mood. So my other big domain is depression and mental disorder. I also have a startup company that focuses on depression proper. But I won’t be long, and whatever is relevant I’ll just pull in as the questions arise.

[6:16]  Yeah. Well, this is amazing. The mind wandering, I hope that I’ll be able to do it more. As I’m working at OX, it’s still a startup, we’re all over the place in a good way. But I’d really love to hear more about the mind wandering. But before that, let’s start talking about alert fatigue. Let’s start understanding really the problem. And I would like to start from Matt, from your personal experience, really starting from where you’ve experienced it before. How does it feel to experience alert fatigue, and can you share from your personal experience what does it mean?

[6:54]  Yeah, I mean it comes in a lot of different flavors, right? Depending on your role, the size of your organization, the size of your team, how many people are helping you on this kind of thing. And a lot of times, even big organizations with well-funded security teams, it’s not like you have a lot of people sitting on the bench ready to rotate in and help out and twiddling their thumbs ready to tag in. Even big well-funded organizations, you’re stretched really thin. For this frontline role specifically, I’ll probably use that term a lot today. Front line meaning you’re really eyes on the wall, responding to incidents, detecting incidents, threat hunting, things like this, really protecting the organization. That’s frontline work. And yeah, it’s a generally high pressure job, where things are quiet when they’re quiet and they’re not when they’re not. And so it’s really your job to determine, in whatever you’re looking at, is this a real issue that I need to rally the troops and respond to right now? Or is this just one of many, many hundreds, usually thousands, of “no, this is not an issue, this is just an anomaly,” or “nope, we hired that new guy and he traveled when he wasn’t supposed to, and okay, yeah, we’re not getting hacked, this is a real employee.” That kind of thing. And so you ask what it’s like. You’re generally clicking “no, this is okay, no, this is okay” 99 times out of 100, and then that one time out of 100 you’re like “no, this is not okay,” and then this is about to be a very interesting and different day. And a lot of that time you’re kind of just cleaning your pile off. Okay, I got this pile of alerts that I’ve got to go through before I take my lunch break or before I sign off and cook dinner for my family today. And it’s like, let’s get through this. And you have to kind of keep alert during that whole time for, “oh nope, I’ve got to make sure I catch that one out of 100 and not just push this off my plate.”

[9:06]  It sounds really difficult, especially the fact that the alert doesn’t stop. Alerts constantly keep coming, and because they’re constantly coming, we’re seeing that it’s not just, as you said, some peak days, but it’s constant. It’s constant pressure, constant alerting. And I’m wondering, what are the behaviors that you would notice that people are experiencing, or that you can see in people that are being overwhelmed from security alerts?

[9:44]  Yeah. So I’ve experienced this both in alerts, and I know the topic today is alerts and that generally brings to mind a SOC, alerts generated from security tools into a security operation center. But there are also many other types of security tools that generate lots of findings, not necessarily alerts, but like I said SAST and DAST tools are notorious for this kind of stuff. And so what you start to see is, especially, SAST is notorious for this, where it’s really hard for automated scanners to find vulnerabilities in code. And so if you get handed a PDF with 4,000 findings and you’re on finding 3,100 and you haven’t found a single true positive yet, and you’re manually triaging all of these, what do you think is going to happen for the next 900 findings? You’ve already lost confidence in this tool. And there’s this constant push and pull between false positives and false negatives, kind of tuning your security tooling stack, whether it’s alerts or findings or whatever it is. You’re constantly, “okay, too many false positives, got to tune this back,” but then, “oh, maybe the tool will then miss something because you tweaked the rule too far the other way.” And there’s this push and pull, and then the humans are kind of the stopgap between that tuning. So you’re basically turning the knob of how much do I want to put on a human’s plate versus how much do I want to trust this tool. And so you asked, how can you tell someone’s experiencing it? Oh, you’re going to get reports of missed things. “Hey, we found this vulnerability, why didn’t your scanner find this vulnerability?” “Oh, it looks like your scanner did find the vulnerability and someone marked it as not a vulnerability in their triage of 4,000 things this week.” They’re going to start to just exhibit other signs of burnout and stress throughout the rest of their job. And then the other thing that causes alert fatigue isn’t too many alerts. It’s, hey, you actually have to respond to those bad ones, and those are late nights. So, hey, Matt was on the phone on the incident bridge until 2 o’clock in the morning last night, and guess what, he had to get up the next morning and make sure his eyes were back on the screen. So it’s real fatigue paired with alert fatigue. Those are obvious signs as well. It’s like, hey, you can tell Matt’s tired. Matt’s been working till 2 a.m. this whole week.

[12:19]  Yeah, that’s very interesting. And to the audience, I do want to say, if you do have questions for us, feel free, open your Q&A, write those questions, we will address those questions also during the session. And now I want to talk with Moshe. Now imagine that we do have some specific research in cybersecurity, but I think it’s limited. I think that we haven’t even started scraping the top areas, and you have a lot of experience in the broader research. And really what I want you to share with us is, from your own experience and research, how people handle those overwhelming volumes of alerts. If you can start by addressing what you’re seeing from the behaviors of alert fatigue and then how people are addressing them.

[13:09]  Yeah. I do want to start by maybe clarifying some of the terminology. So I don’t want the audience to think that we have in the brain some module dedicated for detecting alerts and for being overwhelmed, and it’s been waiting there for ages until we discovered cybersecurity. This is just one name. Alert fatigue is just decision fatigue, cognitive fatigue. There are many different names for the same purpose, because we are multi-purpose creatures and we use our brain for multiple purposes. So the same problems that you see in cybersecurity and in alert detection, you would see in other domains, even everyday activities like searching for your friend in a club, or in a sports game, or being a subject in our experiments that are not related to cybersecurity but still you have to search for L’s among T’s. It’s the same topic everywhere, and it’s important to know that. On the one hand, we are amazing creatures, we really are, and our brain is just stunning. But at the same time, we’re limited. We’re limited by resources and we’re limited by our state. And when we talk about fatigue, I do want to make a distinction which we haven’t made before. People do need to be aware of something that we call sometimes cognitive load. This type of fatigue, or exhaustion I would say, you’re exhausting resources. I don’t like the comparison with a computer, but you can think about a CPU with a zero-sum game of resources: if you use some of the resources for one thing, you have less for others. So the more you tax this machine, be it for other alerts, or for a phone call from your family requiring you to get some groceries on the way back, or some pressure or some conversation in the background, all these things tax the 100% resources, and at the end you’re left with less. And what we’ve discovered, and others, and I think it’s fascinating, is that the more you narrow the amount of resources we have, it’s not that we do less, we actually change the way we do things. We become less creative. We become, instead of exploratory, we are exploitatory: we exploit familiar templates, familiar knowledge, and we resort to easier, more mundane solutions. So when you limit the resources, it’s not that you’re only able to do less, but you also do it in a different manner. So people that are stressed with information, even if it’s just temporarily, they are different people in a way, because they’re suddenly less creative and less exploratory. The other type of fatigue, and I think this is the focus of our webinar, is what happens with this constant grind of being overwhelmed over and over, day in and day out. What is the toll of such professional activity, both on the productivity, on your performance, and also on your life? So we do know that here again we are limited in what we can do when we are taxing, when we’re repeating the same process. So if you’re doing alert detection for eight hours a day, you’re repeating the same process. It’s different input, but you’re repeating the same process. It’s not like you’re thinking about a task finding synonyms for words, and then you do some sudoku, and then you do some Grand Theft Auto, and then you do some other thing. You’re not diversifying your mental activities, you’re focusing on exhausting the same topic. And this is the source of our being fatigued, because we are depleting a lot. And we can talk about the chemicals behind it if you want, but there’s a good explanation for why we’re exhausted and we kind of change the way we take decisions just because we are exhausted. And the last thing I want to say about how Matt said before, real fatigue versus alert fatigue: that’s also the issue we find with mental illness, that if somebody is bleeding in the living room, everybody will put attention on them, but if somebody is mentally sick inside, nobody’s like, “just snap out of it and go do something with yourself.” Because it’s not out. So the mental fatigue is something that it’s hard for people to appreciate from the outside, because it’s not like you see me drowsing over my keyboard. I’m fully alert, I’m in air conditioning with the best chair possible in front of my screen, but my inside is fatigued, and you cannot see it from the outside. And that’s why I think it’s a good question that you asked Matt, about what are the symptoms, what kind of behavior do you identify to diagnose somebody with alert fatigue.

[18:14]  So really, what’s interesting me, and there are a couple of things that I’ve taken. I know that you didn’t like the analogy between a human and a computer, but I was constantly thinking about overclocking, where you’re putting your CPU and breaking the limits of that, and it’s just crash and burn. And what’s interesting in what you said, I’m constantly thinking about alert fatigue and similarly to how people drive. When we’re driving, we’re focused on the road, and I imagine myself, what would happen if my dashboard was constantly showing me stuff? I wouldn’t be able to constantly drive. I would get crash and burn in that sense. So if I’m taking that more to the personal stage and the personal well-being, Moshe, what would constant exposure to high alert fatigue do to my well-being in that sense?

[19:21]  That’s a big question, and to answer it in a reasonable amount of time, I’ll have to make a big leap here. But if I tell you that chronic stress and chronic load might lead to clinical, I mean I don’t want to scare people, but in the extreme it can lead even to depression and anxiety, to clinical depression and anxiety. But in between it might just simply affect the way you take decisions, and not only at work. I don’t think I’m saying anything new to the audience here by saying that chronic stress is not good. We do need stress in life, but we don’t need it to be too high and too chronic, we want it in breaks. Stress also has some positive effects, but we don’t want it to be overwhelming, we want to be able to succeed. And that, by the way, is one thing: when I said that when we repeat the same process over and over and become fatigued, it’s a matter of depleting some kind of chemicals in the brain. And one of them, there are neurotransmitters, some of them you’ve heard about. Dopamine is very famous, and also serotonin. These are neurotransmitters that express reward. We do things every day, not only at work and not only in security alerts, and we are rewarded with these molecules. When we do this over and over, we’re depleting this reward. So you do the same action, you don’t get reward, you lose motivation, you lose the same incentives. And gradually this actually can carry out to outside of your monitor and affect the way your reward mechanism works. And this might actually, as I said before, lead to depression. And again, I want to qualify, it doesn’t mean that if you do this eight hours a week for two months you become, you start using product, it’s not 100%, I’m just demonstrating the principle here.

[21:17]  So, Matt, from this perspective I want to get more from your experience on this part. Can you share what would be kind of the tipping point between high volume of alerts and a balanced stress, if I may, to when it becomes white noise and, as Moshe mentioned, you’re depleting your dopamine and serotonin reservoirs and you’re flying blind?

[21:46]  Yeah. So first of all, I’m a participant in this webinar as much as the people watching. Moshe, this has been awesome just to hear you talk about some of this stuff and talk about depleting the chemicals. I’m learning a lot. So as far as tipping point, it’s obviously going to depend on the person and on the style of work that you’re doing. Like I said, there’s the false positive alert, then there’s the alert where if I miss something, something might be on fire right now and I’m missing it. That’s a really high stress alert versus a false positive of an appsec scanner or something like that, much lower stakes. Like, hey, I’m just trying to find an L in the pile of T’s, to use Moshe’s experiment. And if I miss it, maybe we’ll find it later. A hacker would also need to find it for that to become a problem. It’s less urgent. Versus an alert that’s like, oh, is this suspicious behavior or not? Is this login from this other country that we don’t expect bad, or is this someone traveling? These kinds of things at much higher pressure. So I think the tipping point is different depending on that, depending on the person. But I really like the thing that Moshe said about diversity of action. That’s, I’m skipping ahead a little bit to mitigation strategies, but this is something I’ve always tried to do on the team: how can you find that tipping point for a single action of no diversity, of looking at a single type of alert for a certain amount of time, and then come back from that tipping point. Say that tipping point’s three hours of just straight chugging on something. Then at two and a half hours, try to make it culturally on your team that we’re going to switch tasks. You’re going to go do something else for a little while, and someone else is going to take over the alert triage. So I think a few hours of doing any of those things, and then there’s all sorts of things that impact that, especially how well is that person sleeping the night before, how many hours are they putting in over a period of time chronically, like Moshe said. And how can we cycle those hours off? I’ve got stories I could tell about people that have really pushed that to the limits, and some people are built different and are neurologically a little different. Some people love it. Some people are like, “do not give me any other task, I would love nothing more than to sit at this pile of toil for eight hours, I don’t want to talk to another human.” And some people are like, “please, can I do this as fast as possible so I can go write some code or do something else more creative?” It really just depends on who you’re working with.

[24:34]  Exactly. And I think you’ve touched a few good points. First of all, the personality and the person, and I think this is more for the managers to verify, because we know that there is alert fatigue, we know that there’s a lot of alerts, it’s not a secret. And we understand that we need to at least balance it. And a lot of people were not focused on, you know, how many hours did you sleep at night? Did you go out and have fun previously? Or ensuring that their people are actually doing something else than just working their ass off until 2 a.m. and then first thing in the morning. Because there’s a lot of responsibility here. And we actually got a question from the audience to your point, Matt, about fine-tuning alerts by either trusting the security tool or burdening the SOC team with more alerts. How do you approach that balance?

[25:32]  Well, a lot of times it’s not up to you. You’re not the developer of a lot of the security tools that you’re running. A lot of the time you’re at the behest of the tool. So for me, I can speak from personal experience here. We’ve used some tools that are kind of industry standard tools in the past that we’ve decided to stop using, because we were not willing to take the trade-off of the burden. So a lot of people, “oh, I use this SAST tool, it’s one of the industry leaders.” Sorry, a lot of recent memory is appsec related, so I keep coming back to SAST, but, oh okay, so this tool running on code, it spit out 4,000 false positives and no true positives, we’re going to not use that tool anymore. That’s not a trade-off we’re willing to make. If you are in the position to tune some of your own alerts, maybe you’re developing a threat hunting team, detection engineering team, you absolutely have the power, you try to drive false positives to as close to zero as possible without missing anything. And that’s detection engineering’s full-time job as far as I’m concerned, just constantly striking that balance. The other part I want to mention here, because we’re dancing around this idea: a lot of times I’ve been on teams that don’t have a big bench to dip into for this. So either they do, or they have some super performers. There are two scenarios I see here that are super common and important to contextualize. One, you’re a one or two person shop, you’re it, you’re Matt the security guy and that’s it, that’s all we’ve got. I’ve been the sole security employee for a security company. Then even at bigger shops, say you have a 40 person security team, 50 person security team, but you have two people that are superheroes on the team and they’ve become single points of failure, and no one has really built the skills up because they have that superhero. And I actually have written about this in the past, avoiding superhero culture. If every incident that ever happens, you call that one person, no one else is ever going to learn that muscle memory of how to do anything. So now, without even realizing it, maybe you have a fully staffed team of 40 people, but you really have one or two people that are the pillars holding up the rest of it. And because you’ve ignored that and not created or forced the distribution of some of this work, you’re going to burn two people out, even though you’ve got the staff for it.

[28:21]  So I really want to continue talking about not just achieving the balance. I want to continue about, you mentioned before, mitigation strategies, and I think this is really important, because we’ve understood the problem, and I think a lot of us are experiencing that in different types of areas, not just alert fatigue from security tools. Everyone has their own experience of fatigue, and not just physical fatigue. For example, there are a lot of examples where you want to eat healthy, and you go through your day making a lot of healthy decisions, and by the time you get to dinner, you’ve depleted all those healthy decisions and you’re ordering a pizza and just swallowing the entire thing. So I want to talk with you, Moshe, really about mitigation strategies, and from your understanding and research, what are the most effective techniques that people can start implementing today to start really combating this fatigue. So we understand what it looks like, but what are we doing with it?

[29:36]  Yeah. Okay. So first I’ll start, Boaz, with a personal suggestion for you. I would start the day with the pizza. Start the day with the pizza and then go only healthy from there.

[29:48]  It’s a good one.

[29:49]  But yeah, so for the mitigation, I have a little list that I prepared here based on what you expressed earlier, and I think it’s a great motivation to offer some suggestions. So the first one actually has to do with a very cool finding that’s recent that I’m going to share with you and the audience, and I’ve digested it such that I think it’s easily comprehensible also for people from outside the field, but it’s really cool. So, neurons, when they fire, neurons are basic cells in the brain. When neurons fire they generate what we can call metabolic waste. There’s a metabolism there, they take glucose, they create energy, they fire, and then there’s waste there. And we have little soldiers in the brain that go there very quickly and remove this waste. And then the neuron can continue firing whenever it needs to fire. Waste removed. When you keep using the same neurons over and over, like you do with repetitive attention to alerts, what happens is that this waste accumulates and there’s just too much waste to be removed on time. So people walk around with this waste, and this is documented with imaging, with very meticulous studies done by a group in Paris one or two years ago. And the idea here is that, first of all, my first suggestion for mitigation, I think it’s a little less realistic but maybe we can make it realistic, which is sleep. The removal of this waste is most efficient during sleep, not only rest and not only breaks, but actual sleep. So I know you can’t, somebody works in your company, after two hours they look depleted, you can’t go home to sleep and come back in eight hours. So maybe you think about nap pods in companies, the capsules, put people to take naps. But sleep is the most efficient way to clean up this waste that actually makes the specific neurons required for the repetitive process less efficient. The other thing is what I said before, and I’m happy it also resonated with Matt, which I’ll call alternating domains. So just do things that are diverse. We call it sometimes task switching. And you know what happens for us in the lab, and also when we try to look at people with depression, it actually improves mood. When you make people change, we call it task switching, do this and then do that, back and forth, rather than doing this and finishing this half a day and then moving to the other one. Just keep flipping between them. That’s a method for improving mood, so not only for reducing this depletion and exhaustion. So if you can find, some people, their entire job is one domain and you cannot diversify, too bad, but if you can find a way for them to do this, it will affect both this metabolism waste I talked about and also their interest and reward and mood. The other thing I can say is that, in the world of animation in Hollywood, they used to do, I did my PhD in Los Angeles and I was close to how they were doing these things, and it fascinated me. So when they need to make a cartoon, they go by seniority. The top person that was responsible, that had the most experience in animation, they would do every 100th frame. They draw the first frame, and then the 100th frame, and the 200th frame. And then comes the person one level less senior, and it does every 50 frames. So the more seniority, you do “here’s the beginning of the scene, here’s the middle, here’s the end,” the next person comes and does the 10th frame, the 20th frame, and the least senior person fills up everything. So there’s resolution of going over the information, but it doesn’t have to come from seniority. So imagine, I don’t have experience with alert detection, and maybe in this case it might be a benefit because I’m suggesting something completely off the wall, but imagine that you can divide the examination of alerts to different resolutions, so that somebody can examine things in broader terms and another person will do it in local terms, and maybe swap between them. This could be another way of task switching. Just a crazy idea, but maybe. And of course, we can sit down and think about optimizing protocols for, taking into account what I said about cognitive load, about fatigue that has to do with chemical waste and depletion of glucose and other chemicals in the brain, neurotransmitters, and see what is the best protocol. Maybe 15 minutes this, 10 minutes that, go play volleyball for five minutes and come back. We can think about the optimal. And different people respond differently obviously, so we have to take the individual differences into account. The last point I want to suggest, and it’s really not my expertise, but I had to help my niece in her high school project and she was working about nutrition, and I said, hey, nutrition affects the brain big time. Let’s think about how nutrition can help us. And it turns out that there are recommendations you can look online, and basically the knowledge is there that there are some nutritional considerations that one has to take into account, both for slow release of glucose, steady supply of energy, omega-3 fatty acids for support of stress, hydration definitely plays a role here, antioxidants to counter this inflammation that comes with accumulation of waste. So nutrition is something that needs to be taken into account, but again this is not a prescription, one needs to decide that that’s what they want to do, and then this can be developed further. So these are my mitigations.

[37:02]  I love it. It’s amazing. It really is amazing, because it’s out of everything that we’re usually, and Matt probably you can agree with me, we’re usually talking about. When we talk about security and alert fatigue, we talk about having better tools or better processes or better prioritization. We don’t talk internally about what we eat, when we sleep, how we sleep, and there are so many other things. Super interesting. I didn’t want to stop you in anything that you said. So it really is interesting, and I’m now thinking really really hard on, okay, so get teams to sleep better, to eat better, and provide a recommendation, bring a dietitian or some experts in other fields to give, build plans or just educate on those things as a way to combat alert fatigue. And Matt, I really want to hear from you, first of all, what’s your take on what Moshe said, and the next thing is, what are your successful practices and mitigation strategies you see specifically in security?

[38:22]  Yeah, first of all, I love everything I just heard, that’s so cool. And I can echo from my personal experience. Taking a step back from what you could do for your team, just personally with my journey in mental health and all that kind of stuff, the times that I’m doing the best, and like stress and depression and anxiety are like I’m winning that battle, are times that I’m eating right, times that I’m exercising, times that I’m sleeping at least seven, eight hours a night. When I have a lot of those things in line, a lot of things kind of line up and get a lot easier after that. One of the other things, I’m really glad that we’re not spending too much time on, is, a lot of times these conversations when it talks about stress mitigation and things to help your team with stress, you start talking about breath work and meditation and all this kind of stuff that obviously is super helpful, but at the end of the day, if you don’t remove the great chronic stressor from your life, you can’t out-breathe your way through being up at 2 o’clock in the morning fighting some incident every day for a week. You’re not going to go to a nap pod and cure that. So all those little hacks are the details, that’s the fine-tuning, but you really have to take care of the big rocks. So we talked a lot about that diversity, the diversity of action. I’m huge on that. If I’m leading a team, I try to just switch what people are working on, what they’re passionate about. You want someone to not be depressed or anxious, figure out what they like working on and give them time to do that in a given week. Okay, yeah, do the monotonous stuff that we all have to do, but, hey, make sure you’re getting 20, 30, 40% of your time on your passion project, if that passion aligns with value for the team, which a lot of times it does. You can find the Venn diagram of value to your team and passion in your team. You obviously can’t prescribe diet or sleep or exercise to your team, but I really like what you said, Boaz, in terms of bringing those resources to light, hey, maybe you can work with HR on a benefit for a gym membership payment, and bring in some guest speakers to help on nutrition or whatever it is. For me, I liked to encourage, for myself even, to take walking one-on-ones. If you do one-on-ones with your team, hey, we don’t have to do it here, there’s nothing special about Zoom, just pop your headphones in, let’s walk around the block for 20 minutes and get outside, get some sun, and not be in the cave that is our SOC all day. You can encourage that. But the diversity of stuff, then the other bit that I haven’t brought up yet that I’ve always really liked to do where possible: a lot of times we’re talking about people that are on call. And on call is a really interesting term. On call sometimes means nothing, and sometimes means you got woken up at 2 o’clock in the morning because the pager went off because some alert was whatever, or it’s a holiday. US holidays, when I was incident response, every US holiday I had an incident, because the adversaries know that a bunch of people are taken off of work and it’s a high-risk time. I like to go camping on July 4th weekend every year, I was at the campsite searching for cell service trying to respond to an incident bridge. So on call is really interesting. So I like to set up the on-call schedule so that, A, no one’s on call for too long of a period of time, and then once you’re on call you’re not on call again for a significant amount of time. And then for any time that you are activated on call, I like to force time off. So if you were on call this week and you did get paged and you did have to do after-hours work, or even not necessarily after-hours work but you got paged a lot during your day job and you didn’t get to do any of your normal day job stuff because you were putting out fires all week, I like to do a one-to-one, if possible, of how much on-call time you were activated, to force time off. So you had three days of firefighting this week, cool, you’re taking three forced days off next week. And as soon as possible, hey, go rest, go put your feet in the water, go touch grass, whatever rest looks like for you, go do it. And usually leadership has the ability to force that, hey, you’re not allowed to come in next Monday, Tuesday, Wednesday, go away. For major incidents you can even do longer than that, hey, you go book a beach vacation for two weeks because we have been fighting hell here for two weeks. Certain incidents are really hard, because you’ll have the incident, then you’ll have the postmortem, then you’ll have lawyers maybe crawling around, maybe you’ll have auditors crawling around, it’s really stressful, really long times. So it’s your job as leaders to force the “hey, no, get out of here. Thank you for all the hard work, go take some time off,” because a lot of times your top performers won’t take that time themselves. They’ll think that they need to be there, they think that they need to be tough. So it’s important for you to be like, no, I’m not holding this against you, you’re not going to miss out on that bonus or whatever it is, get out of here.

[43:48]  That’s amazing. I think this is one of the best advices that any organization can have, because first of all the accountability and the ownership is on the leadership. It’s not on the people themselves. Let the people work, let them do what they do best, and you have to ensure that they’re getting enough rest, that they’re getting diversity, that they’re not stressed out, and they’re not getting chronically ill. Because I know how it is with high performing people, they don’t stop. And if you don’t tell them to stop, they will continue doing that until they crash and burn, which at that point it’s really too late. A lot of people have a hard time to just stop and say, okay, next week I’m going to take two days off. They’re not going to do it. I haven’t met one person in their position. And I see that Moshe has something to say about it.

[44:47]  I’m not sure I should mention this book. I’m on vacation now and I took a book called The Denial of Death, and the whole idea there is about how our culture teaches us and indoctrinates us to be heroes. And we mentioned, one of you, the word hero, and I think this mentality, the moment you mention naps and you mention nutrition, it sounds like wishy-washy, no, we’re tough guys, we’re going to go through this without being overwhelmed. Okay, we’re tough. But I agree with you. The mentality of “no, I’m not going to take a break, I’m going to go all the way,” what we need to remember is that, and I’m not sure I’ve emphasized it enough, the load, the fatigue, doesn’t only affect the quantity of what you need to do. You’re doing things differently. To put it in dramatic terms, it’s as if you replace your employee. So you hire somebody who’s creative, who’s exploratory, who’s able to do things, a top gun, and then you overwhelm them, you’ve got somebody who’s narrow, stressed, and opts for the easiest solutions, for decisions that require less deliberation, decisions that are more automatic. So it’s just a different person. So we, I think as a society, based on this book but specifically a security alert, people that work in security should probably consider the idea of, hey, if I’m going to be a hero, I’m going to be a different person, I’m just working as a different employee. So it’s qualitatively different work, it’s not only quantitatively.

[46:38]  That’s super interesting. I really love that comment. It immediately reminded me of something that I read about Woody Allen, which is not relating to that. Woody Allen said that he’s not afraid about dying, he just doesn’t want to be there when it happens. So in that sense, it really is interesting, because what you’re saying, and I really relate to it, is that you hire a person to do something that you feel that they’re the best person to do it, but you’re not getting them enough rest, or you’re not ensuring that they have diversity, or they’re just burning themselves, where you get a different person. We’re seeing people that are not doing the same at work. For example, now a lot of things start to click for me, when I’m seeing a person that started working, they were amazing, they were the best, and at some point they started to decline, and people don’t really understand why, or they don’t think about alert fatigue, they don’t think about what we’ve talked about. So it really is interesting. I want to talk about something and then address a few questions that we have in the Q&A. And maybe Matt, we’ll start from you, and then Moshe on your side. When we look at AI today, AI is a buzzword, it’s a big thing, but the area of AI actually increases the amount of information and alerts you’re receiving. So it’s exponentially bigger, not just because there is a multitude of AIs doing multiple things together, it’s because I can now in a single prompt do the work of 10 people, but I have to do something with that. So, for example, in development I can write 10 times, 100 times, a thousand times more code, and usually it’s not secure code, and then I get more alerts even with fine-tuned tools, as you’re saying. So I want to hear your take, maybe Matt, on if AI will help us in that sense or it will just get things worse.

[49:06]  I think the “just get things worse” part is yet to be realized, so far. I don’t think a lot of vibe code stuff has made it to critical infrastructure production or anything like that right now. Maybe little startups and stuff, but we’ll see. I haven’t seen a ton of good stats on how much vulnerable code is being pumped out by these things. I’m interested, but haven’t necessarily seen an uptick in vulnerable code because of the AI coding tools. I will say I’m optimistic on it helping, especially in the SOC, in the area that we’re talking about right now. First of all, AI is not a superpower, it’s not a magic wand or anything like that. But it is good at certain things, very good at certain things, and one of those things it’s very good at is natural language, being able to interact with an AI the way that we are interacting right now and then the AI translate that to some other syntax and machine and then provide you information. So one of the examples I can give here that I’m excited to see realized as products roll this in: my last job, like I said, I wasn’t a full-time SOC, I wasn’t eyes on the wall triaging alerts all day, but I was a lead incident responder. So I’d get called in when things were on fire. So when you’re not living in these tools day in, day out, you actually kind of don’t have the muscle memory of, hey, what’s the syntax for, I need to look up every time we’ve seen this IP address over the last six months, how do I search that query? And each tool has a different way that you have to answer that question. So you’re in your cloud environment, you have to answer that one way, you’re in your SIEM, you have to answer that a different way. So that’s something that I’ve seen some hints of AI going in that direction of, okay, you now have a tool that I can say exactly what I just said to you, hey, when was the last time we saw this IP address in the last six months, and it just gives you that answer, and you don’t have to remember that syntax at 1 o’clock in the morning when you got paged and things are on fire. So I’m kind of excited about that, the natural language interaction with security information. As far as AI reducing the false positives, could it do some of this level one triage stuff for you? I’m starting to see some startups go there. A lot of tools have been claiming this for years, but they just called it machine learning instead, machine learning your way through false positives. Again, it just becomes another layer to tune: at what level does the AI decide if this is good or bad? Yes, AI is good at adding a lot of context. But I have seen some promise, I’ve seen some tools out there that are using AI to correlate multiple different sources of information and then try to make a more informed decision before it alerts to that SOC engineer. So you’re hopefully getting more qualified stuff, but we’ll see, we’re so early on a lot of this. I think the next few years will be the proof in the pudding on that.

[52:36]  That’s fantastic. Moshe.

[52:38]  You’ll have to remind me what the question was.

[52:43]  So I’ll remind you. What I want to know from you is, in the area of AI, now AI is everywhere, it’s not just cybersecurity, everybody’s using it. I know people that are searching in ChatGPT and Claude and Perplexity and Gemini and whatnot, they’re not opening Google or any other search bar. In that sense they’re actually asking a direct question and expecting an answer from AI. And from your understanding and research, have you started to look into the effects of AI on alert fatigue, or the ability to mitigate alert fatigue in that sense?

[53:23]  No, this is too early, and I can’t say that I am aware of any such attempts. But like Matt, I’m optimistic that AI is going to help. I think it’s like in any other domain, it has to be put to use in the right way. And what maybe the implementation of AI in alert fatigue could borrow from other domains, where such attempts have not necessarily been successful in the past, is domains where, just like here, expertise also entails some implicit knowledge. So if you have to tell me how you are doing, like, I’m new to the company and an expert alert detector, whatever the job title is there, is going to brief me on how to start working, and if he tells me stuff verbally and even gives me some rules, it’ll be great, but this will be half of the job, and a lot of his expertise is things that he cannot verbalize. And tapping that, this will be the critical aspect of how you help implement AI in a way that will be beneficial for you, and not just the things that can be verbalized, and hopefully machine learning can do that.

[54:43]  Yeah, that’s very interesting, because as I’m thinking about it, as Matt mentioned about tier one, if I have some sort of an AI agent that’s connected to all of my tools and it’s going to do what every tier one person is going to do, search for information and correlate it, so really from that perspective I would say that they can take a lot of the load. Getting the AI to share the load, in order to reduce a lot of the alerts that are coming. It’s not going to solve everything of course, but it’s definitely going to be a step in a good direction. I’m also very excited about it.

[55:31]  Yeah, if I can interrupt before I forget, I would look to use AI also for what we talked about in the earlier part of the webinar, which is to detect fatigue. So can you have AI parasite on top of the way I function, to give me alerts that my current decisions suggest that I may be entering a fatigue zone and go home and do some useful things.

[55:59]  I love it. We should have a startup that’s doing that.

[56:04]  Yeah. I got funding just as you said it out loud, funding showed up in my bank account.

[56:11]  Exactly. Yeah. That’s amazing. We have a few questions, I’d like to address them really quickly. From that perspective, as a control of security in my organization, I’m petrified to do significant tuning of false positive and junk alerts in case we miss a critical issue alert, which you’ve touched, Matt. I’m worried a failure may result in outsourcing or company collapse. How do you push past that feeling?

[56:41]  Defense in depth, right? So if you miss one alert, it should not mean your organization has failed forever, and you’re hacked and all the data is out the door. There are a lot of layers of things that have to go wrong before the attacker’s actual goal is met. A friend of mine used to say, zero days aren’t invisibility cloaks. So even if an attacker uses a zero day, a thing that you could not have patched beforehand, they should, once they get in, make a lot of noise, and you should have a lot of other things bubbling up. And so then missing one alert doesn’t matter. Now, did you miss 15 alerts because of all the subsequent behavior that the bad guy did? At that point it’s a systematic thing. And then the other bit, how to dispel this feeling, is just plan and plan and plan, and practice and practice and practice. We brought up the pizza: oh, I had this nutrition and then I was exhausted and I ate the late night pizza. The way to combat that is to plan everything you’re going to eat for the day, stick to that plan. You’re never making the decision while you’re tired, you made the decision yesterday, or you made the decision this weekend when you cooked all week.

[57:54]  I like this metaphor, it’s a really good metaphor.

[57:57]  So same thing. If you are planned, then that one alert, you’re not going to be stressed out about missing that one alert, because you’re not worried about the decision in that moment, you’ve got a plan. You’re just operating according to your plan. And you should be able to swap team members in and out to execute on that plan, so you’re not exhausting anybody. I will say this, and I haven’t mentioned this yet, there’s pre-planning for just standard operating procedures, and there’s also a plan for recognizing that something has failed miserably, and then you need a plan to hit the big red button to say, hey, this person’s burned out, or this process is failing, we better do something right now. It’s the emergency planning, recognizing it and then executing on that.

[58:34]  Exactly. Okay, final question. A lot of teams are now seriously looking into AI advancement, as we talked about, and integrating with their processes. I wonder, is there anything new in the agentic AI space worth noting that may assist teams with alert fatigue? Have you seen anything? We talked about vibe coding, but it’s still extremely early in that sense.

[59:03]  Like I said, yeah, I’m seeing some startups, I’m definitely seeing some startups bubble up. And then the other thing I’m excited about is some of the big SIEM vendors are starting to build some AI tools on this. So just keep an eye on what your tool stack is and the SOC AI type vendors that are popping up that you might be able to kick the tires on. We’re super early on this.

[59:33]  Yeah, I agree. Well, we’re at the minute, and first thing, I’d like to thank you both. It was an amazing conversation. Moshe, super incredible. Matt, thank you very much for contributing from your experience and the examples you provided. This has really gotten me, and I think my audience as well, to think differently about alert fatigue. It’s not necessarily tool consolidation or doing something with the alerts. It’s actually looking at the process, looking at the people, nutrition and sleep and other measurements that personally I knew them but I haven’t thought about them in a way to combat alert fatigue. And I think that this was an extremely interesting conversation. So thank you both for that excellent conversation. Thank you to the audience who’ve been staying with us and asking questions. There are more questions that we haven’t answered, so feel free to hit us up. And of course join the other webinars that we’re planning for you. So thank you very much and have a great rest of the day. Bye.

[1:00:37]  Thank you. Bye-bye.