Bracing for the Future of Tech with Fastly's Next Gen WAF

In this conversation, Daniel Corbett shares his journey into IT and cybersecurity, starting from being self-taught and working in data entry to becoming a Linux system administrator. He recounts discovering a backdoor password in an SSHD binary and how it led him to transition into a security role. Daniel discusses the benefits and challenges of working with Linux and the importance of passion and continuous learning in technology. He also explains his role as a product manager at Fastly, focusing on the next-gen WAF and its unique approach to web application security. The conversation highlights the flexibility and ease of deployment offered by Fastly's WAF, as well as the importance of balancing information and overwhelm in security solutions. The conversation with Daniel Corbett from Fastly focused on the challenges of alert fatigue, Fastly's approach to evolving threats, simplifying product usage, and the future direction of Fastly.

Fastly: https://www.fastly.com/
LinkedIn: https://www.linkedin.com/in/djcorbett/

Transcript

Speaker 1: 0:00

How's it going, dan? It's really good to finally have you on the podcast here. I think we have been planning this thing for quite a while. At this point, it's a relief and it's exciting to have you on. Yeah, thanks, joe, for having me.

Speaker 3: 0:16

I greatly appreciate it and, yeah, it has been something been being planned for a while. I'm definitely excited to be here.

Speaker 1: 0:22

Yeah, absolutely. I mean, that's sometimes just how the schedules work out and you know everything else. Like that, when you're juggling as many things as both of us are, it's difficult sometimes to find that little sliver in a week a couple of months away. Absolutely so true.

Speaker 1: 0:43

Yeah. So, dan, I always start everyone off with telling how they got into IT, how they got into cybersecurity, and the reason why we start there is to really help out anyone that's listening that's maybe coming from a similar background, right? Because I feel like it's always helpful for people of similar backgrounds to hear that story, to be like, oh okay, maybe this is something that I could do, maybe this can happen for me. So where'd you come from with this?

Speaker 3: 1:13

Yeah, I definitely agree. I just want to start off by saying I guess, that I've 100% self-taught, for better or worse. I dropped out of high school and got my GED. Right out of high school. I was working as data entry at a pharmacy that was taking in orders from an internet website and I was just writing up the labels that would go on those prescription bottles and typing those up hundreds of them per day, and I had an opportunity. Well, I just want to take a step back.

Speaker 3: 1:58

At the same time, I've always just been generally interested at that time in technology and I was using Linux a lot at home and free BSD servers and all of that. But I started Office doing data entry. Then I had an opportunity to go work for the website who was sending the pharmacy. I was working at the orders. I got to go work there. I had an opportunity to go work there doing helpdesk just standard IT helpdesk managing Active Directory servers and exchange and all of that. However, this is just my personal opinion. It's always been difficult to do helpdesk for Windows servers and whatnot, mostly because my biggest challenge was always when you encountered an error in one of those environments and you would go to search for that error, you would have like 100 different possibilities that related to that error and I always felt differently when it came to Linux and Unix-based systems. Where you go to look for an error. You're probably going to find one or two things that are your culprit and it's easier to narrow it down, which just meant easier to fix right.

Speaker 1: 3:16

That's a great point and observation that is so true. Whenever I start encountering an issue with a Windows server, it's like my eyes glaze over for a minute because it's like, oh no, this can go anywhere at this point. There's no telling how long it's going to take me to solve this thing, at least with Linux. I know that if there's an error code, like you said, it's like two things that it could be and it's like you just use process of elimination and try out one of them. The difference is so extreme? Exactly.

Speaker 3: 3:58

And so that was just always for me the difference and so I just always yearned for using Linux and being a Linux administrator. But at this organization it was just all Windows. Again, I was just doing help desk. Fast forward, and I had another opportunity to go work as a Linux system administrator for a hosting company under a corporation that was like had many different entities, let's say, and they had their own hosting company. And so I went to go work over there and I was super stoked because I was going to be working primarily on well, exclusively on Linux. It was going to be my first career or day-to-day job using Linux. And so I went to go do that and it was great and I got thrown in headfirst in a way, because the organization had a lot of high traffic websites and so I went from never administering Linux to administering Linux for high traffic websites. So I got a lot of experience from that.

Speaker 3: 5:07

But I've always just been generally interested in security and looking at different things and I don't know what led me down this path.

Speaker 3: 5:17

But I had one day I was like analyzing. I was looking at the strings command output of their SSHD binary on one of our servers that we administering. Don't ask me why I chose to do this. It was just something that I would do sometimes out of kind of boredom, I guess and when I was looking through this binary using the strings output, I noticed a string in there that just seemed odd. It didn't seem like it belonged, it just stuck out and it actually ended up being a backdoor password to the SSHD binary.

Speaker 3: 5:54

I was able to SSH as route to the local server, popped that string in and I had a route shell and I was just like, oh wow, this is not good. So I alerted my manager and we came up with a way to search all of our servers we had like over 400 of them and we found that just about every server was compromised and had this same compromised SSHD with this backdoor password within it. So at that moment they were asking me if I know how to help them fix that and whatnot, and so they put me in charge of moving more into a security role and helping them to kind of build some secure infrastructure and try to prevent those things from happening in the future. Wow.

Speaker 1: 6:47

Yeah, that is. That's a really interesting find. It's like I mean, you're not going to see that every day, right. We always hear about backdoors. We always hear about these kind of I don't want to call them elusive attacks, but they're the 1% attacks, right. You don't hear about that, and it's kind of like always that it's like fact or fiction, right Of cyber attacks, like what do you see? Most often You're not going to see a backdoor that often. Did you ever figure out where it came from or who planted it there or whatnot?

Speaker 3: 7:28

Yeah, what we ended up finding out was that these were some really persistent attackers and in fact they actually knew more about the infrastructure than some of the people working on that infrastructure on a daily basis. They were very in tune with the network. We used to joke that they had printouts on the wall of the network diagrams and they knew everything and where everything was, and all of that. And in the beginning, truthfully, it was really difficult for us to mitigate, just due to the vast knowledge they had of the network. But we just started implementing some secure architecture and specifically, we were building secure kernels at the time using GR Security, which used to be free, open source hardened kernel, and we were using that. It also allows you to create secure CHRUD environments, so we were moving all of our web properties into these CHRUD environments. We had a lot of tools in place for alerting and logging when something went wrong. For example, like in a CHRUD environment, for those of you who are not aware, you only have limited access to specific commands and it's only the commands that are put into that environment. So, for example, if your application doesn't use Wget or curl and you don't put those in that environment, then the attacker won't have access to those. Well, what we did as well is we took it a step further and we patched our shell command. That sometimes would need to be in these environments and it would alert us when a command would try to be executed that did not exist in that environment. So that was one of our ways.

Speaker 3: 9:11

We had a lot of logging. We were logging everything all the time and to answer your question is like yeah, at a later time the attackers also got frustrated and they reached out anonymously to the owners of the company and they were like hey, we'll help you patch more things if you pay us money. But it took a lot of us catching them and we just kept blocking them. We had all the logs every time they would find an access point. Again, it was a host of company hosting like thousands of websites. Every time they would find some vulnerable code and they would get in, we would have all the logs and we would just go back and undo everything that they did within a couple hours, and so it just became even challenging for them. But it was like continuous cat and mouse game, like everything in security, right.

Speaker 1: 10:00

Right, that is. That's actually pretty wild. It sounds like you almost stood up a completely separate secure environment and started to kind of rebuild everything over in that one Is that that's what you did Exactly.

Speaker 3: 10:17

Yeah, basically yeah.

Speaker 1: 10:19

Because at that point it's like what else can you do? You know, if they're already everywhere? Even if you start patching it, they have access to undo the patch. Yep.

Speaker 3: 10:27

Exactly, and that's what we did. We started bringing up new servers with the Hardin kernel CHRude environments. But one of the features of GR Security as well as it has role-based access control that essentially even allows you to control access at a per application level. So I can say, like this application can only touch these directories, it can only access these IP addresses on these ports, and you can get very granular with that. And so we just started building up, like you're saying, completely new clusters of servers that had our secure architecture that we defined and that was the only way to get rid of it. And we ended up finding forum posts later on from folks who were basically had previously been attacking us, who were they are praising us for our change in security policies and saying that like wow, it's like way harder than it used to be to compromise these sites.

Speaker 1: 11:27

Did that architecture or design start being used across other companies that are in the same space? Do you know or have you heard anything about that? Because I would think something like that, like if you write a white paper on that or something like that, other companies would jump on that immediately.

Speaker 3: 11:46

Yeah, I'm not sure. We were kind of operating a little bit in a bubble at that time and we did have some sister companies that did start trying to implement some things like that. We were mostly just operating in a bubble and this was early, early 2000s, so there wasn't. Nowadays there's a lot more information, let's say, out there being publicized on social media and whatnot. Back then it was just a little quieter. I guess we just lived in a cave, we lived in a bubble and we just yeah.

Speaker 2: 12:22

Yeah, it's a lot more of a community now, exactly yeah.

Speaker 3: 12:26

That's definitely a lack of community back then. That's a good way to put it.

Speaker 1: 12:31

Yeah, it's really. It's fascinating when I talk to other people that kind of got their start with security with Linux. The reason is because with Linux you have to, you're starting from zero. It can be as vulnerable or as secure as you want. That's the hard part, because you have to learn so much when you're a Linux admin. I was a Linux admin in the beginning of my career, without the title of Linux admin, but all I did was Linux. I might as well have been logging into a Linux computer because I'd signed into my Windows desktop and the first thing I'm doing is open it up, a shell. That's what I'm doing. But going through that process and being like, oh, this thing doesn't work because I didn't add this entry into the firewall rules, or this thing doesn't work because SC Linux just blocked it because I didn't make this specific rule for this specific action across this port or whatever it might be Going through that process, it ingrains that information into you in some ways. Is that what you felt as well?

Speaker 3: 13:49

Yeah, I like to think of it as it's like computing without training wheels. It's amazing, and there's an unlimited amount of flexibility within that as well. Over time you get spoiled by some of the. In the beginning I was telling you about the logging information and being able to debug and solve problems. It's difficult to understand some of that in the beginning, but once you get an understanding of those tools, those debugging tools and things like that, and then you go into another environment like a Windows environment, you're like wait, I don't have these tools that I'm used to having available.

Speaker 3: 14:30

It's much different. One of the ones for me in Linux I lived in breathed S-trace. Any kind of issue that I'm having in Linux that I can't figure out through logs, I'm popping in S-trace and Windows. That just is not possible, at least that I know of, and at least at that time I haven't stayed up to date with some of the changes in Windows. I know that it also now has the Ubuntu subsystem and things like that Linux subsystem. It's difficult in the beginning, like riding without training wheels, but once you learn how to ride the bike and you get an understanding of these tools, it's hard to go back to anything else.

Speaker 1: 15:11

Right, yeah, it is very difficult. I even find myself now even deploying some Linux OSes on my home lab and just playing around with it. I don't know what it is. It makes me feel like I'm back in that kind of weeds and actually doing real IT work.

Speaker 3: 15:35

Yeah, no, I hear you. Yeah, I have several servers at my house that I have spun up and I'm always poking on them. I think it's important and technology we have to keep practicing or we're going to lose our touch, so very important.

Speaker 1: 15:52

Yeah, that's a really good point too. Technology, the skills that you build, they're all, for the most part, pretty depreciable. There can be a situation where you even remember vaguely you doing this action, like me configuring Se Linux. I can remember me configuring Se Linux. I can remember going through the research process and the testing process and everything else that I had, but I haven't touched Se Linux in five years.

Speaker 1: 16:27

And so if you put Se Linux in front of me on a terminal and said, here you need to configure for this app, I wouldn't be able to do it. Now I would be able to pick it up much quicker than I did before, but I wouldn't be able to sit there and just dive into it like I used to be able to. They kind of forget that these IT skills are actually depreciable. Like you know me with learning Python like the running joke at this point on my podcast is that I've learned Python about five or six times now and maybe the seventh time it's going to stick. You know like I've gone through these courses, I've done everything that you could possibly do and I just don't use it enough on a daily basis and I can't remember all of it to save my life.

Speaker 3: 17:12

Yeah, no, exactly. And not only that, it's not only the things that you've learned and used, but also, I mean, technology is constantly evolving, right Like now a lot of organizations are moving into, have been moving into not only cloud native environments, but Kubernetes environments specifically, and so now you have to really understand how to work with containers and how to observe and manage and debug and all of that, and so it's just like. So it's not only the skills that you learned that are easy to forget, but it's also the way the technology is shifting and evolving. And staying up to speed with that least like maybe not having like an expert level knowledge or opinion on something, but like at least being able to understand from a high level what folks are talking about when they're talking about certain categories of technologies.

Speaker 1: 18:02

let's say yeah, I guess that's like a hidden benefit of this podcast. I bring on so many different people from so many different you know specialties and whatnot. Like I learned I learned a little piece of something new every single time that I'm able to take that and bring it into other conversations. A couple weeks ago I talked to someone who worked on securing the F-35 when the F-35 was in the development phase.

Speaker 1: 18:31

Oh wow, the government and how they used AI to correct the code, quite literally mid-flight as it was being attacked, to remediate these issues to ensure that the plane wasn't going to nose dive randomly or whatever it might be. And I don't know anything about AI, right, I can spell it and that's it. You know like it's, it's. There is so much to that that, like, when people come on and start talking about it, it's like, oh boy, you know, I'm, I'm taking notes, you know, pretty fervently on the side, just just so I can learn, just so I can look things up later on. But if you don't, if you don't have that passion, if you don't have that you know, that drive to learn this stuff, even just little tidbits, like you're probably not going to be too successful in security, I feel.

Speaker 3: 19:25

I 100% agree. Like for me, I would say, not even in security, just broadly in technology. You know, I always tell people like, because you know there's people who they they're interested in getting getting in the technology, mostly for the money and things like that. And I this is my opinion I've tried to tell people that you have to be really cautious with that because it can be really hard. We've all hit those problems where you're banging your head against the wall for a week or whatnot, and and, and it's really difficult to get through that. And then, of course, when you finally have the breaking point, we're like ah, that's like what I've been looking for, that's my solution. It feels amazing, right, but there's a lot of pain to get to that solution and I think, like I agree with you that like passion is key. Be having that passion to like continue, keep wanting to learn more and all of that. That's how, that's how you like become successful, just in technology in general. So 100% agree with you.

Speaker 1: 20:27

Yeah, it's a really good point. You know when, when, when people approach me and they start talking about, like you know, they want to get into cybersecurity for the money and it makes good money and this, and that you know, yeah, we, we, we make good money. But you got to understand the stress and the pressure that is on these positions. You know I used to have hair when I started in security. I had hair, right, I don't have any hair now. That's a huge thing.

Speaker 1: 20:55

Like I remember, you know I had just gotten into security and I had just started dating. You know my wife at the time and you know she she mentioned to me one time she was like, hey, like I don't, we don't even hang out, like we're here together but you're working the whole time. And I was like you know, what are you talking about? Like we're hanging out. You know she's like no, like you're on the computer, you're doing this stuff. Like you know, when are we going to actually, like you know, spend time with each other?

Speaker 1: 21:26

And it like it kind of, like you know, creaked open the door a little bit for me to be like, oh, okay, maybe I need to step away from the computer or not not worry about this issue so much or, you know, handle it more efficiently. And even today, right Like it can be so extraordinarily frustrating, maybe not even from a technical problem perspective, but from getting buy in from the right people, from having you know the right conversations with the right people and catching them on the right day right. There's so many dynamics to this field and it's like if you're not willing to go through that, that fire, you know you probably should not like waste years of your life getting into this field because when you get into it you're going to be so miserable.

Speaker 3: 22:17

Yeah, I could definitely relate with what you're saying. Even I think working remotely kind of also exasperates some of what you're saying, because when you're working in an office people start shutting down at end of business. You start seeing the signs of everyone's packing up and leaving and then you pack up and leave the office. But I work remotely from my home and so I don't always have those signs, which sometimes leads me to staying at my computer probably a little bit longer than I should, because I just enjoy what I'm working on.

Speaker 3: 22:51

I'm working on solving hard challenges, but even outside of that you're mentioning your partner, like hey, you're not paying attention to me or whatever. You're focused on your computer. But even outside of that, even when you step away from your computer, sometimes those problems they come with you in your head. So whatever you were trying to figure out, you might be at dinner and you're still thinking about that same problem. But yeah, I could definitely relate to those challenges about, like, just stepping away from the computer, because when you're passionate about what you're working on, it's hard to do that. I mean, I am guilty as charged of that.

Speaker 1: 23:27

So yeah, I have found maybe one of the best ways for me to kind of detach is to do like really hard cardio. You know I have to go and like run for an hour and somehow that like rinses out my brain and allows me to like do other things other than whatever problem I was faced with, for you know, 10 hours right. But you know, dana, I want to dive into what you're doing now, where you're at now. You're with Fastly and I think you have a pretty interesting, you know, role there, right? Can we talk a little bit about that?

Speaker 3: 24:02

Yeah, absolutely so. I work at Fastly. It's a. We have three different product lines, but I work specifically within the security products I work on.

Speaker 3: 24:16

Fastly has what we call a next gen WAF or web application firewall, helps to protect websites from common attacks and vulnerabilities and things of that nature. But specifically, I'm a product manager on the security products team and I work on the core engine of the WAF. What that means is I work specifically with the engineering teams who are responsible for improving our overall detections, working on the detections being able to detect SQL injection and cross-site scripting and all of that. We also have this concept that we call signals within the product, so it would be that allow you to tag traffic that match certain patterns. We also have rules, and so I work on anything that we call a signal or a rule in the detection engine. So I focus a lot on that and you know, because I come from a very technical background, it has actually really helped me in this role because a product manager, one of the primary responsibilities of a product manager is to prioritize.

Speaker 3: 25:29

We can do all these things. 10,000 things are in our queue or in our backlog. Which things should we be working on now and what should come next and what does that look like. And so the fast lead next to an WAF is extremely powerful, but also technical, right, because it's doing WAF in a different way than most competitors. Every other WAF on the market usually uses regular expression pattern matching and we don't. And so because of that, it could be difficult to prioritize if you're not familiar with the technical nuances that are being presented to you. Right, the engineering team will come to me and say, hey, we have these three things we could do. Here they are. Which one do you think we should do first? And I'm able to talk through that with them and actually establish a priorities list on how we should work on these things. And so, yeah, that's kind of what I'm doing now, just mostly focused on the detection engine within the WAF.

Speaker 1: 26:29

Yeah, that is. You know it's really fascinating how you guys approach the WAF kind of domain of the security stack. Right, because the legacy WAF deployment is, you know, when I think of a WAF and when I try to picture it in my head, it's this appliance that sits at the edge in front of your applications that you're now going to have, you know, a significant amount of configuration with it, right? I mean, I've heard of some WAFs taking 18, 24 months to deploy, to actually get working in front of the applications. And then, you know, because you have so many hiccups and issues along the way, your development teams and your app teams they don't really want it and so they get pretty hesitant with it.

Speaker 1: 27:22

You know, I had the opportunity to actually test out the FASTLE WAF fairly recently and it's a totally different way of thinking about a WAF. I can't explain it any other way. You know like your WAF solution technically works with other WAFs even you know it's like oh, if you have AWS WAF, that's great, you could still throw FASTLE on it and you get an added layer of protection which is a completely like foreign topic. You know like I find that I have to explain that to other security professionals, because we're so used to that legacy deployment, right Of you know, it's this one-stop shop for a WAF solution. It's terrible to work with. We have a whole team that's dedicated to it, right?

Speaker 1: 28:17

I mean, I worked at a place that had 12 people on their network security team and I think eight of them focus on the WAF alone, right? So let's talk about how you do it a little bit differently, because I do actually think that that's really important. Like, yeah, you know, fastle is obviously, you know, sponsoring this podcast or whatnot, this episode, and but but it's not about, it's not just about that, right? I actually dug into the tech and I like it and it's really interesting and it's a different way of thinking about it. So, you know, let's talk about what you guys are doing differently, really, with the more agent-based deployments.

Speaker 3: 28:57

Yeah, you know, we, we one of the biggest pain points like when, like you're you're saying, when it comes to managing WAFs, is the sheer amount of false positives that security teams are dealing with on a daily basis. I mean, I in a in a previous career, I had to spin up other WAFs, like you're saying, and in fact we became so frustrated with it that we just turned off all the rules. We wrote our own. We wrote our own that would just focus on exact attack patterns and then we would just broadly block those things. We didn't, we just wanted to focus on, like you know, like roughly like the Etsy password payloads and things like that, and you know it. And we found that was working better than the 3000 rule rule pack that we are given. And you know it was just too cumbersome. Right, we spent and like as security, you know, teams, we want to, we want to get deeper into the attacks. We want to understand the attacks, what, what's the infrastructure that's being targeted, you know what is the attack surface, look like all of that. But if you're spending your whole day, you know, playing whack-a-mole with false positives, that's not fun at all.

Speaker 3: 30:07

And so, you know, fastly's WAF came from the acquisition of a company called Single Sciences and it was, you know, a long time in the making, where, and I've actually I coincidentally ended up working as a product manager on the product. But I was a fanboy of this technology, going back to like 2013, right, so for me it's like I've known about this technology for a long time, I've been a huge fan of it, and so it's like my dream to be working on this product. And you know, essentially we, the WAF, looks at attacks contextually and it basically parses out, it tokenizes the input and, based on the tokenized input, it's able to determine whether that's a valid SQL injection attack or not. And this is, like really important. And you know, just to like level set here, when I'm like when we're prioritizing, like expanding our detections or looking into false negatives or false positives, because I'm just going to put it out here clearly, no WAF in the world is perfect. I will never claim that our WAF is perfect.

Speaker 3: 31:19

There's a concept called WAF efficacy, which measures your false positive rate against your false negative rate, and things like that. We actually have an open source tool that helps with that. But when we're looking at improving our WAF and, you know, maybe we identify some false negatives that we need to cover, or suspected false negatives, the first thing that we look at is whether the attack would actually work. It may look bad, but what does it work? Is it a valid attack? And we have a lot of vulnerable environments internally that we use for verifying this. And that's where we start, because the problem is is, once you started counting for things just because they look bad that don't actually work, that's your slippery slope to false positives, and so that's how we keep our false positive rate low we focus on real, valid attacks.

Speaker 1: 32:12

So yeah, that's another thing that I didn't really even mention was the false positives. Right, with a solution like this well, not with a solution like Fastly's, but with a solution like a WAF you know, as soon as I hear that, I'm thinking, oh, this is going to take a lot of work to tune it right. And anything that gets around that is kind of seen as like snake oil. Almost you know like it has to be proven out. Almost you almost have to like beat someone over the head with it. You know to to like get it into their head like hey, this is how it's done. And once I figured out that I don't have to learn rejax again for it, I was, you know, I was over the moon for it. You know like learning rejax is like my least favorite thing. I'd rather try and learn Python for a seventh time compared to learning rejax again. Like that is such a terrible thing to put on.

Speaker 3: 33:19

Yeah, for sure, and I've. You know, everyone has had to deal with it and it's a it's. It's a bit of a learning curve, though you know, I I deal with customers coming onto it every day who they almost have PTSD from, from other WAFs. And they come on and they're like I just got done tuning my other WAF for eight months. How long is it going to take me to tune this and things like that? And you know, or they come in and they're like where are my 3000 rules to manage? And so it's another. It's, it's just a different way of thinking.

Speaker 3: 33:53

And then the other thing that's like kind of helps differentiate us in the market is like, most of the time with WAFs, when it comes to blocking, it's a binary decision. You go to blocking mode, that's it. It's either block or allow the traffic. But with our solution out of the box, we we employ a thresholding approach where an attacker, we have to observe a certain number of attacks within a period of time to cross a threshold before we start blocking those attacks. Now some security professionals, they hear that and they they're like what, that's crazy, the WAFs going to let an attack through? Oh, you know, but it's not, it's not. It's not necessarily that you keep this threshold base blocking on all the time. It's to get you to blocking mode faster Because otherwise, like you said earlier, you're in logging mode for 18 months and you're just scared to impact revenue.

Speaker 3: 34:45

You're scared to turn on the thing, but with a threshold based approach you could turn it on very quickly and you know that someone has to send a certain number of things identified as an attack before they get blocked. That gives you time to build your confidence in the product, and not only your confidence, but your development teams confidence, your network engineering teams and anyone who's responsible for that infrastructure. Gives that time to build the confidence while being somewhat protected. And then, as you build the confidence, you start lowering your thresholds and lowering your thresholds until you say, hey, we're good to go, let's go instant blocking, we finally built our confidence. But otherwise another WAFs is just like. You go to blocking mode and it's a lower block and that is very scary for many organizations.

Speaker 1: 35:27

Yeah, when you start talking about blocking mode, you know that's a quick way to get the CISO a heart attack. You know, because to them, to the application team, to the devs, right, it sounds like it's just going to block every single thing unless you make an exception for it, which you know with a lot of WAFs, that's what it typically is and that's why it takes, you know that, 18 months to really refine it and whatnot. I think I was at a place where their WAF had something like seven or 10,000 rules, right, and my question to them was like, how in the world do you even manage it? Like, how do you know what rule to create that you haven't already created, that is going to do something differently than what your other 10,000 rules you know aren't already doing? You're not catching by now, you know, and that was a very real issue, and that's, it doesn't even take an extremely large organization to create something like that. You know, like I think, right, like Ford is over half a million employees, right, it doesn't take a Ford level size I was just looking at Ford trucks, that's why I'm thinking of Ford but it doesn't take a Ford size company to create that sort of, you know rule set right, which is really refreshing.

Speaker 1: 36:52

But you know, I want to talk about, like, the deployment methodology too. The reason why I want to do that is because the cloud is potentially an environment that is something completely different from what people are used to, you know, with legacy on prem data centers, and the reason why it's so different is because you could have ephemeral workloads. You could have serverless workloads, no code, low code, you know all these new things that pose a lot of really hard security scenarios. But with the Fastly WAF you guys are able to really just deploy it wherever the customer is, wherever you are. You know, you guys kind of meet us where we are, which I always thought was extremely valuable because you know it's different when a vendor meets you where you are, rather than you having to change things and, you know, do a different DNS name and all these other things, right. So, like, what was the thought behind that? Were you thinking through like, hey, we won't be able to keep up with the cloud deployment methodology if we don't go this route, or was it something else?

Speaker 3: 38:06

Well, you know, we just realized that every organization is different and some organizations want you to host the entire WAF and do the DNS change, like you're saying. But some organizations they want to host everything, they want to manage everything yourself. And you know, we just felt like if we're calling ourselves the next gen WAF, we can't be limiting you to the deployment types that like what we would call like a legacy WAF would limit you to. And you know we have some customers who they deploy both some of their architecture. They want to do the cloud based deployment. Where we're hosting it, they point the DNS and they'll use that to protect their North-South traffic, right. But then they may also be in Kubernetes and using service mesh and they want to protect that East-West service traffic as well. So in that case they'll have the WAF installed locally within their Ingress controller or reverse proxy within their service mesh to protect that traffic and they'll have that hybrid where again, the cloud is protecting North-South and then the on-prem agent is collecting East-West, protecting East-West. So you know, we just have this like motto, that and it's you said it similar to how I like to say it, but it's like we want to be allow our customers to deploy in any environment or architecture of their choosing.

Speaker 3: 39:31

And it's like I say that specifically because, like, not only do we support, you know, every kind of like load balancer or reverse proxy from like Envoy proxy and HA proxy and Nginx and Apache, we even support IIS right, you can deploy us within IIS but then we go further than that, where we've added support for AWS serverless Lambda, and then, if we can't find an integration point, we have our own reverse proxy or we allow you to hook in through your code. You know we support a wide variety of code languages and integrations through libraries. And then I mentioned architecture, because we've started releasing support for ARM. We're seeing more and more organizations move to ARM 64 for cost savings, specifically in cloud native environments, and so you know we just want to be in the environment or architecture where our customers are, and it's just important, right? So we actually get asked this a lot and it is a huge differentiator for us in the market.

Speaker 1: 40:32

Yeah, I would definitely say so. Even you know just the thought behind you not having to make a DNS change, right to utilize this product, because a DNS change, that's the easy change that you get around. You know that, like I'm a terrible hacker, you know like I am not good at hacking, but I can get around a DNS change Like it's nobody's business. You know, and that's not an impressive thing Like there's hackers, you know, listening to this podcast they're going to hear that be like, yeah, anyone that can spell the word hacker can get around a DNS change Exactly, yep, that's a huge, that's not a small. You know configuration change that you can't just, you can't just overlook it. You know, like that's something that's major. That is that comes with these legacy solutions that you typically have to do. That people start to get, you know, a little bit nervous because what if you're using a different CDN? Right? What if you're using you know some other DNS provider and it's just a hassle, right? You're really kind of removing all the difficulties with actually deploying this technology. Exactly.

Speaker 3: 41:49

Yeah, and then you know it even allows you to employ like a multi-layered WAF approach. You could have a cloud-based WAF and then also have the on-prem thing to protect in case someone goes around. But having both would give you like one protection closer to the edge, closer to your end user, but then also having it locally in case someone does bypass it that you bypass through a DNS change, that you can block them there. But then that's why, when you start talking about, like you know, like in on-premise install, you know historically WAFs that are installing on-premise you mentioned, it's like an appliance, whether it's a hardware appliance or a virtual appliance, and they just don't fit within a cloud-native world or that. You know it's going to be difficult for me to integrate my virtual appliance within my Kubernetes environment that could protect my containers, you know.

Speaker 1: 42:45

Yeah yeah, it gets pretty crazy, like, especially with the cloud, because it's so diverse, you know, and that's the thing that I couldn't quite wrap my head around at first was the fact that, you know, every cloud provider has their own WAF solution, right, and so most cloud customers start with that cloud-native WAF and then they start to explore other solutions and whatnot. And so, you know, the biggest thing for me was okay, we already have this cloud-native solution right. To me, as a security professional, it's terrible, it's absolutely horrible. I refuse to work with it. It is not that fun, right.

Speaker 1: 43:29

And I kind of stumbled upon the fact that, oh, I can easily deploy Fastly alongside this other solution and whatever that other solution is missing. Fastly is going to catch it. And all these other attack avenues that this, you know more static WAF is, you know, kind of catching for or looking for. It'll catch those. But then all the dynamic stuff that's going around, that static rule set or that static WAF right, we're calling a new term here you know, Fastly is going to catch it, based on the deployment model, which I found to be really interesting. You know, like, as a security engineer, that's not something that you typically see, especially in a product like this.

Speaker 3: 44:19

Yeah, I definitely agree. It's 100% unique in that way and it's just like another avenue for dipping your toes in the water right, checking it out, being able to easily download it, trial it, see how it works for you and how it fits in while you go through even a transition period where you're like well, I want to migrate from this WAF to this WAF, but I don't want to be like unprotected in the process and I want to figure out because, even if, like you know, ultimately, like within WAF, waf generally focus on protecting, like you, from attacks and anomalies and things like that, but always within your organization, you're going to have some business logic rules that you want to protect your organization from and these are something that the WAF doesn't know your business logic that needs to be implemented and so while you're porting those things over, that business logic over, you don't want to have any gaps. So being able to run side by side or in line with each other is super helpful for that transition period.

Speaker 1: 45:23

Yeah, absolutely. That's the biggest thing, too with security engineering when you're trying to deploy a new solution, figuring out a way to not decrease your security posture but potentially even increase. It is like it's. One of the most difficult things that you can do in this job is because there's going to be times when you take out one tech and you put in another right, and you shouldn't be configuring this next technology for 18 months before you put it in, before you're considered fully secured.

Speaker 1: 46:05

Why am I losing a security posture that I've worked hard to establish? I'm losing that for the next 18 months as I get this thing up to speed. And with something like a WAF, if you're trying to, let's just say, migrate over to another solution, that's still pretty difficult because you're not getting the real attacks most of the time. You're not getting the real attacks because you have this other legacy WAF that you're migrating from in front of it, and so you're kind of guessing on the configurations that you have to make. So it absolutely makes the migration phase a whole lot easier and quicker, in my opinion.

Speaker 3: 46:49

Yeah, yeah, absolutely. And one of the things that we've found is, as you know, a lot of times with legacy WAFs you have two modes usually Either A I'm overloading you with information that you don't even know what you should look at or not look at. Or B the WAF's not providing you any information or very little. And you know, we have found like a balance in the amount of information we're able to provide, and I mentioned that because we have customers who will be transitioning to our WAF, who they're like oh wow, I am seeing things here that I didn't know were happening, and but I'm also not being overwhelmed with the information, like it's digestible, I can understand it, but they were like my other solution I just turned off the logs because it was just too much or they didn't have logs and weren't providing me anything, and so we kind of found a balance between how much we're showing you to not overwhelm you but also give you insights into some of the anomalous traffic that may be coming to your sites and applications.

Speaker 1: 47:57

Yeah, that's a huge thing too, you know, because not everyone can have a giant security team with several people dedicated, you know, to this one solution and responding to the alerts. You know, I was earlier on my career, I was working with a it was another network appliance, sort of like a SIM, but somewhere in between, and the amount of alerts that I would get within 24 hours it was several thousand. Right, and we're a financial institution. So we have to have a reasoning and a logic behind each and every single one. You have to have, you know, reasons why you're quarantining this incident while you're flagging it as a false positive.

Speaker 1: 48:40

Everything because the auditors, they would go, you know, thoroughly through our systems and make sure that we were doing it, and I mean that alone would take one or two people full time, you know, to just go through and even then you're not doing a very good job of actually responding to the alerts, because you know you're not just going through alert fatigue, but now you're just going through, like you know, this numbness of alerts where it's like well, I've seen this cross site scripting alert, you know, a thousand other times. It was benign every other time. Why would I think that this one is legit. You know, like that's the, that's the issue that you start falling into, and so whenever I see a solution that really manages how many alerts I get, like it's. It's like relieving to me almost right, because I went through such such trials and tribulations earlier on with those thousand thousands of alerts a day.

Speaker 3: 49:39

Yeah, absolutely. And not only that, but it's like one, it hides things that might be important for you to look at from a threat perspective. But two, it also hides things that might be important from a false positive perspective. Again, you're like overwhelmed with information and you can be accidentally blocking legitimate traffic, impacting your organization's revenue, and not even realize it right. In some of those cases you're waiting on an end user to get in contact with your support and try to convince them that they're being blocked by your policy somehow, and that's just super painful, and so it'll just take you longer to identify that. But you're right, it's like that. Alert fatigue is very real, very real.

Speaker 1: 50:25

Yeah, absolutely. So where do you think Fastly is going in the near future, maybe a little bit farther out right of developing the solution? The reason why I asked is because this is already a solution that is redefining how you think about WAF overall right. So where are you guys going in the near future and where are you evolving?

Speaker 3: 50:53

Yeah, there's a few things. One, we're always on the lookout for evolving threat vectors and attack methodology and things like that. One of the things that we did is, when the log for shell vulnerability and exploit came out, we released a virtual patch to our customers and, to be quite transparent, our core product doesn't use regular expressions, but we do leverage regular expressions in these virtual patches because they allow us to get a protection out to our customers very quickly. But, like we saw with log for shell, it was evolving by the minute on social media and so behind the scenes we realized we were like, hey, this is a perfect opportunity for our detection engine, which we call smart parse, to be able to detect this, and so we expanded smart parse to have the ability to de-office gate those log for shell payloads and distill them down into their most basic form, which allowed for simplistic detection. So I say that because we're always monitoring for new threat vectors where we can expand our detection engine, smart parse.

Speaker 3: 52:05

We're also always just looking for ways as well, because sometimes it's not only around the detections but simplifying the usage of the product right, having something that's simple to use and powerful is immensely valuable and I've been focused on a lot of that, on some of the work that I'm doing is we've been trying to stand in our customer's shoes and see what is it like to lower thresholds, what is it like to get to instant blocking mode, what is it like to implement a rate limit rule and all these things, and so we've been taking that in and working to simplify the product.

Speaker 3: 52:44

But we also have an internal security research team, who just is an amazing team, and they're constantly doing a variety of research that we're able to then take and pull into the product right. So we're just like we're trying to stay on, maintain a balance between improving the detection engine, making sure it stays cutting edge, but also making sure that it's just easy for our customers to use, because everything we talked about in this conversation and the points that you noted, like you appreciated, about the product all boiled down the simplicity, simple to manage, simple to observe, right and all of that. So that's kind of where we're focused is how do we keep making this powerful but also maintain the simplicity and usage and observability of it as well?

Speaker 1: 53:41

Yeah, it's really interesting. It's going to be interesting to see where the product evolves over time. As a tech person, right Like that's always something that I'm looking at or I'm thinking about. Well, dan, unfortunately we're at the end of our time here, but before I let you go, how about you tell my audience where they could find you, where they could find fastly, if they want to learn more? I'll obviously have the links in the description of this episode, so if anyone's interested, then go ahead and check it out.

Speaker 3: 54:15

Yeah, thanks, so you can find fastly at Fastlycom. If you're interested to follow me on social media, currently you can. My handle is Daniel Corbett, underscore. Alternatively, if you just navigate to my personal website, which is D Corbett, d C O R B. As in boy E, as in Edward T, as in Tom T, as in Tomcom, you will find my personal website with some of my contact information and I enjoy, greatly appreciate you having me on the program. It was an absolute pleasure.

Speaker 1: 54:48

Yeah, absolutely. I mean, it was a fascinating conversation. I always love it when we can dive into a product or an area right, and talk about these different nuances, because it's not it's not every day that my listeners get to hear that from a perspective of someone that went into this with that legacy mindset, right? Well, yeah, I hope everyone listening enjoyed this episode and if you're interested in learning more about fastly, go ahead and check out the links below.