Security Unfiltered
Security Unfiltered
Navigating the Cyber Threat Landscape with Chris Hale
Ever wondered what it takes to stay one step ahead of cybercriminals? This episode, featuring cybersecurity expert Chris Hale, promises to unravel the complexities of safeguarding digital fortresses while sharing invaluable lessons from the frontlines. Chris’s journey from a help desk technician to the founder of his own cybersecurity firm is nothing short of inspiring. His early interest in computers, paired with a dual major in Exercise Sport Science and Computer Information Systems, laid the foundation for a career that would see him tackling email viruses at Sports Authority and defending against sophisticated malware and ransomware attacks.
The conversation shifts to the high-stakes world of incident response teams, where Chris recounts a harrowing ransomware incident caused by the absence of two-factor authentication on a global admin account. The relentless effort required to handle such crises, including long hours and meticulous post-mortem analyses, underscores the critical role of managed service providers (MSPs) and managed security service providers (MSSPs) in maintaining robust security practices and compliance. Chris’s firsthand experiences highlight the importance of hands-on training and continuous learning, offering listeners a realistic glimpse into the demands and rewards of a career in cybersecurity.
We also navigate the evolving threat landscape, discussing the necessity of quarterly audits, penetration testing, and consistent security practices across global enterprises. Chris shares insights into the importance of continuous cybersecurity training for all organizational levels, using tools like Breach Secure Now to keep security awareness sharp. The episode wraps up with a discussion on the recent CrowdStrike update debacle and the challenges of choosing reliable Endpoint Detection and Response (EDR) solutions. Through Chris’s expert lens, listeners gain a comprehensive understanding of the current issues and best practices in cybersecurity, making this episode a must-listen for anyone invested in protecting their digital assets.
Follow the Podcast on Social Media!
Instagram: https://www.instagram.com/secunfpodcast/
Twitter: https://twitter.com/SecUnfPodcast
Patreon: https://www.patreon.com/SecurityUnfilteredPodcast
YouTube: https://www.youtube.com/@securityunfilteredpodcast
TikTok: Not today China! Not today
How's it going, Chris? It's great to finally get you on the podcast. You know, I think we've been planning this thing for a while, but I'm very excited for our conversation.
Speaker 2:Yeah, me too. It's been a few months. We've been going back and forth and finally getting on here. It's been really good.
Speaker 1:Yeah, yeah, absolutely so. Chris. You know, I start everyone off with telling their background, right, what made you want to get into IT, what made you want to get into cybersecurity? And the reason why I start everyone off there is because there's a portion of my audience that might be trying to get into IT or security for the very first time. Right, and hearing everyone's background, maybe it lines up with someone else's and they can say, oh, you know what, if he did it, maybe I can do it too. So where did you get your?
Speaker 2:start. What was that like? I was really into computers as a kid. I mean, I was around back when the old text like texas instrument machines were out there trying to trying to uh code stuff in to make a little guy dance on on the screen when I was a young kid and then college kind of got into. I was an exercise sport science major and then also a computer information systems major, so I was a double major, got on with used to be guard brothers, became sports authority in their help desk.
Speaker 2:Starting off the beginning, did some AS400 work for them, moved into a network engineer and then started with some managed services providers, kind of learning all the different types of technology out there. After working about six years of that, I started my own company and the security piece kind of just came into because our clients really needed it and we were kind of we started in 2010 and security was important, but not to the level it is now and we just, I mean, as time went on, it was a necessity to learn and get into it, really understand it, to keep up with the needs and actually a lot of the the threads that were out there. So yeah, yeah.
Speaker 1:It's. It's interesting, you know, because the the one thing, that kind of that, I pick out right when, when you were just describing your background, is that curiosity, right it's. It's kind of that that. It's like that hunger that can never be, you know, fully fed right. It's an interesting part of our dynamic in cybersecurity where, like the successful people that stay in cybersecurity for long term, you know they're very curious, they really want to understand how something works, what makes it tick right, and I feel like that's a critical piece for anyone to get into cybersecurity. Would you agree with that? Or maybe there's other pieces.
Speaker 2:No, I think you definitely have to be curious. You've got to be curious, you got to be up for a challenge and you've got to really want to do a lot of research and a lot of looking into things, investigation, all the pieces that go with that and I think I think it lends coming from straight it and security kind of lead into each other, because back in the day, uh, having to look up what would cause an issue, you didn't understand what was going on.
Speaker 1:I think that's led into the security piece, where you start to understand the different pieces of trade, craft and different things that uh, the malicious actors are doing and the things that they're they're using out there to make things work in their favor against the average user, the average company yeah, when you were, when you were in it and kind of going down that security path right, kind of leaning towards it, was there ever an event or breach or something of that nature that really kind of, I guess, sparked your interest or kind of opened up your mind to it?
Speaker 1:I'll give you an example from my own right. So when I was trying to get into security and I was on help desk, I started reading about the complexity of Stuxnet and how Stuxnet was able to, you know, wait for the perfect time and then it would strike and make it look like nothing ever happened and then, you know, basically get rid of itself throughout the network. Right, that, that complexity, that kind of logic behind it, is really what sparked my interest in security. Like, oh wait, if that's possible, then it's basically endless possibilities, right? So how do you stay on top of that? And the curiosity part kind of played into that. Was there anything that kind of sparked your interest or desire in that way, your interest or desire in that way?
Speaker 2:I think some of the big things it was the early days of the different type of of male viruses or whatnot, where they were sitting up mask mailers and things like that, and I believe we had like one of those happen with sports authority where we had to be deeply involved figuring exactly what happened, how we were getting it, because they were getting blacklisted all over the place, and that kind of started me going a little bit down the security path, because it actually was.
Speaker 2:It was more that I was tasked with figuring out what was going on and how to get that situation and then later on, as we went further down, got more into the malwares and the ransomwares that came along afterwards and we've had some really interesting, crazy uh situations we've had with clients that have really been, while the time, very stressful but very uh illuminating and very uh. I mean, to be absolutely honest, it's interesting and kind of fun to go through those things. It's super stressful, but once you get out of it and everything like that, I mean it's really really kind of fun doing the investigation piece and shutting down the channels that they're using to get in and those kind of things. So I think it was really just the process of dealing with the first small issue that came out really sparked me in wanting to get into the bigger ones.
Speaker 1:Yeah, that is. It's really interesting. You know, I try to always, when people are asking me if they should get into cybersecurity or what it's like, you know, I almost kind of try to talk them out of it, right, because if I can talk you out of it in just a conversation, then when you get into an incident or you know a major investigation like that, right, you're going to be convinced pretty quickly like, hey, this isn't for me, this is not what I wanted, or anything like that. Right, you're going to be convinced pretty quickly like hey, this isn't for me, this is not what I wanted, or anything like that. So I try to save people the time in the front end, right? What kind of advice do you give to people that are trying to get started in security in IT if this is the path that they want to go?
Speaker 2:down. You got to be willing to work some long nights and weekends for sure. I mean that's one of the pieces I don't think people understand. I know we've had employees that have come on not understanding on-call and the pieces that you have to go into, the different work that has to be done after hours. When you start talking about cybersecurity incidents, you can be all hands on deck for a few weeks straight across. So you've got to be willing to sacrifice some free time and things like that and you've got to have a hunger to learn. Those are the big pieces.
Speaker 2:I mean, we had an incident a couple of years ago where we had one of our major clients that got hit. They had a South American company. Well, they have a South American branch and they had a South American MSP working with them that didn't 2FA their global admin account into their 365. And they got in and they were able to push out ransomware to all the machines and it was just a huge, huge mess. But it took us solidly for all hands on deck, including the majority of their IT department.
Speaker 2:Three weeks we were working. Weekends we were working I think my group was sleeping three-hour shifts where we were sitting for three hours, then swap out and then going through everything and getting to the point where meetings were just so messed up I mean, they were using Intune to push out stuff all over the place and so it was a massive, massive issue and we were lucky that we were able to to catch it when we did and we had some things already put out there that that stopped some of the news from being pushed out and things like that. But it definitely was a huge issue and I think one of the things you got to be willing to do is sacrifice some time. I mean, you can't just walk away from an issue like that and be like well, I worked eight hours, so I'm going home, so yeah, yeah, that's a really good point and I feel like it's different.
Speaker 1:Time requirements are different depending on the domain in security, right? Like incident response. If you're on an incident response team and you're working for an incident response based company, I mean you can just expect, you know weekends are going to be destroyed, birthdays, holidays, things like that, right, but you're not going to cut your teeth in security at a better place. So there's a trade-off, right? Like you know when you're younger, that's probably the best place. Like you know when you're when you're younger, that's probably the best place that you can possibly be, because when you leave there, you're going to have so much experience, you'll be able to see so many different things and you'll have touched so many different environments that you know that experience will pay off. It'll be worth its weight in gold, I guess yeah, 100 trial by fire.
Speaker 2:I think that goes for it as a bolt. I think that's one of the pieces you got to be willing to do is put yourself out in the unknown and go out and learn through experience versus, uh, I mean, you can read every book there is out there, but until you actually put yourself in that situation and deal with the things, you're not going to learn it the same way. I mean, I I cut my teeth 100% on massive server outages or cybersecurity attacks that, like going into it you're like, oh crap, I don't know enough to deal with this, and then by the end of it you've learned quite a bit.
Speaker 1:Do you have a process for because it sounds like you're drinking from the fire hose quite a bit. Do you have a process like an after action report, almost like something like the military does right after a mission, where you kind of sit down and digest what happened?
Speaker 2:and everything like that. We always do a post-mortem after everything we do. We do post-mortems after almost everything. We do it after onboardings, do it after incidents, we do it after any kind of major failure out there so that we can look at it and see what we could do better. Um, we can, we can change our documentation to adapt to maybe some of the places there were holes, those kind of things. So, yeah, we always do it post post-mortem on everything.
Speaker 1:So I guess we're kind of dancing around it a bit, right. Why don't you tell us you know what, what the company is, you know what you guys specialize in and everything like that, and we'll go from?
Speaker 2:there. So we are a managed service provider. We do IT and base level cybersecurity for small to medium businesses. We actually work with a couple of primary MSSPs, which is just the security providers themselves. So we have some companies that work with them that do some of the larger stuff. But we can do the stuff they can do. We're just not specialized in it the way that they are. I mean, that's all they do. We do also just the normal IT stuff. We do consulting, business planning through technology, those kinds of things. We do a lot of stuff for the construction industry right now.
Speaker 2:But one of the big pieces we are largely involved in is compliance, cybersecurity documentation, compliance documentation. We help a lot of companies get their SOC 2 and do their SOC 2 audits. We do a lot of audit situations. The big piece that we do work with a lot of people on what we do with our third party vendors is like PCI things like that things. We're going to need a little bit bigger, little better tools and a little bit more education. On the PCI compliance piece of it. It's a little bit more difficult. There's a lot of pieces to it, but we go and we do instant response for all of our clients. We've done some instant response outside of that, if we have to go outside and work with the third parties but we don't always have to do that A lot of times we're able to handle it ourselves.
Speaker 1:Yeah, you know from from what you were saying previously. Right, with that MSSP not enabling 2FA on the global admin, that's a pretty egregious security violation, right? How do you keep the MSSPs in check if you're a customer? And then, if you're a third party like yourself, how do you handle that? Because there's a lot of companies out there institution and they had an MSSP that, to be quite honest, I hated, hated every experience I ever had with them.
Speaker 1:I did not want to get on the call with them ever and God forbid I escalate something to them. You know, it's just a terrible experience overall, right, but my biggest issue was how do we know that they're doing what they're claiming that they're doing and that they're actually making our environment? You know more secure that they're doing what they're claiming that they're doing and that they're actually making our environment. You know more secure that they're actually doing their job. We have no way of checking that. Um, how do you, how do you keep them in check and ensure that they're not, you know, creating more security holes or gaps in your environment? Um, we do audits.
Speaker 2:So we do quarterly audits on everything and we we have a we have a pen testing tool that we use ourselves, that we go out do a full audit. We do an audit of the 365. In fact, that global admin had been caught that week, that it wasn't MFA and we had brought it up in our security meeting and that was one of the things that they were going to go back and make sure it was set. They'd already gotten in by that point. When you look at it, I think one of the big issues, especially when we start talking about holes in security, is a good hacker is going to get into your environment and stay there for a long time before they really do anything that you're going to really detect directly. I mean, they're not going to kick off ransomware day one. Well, they might. A bad one will, but a good one's going to be out there for a while. They'd been in for at least two weeks by that point and then, once they figured out that we knew they were there, that's when they started to kick everything off. So I think the auditing is the real good way that we do that. We try to do quarterly. Sometimes it depends on the client. If we have to, we'll do more that client.
Speaker 2:For a year we did monthly auditing after the attack to make sure everything was set the way it was supposed to be. The good thing about using somebody else like a third party is you're not checking your own work. Somebody else is checking your work and you're checking their work a lot of the times. So we're doing audits on our own. Also on the outside, we're making sure that everything fits within what our standards are, and so that helps a lot within that. But so far we've had very few problems with anything. Any MSSP that we've ever worked with. This one was not within our control. It just happened to be that their South American group weren't using them.
Speaker 1:Yeah, it's always interesting how global companies kind of dice up their IT departments and figure all that out. I feel like there's no right way to handle it necessarily. But all of the ways that I've seen are like they're fairly poor, right, like you always try to go with, like the lowest bidder, because you'd have this insane you know bill right at the end of every month if you don't go with the lowest bidder. But then when you go with the lowest bidder, you're going with a local company typically, and if it's a local company in a, in a, you know, I guess, a lower standard country, right, you're not getting the best talent, you're not. You're not, um, getting the best, the best security or the best resources and whatnot, and so it's, it's a, it's a double-edged sword almost yeah, I think there's a lot of uh, a lot of that mindset comes from a lot of these other countries.
Speaker 2:You pay a lot less for your licensing, you pay a lot less for everything, so then when you pay for your services, you pay a lot less too, and and it just all correlates together. I think, yeah, that's one of the big mistakes they made. I mean, after that attack that that particular company was, when we first started with them, they were like their network was a mess, everything was a mess. So we basically got them up to a really secure point and we'd gotten Intune rolled into them to be able to roll everything out and do all the pieces that needed to be done. The irony of that whole situation is they hadn't rolled out Intune to South America, so only their American sites got hit by the attack because that's when they had Intune. But that's what the, the hackers, use, and the other piece that that comes along with once you're attacked is now they're always a target, they're constantly attacked all the time, so they have to be extra secure yeah, I, I feel like that is.
Speaker 1:That's something that a lot of companies you know misunderstand. Almost right. Once you get hit, once you know you're, you're that that target is kind of increased on you, because now they know, hey, we got, we got a ransom word and we got this amount of money out of them before I bet we could do it again, you know, when they come back six, seven, eight months later and they're doing it again. So if you don't, if you don't just patch the holes that they use to exploit to get in, you know, and you have to do a significant amount more on top of it, right, because now you are the directed target.
Speaker 2:And then you add in the piece that they got. If they got in and were in for any period of time, they've gained intelligence. They know who is who. They know the email addresses. I mean. They can use much simpler attacks. They can do spear phishing really super easily because they're able to get in there. They can do brute force attacks on passwords. They can buy a password list and find that particular user is on another account and then you figure out what their password is there. There's all sorts of these pieces that flow there that we deal with and have dealt with for the last two years. We're shutting down 365 accounts and PCs. We work for a while there pretty much every week.
Speaker 1:What does the threat landscape look like? Just for the cloud? Overall? Right, Because it seems like more and more of IT is moving to the cloud overall. Right, Because it seems like more and more of IT is moving to the cloud, but that really it increases the threat landscape in a lot of different areas. Right, we touched on one where an MSSP didn't configure the global admin account correctly. I mean, for a cloud security expert, you know, like myself, right? I mean that's the first thing I'm looking at when I go into any environment, that's literally the very first thing I'm looking at. But for them, they didn't know it, they didn't act on it, right? So what does that threat landscape look like now?
Speaker 2:I think it's pretty high. I mean it's a pretty nutty landscape, but I think there's a lot of good tools like a SASE environment, things that you can put over the top of your cloud environment that allow you a lot more security and allows you to do things like network zero trust and a lot of pieces where you're breaking things down that you're mitigating the access the user has, therefore mitigating the access that the attacker would have, doing that through different types of technologies that you can put over the top of your cloud environments, and I think that's a big one that's helping out a ton, and it, I mean the great thing about cloud is it allows you to work from anywhere, allows people to do all sorts of things. Companies can don't have to have people just from the same state. They can have them all over the world. You just have to have the things to protect it. And I think they're just easier to attack if you don't put in the right buffers and fun things, if you aren't putting in the right technologies and the right well, it's technologies, it's policies and processes and education. Education is the biggest piece. You're not educating your employees what a phishing email looks like or not to put out a code or not doing anything like that, then that's your first point of failure. That's the low-hanging fruit. So you could have all the technology in the world and somebody goes out there and puts out the wrong thing or allows somebody in, you're going to have some problems.
Speaker 2:I mean, some of these things like SASE will prevent that, but there's pieces that you could definitely get in. That's why, on one side, you mitigate their access to just the things they need. They need access to these folders, this printer, this, so that you can't get that lateral spread going across your network and your cloud. But we just had a client and it wasn't a direct cyber attack, but they had a client they had. It wasn't a direct cyber attack, but they had a client. They had a vendor that was attacked. They got banking information for them, called their financial manager and he gave away their passcode into their bank and they drained their bank accounts. And that was just a couple weeks ago, so it wasn't like the most technical attack in the world, but yep, did that. So I mean wow, yeah.
Speaker 1:Yeah, I've seen it where, uh, you know, the attackers will target, like, the executive's assistant or something like that, and really do a good job of impersonating the executive and have them, you know, transfer a certain amount of money. It's uh, it's frustrating, right. And with that, you know, like you said, the training becomes critical. The training becomes, you know, really the backbone of everything that you're doing. Have you found a good training platform or a good you know certification path to go down that provides, you know, the right amount of training, the right, a good you know certification path to go down that provides you know, the right amount of training, the right amount of you know background that these that these people in these roles need?
Speaker 2:Um, there's a couple of different ones. I mean, there's the ones that we use just for employees as a whole, that are that go across the board, which I think are the more likely ones to have a problem, like Breach Secure now, and then I can't remember what the other one was that we use off the top of my head. I think there's some decent stuff. You can go out there to train some of the higher-ups. I think the higher-ups tend to be the more—it's either the really low-level people or the really high-up people that tend to be the ones that make the biggest mistakes and tend to be the ones that let out the most information information. I think what we just we basically do is we do some training ourselves and we use some of the tools that are out there like Breach Secure Now and things like that that go out and actually send out training videos, send out fake phishing emails and we can tell who we need to train to and things like that.
Speaker 2:The big piece of it is it's just got to be top of mind awareness. It can't be something you just train once and go away with. You've got to just constantly be pushing this into people's heads. I think I just got a phishing email. While we were on this that popped up that was some sort of here's a payment via ACH. I don't know who the company is, so I'm not clicking anything. So those things pop up all the time and you've just got to teach people to stay away from them and understand what they are and the type of problems they can cause yeah, I, I always uh, have issues, you know, trying to find the right solution, right that that trains people up and that have you found a good solution to to do that?
Speaker 2:I think just uh, good is relative. We found ones that definitely help, like I said, like I mentioned Breach Secure Now earlier, it's good, it's simple. I think the biggest problem you have training people is they can't be long-winded training sessions. Maybe something short that they understand doesn't affect their productivity, because people aren't going to do them if they're long and take forever and so they're kind of worthless. On that point, um, I think you just have to understand the human mind and how it works and the attention spans are short. So I always look for things that are short and to the point and can get things done, and they're a lot. Of. Them are kind of like cartoony videos that are three, four minutes long and get people through that and then they go through their little testing piece and then we send out the phishing tests and see who really listened to it and who didn't.
Speaker 1:Yeah, I, I know, when I used to, when I used to run the phishing tests for a company, I would, I would. I would probably be a little bit unfair, but it's not like the attackers are going to be, you know, fair with it trying to say like, oh, we can't send an email about their 401k. It's not like the attackers are going to be, you know, fair with it trying to say like, oh, we can't send an email about their 401k. It's not going to be. That's not fair. You know, to mess with someone's retirement or anything like that, they're not going to do that.
Speaker 1:And I, my boss, would get so many complaints about, you know, my phishing test, saying like, oh, that was unfair, that wasn't right for him. To you, you know, create this phishing test that used my 401k and everything else like that. Or talked about my bonus during bonus time. Um, you know my response was okay, well, are the hackers gonna count that out? Right, because if they're looking at us, they know when the bonus is hit, they know when bonus time comes around. I mean, you could take a wild guess and say it's the end of the year and you're probably right. Are they going to, you know, say that that's unfair. They don't want that money, they don't want that. You know, that 20 grand bonus, that 50 grand bonus, right, it's a, it's a double-edged sword, right, and my, my boss, would constantly have to, you know, kind of kind of weigh the battle on either side, right, kind of go back and forth on it. I guess. Yeah, I guess I didn't last too long with testing, with doing the phishing test, but I felt I did a great job.
Speaker 2:Well, I think cybersecurity generally gets a lot of pushback. I mean we always walk the line of productivity versus security. I mean you got to be able to work and be able to do it, but that's the same idea. Like you've still got to push the envelope a little bit and make sure people understand how much these things can be attacked because you can have. I mean, we've had clients where the employee was attacked because they couldn't get all the issue for the company, but they were able to get the banking, get in and get banking information on the employer and they've been attacked and had to do a bunch of things. So it's in their best interest not to, and especially if you start talking about people that are working with HIPAA environments and things like that, they could be held for the things that go down and they don't really. I mean people don't understand all those pieces that go into it. So, yeah, I think the more brutal you are with your testing, the better it's going to be in the long run for the users.
Speaker 1:Yeah, it gets them used to seeing you know questionable things that are asking for a little bit too much or asking for weird things that they shouldn't be. You know, you know what's a good path to get into incident response. I know we kind of talked about you know the experience side of it where you go into it fairly early on in your career or whatnot. But is there different certifications or different training that you're looking for to bring someone in? Uh?
Speaker 2:CISP like uh, there's a couple of, but there's no real good way to get a full fledged cybersecurity certification like that. I mean you can do like NetPlus and those are good places to start to be on a low-level piece of it, but to get into the major parts you're going to have to have experience. I think about the CISP is like five years I think, I don't know. It's hard to get into the major incident response pieces of it. But you start with your NetPlus, you get in with a company that's willing to put you on maybe an incident response team, moving up from help desk to something where you're a helper within that level you're going through and you're involved in their tabletops. We do a lot of tabletop exercises with our clients, which is really helpful and it's good to get some of the lower level people involved with those so they kind of understand your documentation and things like that. But it's really just letting your getting into an it department and letting your manager know that you're interested in cyber security.
Speaker 2:Get that net plus, security plus done out of the way and then start working your way through and get the experience, because the experience is going to be what's key to getting into some of the major pieces and then becoming like becoming a security officer for a company that's I. That's a pretty good job. It's a little stressful, but if you can manage an incident response team, it's a pretty good situation. You're in there making the decisions on how things work. They're a little bit of a pain in the butt for the IT people sometimes. But yeah, I mean, you just got to work your way up there. I think experience does better than any kind of certification. If you have a job that allows you to get some experience, that's the key way to go.
Speaker 1:Yeah, I always approach it from like a multi-prong approach right, like getting that experience, getting the certifications, getting the right education right. I want to eliminate as many questions in that hiring manager's mind as possible, you know, with doing everything that I possibly can to get the right skill set, to get the right knowledge and whatnot to move up right. It's just, it's an interesting market right now right, where I'm hearing from a lot of people that you know it seems like a lot of companies are posting open roles but no one is really hiring for them, and so trying to find ways to break down, you know, those unseen or unspoken barriers I think is always helpful.
Speaker 2:I think on the other side is the hiring companies. It's hard to hire somebody security. I mean we hired last fall we hired a security person, brought them in and they were terrible. I mean they were, they had all the certifications, everything, but they just they like weren't real world, experienced enough to really to do the things we needed to do. I mean we had a uh, we had a phishing attack on the executive team for one of our larger clients and my help desk guys were 10 times more helpful and 10 times better than he was.
Speaker 2:In fact, we got a complaint from the CEO on that guy just because he didn't know how to handle things and he wouldn't do the things we needed to do. He just wouldn't. He wasn't, wouldn't do like shutting off accounts right when they needed to be, and he wasn't. He wasn't super, his customer services skills were really terrible and he didn't know how to communicate these pieces. And you're just like holy crap, so we had to get rid of him. I mean finding somebody that can do the job and knows how to talk to people, and I mean you can't, can't be a developer and be a security engineer. I mean you're gonna have to talk to people you're gonna be dealing with people in their high stress situations, those kinds of things. So I think it's hard to hire in that position too. It's not like a job where you don't talk to people, that's for sure.
Speaker 1:Yeah, yeah. I feel like that's a part of the job that a lot of people don't realize is a part of it, right, when they're trying to get into it. I remember when I was in help desk and being in that stressful situation where you know the application that you're providing help desk for is a critical application. Right, it's a 911 software that overlays your PBX and gives you enhanced information and whatnot. And so usually when people are calling you, it's a system down system degrading situation. It's already high stress.
Speaker 1:Maybe it's the very first call in the morning and maybe it's, you know, your only call of the day that it just goes all day long. Um, and you know they're they're not, they're not happy most of the time, they're not very kind, most of the time they're not. They're not happy most of the time, they're not very kind, most of the time they're usually yelling at you, inducing that stress right off the bat. You know, in a situation where you know you need to be on top of your game and you just started your day, right, it's monday morning, you know and uh.
Speaker 2:So either monday morning or friday afternoon was my memory.
Speaker 1:What was it? Uh, crowd strike, push that update at like 4 pm on thursday. Or what was it? Crowdstrike pushed that update at like 4 pm on Thursday, or something. It's like, guys, what are you doing? We were lucky.
Speaker 2:Only one of our clients had just the CrowdStrike on them and it was only something like 30 machines. We were lucky. I remember my operations manager. His comment was imagine being in charge of a thousand machines at this one, because you're going to touch every one of them.
Speaker 1:Yeah, so yeah yeah, I, I mean there's. There's airlines out there that are still recovering from it. Yeah, and the only, the only reason southwest was unscathed is because they're still on windows 3.1 or something like that too cheap to buy crowdsourcing.
Speaker 2:It's not the cheapest piece of software yeah, I'm sure that it.
Speaker 1:Uh, it made a lot of people question if they're going to have auto updates on. You know going forward, yeah, because, like you know for for for a security person to say, yeah, let's have auto updates on you know, all the time for an edr software that's a big thing and so you really go through and you put that solution through its paces, trying to build that confidence. You know, right off the bat so you can have it on auto update as quickly as possible. With other solutions, you know you do other things right when you kind of have, you know, test groups of systems where the auto update is turned on and everything else is not. Well, now, you know, I was even just thinking about this earlier today for my day job where it's like, man, should we really even have that thing on Because we have like 600,000 machines?
Speaker 1:in our environment. What if you know? What if someone you know has that enabled and the group is larger than like 2000 machines? What are we? What are we going to do? Right, yeah.
Speaker 2:We're probably calling an incident response company at that point you are with like 600,000 machines with that crowd Cause you had to go in and boot that with the machines into safe mode to get that resituated. Yeah, so I mean it was. I mean mean it took my guys on 30 machines. It took them all day on friday to get that fixed and that's just. I mean that's why I'm touching every one of them and sometimes, like, a lot of these people are remote so you had to call them and walk them through how to do it oh man that's brutal yeah because you're not always getting the most technical person.
Speaker 2:Yeah, man, what do you?
Speaker 1:think, what do you? What do you think is gonna come of this? You know, because delta already you know so that they're probably gonna seek. You know charges or not charges, but you know payment for the hours spent and whatnot. Right, what do you think is going to?
Speaker 2:happen with this? I don't know. I think it's a. It's a. It's a weird situation got to figure out like, what was their due diligence on? Like, how bad, how well did they test this update? Like, are they just pushing out things? They shouldn't be pushing them? I think it's going to come down because it's not like. This hasn't happened with windows before, like microsoft this used to be a problem with microsoft. All the time you push out a critical update, next thing you know servers are blue screening all over the place. So I think it's going to be interesting to see how that shakes out in court and see who's who's at fault. And I mean, they're suing both microsoft and crowdstrike is what I read yeah, it's.
Speaker 1:You know it's a weird situation because CrowdStrike is the number one by far EDR on the market, right, so it's like, okay, who else am I going to go with? Am I going to go with SentinelOne, like, am I going to go with? I mean, at that point I don't even know who else to bring into the picture, right, because Carbon Black used to be a big player. Bring into the picture, right, because carbon black used to be a big player and ever since they got bought out three, four times, they haven't really been a player in the space at all. I mean, you try and google them and it goes to another website, right, like there, there used to be, you know, cyber reason in the mix, but that kind of died out fairly quickly, and so you have all these companies going with a top tier product, right, like I can tell you for sure we sold it internally as it being the premier edr solution.
Speaker 1:You know, 70 of the companies in the world, like are using this thing, right, um, a hundred percent of the companies of our size are using it. So, like, where do you, where do you go, you? You kind of you take this outage and then, wherever you go from here. It's a downgrade and so now you're dealing with an inferior product. So it's like yeah such a.
Speaker 2:It's a headache it is, I mean, but something like this. I don't know if you remember the Kaseya breach that wiped out all the MSPs that they got breached because they were throwing out updates to the people that were running non-cloud instances. They didn't put the right update out, but Kaseya survived that and I thought they were done for when that went down Because they got something like I don't know, thousands of companies ransomwared through that whole thing. Wow, I mean, they got something like I don't know, thousands of companies ransomwared through that whole thing. So, wow, I don't know. Yeah, you can definitely have a mistake made and still survive and it's just going to be.
Speaker 2:It's going to take a little bit of time to repair, but CrowdStrike's phenomenal. They're expensive and that's one of the key parts of why it was big companies getting hit with it and not small ones, and I think one of the big issues. I think that comes down that those people create their entire security stack through CrowdStrike. So their SOC is CrowdStrike and everything is CrowdStrike and then when something like that happens, then they're completely screwed and if they want to go out and find somebody else, they've got to replace all those pieces with multiple different vendors. So I don't know. We tend to diversify our stack a little bit just for those reasons alone. But I think CrowdSt we tend to diversify our stack a little bit just for those reasons alone. But yeah, I think CloudStrike will come out being okay, but it's going to take a little while of repairing. We'll see how the court cases get ugly. That might be what hurts them, because that's going to change the public perception on things.
Speaker 1:Yeah, yeah, delta's I think it was their CEO was saying you know, you never hear of Apple having an outage like this or anything like that. But at the same time, apple isn't providing the same product. You know they're. They're providing. They're providing. You know, end user, customer centric. You know products that anyone and everyone can can buy right, and they're. They're not creating server os's anymore. They're not creating server OSs anymore. They're not creating servers. They used to, but that was like 15 years ago and so it's a totally different ballgame. I'm an Apple fanboy, I'll admit it. I have an Apple phone, laptop. The only thing that isn't Apple in my house is my desktop, because I game on it. When I told Apple that, when I interviewed for them probably a decade ago, they were like why don't you game on a Mac? I'm sitting here like guys. Are you that out of touch that you literally don't realize that your platform is not for?
Speaker 2:gaming. Apple has had its problems. Os X has had tons of problems where they released a new version and all of a sudden, three quarters of your apps don't work, and so we run into that. I mean, anytime we're dealing with any kind of updates on their operating systems, we always tell our clients like, hey, wait three to six months, because you go on there and all of a sudden your QuickBooks isn't working and it's a primary thing that you're using. That's not going to work out well for you.
Speaker 2:So Apple does have those issues. It's just not as big of an of an issue. I mean you can always go back to your previous version and things like that. But everybody, everything, has its issues. But I think, yeah, it was just such a huge hit with CrowdStrike. I mean it stranded people, banks were down. I mean it was pretty nuts. So I think that's where the big piece comes. There, I mean, you start talking to like I've never seen this apple. Well, apple has its problems. I mean ios. There's a reason why you back up your phone before you go to the next level and I mean there's all these little things that pop up, I mean, and so they happen. It just, uh, I think what? What hurts crowd strike? Is they forced the?
Speaker 1:update out. That's a huge thing as well. You know, like even on my phone it hasn't happened in a while. But you know, I think it was related directly to like the Pegasus exploit or the Pegasus malware Right, where Apple was like you're updating your phone right now and there's no option to disable it or anything like that. It makes me nervous, then, because it's like all right, well, how much testing did you put into this? If you're forcing it on us? It kind of sounds like you wrote it three hours ago and you're like works good on my phone, let's just push it.
Speaker 2:Yeah, exactly, I mean, that's what happens a lot of times too, and so you never know what other little things it's going to break. I mean, I'm doing the beta with ios 18 right now and it was like, yeah, I might not, I don't know, I'll do it and checking it out, I always want to see the newest, the latest and the greatest and play around with it. So I, uh, right now at work, I run purely on pcs, but I, for the longest time I was writing, was running a Mac with a parallel running a VM when applications wouldn't work within the Mac environment. So I'm a Mac fan. I've used a lot of Apple. My wife works for Apple.
Speaker 1:Do you think companies will rethink their tech stack after an outage like that? Will they start maybe looking at Linux a little bit more seriously or other architectures like Oracle, potentially?
Speaker 2:Even Macs. None of the Macs got hit by the cross-track issue that same company I was talking about. They run half well. They run about a third Mac and two-thirds PCs because their developers use Macs. So none of the Macs can hit. So that was I mean. You can see, apple could definitely take advantage of that and the Fortnite CrowdStrike runs great on Macs. So one of the big pluses of CrowdStrike is how well it does on across platforms.
Speaker 1:Yeah, it's going to be interesting to see, I think, where the court case goes and how effective that is against CrowdStrike, right, and it'll be interesting to see the changes that take place in the industry, even from the outcome of that court case, because I feel like a lot of other. You know security vendors that are critical pieces of software in anyone's security stack, pieces of software in anyone's security stack. You know, I think it's easy for us to forget where these applications or where these agents sit on our systems, right like they're, they're like the most critical part of the system. If this thing has issues, your entire system has issues and I feel like people are kind of just now reawakening to that that logic.
Speaker 2:Well, I think that comes back to like secure, like productivity versus security type situation. I mean you can't overdo it on the security side and have issue, and that security has issues. And they say you know you have no productivity, so that's been a big piece. We run, we run an application. It's a zero trust application that just does't. I mean basically only allows applications that are wireless to run on the machine. And with developers we have a ton of problems because there's all sorts of things that can cause it to flag things and not allow it to run. And so we've had that productivity battle all the time. There can be a switch. There could be just a switch within some line of code that will cause it to freak out and block it. So, yeah, what were you gonna say? I would just say it just. It's that that security versus productivity piece, and that's, I think, what. What we saw with ground strike is how deeply it affected productivity. I mean it stopped half the world. I'm in one big shot. Yeah, yeah, it stopped like half the world.
Speaker 1:I'm in one big shot. Yeah yeah, it stopped like half the world, excluding, you know, china and Russia, where it's been probably right, exactly Well. Well, chris, you know this has been a great conversation. Before I let you go, though, how about you tell my audience where they can find you if they want to reach out and connect, and where they can find your company?
Speaker 2:You can come to our our it's technologyresponsecom. Our phone number is 720-420-1589. Or you can reach out to me, either at info at technologyresponsecom, or you can email me directly, chris at technologyresponsecom.
Speaker 1:Awesome. Well, thanks, chris for coming on. I appreciate it. Yeah, absolutely Well, thanks everyone. I hope you enjoyed this episode.
