Unveiling OT Security: Expert Insights and Real-World Stories

Overview

OT experts and runZero customers Jens Baetens (Deloitte) and Tim Pryor (formerly Meta) dive into the unique challenges of discovering and managing OT environments, and the pivotal role that effective asset discovery plays in safeguarding critical infrastructure.

Our panelists discuss OT security strategies and best practices that you can leverage in your own environment:

• Actionable insights into the rapidly evolving landscape of OT security, including key considerations to inform planning and security
• Real-life experiences and lessons learned from securing various environments, such as industrial, manufacturing, and Smart Buildings, in OT
• Analysis of the various ways to get an initial asset inventory with active scanning and passive discovery

Transcript

(00:02) Carolynn Mallozzi: So good morning. Good afternoon. Good evening, everybody. Thank you so much for joining today’s webinar titled: Unveiling OT Security: Expert Insights and Real-World Stories. Today’s webinar is being presented by Huxley Barbee, security evangelist here at runZero.

Before I introduce our two speakers. I would like to cover a few housekeeping items.

Today’s webinar is being recorded. You will be able to share a link – we – will be able to share a link with you after the event is complete, we welcome you to revisit the content yourself at your leisure.

We also invite your comments and questions. Please look at the QA chat box on your screen. If you think of a question for the speakers at any point. Just type it in there, and I will either pose it to our speakers at that time, or hold it for the discussion portion at the end of the event. We are joined today by a robust customer panel.

Our first customer panelist is Jens Baetens, Manager of OT/ICS Cyber Security at Deloitte, Belgium. He is focused on industrial and manufacturing environments at Deloitte, Belgium and has spent the last 3 plus years building a cybersecurity framework for the manufacturing sites of a large automotive company.

Our second panelist is Tim Pryor, former ICS/OT Network Engineer at Meta. After a short tenure at Meta, he is now working with a diverse team at Amazon on an electrical preventative maintenance program.

At this time I’m going to hand the floor over to Huxley Barbee, who is going to start today’s presentation. Huxley, the floor is yours.

(01:54) Huxley Barbee: Alright. Thank you everybody. Welcome to this runZero webcast on OT security. Today we have two distinguished guests that will educate us on some of the critical challenges of securing OT environments.

So, Jens, let me start with you. You’ve worked with a lot of customers to help defend their ICS environments or to help them recover from incidents. And so you’ve seen a lot in your engagements, including how the adversary gains entry into various systems. So interestingly, I was giving a talk about a month ago, and somebody came up to me afterwards and said, “Hey, you know, with these OT systems, these IT systems. They’re fairly old. Isn’t it easier to do buffer overflow than compared to its systems?” And my response was “Yeah, probably true. But with so many ICS systems with default, user names, default passwords and default configurations, it’s just easier to do the easy thing rather than trying something really hard, like a buffer overflow.” But what are your thoughts on that?

(02:58) Jens Baetens: Thanks definitely. I hope you can hear me. Yes, yes, all right. A little bit of technical difficulties getting in. So I think the thing there is that it’s a little bit of a historical question. So historically, a lot of these systems don’t really have security in mind. I think if we pose it as a question, how easy would it be to get into an IT system. If there were no Security administrators would be quite easy as well. So I don’t think there’s really a difference there.

What we do see is that all of these systems were built quite some years ago, or have been running for 10, 15, 20 years. So they were also built before really, this interconnection and convergence of IT and OT. Where we really see that everything is a lot broader in connection now. Everything needs to be talking to the cloud. People want to collect a lot more data. So we really see that these systems are now a lot more exposed, and it’s not always known to those that have the systems that these systems are exposed.

So really see a lot more availability, I think, on the question. Should you exploit them? You only give a bit of the answer. Of course, if we can take an easy route, you can take an easy route. And it doesn’t need to be more difficult than it really is.

(04:19) Huxley Barbee: Yeah, yeah. And it’s very interesting how this OT convergence from, you know, starting like 2005 or so has to really lead to this. So exposure of these ICS systems over the Internet, making it easier for the adversary because you used to have to, just like, you have to walk up to it in order to compromise it right.

Alright. So, Tim, turning, turning to you now your OT experience lies with what is known as FACNET or facilities net, and technically this does not fall under the definition of critical infrastructure and key resources. But nevertheless, I feel like it’s probably just as important to us as some of these other types of OT environments, cause we’re in these buildings. But tell, tell us more right, what is FACNET really? Is it just heating and ventilation? Or is it like something more than that?

(05:14) Tim Pryor: Yeah, no, thank you. I think you hit it right on the head. And this is the biggest challenge is is a lot on saying as well, too, is that these these systems? They’re not secure. It’s a 100% trust network. Right? You went out. You walked out, you programmed something, you plugged it in. You wired it in. Why wouldn’t it work the right?

Mindset of the building when we built it. But you’re absolutely right. These systems are all over the place, right? And then security. You’ve got power. You know. You got fire. You got elevators. You’ve got HVAC, of course, and access control. card readers, video cameras.

You know the meeting gear. A lot of people moved all their AV equipment into the facilities network, or the OT, or IoT, or IIoT, and they’re all kind of. The greatest one I’ve heard now is well, BMS is like, like, heritage. IoT.

I’m like, okay, right? Well, we’ll take it. Yeah. but you know. And we have exercise equipment and gyms and things like that. So you have this incredibly diverse set of equipment that sits inside of a space that people assume right, that it’s secure. And then it’s not important until it is. And it’s and it’s far too late, right? By that point you’re in deep trouble, right?

So yeah, I mean. And then we look at these systems today and Jans kind of spoke about it, too, where they were they were living in these very isolated places. You know, the Internet wasn’t a thing yet. Right? Late, not for most of us. You know, these systems are expensive, and they’re designed to last 20-25 years. And you know the PC world, every 20, you know, data center worlds, 18 months. Right? We’re flipping this thing out. And so there’s there’s a vast difference in these 2 in these 2 worlds where this physical hardware that has no inherent protection. I mean your laptop, your phone, right? All this stuff has some level of inherent protection depending on, you know, username and passwords, firewalls, all kinds of fun stuff that’s in there, right? But this equipment has nothing, and it’s very, very simple to exploit if you get into that space. So yeah, a huge challenge, and unfortunately, most companies don’t see that as being actually more critical in some states, in some depending on what you do right? Depending what your businesses it can be more critical than email in the day to day until too late.

(07:38) Huxley Barbee: Yeah. Your comment about how there’s just a lack of controls on the OT side is spot on. Every time I’m dealing with the OT environment I feel like I jumped into the Delorean with Marty Mcfly, and I go back to the future right? Go back to a world where nothing was encrypted, there might not even be authentication. There are no security controls on anything like so yeah, it makes me feel like a young man again. Alright. So back to Jens here.

You know we’ve all heard about the Purdue model, right? And how risk is supposed to be stratified across these different layers, and how you’re not supposed to really like jump from one layer to a non-adjacent layer. And all this stuff. And in theory, if the adversary were to get to own a PLC and the nested devices. Supposedly they would have to go through those upper layers first to get down to the bottom.

But here’s the thing you know. How often do you see in the field? The adversary directly filtrating the ICS environment rather than rather than you know, going through all the layers on the IT side.

(08:50) Jens Baetens: Yeah, I think that also depends a bit on how you see directly infiltrated. I think a lot of the latest attacks that we’ve seen on the ICS environment through a TV session. A VPN session that is compromised is the question there. In my opinion, that is a direct compromise of the ICS environment. Because if I get access to a VPN or a team viewer session, it drops me into the ICS network. It’s a direct attack on the ICS network. Maybe some IT controls are involved. But, for example, a VPN. But not really, not really that much. I think if we really look at attackers that are interested in these in this ICS space, that’s a very limited group. It’s only the very mature. I think if we, even, you know, look mostly at Nation State detectors that are interested in in attacking ICS controls really by intent.

Others will reach it by accident. But I think the majority of impact is still classic. And somewhere of course, if you have an OT network and it has a Windows 7 PC sitting somewhere it will also get encrypted and you will also have impact on your ICS systems.

That’s really, let’s say the thing we have there, and maybe on that topic as well as going into the ICS network. If you have control over an ICS network as an attacker I doubt that anyone is really exploiting it at this point, because if I would have, let’s say, control over an electricity grid or a water treatment plant. What do I gain with stopping it? I will likely get caught, and if I really instigate anything with this that should be of intent, and I said. In my opinion, the Nation States that might have this access will probably be sitting waiting in kind of a cold war situation.

(10:45) Huxley Barbee: Yeah, yeah, that’s an interesting point, you know. Just speaking, IC systems are so insecure. It doesn’t mean that the adversary is necessarily going to exploit that insecurity, at least not right now. Which to my mind means like things are actually a lot worse than they appear is the way I take it. But you know I’m a little more pessimistic since then than than some.

Speaking of pessimism, Tim. I’m not very familiar with with facilities net that but what do you think is the biggest risk for facilities, or, to put another way. If you were the adversary and you can’t come upon a new facilities network like, what are the first things that you would check for to try and infiltrate?

(11:30) Tim Pryor: I think it’s in a facilities environment, FACNET environment. It’s much the same right as what we would think of a true OT or an IoT network, right? Something industrial in nature, very much the same.

Probably the same reason that something got exposed when somebody asked for an IP address does it need to be outward facing or exposed to the Internet. Yeah, yeah, we gotta go remote into it. Okay, cool. Here’s your IP boom. No rules. Put around it.

That’s where you know. I mean, we’ve done some, you know, discovery before. Look basically looking for our own stuff to make sure of things, you know. But a simple global controller from a Billy automation system can be sitting on the Internet. And of course, what’s the first thing you hit right, default passwords? Boom! Boom, you’re in most of the time, and that’s direct. You bypass everybody, because you have a simple misunderstanding of actually what.

This is a challenge. Right. You have OT folks, which is, you know, especially building automation folks. And there’s some really smart guys. But they’re not. They’re not IT network guys. So they use terminology, meaning one thing, IT takes that terminology under the perceived understanding, and gives them what they’re looking for without maybe sometimes the full, the full, you know the full exposure or the full discussion to happen. And you have things like this happen.

Another beautiful example is, cellular gateways right? I don’t want to throw names out there because we’re not gonna be trashing people, but cellular gateway, right? Whether it’s a backup you know, you have a generator. You got a pump. You got a chiller, its reporting its stuff back to the manufacturer for maintenance and things. And all the right reasons. Right? We? We need to really pull that data. And Jens was speaking about earlier. But these things are far from secure. Right? So that that’s some of the what I would say the typical like kind of innocent mistakes, because IT people are not OT and OT people are not IT.

And those worlds converge daily, and now that the OT or the BMS world lives in the IT world, even if it is its own infrastructure, so lives in that in that the ethos of the Internet, right? And we’ve got, you know, cloud to cloud. We got multi clouds. We got stuff running all over the place and some things happening. And there’s ways to do that. And there’s ways to do exceptionally, exceptionally secure. But I think it takes a lot of time. We don’t see it, because what we talked about earlier and that is the BM OT network, or our facilities network and more generically is not under the, you know, under the guidelines of critical infrastructure.

(14:13) Huxley Barbee: The big takeaway for me from what you just said is this whole interplay between like, what does this business need to operate? And what does the business need to be secure? Right? That interplay exists, you know very much. Still on the OT side, just as it does on the IT side. So you know, that’s that’s that’s very interesting.

But speaking more on security governance. Jens, let’s think about CIS control number one, right? And know what you know what you have so that you can protect it. When you walk into a new ICS environment. How often do your customers know what they have or think they know what they have? And how often does reality match up to their understanding. And do you use active scanning or passive discovery to verify whether or not they’re correct? And when do you choose to use one versus the other?

(15:09) Jens Baetens: Yeah. So I think most of my clients are aware that they do not really know what they have. And those that do think they know what they have will always run into, let’s say, some issues of that. All of this equipment, or whether it’s in building management or whether it’s in manufacturing environment. If you have a machine and it has a touch screen, it likely has at least one computer in it. So if you connect to the network, that might be 3 different devices that show up while you might think, okay, it’s only one. So that’s the first thing there.

If we really think about what doesn’t match up, I said, usually not active or passive scanning, I think, is one of the most asked questions, especially in OT, where the the industry standard or the the most trusted way currently is still to say, Okay, let’s do it in a passive way. Let’s not intervene with anything. Let’s not touch anything because of these might be sensitive systems, production downtime.

But doing a good passive scan does require quite some, quite some effort. You need to have a good overview of the network to really get into all the nooks. Especially with machines that might have unmanaged switches in them. How will you capture traffic from in there? It’s it’s not really possible to to span it back towards your towards your end. And also, yeah, with passive scanning, you have some type of lead time. I think we’ve had facilities where we have to scan, at least for a week, because some of the processes or some of the things communicate only once a week. So where we really have to sit and watch, let’s say, for quite some time.

Active scanning, on the Other hand, gives you the opportunity to move a lot faster, because you can really go and probe out into the network and also really reach all of the ends of it. You can literally say, okay, I have a 10.00 network and will just scan everything. That gives really a good view on on what is all there. It’s also faster in response. If you want to know something that what is connected now you can just scan.

The question there with the ICS devices is usually okay. Will they respond correctly? In my experience in the field most of the devices are capable of responding correctly. If you talk to it correctly, as long as you’re doing full TCP handshakes, sending it expected messages, not really doing vulnerability type scanning or or send scanning only. That’s where the devices usually tend to go unstable because you’re doing, let’s say, slightly less expected things or completely unexpected things, even. But these devices then, will fail, or will be stuck in a resource allocation mode and really give the thing.

Passive scanning, let’s say, is still the safe and trusted option for most organizations, and is very widely used. In my opinion, we should be using active scanning at least as much because it’s a helpful tool, and it gives you much more responsive way or faster way. Instead of weeks, months of scanning, you can go to hours or days to really get those insights.

(18:26) Huxley Barbee: Yeah, and sort of to rephrase what you said about active scanning they’re like people – if you talk to them nicely they’ll talk back to you nicely, right? But that’s a great story about how there’s some processes that run like once a week. So you don’t. You don’t. You don’t know about them unless you’re listening with a passive collector for a link. I actually once had a customer that had an annual process, and so you had to listen for like 13 months before you could be sure that you were getting all the traffic that you needed.

So the same type of question but to you, Tim, but on the facility side. And you know one thing about your experience that’s really fascinating is you’ve not just done facilities net for just one building, but an entire university campus, right? Higher education has many similarities to a city, a municipality, right? So on. campuses you’ll have offices and housing, athletic centers, medical centers, law enforcement, food and beverage, and so on, and so forth. And then oftentimes some of these campuses even have power generation. And these mini organizations on the campus are often they often have a lot of autonomy, so they could do whatever they want essentially, to, to put it bluntly. So with facilities, how often is it that you seen where the network in reality diverges from the understanding of what the model of what the network should be. And, and, and, would you use active scanning or discovery?

(19:56) Tim Pryor: Yeah, I would say almost almost every time, right truthfully, I mean, there’s there’s gonna be the outliers that somebody has been really tight and had great great processes to keep things super tight. But the reality is that through simple, simple things remodels a piece of equipment being upgraded, or simply innocent human behavior. We talk, you know, I plug my phone into the USB port on my workstation in the control room. And now I, you know, I have a problem on the network potentially right. And things like that, or you know, things have been remodeled and added to it. Different stuff has happened. Some buildings are merging. I mean, sometimes in higher education, the government is probably the same way where things happen in phases.

Maybe we’ve got a significant building, a big, big student housing building, somebody, maybe thousands students live in this building, and it’s being remodeled over a 5 year period. You may at some point have multiple control systems operating together within the same system, because you’re replacing an old system with a new system as you remodel. So very challenging, very challenging situations that come in there, and if it’s a really like micro kind of free model, it doesn’t even have commissioning involved with it. So there’s no official turnover or recommissioning of the control systems and communicating to the various teams who need to know what’s going on right? So it’s kind of like a ripple effect there. So you know. We talk about higher Ed and the government also the challenge there too is that there are some of the only building owners, I think, especially in the United States that own a building for multiple life cycles. And we, we, I think the industry is like 50 years right? It’s like a building shot after that. Right? Well, I mean, there’s usually government buildings that are a couple of 100 years old, and and they’ve had controls, and sometimes they have really like unique and incredibly delicate things in them that are very old, right? That are very important to us as a symbol, very important to us the same. So you have a lot of challenges that you can’t just go, you know. Knock down a wall and run some conduit and all kinds of things like that. So all of these things bring in very sometimes very edgy new things, right? New IoT things that you’re trying to work in there and can be a lot of challenges.

And this is where I think that the active scanning for me I like both for 2 different reasons. I like active scanning, because I wanna know, especially on a new build what I have. Soon as that network is lit up, I can put something in that subnet to begin to scan it. I want to know what’s there. I wanna you know what’s there from day one. And I wanna compare that as new things come on and come off. And so the handover process is happening. I wanna see when everything gets turned off and it’s handed over what’s still there, right? Kind of that ‘as found’ and ‘as last’ state, you know, as we talked about. And also that continuous scanning. I have a baseline of where things are, and that’s for my active scanning and Jens kind of talked about it, and that can really depend on your network stability, right? Especially building automation. Things are very chatty. They talk. And some of them are really bad. Right? They just, I mean just seeing a global whois every 6 seconds, we’ll follow who has right? Just just pinging the network like crazy trying to look for things right. So you with active scanning, and how the scanning is done right, you can begin to kind of fluctuate and adjust some of these settings and stabilize a network, you know, and bring it into and bring it under control.

And I like the passive more for the behavioral, you know, kind of response. I’m learning over time what equipment does. Right? I’m teaching this piece of hardware what’s acceptable, and what’s not, and so it’s looking for new connections like “Whoa, hey? This station is also connected to the Internet, who did that?” right? Different things like that as well as as people randomly adding stuff, because the one thing we haven’t really talked about yet, and has that nefarious player who comes in and plugs in a raspberry pi or plugs in this into your network and unmanaged switches which we’ve talked about is historically in this world they’re online, they’re there, right? And so managing switches needs to be stronger, becoming the new norm. And there’s some great stuff about that, maybe a topic for a different day. But yeah, both active and passive. For me personally, have exceptional value. It can be challenging. True, but I think it’s worth the fight to get it there.

(24:35) Huxley Barbee: Yeah, your note about baselining is very important. I think baselining for threat detection is a very hard problem to solve. We may never, never fully solve it. But at the very least you don’t want to be dealing with baselining when, while all you really want at least to start with is to get a full view of all the assets right building that as inventory, so at least at least get asset inventory right? And then you can figure out the baselining for that detection later.

(25:08) Tim Pryor: I think some of our new systems and I think some of our new systems are newer systems out there. So this is more of the facilities kind of world right here. Your BMS’s, your BPM’s they are understanding that their system actually needs to run more like what we would think a typical IP your IT system right? Certificates and things like that on devices, edge devices that are actually running a full kernel that have a certificate. Right? All there are things that are coming. I think the industry’s understanding it now, and how important it is, problem is there’s a lot more of the old stuff, and it is expensive, right? So it’s tough for owners to drop millions of dollars on something that already works. There’s nothing wrong with it, it works.

(25:56) Huxley Barbee: Absolutely. Alright so let’s head on over to Carolynn to take some questions from the audience.

(26:02) Carolynn Mallozzi: Carolynn, what’s the first one? Alright. That was great, and thank you, everybody for participating today. I thought that was a great conversation. First question we have is, what do you think about lateral movement threats jumping from one PLC to another, or from a PC to a PLC? Can go for both Tim or Jens.

(26:24) Jens Baetens: Right? Yeah. Alright. I think maybe to start with the first part jumping from one PLC to another. PLC’s are well, ‘dumb devices’ to put it that way, they follow leather logic. There’s a very limited set of programming, I think the the concept of infecting one PLC, and then having it spread to other PLC’s is something that I’ve explored myself a little bit. Trying to write let’s say, a self replicating worm that goes from PLC to PLC it’s quite difficult, not super practical, and I imagine the people that are smart enough to really do it will keep it for themselves. But should be feasible

From PC to PLC, is definitely an attack factor. That is a lot more exploited. Because if you can get access to an engineer’s workstation that controls programming to multiple PLC’s. If you can from there push a bad config or just click the reset to a default button then you’ll already have a lot of impact. And you don’t really need to impact these devices. If you can just stop their logic or stop these devices. It will immediately cause an impact. Oh, and I just see that the lights are alright. So yeah, that’s a bit. I think. My thought there.

(27:50) Tim Pryor: Yeah, I would agree quickly there. I think the biggest threat is the PC involved, right? Almost always. It can be from something as simple as a USB drive to download try logs. USB 6 by default are they have malware on them. That’s why they auto start. It can’t be scanned. You shouldn’t use them, right. It’s some of these kinds of things that we have to really get in this world of understanding how really vulnerable things are. And yeah, and I would say, the PC, the workstation to the, to the server is the most, the most damaging right.

Cause, the most, PLCs and BMS controllers, I mean, there’s different logic involved. But then they start pumps to turn things on and off right. They monitor sensors. It’s kind of right. And it’s a rudimentary thing. It’s the same stuff, different logic that runs them different speeds that they run at PLC rules and making widgets, you know. 1,000 widgets a minute. BMS runs in 15 min intervals. Right? It’s all about comfort and different things like that. So the speed is different. But yeah, the real risk there is the HMI part of that and that access to the Internet, right? And behavior around there. But it’s a real thing. And and it has to be this where we come into new devices being introduced to the network instantly knowing about it, and then instantly knowing about when a behavior of a device changes. Two different things right?

And then a known problem exists to be able to query against the entire corpus of your asset management, and instantly get a response so you know how to deal with it quickly, right? Paramount in being successful in that space.

(29:37) Huxley Barbee: I think it’s worth remembering that the adversary is gonna do the easy thing. So if there’s a Windows XP device where eternal blue is gonna work, that’s what they’re gonna go for. So alright? Next question.

(29:49) Carolynn Mallozzi: Thank you so much. Next question we have here is the Purdue model enterprise, IDMZ OT still relevant?

(30:09) Tim Pryor: So I would say, all, all, all the models have their plus and minuses, right. But let’s just put it this way. If you can’t see something, you can’t hack it, right? If you can’t find it. If you cannot find it, you cannot hack it, right? So some of these models, you know, I mean, the reality is, things were created in this world. Do they evolve as fast as the bad player? Never right? And we have to be right every single time. So they get all the good and get lucky once in their end, right? And then there’s problems. So I won’t get into that too much.

(30:52) Jens Baetens: No, I think, the Purdue model it has some parts that are definitely relevant. The fact that you have layers and and technically, let’s say, being able to cut off your OT, and then, like midway point and say, okay, everything will still keep operating if we make this cut.

However, I believe that it’s becoming more and more difficult to do this. Probably be running things in the clouds or running like operating like your Scada systems in the cloud like you. You’re now literally connected to the Internet, maybe because even you have more uptime than the physical server if you run it in the cloud, and your Internet connection is literally better than the energy provided to your building. So running your critical infrastructure somewhere off site is interesting.

(31:55) Tim Pryor: Let’s kick it just a little bit right? So the idea of like seeing something in a network that’s misbehaving and snatching that out of the network. If it’s a PC, a laptop, a printer, yeah, even a server. Okay, there’s services that may go down, or something like that. You snatch something out of the middle of a network in an OT world, right? An industrial world. You could actually hurt people right? It depends on what that device is, and it really doesn’t matter what it’s what, whether it’s misbehavior or your interpretation of its behavior is bad, right? The model may be different, and what it’s actually supposed to be doing. BPMD tables are a perfect thing to drive IT people crazy is. You snatch that thing out of the middle of the network, and all of a sudden that device doesn’t allow me to talk to anybody anymore. And you isolate it. You broke everything.

So to to Jens’ point maybe you’ll get lucky, and the right thing is misbehaving, and you snatched it fine.

(32:47) Huxley Barbee: I think to sum it up, Purdue is a nice model. Keyword being model. Right, next question.

(32:50) Carolynn Mallozzi: Thank you so much. Next question, have you seen more implementation of micro segmentation utilizing VMware, NSXT and Cisco cyber vision in OT environments?

(32:55) Huxley Barbee: Or, or, more generally, are you seeing more micro segmentation on the OT side, if you’re not familiar with those specific technologies.

(33:07) Tim Pryor: One of the things Penn State did in full disclosure was not me. I was fortunate enough to inherit this and learn about this from an incredible team. But Penn State was one of the first to actually deploy a micro segmentation kind of model right to the application layer. Even applications that live on the same server cannot run and basically, you cannot connect to it, unless you know, you have the certificate right? It’s like 3.5 on the IP stack somewhere like where the handshake takes place, and that’s really come back to this idea like you can’t see something. You can’t hack it. You can’t find it right. And things are allowed to move within the network. Your East and West with your micro segmentation is very strong because you said this can talk to that, and that tunnel is okay.

And that helps stop the question earlier. And that is that migration from PLC to PLC or PC to PLC, because those devices are not okay to communicate, and therefore they don’t ever right. So we did, my team had done this with hundreds and hundreds of buildings within the network with the right technology, the right application and stuff done there. It can be done very well and be hyper productive right? And and and actually, things like remote emergency remote, you know, control centers in case you’re building burned down. Yeah, find a plug and a cell phone. And everythings’s got cell phone built into it, your online things like that, incredible technology in my experience.

(34:56) Huxley Barbee: Alright, Jens, micro segmentation in OT?

(35:01) Jens Baetens: Possible. Easy to achieve if you have a green field, I think I think if you do brown field implementations, it becomes very, very difficult, mainly because something we refer to in the beginning as well knowing what you have. And then also knowing what talks to what is very important in this case. And yeah, it’s gonna define whether you succeed or not, because if you start implementing micro segmentation, and then something tries to reach something that it should be talking to and is no longer able to, you will have issues.

So there is a very long period of you having to monitor and learn how your network behaves, and what is correct or not, so doing it on a green field implementation is probably quite easy to do or will be easier to do, let’s say, and a brown field implementation will either require a lot of knowledge about the network that you already gained in some other way or will be quite the effort.

(35:54) Huxley Barbee: Yeah, micro segmentation, very hard to implement great return on investment if you can do it. Alright. Okay. So we got time for one more question, Carolynn.

(36:06) Carolynn Mallozzi: Awesome sounds good with the increase of IoT and the industrial environment and the challenge around patching, etc. How much of this increases the attack surface? And what are some of the solutions you can share from both of your experiences?

(36:23) Tim Pryor: Yeah, so the patching problem is right in there with the default password problem. Right? It’s a massive amount of equipment, and it all depends on the facility, right? How big you get 10,000, 20,000, maybe 100,000 devices that have passwords in them.

There are some new forward thinking individuals who are working on software to deal with this problem and automate it, and manage it and watch it. But part of the problem in my experience with that is, or or the feedback that I get from team members and folks is kind of rolls back to the manufacturer, and this is towards the manufacturer vs the people on the streets. Right. I should be able to patch my device without losing the config file right? Number one problem. If I pass the device I’ve wiped the config file, and if I don’t have it, the current version, or I’m unable to actually back it up right in the process to do that. So now you have a risky situation that the recovery from it is exceptionally expensive in time and loss production, and whatever it may be right. Just the time to, I might actually have to rewrite it if the upload doesn’t work right? Which you know never happens, and it can become a significant problem. So then it becomes a risk reward discussion, right? Is actually patching it ‘because it works’ worth the stress factor that I’m gonna shut down, or am I gonna push it to the outside where I can protect it better from the outside and just hope that nothing happens internally.

It’s a complicated problem. that is not, there’s no easy answer for. In my opinion, in the near future not an easy one there. There’s some answers that are coming, and I think they’re getting there. But, a lot of people are afraid of breaking things right because it happens a lot. That’s my experience.

(38:20) Huxley Barbee: One of the biggest challenges in OT is making sure everything’s always available, avoiding outages at all costs. Alright Jens, your thoughts.

(38:24) Jens Baetens: So I think, indeed, yeah, patching is a difficult topic. I think, indeed, the first question is, should we be patching? If everything works and everything is fine, do we really need to patch it? What is the threat of it actually being exploited? If it’s in a vulnerability that gets fixed where someone literally needs physical access to this device to exploit it does it really matter if people do the patch? Yes or no? So there’s a lot of procedural questions, let’s say, should you be patching? Yes or no?

And then I think there’s also the question, can you patch? Some of these devices purchased come with a Windows 7 embedded inside of it. But these devices are kind of like, if you reset them, they go back to their default, programming state, and everything else is erased. So even if you could patch it, you reboot the device, and it’s back to their original state. You lost your patching status. It’s great from a perspective of backup and great from perspective of of availability, because if the machine is stuck, you click the reset button, it reboots, everything is back to standard settings, and everything works again. But from a patching perspective if you made any tweaks to it, someone clicks the reset button. Everything is gone again, and vendors don’t always allow you to make changes to their systems. Except if you say yes, we’ll do it on our own risk, and we will now no longer have any support on your device, which then leads to of course, all of all the risks that you might not want to accept as an organization.

(40:01) Huxley Barbee: On this one I am going to do a plug for runZero. 1. You can’t patch something if you don’t know about it, and number 2, as you’re prioritizing risks, you know should I patch this? One of the very important things as you said, Jens, is this device reachable? So being able to understand the reach ability of that particular device is also very important as well.

Alright. So that was the last question here. So this recording will go up on Youtube and in the description for that Youtube video, we will have links where you can connect with Jens as well as Tim. And, by the way, if any of you are looking for that same full asset inventory that’s helped Jens and Tim in their jobs, both active scanning as well as passive traffic sampling there is a free trial, and there’ll be a link to that as well in the Youtube video description.

Once again, I want to thank Jens and Tim for their insights on securing OT environments. I’m sure that anyone who’s coming into a security team that protects an OT or ICS environment will learn a lot from this conversation. Thank you, gents, and thank you. Thank you, Tim.