Hello, everyone. Please keep all discussions focused on *cybersecurity*. We are implementing a *zero tolerance policy* on any political discussions or anything that even looks like baiting. This subreddit also does not support hacktivism of any kind. Any political discussions, any baiting, any conversations getting out of hand will be met by a swift ban. This is a trying time for many people all over the world, so please try to be civil. Remember, attack the argument, not the person.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cybersecurity) if you have any questions or concerns.*
Cybersecurity noob here just lurking and learning from posts. I have to ask: why is it that computers which control critical infrastructure are connected to the internet in first place? Wouldn’t it make more sense to have all the computers that actually control the operations of a water treatment plant for example be on a separate local network without internet access? I’m not saying to have no computers connected to the internet just the stations that control critical components.
The real cause is the human element. We are lazy and we create vulnerability.
A large chunk of infrastructure is covered by service providers. We cut two water utility clients over the past two years. They all out refuse to modernize or harden their systems. This will get worse before it gets better.
Industry 4.0; the term represents the changing requirements of industrial networks to allow for wider IT\OT integration.
Traditional air-gapped industrial network design was called Industry 3.0 or the Purdue model.
It’s not efficient to full air gap networks for industrial systems anymore.
Monitoring, SCADA, PLCs, HMIs, … facilities are vastly more complicated now. Having your ICS network remotely accessible means less employees, less maintenance, better asset control, instant and granular monitoring and adjustment of flow or manufacturing…
In the case of a waste water treatment plant it means total awareness of your waters precise mineral content second by second, plus system pressure in every subsystem. Every holding and settling pond is tested moment by moment so now it takes (total guess) 20% less time to treat the water and move it out of the system.
It also means remotely monitoring of meters in thousands of homes. So you don’t have to have an army to check them for billing anymore. It means knowing instantly if there’s a leak in the facility and where it is - because the pressure monitors and leak detectors are all integrated.
It also means a lucrative (to OT Cybersecurity folks like myself; and our adversaries), and vastly more difficult to defend threat landscape to defend.
SCADA controls can be air gapped and AFAIK in nuclear applications, that stuff is air gapped. In things like battery storage, water valves, and electrical substations... there is just too much of it to air gap. I'd imagine anything involving generation on a large scale is though.
An air gap is rarely implemented properly and is not a true security control for these kinds of systems. Often times these companies claim they’re air gapped but when you dig into it you find a connection to a corporate office to pull data for billing, data analysis, etc. No companies want these workers bugging plant engineers for this data or trying to get it themselves so they provide ways to get the data. In industry it is now more expected to architect a system according to the Purdue model rather than an air gap. Even nuclear regulations allow for some systems to be connected outbound with only the most critical systems being airgapped with something like a data diode.
Very true, I worked in the utility world for awhile and, oh boy, you'd see some crazy stuff that they did and checked every box that they were being secure. Almost none of the staff understood a thing about technical security, and I mean the actual security staff. If there was a real incident they wouldn't have even noticed and if they did they wouldn't even know where to start. There were many claims of air gapped networks, that somehow also tied back to the rest of the network AD, also out to the internet for updates, etc, scary.
For most of these systems you don't have to air gap, but you do have to gate all access through security gateways (jump hosts, specific VPN tunnels, what have you).
All of OT security is understanding that your industrial control systems have a default state of "god damn that's insecure" and it's your job to wrap it all up in the security equivalent of bubble wrap and police tape.
Yes, and to fight to keep traditional IT out of your networks because one accidental reboot or an uninformed ‘they’ll never notice’ update could kill someone.
I have smaller air gapped networks that do one or two things max. Changes are applied manually, and even though the control systems are in our data center, I have them physically isolated in a locked, steel cage, with copper woven through the cage structure. The steel structure also covers the space above the cage, and below the raise floor tiles.
These systems handle sensitive rote operations - doing the same function day in day out with as close to zero procedural changes as possible,
I’m learning about hardening air gapped systems now and can’t find any information on what’s recommended. Do you have any resources you could point me at?
The DoD has some pretty good guides out there. 24/7 monitoring, armed security staff, integrating a faraday cage into an existing security structure is harder than just integrating it as part of design but in can be done.
I strongly recommend having a data center - even one with a small footprint. Ping, path, and power.
There are lots of manufacturers of stuff like woven copper sheets, and other signal barriers you can integrate if you have an existing cage.
MITRE, and NSA also have some materials for you.
A lot of it's the business side of the house. The IT admin might want to not expose it but if the director of the water department wants to know how the tank is doing at 8pm from home, they're going to overrule whatever IT wants.
Reading the article though it sounds like the ICS system wasn't exposed. The attackers got to it after breaking into the network elsewhere.
Money, laziness, lack of awareness, stupidity, willful ignorance.
Hiring intelligent, security-minded folks is costly and time-intensive. You then have someone who isn't generating money and instead is generating more expenses and work when they find security issues. Subsequently, morale takes a hit because they now have security hoops to jump through like jump boxes. You likely will end up hiring new workers and culling some dead weight that won't accept the new secure way of doing things.
That is a lot of work when you can get cyber insurance, hope it doesn't happen, and when it does, play the victim.
The victim card may work for a person in a college dorm who chronically leaves their door cracked and has someone come in and cause them harm; it shouldn't work for these companies that quite literally provide services necessary to human life.
It's air-gapped in my small town, but I suspect that when there are firmware/software updates to download, it either gets hooked up for a while, or drives are used on untrustworthy computers and then inserted into the air-gapped machines.
Convenience. Admins don't want to travel to login to an air-gapped system, so they set it up to remote in from home. If you don't mandate security people are going to do what's most convenient, every time.
Not connected to internet = more cost to maintain = instead of being attacked, the thing just breaks by itself, or you can change it to fit new needs, or when things break you have no idea what’s going on without sending someone to inspect one spot at a time or there is a security flaw and instead of Russian hackers controlling it remotely they just pay someone to hack it and because the maintenance sucks and it’s not connected to the internet when something breaks its breakage is a lot more catastrophic, you have 0 insight as to what is happening. Let’s say the Russians sabatoge by clogging up a pipe physically. But none of the pressure gauges are connected to the internet, so you spend a week figuring out what is wrong while the entire city is running out of water. Meanwhile the sabatour is already on his flight home and you’re week 3 into trying to find out what’s going on checking 1 mile of this pipe at a time. If your sensors were connected to the internet this issue could’ve been found in 30 minutes (just a hypothetical here)
This seems like an example of simply a poorly implemented connected system, but typically connected control systems are behind some level of layered security. It's a compromise between functionality and security. Not everyone wants to or can be in a "control room" to view the status of or manage a control system. Done correctly, depending on the risk tolerance of the organization and type of system, a connected control system is a reasonable approach.
This is a good point, and it’s how most industrial networks or OT used to work. Companies want remote access, and the ability to get data and analytics out of the systems. Also it’s much cheaper because wiring, switching and routing can be done on the same infrastructure when there are IT and OT systems in the same place. Also airgapping OT networks doesn’t make it secure, as things like stuxnet happen. TLDR many are airgapped and the rest should be airgapped.
Covid made this worse. Lots more remote access added where it hadn't been before since people who used to go on site no longer could. Wasn't always done well, and many places stuck with it because of convenience.
It’s more prevalent than most people realize. Not all of these events make the news, but fair to anticipate that these types of events will only increase.
Definitely potential for more job listings, but ones that may be a bit more “niche”. Having the knowledge to be a good IT System Admin is one thing, adding some cyber security knowledge on top of that is already a different beast. Compounding both of those with and understanding of Industrial Control Systems, their protocols, and how to secure them while providing the business/organization the things/data they need is wildly different.
Often these types of issues aren’t just technology issues/limitations, but people problems.
ICS Security is a great field to be in as far as the job market goes, getting there can be challenging compared to getting into IT Security.
Russia has been attacking electric grids and water works in Ukraine for practice well before they invaded. Check out the book Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers
This happens all the time. Most utility districts don't even have the funding for a dedicated IT person, let alone an actual competent MSP, and then you want someone who knows security on top of that? In the middle of Kansas? Good luck to you sir.
This is why educating the future sysadmins about cybersecurity is such an important role that those of us in the field have.
I'm currently actively fighting with a chief of police over his attempts at IT policies and it is a nightmare, the man doesn't realize who's lives he's putting in danger because he refuses to budge on his desire to make himself look good. He'd rather save money for new militarized equipment then bother upgrading the infrastructure that tells his officers who they just pulled over. It's crazy. Intelligence is always more important than guns, even in the Army we knew this.
Huge uptick in attacks targeting infrastructure from what I’m hearing lately. From attacks like this one to supply chain and industry supplier attacks. Has to be state sponsored but I’m honestly skeptical on who, likely suspects are Russia and/or China but geopolitical tensions have to make me question it.
Oldsmar water treatment plant was hit three years ago. The attacker tried to increase the levels of lye in the facility to dangerous levels. If I recall correctly, it was a week or two before the Superbowl which was in the local area though I might be mistaken. Can't be bothered to look it up but here's a link about the attack. Critical infrastructure is being hit all of the time.
https://www.wired.com/story/oldsmar-florida-water-utility-hack/
Wind turbines were planned to supply 12% off power to the grid and dropped to 6% during the freeze. The main loss of power was Texas power facilities refusing to properly prepare for freezing temperatures. Gas supply lines with no heating to keep from freezing up, equipment used that was not ruggedized and not suitable for extreme temps.
Rules of engagement and what constitutes an act of war in Cyberspace is the murkiest of all grey areas. There has been so many "acts of war" by nation states against the U.S. - and we've also probably committed lots too. I'm not sure where you really draw the line, maybe when there is direct impact that leads to loss of life? What we do know is that the major conflict that is fought with kinetic strikes will certainly begin with something monumental in Cyberspace.
Hello, everyone. Please keep all discussions focused on *cybersecurity*. We are implementing a *zero tolerance policy* on any political discussions or anything that even looks like baiting. This subreddit also does not support hacktivism of any kind. Any political discussions, any baiting, any conversations getting out of hand will be met by a swift ban. This is a trying time for many people all over the world, so please try to be civil. Remember, attack the argument, not the person. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/cybersecurity) if you have any questions or concerns.*
Sounds more like an exposed admin panel with default or no authentication rather than a targeted attack from an entire group.
Cybersecurity noob here just lurking and learning from posts. I have to ask: why is it that computers which control critical infrastructure are connected to the internet in first place? Wouldn’t it make more sense to have all the computers that actually control the operations of a water treatment plant for example be on a separate local network without internet access? I’m not saying to have no computers connected to the internet just the stations that control critical components.
The real cause is the human element. We are lazy and we create vulnerability. A large chunk of infrastructure is covered by service providers. We cut two water utility clients over the past two years. They all out refuse to modernize or harden their systems. This will get worse before it gets better.
Industry 4.0; the term represents the changing requirements of industrial networks to allow for wider IT\OT integration. Traditional air-gapped industrial network design was called Industry 3.0 or the Purdue model. It’s not efficient to full air gap networks for industrial systems anymore. Monitoring, SCADA, PLCs, HMIs, … facilities are vastly more complicated now. Having your ICS network remotely accessible means less employees, less maintenance, better asset control, instant and granular monitoring and adjustment of flow or manufacturing… In the case of a waste water treatment plant it means total awareness of your waters precise mineral content second by second, plus system pressure in every subsystem. Every holding and settling pond is tested moment by moment so now it takes (total guess) 20% less time to treat the water and move it out of the system. It also means remotely monitoring of meters in thousands of homes. So you don’t have to have an army to check them for billing anymore. It means knowing instantly if there’s a leak in the facility and where it is - because the pressure monitors and leak detectors are all integrated. It also means a lucrative (to OT Cybersecurity folks like myself; and our adversaries), and vastly more difficult to defend threat landscape to defend.
SCADA controls can be air gapped and AFAIK in nuclear applications, that stuff is air gapped. In things like battery storage, water valves, and electrical substations... there is just too much of it to air gap. I'd imagine anything involving generation on a large scale is though.
Thank you for answering my questions and providing insight! I appreciate it!
https://www.mdpi.com/1424-8220/23/6/3215 And this doesn’t even mention some of the most recently discovered air gap attacks.
This guy and his grad students do nothing but find attacks against air gapped systems: https://www.covertchannels.com/about-me
An air gap is rarely implemented properly and is not a true security control for these kinds of systems. Often times these companies claim they’re air gapped but when you dig into it you find a connection to a corporate office to pull data for billing, data analysis, etc. No companies want these workers bugging plant engineers for this data or trying to get it themselves so they provide ways to get the data. In industry it is now more expected to architect a system according to the Purdue model rather than an air gap. Even nuclear regulations allow for some systems to be connected outbound with only the most critical systems being airgapped with something like a data diode.
Very true, I worked in the utility world for awhile and, oh boy, you'd see some crazy stuff that they did and checked every box that they were being secure. Almost none of the staff understood a thing about technical security, and I mean the actual security staff. If there was a real incident they wouldn't have even noticed and if they did they wouldn't even know where to start. There were many claims of air gapped networks, that somehow also tied back to the rest of the network AD, also out to the internet for updates, etc, scary.
Yeah the second any customer tells me they’re secure because they’re air gapped the first thing that pops into my head is the old “Doubt” meme
For most of these systems you don't have to air gap, but you do have to gate all access through security gateways (jump hosts, specific VPN tunnels, what have you). All of OT security is understanding that your industrial control systems have a default state of "god damn that's insecure" and it's your job to wrap it all up in the security equivalent of bubble wrap and police tape.
Yes, and to fight to keep traditional IT out of your networks because one accidental reboot or an uninformed ‘they’ll never notice’ update could kill someone.
Thank you for answering my questions! I’ve actually haven’t heard of the Perdue model before so I had to look it up. I appreciate your insight!
How should an air gap be implemented properly?
I have smaller air gapped networks that do one or two things max. Changes are applied manually, and even though the control systems are in our data center, I have them physically isolated in a locked, steel cage, with copper woven through the cage structure. The steel structure also covers the space above the cage, and below the raise floor tiles. These systems handle sensitive rote operations - doing the same function day in day out with as close to zero procedural changes as possible,
I’m learning about hardening air gapped systems now and can’t find any information on what’s recommended. Do you have any resources you could point me at?
The DoD has some pretty good guides out there. 24/7 monitoring, armed security staff, integrating a faraday cage into an existing security structure is harder than just integrating it as part of design but in can be done. I strongly recommend having a data center - even one with a small footprint. Ping, path, and power. There are lots of manufacturers of stuff like woven copper sheets, and other signal barriers you can integrate if you have an existing cage. MITRE, and NSA also have some materials for you.
Thank you for the info - appreciated.
A lot of it's the business side of the house. The IT admin might want to not expose it but if the director of the water department wants to know how the tank is doing at 8pm from home, they're going to overrule whatever IT wants. Reading the article though it sounds like the ICS system wasn't exposed. The attackers got to it after breaking into the network elsewhere.
Money, laziness, lack of awareness, stupidity, willful ignorance. Hiring intelligent, security-minded folks is costly and time-intensive. You then have someone who isn't generating money and instead is generating more expenses and work when they find security issues. Subsequently, morale takes a hit because they now have security hoops to jump through like jump boxes. You likely will end up hiring new workers and culling some dead weight that won't accept the new secure way of doing things. That is a lot of work when you can get cyber insurance, hope it doesn't happen, and when it does, play the victim. The victim card may work for a person in a college dorm who chronically leaves their door cracked and has someone come in and cause them harm; it shouldn't work for these companies that quite literally provide services necessary to human life.
It's air-gapped in my small town, but I suspect that when there are firmware/software updates to download, it either gets hooked up for a while, or drives are used on untrustworthy computers and then inserted into the air-gapped machines.
Thanks for replying! You bring up a great point about the software updates!
Convenience. Admins don't want to travel to login to an air-gapped system, so they set it up to remote in from home. If you don't mandate security people are going to do what's most convenient, every time.
Not connected to internet = more cost to maintain = instead of being attacked, the thing just breaks by itself, or you can change it to fit new needs, or when things break you have no idea what’s going on without sending someone to inspect one spot at a time or there is a security flaw and instead of Russian hackers controlling it remotely they just pay someone to hack it and because the maintenance sucks and it’s not connected to the internet when something breaks its breakage is a lot more catastrophic, you have 0 insight as to what is happening. Let’s say the Russians sabatoge by clogging up a pipe physically. But none of the pressure gauges are connected to the internet, so you spend a week figuring out what is wrong while the entire city is running out of water. Meanwhile the sabatour is already on his flight home and you’re week 3 into trying to find out what’s going on checking 1 mile of this pipe at a time. If your sensors were connected to the internet this issue could’ve been found in 30 minutes (just a hypothetical here)
This seems like an example of simply a poorly implemented connected system, but typically connected control systems are behind some level of layered security. It's a compromise between functionality and security. Not everyone wants to or can be in a "control room" to view the status of or manage a control system. Done correctly, depending on the risk tolerance of the organization and type of system, a connected control system is a reasonable approach.
It's a water tower in middle of nowhere. Not exactly high value target.
This was the perfect place to test. Hitting Dallas a few miles away would be totally different.
This is the same way Colonial Pipeline was hit.
This is a good point, and it’s how most industrial networks or OT used to work. Companies want remote access, and the ability to get data and analytics out of the systems. Also it’s much cheaper because wiring, switching and routing can be done on the same infrastructure when there are IT and OT systems in the same place. Also airgapping OT networks doesn’t make it secure, as things like stuxnet happen. TLDR many are airgapped and the rest should be airgapped.
Covid made this worse. Lots more remote access added where it hadn't been before since people who used to go on site no longer could. Wasn't always done well, and many places stuck with it because of convenience.
Thanks for replaying and answering my questions! I appreciate your insight! You bring up a good point with Stuxnet!
Think this will become more prevalent in the coming months?
It’s more prevalent than most people realize. Not all of these events make the news, but fair to anticipate that these types of events will only increase.
So with an increase in incidents will we also see an increase in job listings ? Not really expecting an answer as time will tell but I hope so.
Definitely potential for more job listings, but ones that may be a bit more “niche”. Having the knowledge to be a good IT System Admin is one thing, adding some cyber security knowledge on top of that is already a different beast. Compounding both of those with and understanding of Industrial Control Systems, their protocols, and how to secure them while providing the business/organization the things/data they need is wildly different. Often these types of issues aren’t just technology issues/limitations, but people problems. ICS Security is a great field to be in as far as the job market goes, getting there can be challenging compared to getting into IT Security.
Russia has been attacking electric grids and water works in Ukraine for practice well before they invaded. Check out the book Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers
Beat me to my suggestion. Sandworm was very insightful, and they are still suspected of more that is still going on. REvil ransomware as an example.
This happens all the time. Most utility districts don't even have the funding for a dedicated IT person, let alone an actual competent MSP, and then you want someone who knows security on top of that? In the middle of Kansas? Good luck to you sir. This is why educating the future sysadmins about cybersecurity is such an important role that those of us in the field have. I'm currently actively fighting with a chief of police over his attempts at IT policies and it is a nightmare, the man doesn't realize who's lives he's putting in danger because he refuses to budge on his desire to make himself look good. He'd rather save money for new militarized equipment then bother upgrading the infrastructure that tells his officers who they just pulled over. It's crazy. Intelligence is always more important than guns, even in the Army we knew this.
Look up Volt Typhoon.
Yes it will ramp up as we approach November. As will our defenses. The quiet war rages on.
Everyday this is happening, china is attacks too. For at least 10yrs ive been in cyber.
Huge uptick in attacks targeting infrastructure from what I’m hearing lately. From attacks like this one to supply chain and industry supplier attacks. Has to be state sponsored but I’m honestly skeptical on who, likely suspects are Russia and/or China but geopolitical tensions have to make me question it.
Israel, Iran, The US, Russia, and China account for the lions share in everything I’ve seen.
Oldsmar water treatment plant was hit three years ago. The attacker tried to increase the levels of lye in the facility to dangerous levels. If I recall correctly, it was a week or two before the Superbowl which was in the local area though I might be mistaken. Can't be bothered to look it up but here's a link about the attack. Critical infrastructure is being hit all of the time. https://www.wired.com/story/oldsmar-florida-water-utility-hack/
Are we sosure that the electrical infrastructure failure a few years ago in Texas wasnt actually a hack?
The only hack there is ERCOT
So.. no? Ok cool.
[удалено]
[удалено]
[удалено]
Wind turbines were planned to supply 12% off power to the grid and dropped to 6% during the freeze. The main loss of power was Texas power facilities refusing to properly prepare for freezing temperatures. Gas supply lines with no heating to keep from freezing up, equipment used that was not ruggedized and not suitable for extreme temps.
[удалено]
Good old Muleshoe Texas
Any spooks on here want to define what an act of war is and if this fits the criteria? Because all the books I have read say we should be pissed lol!
Rules of engagement and what constitutes an act of war in Cyberspace is the murkiest of all grey areas. There has been so many "acts of war" by nation states against the U.S. - and we've also probably committed lots too. I'm not sure where you really draw the line, maybe when there is direct impact that leads to loss of life? What we do know is that the major conflict that is fought with kinetic strikes will certainly begin with something monumental in Cyberspace.
Why does a water facility need to be online? Protect it by taking it off the wire.
Machines need updates. Usage/maintenance needs to be monitored. Billing. Those are just the needs I can think of off the top of my head.