America’s water and wastewater system is under threat.
A 2024 report by the Environmental Protection Agency’s Office of Inspector General described more than 300 separate water systems that are at risk from cyberattacks, systems serve more than 100 million people.
An attack on any of those systems could disrupt service or cause irreparable physical damage to the system’s drinking water infrastructure.
To help water systems in the United States better protect themselves against a cyberattack, the Cybersecurity for Manufacturing Innovation Institute (CyManII) at The University of Texas at San Antonio, has developed a cyber training program that allows critical infrastructure organizations to practice defending their systems and protecting their information technology and operational technology assets.
CyManII recently launched a pilot program with the water treatment facility in Mission, a U.S. city near the Texas-Mexico border. Mission city officials saw the pilot program as an opportunity to strengthen their cybersecurity posture and give them the tools needed in case of a future cyberattack.
“You see a lot of emergency management programs out there, and you see a lot of conferences that city municipalities attend, but this is the first comprehensive exercises that we did in emergency management specifically focused on cybersecurity,” said Andy Garcia, assistant city manager of Mission.
“I would recommend [this training] to other cities because it allows you to take your disaster recovery plan and put it to use,” Garcia added. “It lets you see how you’re able to stick to your policies and procedures. You want to make decisions quickly. It would be beneficial to every city.”
To help Mission improve its cybersecurity, CyManII offered an a la carte selection of training activities, including simulated cyberattack exercises, a cyber risk assessment and participation in the Cyber Readiness Program.
Quick decisions
Part of what makes CyManII’s program unique is that it focuses on both the technical team tasked with fighting the cyberattack as well as the leadership team, which will have to make quick decisions about who to get involved in the response and when to involve them.
“This is an opportunity to bring out your incident response plan, bring out your playbook, and see how it really plays out,” said Joe Mallen, assistant director of experiential learning for CyManII.
By testing their playbook, an organization will be more prepared when they are attacked.
“It’s not a matter of if they’re going to get attacked, it’s a matter of when they’re going to get attacked,” Mallen said.
With support from CyManII, Mission city staff members tested their existing playbook and also developed a new one based on lessons learned during the exercise.
But the main activity across the two-day exercise was simulating real attacks that allowed the technical team and the leadership team respond in real time.
Simulated attacks
As a training tool, the simulated attacks are unique to CyManII, which arms participants with all the same tools — both software and hardware — they would use on the job.
“I was blown away by what they have to offer,” said Abram Ramirez, IT director for the City of Mission. “To have the technology for hands-on [experience] that’s practical in [an attack] scenario, that wasn’t something I was expecting. Now it’s something I want to look at trying to implement at least once a year.”
The team faced two simulated attacks, one on each day of the exercise, with Mission leadership involved in the simulation on day two.
“A lot of folks don’t ever see a cyberattack until [they’re under attack], so we’re trying to give them an opportunity to see a cyberattack and practice responding to one before it happens on the job,” Mallen said.
When it happens on the job, there’s a lot of pressure to get services up and running, he added. “You have administration on you and customers calling — it’s chaos. Our Cyber Range allows teams to come in and practice, and we’ve seen, like with anything, it’s all muscle memory. The more you do it, the more practice you get, the better you get.”
That was true of Mission’s technical team by the second day. “It was like a whole new team walked in,” Mallen said, adding that even after one day of practice, they felt stronger and more capable of handling the next attack.
Communication is key
Another highlight of the pilot program, said Ramirez and Garcia, was having leadership involved in the simulation on day two.
While leadership often has an emergency response plan of their own — as Mission does — it’s rare that they get to kick the tires on the plan and make sure it will work amid a cyberattack.
More importantly, the exercise gives the technical team and the leadership team an opportunity to practice communicating through an emergency.
“It tested our ability to respond as a group,” Garcia said. “It allowed us to kind of develop not only an understanding of how we communicate with one another, but how we communicate with other stakeholders, partners and agencies.”
Ramirez added: “It’s important because we need to have that communication to be prepared to respond to any type of incident. So, having the executive decisionmakers who were involved with insurance, involved with legal help us, I think now we’re very prepared.”
Providing assurance
In addition to having hands-on training with the simulated attacks, what Garcia found most helpful was having CyManII’s cybersecurity experts on hand to provide real-time guidance.
“[It was beneficial] having the guidance of cybersecurity professionals to help develop best practices and to provide assurances,” Garcia said. “Having that support while making the actual decisions helped make it a little bit easier to execute it if you have to do it in a real-life scenario.”
Now, Garcia and Ramirez agree that city officials and staff are on much better footing should the City of Mission face a real cyberattack.
“I think the whole experience from getting trained to the live scenario [was very beneficial],” Ramirez said. “I’d go through this training again. I already asked CyManII how many times we can do it.”
CyManII leaders say the training is the start of a strong partnership.
“What we accomplished here in Mission is only the first step,” said Ed McCormick, Regional Innovation Officer for CyManII’s Lone Star Cyber Forge. “This effort demonstrates how we can transform training into sustained resilience, ensuring that the technologies, practices, and partnerships we build today become the foundation for safeguarding America’s critical infrastructure tomorrow.”
