Summary: The article discusses the vulnerabilities found in Automatic Tank Gauge (ATG) systems that monitor fuel storage, highlighting the risks posed by their exposure to the Internet. It emphasizes the need for enhanced security measures to protect critical infrastructure from potential cyberattacks.
Threat Actor: Malicious actors | malicious actors
Victim: Critical infrastructure facilities | critical infrastructure facilities
Key Point :
- Multiple critical 0-day vulnerabilities were discovered in ATG systems from various vendors, posing significant risks of physical damage and environmental hazards.
- Many ATG systems remain exposed to the Internet, making them attractive targets for cyberattacks, which could lead to catastrophic consequences if exploited.
- There is a pressing need for organizations to remove ATGs from public access and implement robust security measures to safeguard these critical systems.
Introduction
Industrial Control Systems (ICS) have become a ubiquitous part of modern critical infrastructure. Automatic Tank Gauge (ATG) systems play a role in this infrastructure by monitoring and managing fuel storage tanks, such as those found in everyday gas stations. These systems ensure that fuel levels are accurately tracked, leaks are detected early, and inventory is managed efficiently. Although the typical gas station comes to mind when thinking about fuel tanks, these systems also exist in other critical facilities, including military bases, hospitals, airports, emergency services, and power plants, to name a few.
Recent investigation by Bitsight TRACE has discovered multiple critical 0-day vulnerabilities across six ATG systems from five different vendors. These vulnerabilities pose significant real-world risks, as they could be exploited by malicious actors to cause widespread damage, including physical damage, environmental hazards, and economic losses. What’s even more concerning is that, besides multiple warnings in the past, thousands of ATGs are still currently online and directly accessible over the Internet, making them prime targets for cyberattacks, especially in sabotage or cyberwarfare scenarios.
Bitsight strongly believes in responsible disclosure of vulnerabilities. For the past six months, Bitsight has been collaborating closely with the U.S. Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA), as well as with affected vendors, in order to mitigate these vulnerabilities. This coordinated effort aims to safeguard critical infrastructure and prevent the dire consequences that could result from successful attacks. CISA published remediation advisories for interested parties.
In this blogpost, we will explore the ATG systems, their inherent risk when exposed to the Internet and the several critical vulnerabilities uncovered by Bitsight TRACE. By understanding these vulnerabilities, we hope that the reader can better appreciate the urgent need for enhanced security measures and the steps that need to be taken to protect these systems from exploitation.
What is an ATG system and why it matters
Automatic Tank Gauging refers to a system that automatically measures and records the level, volume, and temperature of products in storage tanks, such as gas stations fuel tanks. It can also monitor leaks, issue high-level and low-level alarms, trigger sirens, emergency shutoff valves, ventilation, fuel dispensers and other peripherals. The ability to control physical processes is made by interfacing with the internal or external relays. This technology helps ensure compliance with environmental regulations and is used to optimize inventory management at a gas station or other facilities that store fuel (hospitals, airports, military facilities).
There are several brands and models of controllers that are commonly used in Automatic Tank Gauging systems. Our research focused on some of the brands and models most commonly found online. It is by no means exhaustive, but we considered a good first approach to the issue.
Part of what makes these devices attractive to security researchers, or a malicious actor for that matter, is the potential ability to control physical processes that could lead to disastrous consequences if they are abused in unintended ways.
What Happens When ATG Systems Are Abused
So what could happen when malicious actors gain access to an ATG system with full privileges? Depending on the abused functions, the actual physical installation of the device, and the lack of mitigating controls, the consequences could be disastrous. It is a concern that vendors acknowledge and some even issued warnings about such consequences in the past. To cite one of the vendors:
“WHAT CAN HAPPEN IF YOU DO NOT TAKE OWNERSHIP OF YOUR NETWORK?
- Rename Tank Information: On consoles using Telnet, hackers find the MAC address, determine whether it is a TLS-350 or TLS-450PLUS and simply change the tank names to something inappropriate.
- Resize Tanks (From 10K to 20K Gallons): It is possible to change the tank size, so it appears the tank can hold more than it really can. The thresholds could also be changed so that overflow alarms appear at a higher level. The potential would be to overfill the tank causing an environmental leak.
- Shutdown Dispensing (PLLD and Relay Settings): The relays could be deprogrammed so that the pump wouldn’t be activated on a hook signal. Additionally, PLLD could be turned off so catastrophic leaks may not be detected.
- Capture Sensitive Corporate Data: By monitoring insecure Telnet connections, observers can gather operations data (delivery, inventory, alarms, etc.) for sale to third parties.
- Shutdown IP Cards / Networking Services: After gaining access to a vulnerable corporate network, hackers could alter TLS-350 Ethernet cards lacking passwords; changing configurations and rendering management systems ineffective. Critical operations could be impacted (hospitals, emergency providers, cell service, power plants, etc.).
- Loss of Compliance Data: Reprogramming the console could result in the loss of compliance data translating to potential regulatory fines.”
While the vendor refers to a specific product and version, these warnings are likely applicable to others, as most ATG systems share many of the same features. An administrator user on another ATG system also has the ability to perform some, if not all, of these actions with similar impact. In fact, most of the vulnerabilities we discovered allow for just that: for an attacker to have full control of the ATG as administrator!
Based on feedback from manufacturers of ATG equipment, we recognize that owners and operators may implement mitigating controls, external to the ATG systems themselves, that would prevent the worst case scenarios from occurring. In certain jurisdictions, for example, owners and operators are legally required to reduce the likelihood of spilling or overfilling – and many may have taken measures to do so. Nevertheless, it is important to share information about the potential consequences of exploitation so that owners and operators are aware of the risks.
Past Security Research
ATGs were already mentioned in the past as vulnerable systems, as far back as 2015. In January that year, H.D.Moore published in the Rapid7 blog an article titled: “The Internet of Gas Station Tank Gauges” based on Jack Chadowitz research on ATGs. The article stated:
“Approximately 5,800 ATGs were found to be exposed to the internet without a password. Over 5,300 of these ATGs are located in the United States, which works out to about 3 percent of the approximately 150,000 [1] fueling stations in the country.”
In August 2015, Trend Micro published “The Gaspot Experiment: How Gas-Tank-Monitoring Systems Could Make Perfect Targets for Attackers.” Instead of probing the landscape, Trend Micro took another approach:
”To better understand the current gas-tank-monitoring system attack landscape, we developed a way to simulate the existence of these devices to check whether threat actors will find them venues attractive enough to go after. […] These are essentially gas tank-monitoring system honeypots, hence the nickname, “GasPot.”
Later on, there were two follow up Rapid7 articles, one in November 2015 and another in November 2016, led by Jon Hart, who also produced an ATG client module for Metasploit which implements some basic TLS-250 and TLS-350 commands.
In November 2022, a company called Cyborg Security raised the alarm again in a blogpost. Their conclusions were clear:
“For an attack surface of critical systems to increase by nearly 120% over 7 years is unacceptable.”
These legacy issues were essentially about the so-called “ATG protocol.” We will share our insights about this protocol and why it is not safe by design before we dig into the other new, vendor-specific, vulnerabilities.
Why We Are Researching ATGs
Critical infrastructure sectors have come to heavily rely on Industrial Control Systems (ICS) to control cyber-physical systems. If you have been following our TRACE research blog posts, you might have already realized that Bitsight has been setting up an ICS laboratory for a while. We hinted at that in the recent ICS introduction article. It serves several purposes, one of which is helping us to properly and safely identify an increasing number of ICS devices that are reachable via the Internet. Ever since I wrote about the different ICS protocols and the thousands of exposed ICS systems, one protocol caught my attention: ATG.
The ATG protocol appeared as having a high prevalence, with several thousand devices online and reachable, which I knew nothing about. This was a perfect target on which to conduct research within our ICS lab.
The Legacy Issues
Before we dive into the new vulnerabilities we found, it is important to stress the impact of the old ones. As we’ve mentioned, security researchers warned about the “ATG” protocol being exposed in different ATG systems. It appears there are references to this protocol described in multiple ways, such as the Veeder-Root protocol, Gilbarco protocol or TLS protocol. In fact, the Veeder-Root TLS series has excellent documentation about the protocol.
The protocol itself seems to have been designed for the serial RS-232 interface, which is normally used to connect the system to a controlling computer. When an ATG system is connected to a network (either via a modem or ethernet card), a TCP/IP port (default 10001) is listening which essentially mirrors the serial interface behavior over a socket. So by directly connecting to the TCP/IP port and calling the desired functions in the right format, an attacker can simply use the system as if he was using an RS-232 cable.
By analyzing the detailed TLS manuals, it is possible to gain a deeper understanding about the protocol features. The Veeder-Root TLS-450 Serial Command Manual has over 600 pages and documents almost as many different functions. It is not exactly clear which devices implement which functions, but we’ve seen many other ATGs implement this protocol to expose some of the functions defined in the manual. There are several functions that allow for settings and parameters to be changed that might have a security implication in some way, shape or form.
Furthermore, the protocol seems to be insecure by nature and cannot be fixed without changing the specification itself, since the only security feature it contains is a code in each command message:
This security code is, however, both optional and insufficient. By default, it is not used, perhaps because some models require manual dip-switch configuration to enable the feature to begin with. There also seems to be conflicting information about what is considered an acceptable security code. The security code is described as a six digit code. It makes sense if you consider that the older models had a numeric keypad. This yields about 1,000,000 combinations, a code that can nowadays be trivially guessed. An attacker able to try a modest amount of 100 combinations per second would need less than 3 hours to iterate over the entire code space. Further research, however, indicates that at least in newer models like the TLS4B, we were able to confirm that these actually accept more than only numbers:
This increases the search space considerably, although the fact remains that it has a maximum of 6 characters.
These are some of the actions that could be performed by an attacker via the protocol:
Network related configurations (DoS, traffic monitoring, firmware updates,etc)
Change DNS server / gateway
Change contact numbers and alarms destination (silent alarms, trigger alarms to paid numbers)
General configurations
Delete/change automatic events
Change product labels (ex. switch diesel for gasoline)
Physical related configurations
Change tank volume/diameter/tilt/limits/other parameters (spillage)
Change pump control devices and parameters
Change relay, relay types and configurations (potential physical impact)
Other actions
Start/stop In-Tank Leak Detection Test
Start/stop pressure line leak detection test
Remote reboot (and DoS by looping the reboot command)
In a nutshell, those actions can lead to the disastrous consequences to which the vendors warned about. This is why it is paramount to disconnect any ATG from the Internet. Still, looking at the last month alone we can find 6,542 devices (excluding GasPots) directly connected to the Internet without any security code at all.
Discovering New ATG Vulnerabilities
All these features make the ATG protocol a potential target for attackers. It looks and feels like a legacy serial protocol that was just given the ability to become online. That alone would make it an interesting target for an attacker.
But what about other ATG systems? ATGs that don’t necessarily communicate using the old “ATG” protocol. What interfaces do they provide? Are they also exposed over the Internet? And, more importantly, are they more secure?
These were the kind of questions that Bitsight TRACE set out to find out, so we leveraged our ICS lab to better understand the risks that these devices pose.
In one single but intensive week, we’ve found multiple vulnerabilities in several different devices. To be more precise, 11 vulnerabilities in 6 different models, each one underscoring the critical need for better security practices. Of the 11, one was found to be a duplicate of an existing vulnerability. Here’s a summarized view of what we found:
It is not the number of new vulnerabilities that is most concerning, and probably not even their criticality, but that they reflect fundamental security flaws that should have been addressed long ago. CISA, in their advisory on Categorically Unsafe Software, describes that focusing on quality and eliminating groups of errors improves security. They cite Steve Christey from MITRE, who published a paper in 2007 describing the “unforgivable vulnerabilities” and the criteria that they obey. In short, all these new vulnerabilities fit these criteria.
We found vanilla Reflected XSS. The authentication bypasses were direct path access. The command injections lacked filtering. There were hardcoded administrator credentials. The arbitrary file read was a direct path traversal access, yielding admin credentials. The SQL injection could be exploited aided by full SQL error logs.
All these vulnerabilities allow for full administrator privileges of the device application and, some of them, full operating system access.
The reader might ask, why do we keep encountering these types of vulnerabilities in ICS devices? Well, there are a number of reasons. Designing and maintaining ICS systems presents additional challenges compared to pure software systems, and even IoT devices. To start, many ICS systems were designed decades ago, long before cybersecurity was a big concern. These systems were built to prioritize reliability and efficiency, not security. As time goes by, they do get new features but usually they are not completely designed from scratch. As a result, they often lack modern protections. In addition, many of them were not designed to be connected to a hostile environment like the Internet. Vendors recently started to integrate them with newer technology to improve efficiency and remote access and this significantly changes their threat model. Of course, there is also a lack of cybersecurity experts that are familiar with ICS systems. It is hard to find vulnerabilities if no one is looking for them.
From DoS to Physical Damage: The Impact of ATG Exploitation
What exactly is the potential impact associated with these vulnerabilities?
Denial of service
The simplest attack that can have a significant impact is simple DoS by reconfiguring the system, deleting values or reflashing the device with faulty firmware. These changes would effectively disable the device, lead to downtime and would usually require human intervention. In fact, these types of attacks are currently ongoing, with claims of exploitation of at least one brand of devices for which we published a vulnerability on just two weeks ago.