"In 2024, nearly every device can be a router." - HD Moore, founder & CEO of runZero
Despite significant advances in information technology, security practitioners continue to face persistent challenges. While secure-by-default designs, zero-trust architectures, and increased security awareness have made strides in protecting networks, issues like network dark matter and segmentation failures remain problematic. These issues are often outside defenders' direct control, making them difficult to address and leaving easy footholds for attackers to exploit. In this blog, we explore complex asset management challenges associated with network dark matter and segmentation decay, leaving devices and networks unnecessarily vulnerable to attacks.
Dark matter: IoT and embedded devices #
The level of attention, monitoring, and updates that network-connected devices receive can be divided into a three-tier hierarchy, a “hierarchy of visibility” as it were.
At the top level, there are devices that humans interact with directly, or form part of a production system: our laptops, desktops, servers, routers, and switches. They tend to have high visibility to the information security team via mechanisms like Simple Network Management Protocol (SNMP) and endpoint management (EPM) software. These systems usually get frequent, managed updates that are often installed automatically and en masse.
The middle tier consists of “limited visibility” devices present in every office: smart TVs and projectors, media devices like Rokus and Apple TVs, wireless access points, and printers. These devices often support updates over the network but may not receive frequent updates and may require manual intervention to apply them. How often do we think about updating the firmware on our venerable Brother printers?
And last but not least, the dark matter of networks makes up the bottom tier. Much like dark matter in cosmology, these devices are present on the network and their influence can be felt, but they are mostly invisible to IT and management tools. These are things like thermostats, smart plugs and lights, aquarium pumps, refrigerators, sprinkler systems, physical access control systems, and so on. These devices often fade into the background and can go relatively unnoticed for years. Updates are likely infrequent or nonexistent, and may require manual intervention if they can even be applied at all.
FIGURE 1 - Device types by visibility.
Devices in the last two tiers often outnumber the “visible” devices, sometimes significantly. Analysis of the runZero cloud data for physical assets (excluding virtual machines) indicates that limited visibility devices make up a whopping 45.46% of discovered devices, with true dark matter devices making up a further 19.09%.
The decay of segmentation #
The premise behind network segmentation is that security can be improved by preventing communication between systems with different trust levels and business functions. A company may offer a wireless guest network to office visitors, but does not want those visitors to be able to talk to critical file servers or security equipment.
Segmentation is the most popular approach to securing unmanaged devices; if we are unable to enforce policies on the device itself, placing that device on a separate network that has little access to anything else can reduce the risk of a compromise. With the massive increase of “smart” IoT devices, even ISP-furnished residential routers now offer segmentation features.
Segmentation is a widely-accepted approach to improving network security, to the point where overlooking network segmentation can violate the requirements of common industry requirements and best practices, such as PCI DSS.
FIGURE 2 - A simple example of network segmentation.
Segmentation as a goal is great, but it’s prone to failure, and often in ways that are not obvious to the team responsible for its implementation. Segmentation assumes that systems are organized into groups based on their business function and that those systems are prevented from communicating with other groups. Cracks appear quickly as the number of systems in the segment grows, primarily for two reasons.
First, each additional system in a network segment has the ability to weaken the security of the whole segment. This happens when a system is added that exposes additional network services or that accepts a different form of authorization relative to its peers. Any compromise of a system in that segment can provide a foothold for additional attacks, many of which are only possible when the attacker is on the same local network. The wider the variety of services and authentication sources available, the greater the chance of a successful attack, and this exposure scales with every addition.
Second, modern equipment is complex and it is rare for any system to have a single network interface. Nearly every device we interact with has more than one way to communicate. The phones in our pockets may support a dozen concurrent network interfaces at once (wireless, Bluetooth, 4G/5G, AWDL, NFC, and more). A modern laptop ships with a half-dozen interfaces out of the box. The humble network printer arrives with wireless, Bluetooth, and Ethernet enabled by default. Segmentation assumes that placing a device on a network limits its access, but that isn’t true when every device has multiple network interfaces, and these interfaces allow an attacker to hop between physical connections using wireless protocols.
Case study: the office printer #
The humble office printer is a great example of the challenges with network segmentation. A typical all-in-one (or multi- function center, aka MFC) printer arrives with Ethernet, Bluetooth, and wireless networks configured out of the box. The printer’s wireless interface is exposed as an open access point, allowing anyone within physical range of the device to connect and communicate with the printer. For many reasons, printers are often connected directly to wired Ethernet and the wireless interfaces are still left enabled.
From a security perspective, any device with multiple networks (known as multi- homed devices) can introduce risk, but how risky is this printer example? If an attacker is able to compromise the printer and relay traffic, certainly that is bad, and while it’s been demonstrated repeatedly (including attacks via the Fax modem), let’s assume that future printer firmware is more secure, and opportunities to directly compromise the printer are less common.
Unfortunately, printers often ship with another tricky feature, and one that isn’t possible to disable: IP forwarding. Many printers act like network routers and offer no way to configure this behavior outside of disabling all but one network interface.
IP forwarding allows an attacker connected to one side of a device to route packets through to a network on the other side. This doesn’t always mean that network address translation (NAT) is enabled, but even one-way packet delivery into a target network can be disastrous, as source-address spoofing can be used to force target devices to reply to an Internet-facing public IP, providing a two-way communication channel into what should be an isolated network. Even single-interface devices with IP forwarding enabled can be abused to force a device to repeat a message from its own MAC address and network.
Printers are not the only type of device that forwards IP traffic between network interfaces by default; runZero tests for IP forwarding during active scans, and has identified this behavior across IP telephones, network storage appliances, media servers, network cameras, DVRs, battery backup units, smart TVs, video game consoles, and even smart light bulbs. Even industrial automation equipment from HVAC controls to programmable logic controllers (PLCs) inexplicably enable IP forwarding.
FIGURE 3 - Unusual devices with IP forwarding capabilities. Box sizes indicate relative frequency in our dataset.
Why does this happen? Oftentimes these devices have virtual network interfaces that are only visible to the firmware itself. To communicate across these virtual interfaces, IP forwarding must be enabled, and no firewall rules were added to prevent the forwarding capability from accepting packets on external interfaces.
This behavior can also surface on servers and workstations. For example, if a developer is using containers for development on their laptop, the container environment often creates virtual network interfaces, and enables IP forwarding for communication across the interfaces. Just like the case of the network printer, if that laptop is connected to both a wired and wireless network, the IP forwarding feature effectively turns their laptop into a router between these segments, since no firewall rule prevents it.
FIGURE 4 - A network diagram showing unexpected network bridging points.
Advancing Security Beyond Segmentation #
Segmentation is still one of the best tools we have to improve security, but its limitations are becoming more obvious with modern equipment. The complexities of managing network dark matter and segmentation decay present worrisome exposures that are increasingly difficult to defend against. Addressing these challenges requires a comprehensive CAASM approach for improved asset discovery, attack surface management, and detection of emerging threats to safeguard networks against advanced threats.
Excited to learn more? Download the runZero Research Report to take a deeper dive into the state of asset security.
Not a runZero customer? Fire up a free trial and gain complete asset inventory and attack surface visibility in minutes.
