The Evolution of Network Scanning: 25 Years of Nmap

With 2022 marking the 25th anniversary of Nmap, runZero hosted a moderated conversation between security industry legends, HD Moore and Gordon “Fyodor” Lyon, the creator of Nmap. They discussed the challenges, rewards, and lessons learned from their work building network scanning technology. They covered everything–from product development to motion picture cameos.

About the speakers

Gordon has been in the networking industry for decades, starting back in his college days when residential Ethernet was introduced. He started Nmap 25 years ago as a personal network project and has become more focused on growing it, alongside other tools and resources for network discovery and active scanning.

HD comes from a more cybersecurity focused background and founded the Metasploit Project. As a pentester and security researcher, HD experienced the challenges of performing internal discovery and asset inventory development firsthand. With this challenge in mind, he eventually cofounded runZero.

25 years of Nmap

Gordon developed Nmap while spending a summer at one of Johns Hopkins youth talent centers. He joked, “I wasn’t one of those [talented youth], but I was a teaching assistant. They gave me a dorm room that I had Ethernet access in and I spent that summer basically just writing Nmap for my own purposes back in ‘97.” After writing the initial version of Nmap, Gordon sent it to Mike Schiffman who was the editor of Phrack at the time.

From there, Nmap was published. While it was originally written for individual purposes, Nmap received a wealth of support and potential patches for bug fixes from across the cybersecurity community. Gordon continued adding features that piqued public interest before those features became mainstream and kept Nmap ahead of the curve. Nmap’s popularity led to the product being featured in several movies, including the 2003 film The Matrix Reloaded.

It’s more than just free product placement in movies that has kept Nmap relevant though. Gordon said he believed Nmap’s staying power came down to timing. “If you had to download some scanner source code and…fix little errors…Nmap really made it a lot easier to scan in a fast consistent way and really understand your network better,” Gordon said. “Nmap was there at the early stage and everyone started using it in their toolbox…we want to trust our tools.”

Nmap became the trusted tool at the right time. Now, Gordon does Nmap work full-time and what started as a personal project during the summer of ‘97 is now one of the most recognized networking tools in the world. HD even said, “I don't think a lot of folks realize how far Nmap and a lot of Gordon's work has gone. Like, nearly every security product on the planet uses something from Gordon somewhere. It's either using Npcap, it's using the Nmap fingerprint library, or it’s using Nmap directly. And it goes well beyond vulnerability scanners.” Which brings us to HD’s journey and starting runZero.

Starting runZero

HD started with a lot of the new 90s networking tools, similar to Gordon, and was excited about the potential of writing his own “stuff.” He said, “I felt like there was a whole undiscovered world of fun [electronic data processing] services out there that we just weren’t scanning early on.”

With security growing as an industry and organizations developing more mature security measures, managed assets were likely to be accounted for. EDRs and basic network monitoring practices developed to support the growth in technology. runZero needed to focus on something missing from the space.

HD has a strong background in exploit development and penetration testing, so he applied those skills to network discovery. HD said, “With exploit development, one of the coolest things about building an exploit is leaking a pointer, finding fun ways to determine the exact target and offsets before you actually trigger the payload. I feel like with network discovery, it's the same thing.”

To HD, it was “that same kind of challenge of ‘how do I take all these little tiny information leaks and build something really useful with it?’” He wanted runZero to start with essentially nothing and be able to find everything. This required taking a new approach to active scanning and starting with the assumption that every device has a V6, has at least one local link. That assumption allows for every device to be scanned by default.

He also commented, “I just really like the idea of taking disparate information from assets and putting it together and building something bigger out of it. Like okay, this packet says it’s this type of OS, but this secondary interface says it’s part of this domain. And then trying to glue all that together into something comprehensive. That’s always kind of been my approach for the pentest and red team work I was doing.” HD took that approach to networking and successfully started runZero.

Future developments for Nmap and runZero

Nmap and runZero both originated as a result of the gaps in scanner technology and have gone beyond simply scanning capabilities, but what does that mean for the future of these tools? The answer is they’re both taking measures to improve with timelines and technical compatibility being the biggest challenges moving forward.

Gordon discussed how, for the September 1 Nmap release, there aren’t a ton of new and exciting feature updates. However, their Npcap has reached a stable state and is something the Nmap team is proud of. He reflected that they “collected a lot of technical debt” while Npcap took up all of their focus and resources. They expect new version detection capabilities to come forward soon now that Npcap is in an acceptable state where it’s not crashing systems. Gordon said, “I hate to disappoint [our users] but on the other hand, if Nmap doesn’t work on your system, or if it crashes your system…then all the features in the world don’t matter as much.”

On the other side, runZero is actively releasing new features based on customer feedback. HD spoke about the development of runZero fingerprinting and targeted asset discovery. runZero does advanced application-level asset identification, going deeper into IP addresses and system detail than other active scanners. One of the stories HD recounted that stemmed from a customer input was, “We had a customer who had a mandate saying ‘we cannot have Huawei equipment on the network.’ Just kind of a standard thing. If you're a federal contractor or anybody who works for federal contractors, you can't have certain manufacturers in the network and they're [the customer] swearing up and down. They did not have a Huawei device in network, but we kept reporting one anyway.”

HD continued, “We were reporting it in the SNMP ARP cache of a printer. So we couldn't scan the device directly, but we could tell you it was there cuz it was hanging off of a printer's ARP cache. The printer could see it, even though our scanner couldn't see it directly. And it was just driving them crazy, they literally tore the entire network apart, rebuilt it from scratch, and the device was still there and they're like, ‘your scanner's busted. You guys are lying.’ Like we're not lying. I swear, it's there. Like unless this printer’s just making up its ARP cache, it is there. Ends up, it was the phone in his pocket the entire time so they had a Huawei peer series phone.”

Gordon chimed in to compliment one of runZero’s capabilities: DCE/RPC fingerprinting. He spoke about some of the aspects that stood out to him like finding ways to “not only remotely determine the OS of Windows systems without authentication, but also [identify] all sorts of other random stuff like network cards.” They discussed the potential of Nmap incorporating runZero’s DCERPC into their own product as well.

Gordon also mentioned the runZero blog as a source of ideation for new Nmap capabilities. He made a point to highlight runZero’s asset version identification. “That was another case where I was just like, you know, reading [the blog post runZero] did on it, I’m like, this is a good idea. We should steal that,” Gordon laughed.

Open source pros and cons

Towards the end of the interview, HD and Gordon were asked about what stories they could share from working with the open source community. HD mentioned that one of the merits of open source communities is that people can bring up contributions anonymously so they can highlight an important problem or solution to a vendor's regardless of where they are employed. “Metasploit has always been…a long-term dumping ground for code because it’s a great way to move the bar and get vendors to react,” HD said. That communication capability is an important part of the open source community.

Gordon focused on the potential for bad actors to take advantage of open source. “I got a call from the FBI and they told me that they thought the Nmap organization systems were being hacked,” he recounted. And the FBI was right. Luckily, Nmap was able to boot out the hackers and rebuild the Nmap systems in time. Gordon said, “I wouldn’t call [the hackers] the open source community by any means, and you know, Nmap has benefitted so much from the open source community, but you have to remember that there are also always the more malicious types.”