Recog development with runZero

(updated ), by HD Moore
Rumble, Inc. is now runZero!

Rumble Network Discovery is now runZero!

Overview #

Recog may be one of the most underrated open source security projects of all time. Recog started off in the early 2000s as the fingerprinting backend for Rapid7’s Nexpose (aka InsightVM) vulnerability scanner. It was released as open source in 2014 and integrated into the Metasploit Framework and Metasploit Pro products. The fingerprint coverage continues to grow through analysis of the Project Sonar data and contributions by our team as part of runZero development.

At its core, Recog is a collection of XML files, each containing a list of fingerprints, with each fingerprint consisting of a regular expression and a series of values to assert on match. These XML files each correlate to a specific protocol response or field. For example, the http_servers.xml database matches the normalized value of the HTTP Server header. To use this database, the fingerprints are processed sequentially, stopping after the first match, and recording the asserted values in the fingerprint. Regular expression capture parameters can be used to extract subfields and assert these in the match values. These capture parameters can be combined with static strings to build up more complicated match values.

The following fingerprint matches the HTTP Server value of Microsoft-IIS/7.5:

  <fingerprint pattern="^Microsoft-IIS/7.5$">
    <description>Microsoft IIS 7.5 runs on Windows Server 2008 R2 (and Windows 7)</description>
    <param pos="0" name="service.vendor" value="Microsoft"/>
    <param pos="0" name="service.product" value="IIS"/>
    <param pos="0" name="" value="IIS"/>
    <param pos="0" name="service.version" value="7.5"/>
    <param pos="0" name="service.cpe23" value="cpe:/a:microsoft:iis:7.5"/>
    <param pos="0" name="os.vendor" value="Microsoft"/>
    <param pos="0" name="" value="Windows"/>
    <param pos="0" name="os.product" value="Windows Server 2008 R2"/>
    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>

Those <param> items result in those specific keys and values being asserted if this match is successful.

This process applies for every other supported match type. New databases can be created for almost anything and can assert one of the standard values, or a custom match value (often in OID format).

Recog development with runZero #

If you would like to get started with Recog development, the runZero Scanner (available in our free tier) is a quick way to get rolling. The --fingerprints (shorthand: -f) option can be used to specify an alternate fingerprint database and the --fingerprints-debug option can by used to write scan log entries for sucessful and missing matches.

Install the runZero Scanner into your path and clone a copy of the Recog repository:

$ git clone

To test the HTTP fingerprints, we can run a new runZero scan, limiting it to just the SYN probe and two HTTP ports:

 $ sudo runzero-scanner --text -o test.log --overwrite --probes syn -p 80,443 -f ./recog/xml/ --fingerprints-debug

Running this scan will produce a bunch of FP-MATCH and FP-FAIL output from the fingerprinting engine:

Aug  5 14:05:01.100 [INFO] loading alternate fingerprints from ./recog/xml/
Aug  5 14:05:01.311 [INFO] writing results to C:\Users\Developer\go\recog-test\test.log
Aug  5 14:10:24.281 [INFO] [recog] html_title.xml FP-FAIL "index"
Aug  5 14:10:24.281 [INFO] [recog] http_servers.xml FP-FAIL "App-webs/"
Aug  5 14:10:24.285 [INFO] [recog] favicons.xml FP-MATCH "7de37347ff2f9277824b3e3cfe4a8ada" to "^7de37347ff2f9277824b3e3cfe4a8ada$" (TRENDnet IP Camera)
Aug  5 14:10:24.285 [INFO] [recog] http_servers.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to "(?i)^Apache(?:-AdvancedExtranetServer)?(?:/([012][\\d.]*)\\s*(.*))?$" (Apache)
Aug  5 14:10:24.286 [INFO] [recog] apache_os.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to ".*\\(Ubuntu\\).*" (Ubuntu)
Aug  5 14:10:24.295 [INFO] [recog] html_title.xml FP-FAIL "Panopticon Human Observer POHO-1984"
Aug  5 14:05:40.744 [INFO] scan completed in 39.4086113s
Aug  5 14:05:41.139 [INFO] identified 11 unique assets through correlation
Aug  5 14:05:41.330 [INFO] artifact generation completed

A list of failed matches can be pulled from the scan log using grep:

$ grep FP-FAIL test.log/scan.log

If the HTML title of Panopticon Human Observer was a reliable match and the device model was POHO-1984 we could create a new fingerprint at the end of html_title.xml:

  <fingerprint pattern="^Panopticon Human Observer (POHO-\d+)$">
    <description>Panopticon Human Observer IP Camera</description>
    <example>Panopticon Human Observer</example>
    <param pos="0" name="hw.vendor" value="Panopticon"/>
    <param pos="0" name="hw.device" value="Web cam"/>
    <param pos="1" name="hw.product"/>

This fingerprint captures the model name in the regular expression and then asserts that value as the hardware product.

Saving this XML file and rerunning the scan should now show:

Aug  5 14:25:19.185 [INFO] [recog] html_title.xml FP-MATCH "Panopticon Human Observer POHO-1984" (Panopticon Human Observer IP Camera)

To verify that the XML fingerprints are well-formed, there are two tools available, recog_verify and rspec.

First, make sure a recent version of Ruby is installed and configure the dependencies:

$ cd recog
$ gem install bundler
$ bundle install

To use recog_verify, specify the XML file as the argument:

$ ruby bin/recog_verify xml/html_title.xml

The full test suite can run be via rspec:

$ rspec

To normalize the vendor, hardware, operating system, and service names, you can use recog_normalize (also in bin).

The CPE mappings can be regenerated via the script.

Once you are happy with your changes, you can submit this to the upstream Recog project by following the contribution guide.

Acknowledgements #

Recog is an open source project, but didn’t start off that way, and many of the contributors from the early days deserve credit for what the project became. The Rapid7 Nexpose team did a phenomonal job of building a future-proof fingerprinting system. The late Jon Hart was critical in shepherding Recog from a Nexpose asset to a successful open source project. The entire Rapid7 team has been instrumental in keeping this project alive and fed with the latest Sonar data. The folks behind the NICER report did an incredible amount of work assessing Recog’s coverage and filling the gaps where needed.

Outside of the Rapid7 team, the Metasploit & runZero communities have been amazing to work with and helpful in identifying bugs and missing fingerprints. Recog has become a great example of how open source can bring value to both commercial and community projects through global collaboration.

HD Moore
Written by HD Moore

HD Moore is the co-founder and CTO of runZero. Previously, he founded the Metasploit Project and served as the main developer of the Metasploit Framework, which is the world's most widely used penetration testing framework.

Similar Content

August 4, 2020

Rumble 1.10: Continuous Scans, Site Defaults, and More!

Rumble Network Discovery is now runZero! Overview # Rumble 1.10 is live with continuous scanning, user interface updates, an event log, updates to the scan engine, additional fingerprints, and a new way to keep recurring scans in sync with their sites! Continuous Scanning # …

Read More

July 7, 2020

Rumble 1.9.0: Scan Engine Updates, Reports, and More!

Rumble Network Discovery is now runZero! Overview # Rumble 1.9.0 is out with major updates to the scan engine, reports, fingerprinting, user interface, documentation, and much more! Scan Engine # Folks who scan external assets using their hostnames will now see asset …

Read More

May 4, 2020

Rumble 1.7.0: Reporting, Fingerprints, and More!

Rumble Network Discovery is now runZero! Overview # Version 1.7.0 of Rumble Network Discovery is live with big updates to reporting. The Analysis Reports introduced in version 1.6.2 are now joined by a new Subnet Grid Report, linked off the main Subnets Report under the …

Read More