Recog development with runZero


Overview #

Recog may be one of the most underrated open source security projects of all time. Recog started off in the early 2000s as the fingerprinting backend for Rapid7's Nexpose (aka InsightVM) vulnerability scanner. It was released as open source in 2014 and integrated into the Metasploit Framework and Metasploit Pro products. The fingerprint coverage continues to grow through analysis of the Project Sonar data and contributions by our team as part of runZero development.

At its core, Recog is a collection of XML files, each containing a list of fingerprints, with each fingerprint consisting of a regular expression and a series of values to assert on match. These XML files each correlate to a specific protocol response or field. For example, the http_servers.xml database matches the normalized value of the HTTP Server header. To use this database, the fingerprints are processed sequentially, stopping after the first match, and recording the asserted values in the fingerprint. Regular expression capture parameters can be used to extract subfields and assert these in the match values. These capture parameters can be combined with static strings to build up more complicated match values.

The following fingerprint matches the HTTP Server value of Microsoft-IIS/7.5:

  <fingerprint pattern="^Microsoft-IIS/7.5$">
    <description>Microsoft IIS 7.5 runs on Windows Server 2008 R2 (and Windows 7)</description>
    <param pos="0" name="service.vendor" value="Microsoft"/>
    <param pos="0" name="service.product" value="IIS"/>
    <param pos="0" name="" value="IIS"/>
    <param pos="0" name="service.version" value="7.5"/>
    <param pos="0" name="service.cpe23" value="cpe:/a:microsoft:iis:7.5"/>
    <param pos="0" name="os.vendor" value="Microsoft"/>
    <param pos="0" name="" value="Windows"/>
    <param pos="0" name="os.product" value="Windows Server 2008 R2"/>
    <param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>

Those <param> items result in those specific keys and values being asserted if this match is successful.

This process applies for every other supported match type. New databases can be created for almost anything and can assert one of the standard values, or a custom match value (often in OID format).

Recog development with runZero #

If you would like to get started with Recog development, the runZero Scanner (available in our free tier) is a quick way to get rolling. The --fingerprints (shorthand: -f) option can be used to specify an alternate fingerprint database and the --fingerprints-debug option can by used to write scan log entries for sucessful and missing matches.

Install the runZero Scanner into your path and clone a copy of the Recog repository:

$ git clone

To test the HTTP fingerprints, we can run a new runZero scan, limiting it to just the SYN probe and two HTTP ports:

 $ sudo runzero-scanner --text -o test.log --overwrite --probes syn -p 80,443 -f ./recog/xml/ --fingerprints-debug

Running this scan will produce a bunch of FP-MATCH and FP-FAIL output from the fingerprinting engine:

Aug  5 14:05:01.100 [INFO] loading alternate fingerprints from ./recog/xml/
Aug  5 14:05:01.311 [INFO] writing results to C:\Users\Developer\go\recog-test\test.log
Aug  5 14:10:24.281 [INFO] [recog] html_title.xml FP-FAIL "index"
Aug  5 14:10:24.281 [INFO] [recog] http_servers.xml FP-FAIL "App-webs/"
Aug  5 14:10:24.285 [INFO] [recog] favicons.xml FP-MATCH "7de37347ff2f9277824b3e3cfe4a8ada" to "^7de37347ff2f9277824b3e3cfe4a8ada$" (TRENDnet IP Camera)
Aug  5 14:10:24.285 [INFO] [recog] http_servers.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to "(?i)^Apache(?:-AdvancedExtranetServer)?(?:/([012][\\d.]*)\\s*(.*))?$" (Apache)
Aug  5 14:10:24.286 [INFO] [recog] apache_os.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to ".*\\(Ubuntu\\).*" (Ubuntu)
Aug  5 14:10:24.295 [INFO] [recog] html_title.xml FP-FAIL "Panopticon Human Observer POHO-1984"
Aug  5 14:05:40.744 [INFO] scan completed in 39.4086113s
Aug  5 14:05:41.139 [INFO] identified 11 unique assets through correlation
Aug  5 14:05:41.330 [INFO] artifact generation completed

A list of failed matches can be pulled from the scan log using grep:

$ grep FP-FAIL test.log/scan.log

If the HTML title of Panopticon Human Observer was a reliable match and the device model was POHO-1984 we could create a new fingerprint at the end of html_title.xml:

  <fingerprint pattern="^Panopticon Human Observer (POHO-\d+)$">
    <description>Panopticon Human Observer IP Camera</description>
    <example>Panopticon Human Observer</example>
    <param pos="0" name="hw.vendor" value="Panopticon"/>
    <param pos="0" name="hw.device" value="Web cam"/>
    <param pos="1" name="hw.product"/>

This fingerprint captures the model name in the regular expression and then asserts that value as the hardware product.

Saving this XML file and rerunning the scan should now show:

Aug  5 14:25:19.185 [INFO] [recog] html_title.xml FP-MATCH "Panopticon Human Observer POHO-1984" (Panopticon Human Observer IP Camera)

To verify that the XML fingerprints are well-formed, there are two tools available, recog_verify and rspec.

First, make sure a recent version of Ruby is installed and configure the dependencies:

$ cd recog
$ gem install bundler
$ bundle install

To use recog_verify, specify the XML file as the argument:

$ ruby bin/recog_verify xml/html_title.xml

The full test suite can run be via rspec:

$ rspec

To normalize the vendor, hardware, operating system, and service names, you can use recog_normalize (also in bin).

The CPE mappings can be regenerated via the script.

Once you are happy with your changes, you can submit this to the upstream Recog project by following the contribution guide.

Acknowledgements #

Recog is an open source project, but didn't start off that way, and many of the contributors from the early days deserve credit for what the project became. The Rapid7 Nexpose team did a phenomonal job of building a future-proof fingerprinting system. The late Jon Hart was critical in shepherding Recog from a Nexpose asset to a successful open source project. The entire Rapid7 team has been instrumental in keeping this project alive and fed with the latest Sonar data. The folks behind the NICER report did an incredible amount of work assessing Recog's coverage and filling the gaps where needed.

Outside of the Rapid7 team, the Metasploit & runZero communities have been amazing to work with and helpful in identifying bugs and missing fingerprints. Recog has become a great example of how open source can bring value to both commercial and community projects through global collaboration.

Written by HD Moore

HD Moore is the co-founder and CEO of runZero. Previously, he founded the Metasploit Project and served as the main developer of the Metasploit Framework, which is the world's most widely used penetration testing framework.
More about HD Moore
Subscribe Now

Get the latest news and expert insights delivered in your inbox.

Welcome to the club! Your subscription to our newsletter is successful.

Related Articles

Product Release
Introducing the customizable dashboard, Wiz integration, and more!
Introducing the customizable dashboard, Wiz Integration, and other Q2 2024 enhancements to the runZero Platform.
Product Release
How to integrate your SIEM platform with runZero to create an actionable asset inventory
Learn how to combine runZero's real-time asset inventory with SIEM exports for comprehensive asset tracking and historical data analysis..
runZero Insights
Celebrating Women’s History Month with trailblazers & innovators
It’s Women’s History Month! runZero is celebrating all month long by highlighting innovative women who have been technological trailblazers.
Upcoming NYDFS regulatory requirements on asset inventory and vulnerability enumeration
Is your business prepared for the approaching deadlines for complying with the latest version of the NYDFS Cybersecurity Regulation (23 NYCRR 500)?...

See Results in Minutes

Get complete visibility into IT, OT, & IoT — without agents, credentials, or hardware.

© Copyright 2024 runZero, Inc. All Rights Reserved