Overview #
Recog may be one of the most underrated open source security projects of all time. Recog started off in the early 2000s as the fingerprinting backend for Rapid7's Nexpose (aka InsightVM) vulnerability scanner. It was released as open source in 2014 and integrated into the Metasploit Framework and Metasploit Pro products. The fingerprint coverage continues to grow through analysis of the Project Sonar data and contributions by our team as part of runZero development.
At its core, Recog is a collection of XML files, each containing a list of fingerprints, with each fingerprint consisting of a regular expression and a series of values to assert on match. These XML files each correlate to a specific protocol response or field. For example, the http_servers.xml database matches the normalized value of the HTTP Server header. To use this database, the fingerprints are processed sequentially, stopping after the first match, and recording the asserted values in the fingerprint. Regular expression capture parameters can be used to extract subfields and assert these in the match values. These capture parameters can be combined with static strings to build up more complicated match values.
The following fingerprint matches the HTTP Server value of Microsoft-IIS/7.5:
<fingerprint pattern="^Microsoft-IIS/7.5$">
<description>Microsoft IIS 7.5 runs on Windows Server 2008 R2 (and Windows 7)</description>
<example>Microsoft-IIS/7.5</example>
<param pos="0" name="service.vendor" value="Microsoft"/>
<param pos="0" name="service.product" value="IIS"/>
<param pos="0" name="service.family" value="IIS"/>
<param pos="0" name="service.version" value="7.5"/>
<param pos="0" name="service.cpe23" value="cpe:/a:microsoft:iis:7.5"/>
<param pos="0" name="os.vendor" value="Microsoft"/>
<param pos="0" name="os.family" value="Windows"/>
<param pos="0" name="os.product" value="Windows Server 2008 R2"/>
<param pos="0" name="os.cpe23" value="cpe:/o:microsoft:windows_server_2008:-"/>
</fingerprint>
Those <param> items result in those specific keys and values being asserted if this match is successful.
This process applies for every other supported match type. New databases can be created for almost anything and can assert one of the standard values, or a custom match value (often in OID format).
Recog development with runZero #
If you would like to get started with Recog development, the runZero Scanner (available in our free tier) is a quick way to get rolling. The --fingerprints (shorthand: -f) option can be used to specify an alternate fingerprint database and the --fingerprints-debug option can by used to write scan log entries for sucessful and missing matches.
Install the runZero Scanner into your path and clone a copy of the Recog repository:
$ git clone https://github.com/rapid7/recog.git
To test the HTTP fingerprints, we can run a new runZero scan, limiting it to just the SYN probe and two HTTP ports:
$ sudo runzero-scanner --text -o test.log --overwrite --probes syn -p 80,443 -f ./recog/xml/ --fingerprints-debug 192.168.0.0/24
Running this scan will produce a bunch of FP-MATCH and FP-FAIL output from the fingerprinting engine:
Aug 5 14:05:01.100 [INFO] loading alternate fingerprints from ./recog/xml/
Aug 5 14:05:01.311 [INFO] writing results to C:\Users\Developer\go\recog-test\test.log
Aug 5 14:10:24.281 [INFO] [recog] html_title.xml FP-FAIL "index"
Aug 5 14:10:24.281 [INFO] [recog] http_servers.xml FP-FAIL "App-webs/"
Aug 5 14:10:24.285 [INFO] [recog] favicons.xml FP-MATCH "7de37347ff2f9277824b3e3cfe4a8ada" to "^7de37347ff2f9277824b3e3cfe4a8ada$" (TRENDnet IP Camera)
Aug 5 14:10:24.285 [INFO] [recog] http_servers.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to "(?i)^Apache(?:-AdvancedExtranetServer)?(?:/([012][\\d.]*)\\s*(.*))?$" (Apache)
Aug 5 14:10:24.286 [INFO] [recog] apache_os.xml FP-MATCH "Apache/2.4.29 (Ubuntu)" to ".*\\(Ubuntu\\).*" (Ubuntu)
Aug 5 14:10:24.295 [INFO] [recog] html_title.xml FP-FAIL "Panopticon Human Observer POHO-1984"
Aug 5 14:05:40.744 [INFO] scan completed in 39.4086113s
Aug 5 14:05:41.139 [INFO] identified 11 unique assets through correlation
Aug 5 14:05:41.330 [INFO] artifact generation completed
A list of failed matches can be pulled from the scan log using grep:
$ grep FP-FAIL test.log/scan.log
If the HTML title of Panopticon Human Observer was a reliable match and the device model was POHO-1984 we could create a new fingerprint at the end of html_title.xml:
<fingerprint pattern="^Panopticon Human Observer (POHO-\d+)$">
<description>Panopticon Human Observer IP Camera</description>
<example>Panopticon Human Observer</example>
<param pos="0" name="hw.vendor" value="Panopticon"/>
<param pos="0" name="hw.device" value="Web cam"/>
<param pos="1" name="hw.product"/>
</fingerprint>
This fingerprint captures the model name in the regular expression and then asserts that value as the hardware product.
Saving this XML file and rerunning the scan should now show:
Aug 5 14:25:19.185 [INFO] [recog] html_title.xml FP-MATCH "Panopticon Human Observer POHO-1984" (Panopticon Human Observer IP Camera)
To verify that the XML fingerprints are well-formed, there are two tools available, recog_verify and rspec.
First, make sure a recent version of Ruby is installed and configure the dependencies:
$ cd recog
$ gem install bundler
$ bundle install
To use recog_verify, specify the XML file as the argument:
$ ruby bin/recog_verify xml/html_title.xml
The full test suite can run be via rspec:
$ rspec
To normalize the vendor, hardware, operating system, and service names, you can use recog_normalize (also in bin).
The CPE mappings can be regenerated via the update_cpe.py script.
Once you are happy with your changes, you can submit this to the upstream Recog project by following the contribution guide.
Acknowledgements #
Recog is an open source project, but didn't start off that way, and many of the contributors from the early days deserve credit for what the project became. The Rapid7 Nexpose team did a phenomonal job of building a future-proof fingerprinting system. The late Jon Hart was critical in shepherding Recog from a Nexpose asset to a successful open source project. The entire Rapid7 team has been instrumental in keeping this project alive and fed with the latest Sonar data. The folks behind the NICER report did an incredible amount of work assessing Recog's coverage and filling the gaps where needed.
Outside of the Rapid7 team, the Metasploit & runZero communities have been amazing to work with and helpful in identifying bugs and missing fingerprints. Recog has become a great example of how open source can bring value to both commercial and community projects through global collaboration.
