Self-hosting runZero

Enterprise

Background

The self-hosted version of runZero allows you to run the entire platform on-premises or within your own cloud environment. This platform is functionally identical to the hosted service, provides a fully-offline mode, and does not send any inventory data back to runZero.

While self-hosting is less common, here are a few reasons your company might choose to:

  • SOC2 compliance requirement
  • ISO compliance requirement
  • Other compliance requirement
  • Prefer data on-premise

Self-hosting requires an enterprise license and must be explicitly enabled for your account. Please contact your runZero sales representative for further information.

Requirements

Before you get started, make sure your system meets the following requirements.

Hardware requirements

  • 4 CPU Cores at 2 GHz or faster
  • 100 GB of local disk storage
  • 32 GB of RAM (more for large sites)

Software requirements

  • PostgreSQL 12 or newer (Our installer can install this for you)

Supported operating systems

  • Ubuntu 20.04 x86_64
  • Ubuntu 18.04 x86_64
  • Red Hat Enterprise Linux 8.x x86_64
  • Red Hat Enterprise Linux 7.x x86_64
  • CentOS Linux 8.x x86_64
  • CentOS Linux 7.x x86_64
  • Oracle Linux 8.4+ x86_64 with UEK 5.4 or kernel 4.18+
  • Oracle Linux 7.9+ x86_64 with UEK 5.4 or kernel 3.10+
  • Debian Linux 11.x x86_64
  • Debian Linux 10.x x86_64
  • Debian Linux 9.x x86_64

Note about Debian

By default, Debian Linux uses the su command instead of sudo. It also requires that you use the command su - to become root, not just su, in order to update the PATH to include system administration commands such as useradd. If you receive an error during installation saying executable file not found in $PATH, this is the most likely reason.

Connectivity

The self-hosted runZero platform requires the following connectivity for online updates:

  • Outbound access to console.runzero.com (13.248.161.247, 76.223.34.198) on TCP port 443 for HTTP over TLS.

The system running the runZero platform should have a static IP address.

The Explorers need to be able to validate their HTTPS connection to the platform via a TLS certificate. By default, the runZero platform installer will set up a self-signed certificate for the console’s IP address, and downloaded Explorers will be preconfigured with the appropriate URL and certificate so they can make a verified connection.

Note: Some components of the application still reference the name "Rumble" for backwards compatibility. The documentation will be updated as these are changed.

If instead you plan to use a real certificate from an internal or public CA for your runZero console deployment, you will want to assign the console server a DNS name, and edit RUMBLE_CONSOLE in the configuration to contain the fully qualified domain name. The TLS_CERT and TLS_KEY should be set to point at the appropriate certificate and private key files. The console should then be restarted via rumblectl restart. Newly downloaded Explorers will then be preconfigured with the right URL and certificate.

Because Explorers are preconfigured with the console address during download, if the IP address and/or DNS name are changed at a later date, the Explorers may need to be redeployed to receive the updated address.

Offline mode

The self-hosted version of runZero has the ability to run in offline mode. In this mode, the runZero update service is not used and offline updates must be applied manually. Enable this mode if you’re in an isolated network or you don’t want your self-hosted runZero console to make any connections to the internet. In addition to disabling online updates, offline mode also disables certain DNS probes that could reflect responses to the internet during a scan.

Installation steps

For offline installs please see offline installation.

For installs that use your own database credentials see Installation with your own PostgreSQL database.

Here’s what the installation process will do:

  • Set up PostgreSQL and create a passworded user.
  • Generate TLS certificates for your IP address located in /etc/rumble/certs.
  • Generate a configuration file at /etc/rumble/config and set some defaults.
  • Create a systemD service for the runZero platform.
  • Create all the necessary cron jobs required for the Rumble platform.

Step 1: Download and run the installer

Step 2: Initialize the admin user

After you’ve installed the runZero platform, you will have access to the runZero CLI rumblectl.

To initialize an admin user, run:

rumblectl initial [email address]

Step 3: Log in to your self-hosted console

If everything is set up correctly, you can log in to your console at https://YourInternalIPAddress.

Note that you may need to enable HTTPS to pass through the Linux system firewall. Example commands:

Ubuntu Linux: sudo ufw allow https/tcp

RHEL/CentOS/Oracle: sudo firewall-cmd --add-service=https

To make a firewall-cmd change permanent across reboots, run the command a second time with the --permanent flag added.

Installation with your own PostgreSQL database

runZero defaults to installing and configuring a PostgreSQL user and database for you. If you would like to provide your own details this option will allow you to override that behavior.

Requirements

  • PostgreSQL 13 or newer.
  • Password authentication must be enabled.
  • Two extensions are required: pg_trgm and uuid-ossp. These may be supplied as part of a contrib package rather than as part of the main PostgreSQL server install, depending on where you obtain your PostgreSQL packages.

PostgreSQL example to enable extensions and add a database/user

CREATE EXTENSION IF NOT EXISTS pg_trgm; CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE DATABASE rumble;
CREATE USER rumble WITH PASSWORD 'YOURPASSWORD';
GRANT ALL PRIVILEGES ON DATABASE rumble TO rumble;

Steps to install the self-hosted runZero platform with your own database credentials

  1. Run this install command as root:
./runzero-platform-[VERSION]-linux-amd64.bin install --manual-database
  1. Edit your runZero configuration and add your database details in /etc/rumble/config. The line you need to edit is:
DATABASE_URL=postgres://rumble:{DB_PASSWORD}@127.0.0.1:5432/rumble?sslmode=disable

Change this to match your credentials. You need to set the user, password, host, port, and database name. Here is the format:

DATABASE_URL=postgres://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}?{DB_OPTIONS}
  1. Verify the self-hosted runZero platform can connect to your database with this command. sudo rumblectl database verify
  2. Once your database is configured and verified you can restart the self-hosted runZero platform service sudo systemctl restart rumble-console

runZero updates

For offline updates please see CLI update with offline mode.

The self-hosted runZero platform must be updated prior to first use. The rumblectl command can be used to download the update and then restart the service after the update is complete.

You can update the platform and scanners at the same time or separately with the CLI update management commands.

Managing users

You can manage users inside your self-hosted runZero platform console at https://YourInternalIPAddress/team or via the runZero CLI

Some things you can manage:

  • Adding, deleting, and listing users
  • Resetting passwords
  • Changing default roles
  • Viewing details
  • Changing organization roles

CLI service management

Start the runZero service

Starts the runZero platform service.

rumblectl start

Stop the runZero service

Stops the runZero platform service.

rumblectl stop

Restart the runZero service

Restarts the runZero platform service.

rumblectl restart

Install the runZero platform

Install the runZero platform service and all required dependencies such as PostgreSQL. Creates a SystemD service, generates cron jobs, and generates a configuration file in /etc/rumble.

rumblectl install

Uninstall the runZero platform

Stop and remove the runZero platform service from SystemD and removes the generated cron jobs. This does not remove your PostgreSQL database, and it retains your data.

rumblectl uninstall

Purge the runZero platform

Stop and remove the runZero platform service from SystemD and removes the generated cron jobs. This will delete your runZero database and remove all the runZero directories /etc/rumble and /opt/rumble.

rumblectl purge

You can uninstall and purge everything except the database and your PostgreSQL settings with this flag:

rumblectl uninstall --purge --ignore-database

Run the runZero platform manually

Starts the runZero platform manually. Logs will be written to standard output.

rumblectl server

Verify your database is reachable

Attempts to connect to your database using your self-hosted runZero platform configuration. It will either succeed or display an error.

rumblectl database verify

CLI update management

Update the runZero platform and scanners

Updates the runZero platform service and runZero scanners. You can use the optional parameter force to force the update even if the current installation is the latest version.

rumblectl update [--force]

Update the runZero platform

Updates just the runZero platform service. You can use the optional parameter force to force the update even if the current install is the latest version.

rumblectl update-platform [--force]

Update the runZero scanners

Updates just the runZero scanners. You can use the optional parameter force to force the update even if the current installation is the latest version.

rumblectl update-scanner [--force]

CLI user management

Create the initial administrator account

Creates the initial admin user for a new installation. You must provide an email address.

rumblectl initial <email>

List user accounts

Lists all the users along with their email address, full name, and current roles.

rumblectl user list

Add a user account

Creates a new user account under the initial administrator user. You must provide an email address.

rumblectl user add <email>

Delete a user account

Deletes a user account. You must provide an email address. This cannot be undone.

rumblectl user delete <email>

Get user details

Gets the details for a user account. You must provide an email address. Provides information such as full name, date created, last login IP, last login time, last activity, default organization role, and their current roles. You must provide an email address.

rumblectl user details <email>

Set a user role

Sets a user’s role to the role provided. Email and role must be provided. The organization is optional. If the organization isn’t provided this sets their default role.

rumblectl user set-role <email> [organization name or organization ID]:<role>

Reset a user password

This will generate and apply a new password for the specified user. The password will be printed to the terminal. You must provide an email address.

rumblectl user reset <email>

CLI organization management

List all organizations

Lists all the organizations by their name and ID.

rumblectl organization list

Advanced configuration

The file at /etc/rumble/config can be modified to support a wide variety of configurations.

After making changes, apply them by running rumblectl restart.

Email server

runZero uses an SMTP server for user account invitations and notifications. The default configuration assumes that a SMTP server is available on localhost that does not require authentication:

SMTP_SERVER=127.0.0.1:25
SMTP_AUTH_METHOD=none

runZero will automatically use STARTTLS with plaintext SMTP servers and validate the certificate. In internal environments where the SMTP server is not using a valid TLS certificate, verification can be disabled by setting:

SMTP_TLS_NOVERIFY=true

Transport-layer TLS (instead of STARTTLS) can be configured with:

SMTP_TLS=true

If authentication is required, the following three settings should be configured:

SMTP_AUTH_METHOD=plain
SMTP_AUTH_USER=YourUsername
SMTP_AUTH_PASS=YourPassword

Emails are sent from noreply@rumble.run by default, but this can be changed by setting the FROM_EMAIL option:

FROM_EMAIL=runzero@yourcompany.int

Hostname and port

The RUMBLE_CONSOLE variable is used for creating inbound links, configuring deployed Explorers, and generating the default self-signed TLS certificate. This setting is how both users and deployed Explorers connect to the platform. Changing this setting may require regeneration of the TLS certificate and redeployment of Explorers.

RUMBLE_CONSOLE=https://{IP ADDRESS OR HOSTNAME}:443

runZero can be configured to run on a different port with the CONSOLE_PORT setting. This port defines where the console listens, but users and Explorers still connect to the RUMBLE_CONSOLE value. In most cases this should match the port specified by the RUMBLE_CONSOLE.

CONSOLE_PORT=443

TLS configuration

runZero will generate a self-signed TLS certificate and serve all web requests using HTTP over TLS. The standard configuration uses a self-signed certificate stored in the filesystem:

TLS=true
TLS_CERT=/etc/rumble/certs/cert.pem
TLS_KEY=/etc/rumble/certs/key.pem

The certificate and key file are PEM encoded and can be replaced with any valid certificate. Please ensure that any new certificate lists the value of RUMBLE_CONSOLE in the list of Subject Alternative Names.

Note that the key file must be unencrypted (not password protected). If you have a passphrase set, you can remove it using OpenSSL. For example:

# RSA key
openssl rsa -in key.pem -out newkey.pem
# DSA key
openssl dsa -in key.pem -out newkey.pem

If a TLS-terminating reverse proxy is used (AWS ELB, nginx, etc), TLS can be disabled at the application level:

TLS=false

Please note that while the web interface can be accessed over plain HTTP in this scenario, Explorers will refuse to connect to a plain HTTP port, and features like WebAuthn MFA will not work unless the site is accessed through TLS.

Specific TLS versions and ciphers can be configured.

TLS versions are chosen by minimum and maximum:

TLS_VERSION_MIN=1.2
TLS_VERSION_MAX=1.3

TLS ciphers may be chosen by profile name:

  • default: A set of strong ciphers, great for most configurations
  • nist80052: A set of strong ciphers, approved in NIST 800-52r2.
  • nist80052-aes256: A set of strong ciphers, approved in NIST 800-52r2, restricted to AES-256 variants

Please note that TLS 1.3 ciphers work differently and if a specific set of ciphers is required, both TLS_VERSION_MIN and TLS_VERSION_MAX should be set to 1.2. For example, to restrict Rumble to only NIST 800-52r2 approved ciphers using AES-256, the following configuration should be used:

TLS_VERSION_MIN=1.2
TLS_VERSION_MAX=1.2
TLS_CIPHERS=nist80052-aes256

TLS ciphers may also be chosen using comma-separated list of cipher constants.

Database configuration

runZero uses a PostgreSQL database for all platform data, except for raw scan files, change reports, and images processed from scans. By default, runZero will configure a local PostgreSQL server on the same system, with a random password, and without TLS encryption:

DATABASE_URL=postgres://rumble:{DB_PASSWORD}@127.0.0.1:5432/rumble?sslmode=disable

If separate database is preferred, any PostgreSQL server running 12.x or newer should work. TLS (sslmode=require) should be enabled when a non-local database server is configured.

The default database pool (connection count) can be modified for high throughput environments:

DATABASE_POOL_COUNT=50

Proxy configuration

runZero makes outbound connections to receive platform updates (in online mode), to connect to third-party APIs, and to delivery webhooks for notifications. If a proxy server is required, it can be configured with this setting:

HTTPS_PROXY=host:port

Storage configuration

runZero uses local file storage to store raw scan data, change reports, and images retrieved from assets. This storage directory must be owned by the rumble user and be mounted below the /opt/rumble path.

RUMBLE_STORAGE_MODE=local
RUMBLE_STORAGE_PATH=/opt/rumble/storage

Files within the storage directory are split up into two groups, assets and scans. The names of these can be changed by setting:

ASSET_BUCKET=assets
SCAN_BUCKET=scans

To use AWS S3 for file storage instead, the following configuration can be set:

RUMBLE_STORAGE_MODE=s3
ASSET_BUCKET=company-rumble-assets
SCAN_BUCKET=company-rumble-scans

If S3 is used, AWS must also be configured.

Secret configuration

runZero uses three randomly generated secret tokens to secure the platform. These keys are hexadecimal strings generated by 16 bytes of random. Compatible values can be generated by OpenSSL:

$ openssl rand -hex 16

The authentication key used for local storage HMAC operations. This key can be rotated as long as the service is restarted afterwards:

RUMBLE_STORAGE_KEY_SECRET={SECRET_32_HEX}

The session secret key is used to sign and validate browser session cookies. This key can be rotated, but doing so will invalidate all existing web logins:

SESSION_SECRET={SECRET_32_HEX}

The DB key is used for encryption of sensitive fields (user password bcrypt hashes). This key cannot be rotated, as password authentication will no longer work. If this key is changed, users must reset their password from the command-line or web interface using email before they can sign back in:

DB_KEY={SECRET_32_HEX}

AWS configuration

The AWS region is required:

AWS_REGION=us-east-1

The Access Key ID and Secret must be valid and correlated to a user with read-write access to the S3 buckets and read-only access to Secrets Manager.

AWS_ACCESS_KEY_ID=AKIA....
AWS_SECRET_ACCESS_KEY=SECRET....

AWS Secrets Manager can be used to retrieve almost any configuration setting at startup. The Secrets Manager entries should match the key names of the configuration file. The secret name can be defined with:

AWS_SECRETS_MANAGER_KEY=rumble/production

The location of the Explorer and scanner binaries can be changed with these settings. Note that these should still live under /opt/rumble or the service will not be able to load them:

RUMBLE_RELEASE_DIR=/opt/rumble/agent/
RUMBLE_SCANNER_RELEASE_DIR=/opt/rumble/scanner/

Content Security Policy

In the case of a non-standard S3 configuration (or S3-like deployment, such as minio), the Content Security Policy headers need to be configured to allow external image loads.

The CSP_IMAGES setting can be used to specify one or more (comma-delimited) external image sources:

CSP_IMAGES=https://*.custom-storage-backend.lan

In additional to CSP_IMAGES, the following additional CSP settings are available:

CSP_SCRIPTS=https://*.myscripts.lan
CSP_FONTS=https://*.myfonts.lan
CSP_STYLES=https://*.mystyles.lan

Logging

The self-hosted runZero console sends its logging output to standard output. Typically this is picked up by Linux SystemD, and stored in the journal where it can be queried with the usual journalctl tool.

runZero’s logs are output in CEE-enhanced JSON format. This is compatible with rsyslog, syslog-ng, lumberjack, DataDog, ElasticSearch, and other common tools.

The environment variable LOG_MAX_LENGTH can be set in runZero’s config file to apply a limit to the length of each log line, in bytes of UTF-8 text. A value of 0 means no limit, other values below 480 are treated as 480. runZero will attempt to preserve the most valuable logging fields when truncating log output. Note that the length limit is applied before any additional information SystemD or syslog adds to the start of the line.

The environment variable LOG_FORMAT can be set to text to disable the CEE and JSON format, and log in plain text.

LOG_FORMAT=text
LOG_MAX_LENGTH=512

HTTP Timeouts

The self-hosted runZero Console allows you to change the built-in web server’s HTTP timeouts. These can be changed through three configuration variables.

You are able to set the HTTP idle, read, and write timeout. The default settings are below and don’t usually need to be changed.

HTTP_IDLE_TIMEOUT_MINUTES=3
HTTP_READ_TIMEOUT_MINUTES=30
HTTP_WRITE_TIMEOUT_MINUTES=90

Concurrent Processing

The runZero Console can process more than one completed task at a time if the RUNZERO_CRUNCHER_INSTANCES option is set to a value greater than 1. Tasks are only processed concurrently when they exist within different organizations, or the tasks are within the same organization, but different sites, and the task itself does not require cross-site asset merging. Most third-party connectors and integrations require cross-site merging and are not able to take advantage of concurrent site processing within the same organization. Please note that the resource requirements for concurrent task processing scale linearly with the specified instance count.

The example below configures the console to process up to four concurrent tasks across all organizations:

RUNZERO_CRUNCHER_INSTANCES=4

Permissions

The self-hosted platform requires root access to install and manage from the command-line.

The platform service (rumble-console) runs as root and spawns a worker subprocess that runs as the rumble user account inside of a chroot environment (/opt/rumble). All substantive work happens within this isolated subprocess.

The following filesystem locations are used by the self-hosted platform:

/etc/rumble

Path Owner Permission Notes
/etc/rumble root 0700 Configuration files and certificates
/etc/rumble/config root 0600 A plain-text configuration file
/etc/rumble/certs root 0700 A directory containing the TLS certificate and key
/etc/rumble/certs/cert.pm root 0600 The TLS certificate in PEM format
/etc/rumble/certs/key.pm root 0600 The TLS certificate private key in PEM format

/opt/rumble

Path Owner Permission Notes
/opt/rumble/tmp rumble 0755 A temporary directory
/opt/rumble/storage rumble 0700 Contains asset and scan artifacts
/opt/rumble/console root 0755 Contains the platform executable
/opt/rumble/console/rumble-console.bin root 0755 The platform executable
/opt/rumble/agent root 0755 Contains the Explorer binaries
/opt/rumble/agent/rumble-agent-* root 0755 The Explorer binaries
/opt/rumble/scanner root 0755 Contains the Scanner binaries
/opt/rumble/agent/rumble-scanner-* root 0755 The Scanner binaries
/opt/rumble/proc root 0755 Contains copies of system /proc files
/opt/rumble/proc/cpuinfo root 0644 A copy of /proc/cpuinfo
/opt/rumble/proc/meminfo root 0644 A copy of /proc/meminfo
/opt/rumble/proc/version root 0644 A copy of /proc/version
/opt/rumble/etc root 0755 Contains copies of system files
/opt/rumble/etc/resolv.conf root 0644 A copy of /etc/resolv.conf
/opt/rumble/etc/ca-certificates.crt root 0644 A copy of the system root CA store
/opt/rumble/etc/rumble rumble 0700 Contains instance identifiers
/opt/rumble/etc/rumble/cruncher.id rumble 0700 A unique ID to identify the cruncher instance
/opt/rumble/etc/rumble/hub.id rumble 0700 A unique ID to identify the hub instance
/opt/rumble/config root 0700 Unused today

Backup and restoration

Your runZero installation and data can be backed up and restored to preserve your configuration.

runZero data backup

A backup of a self-hosted installation can be obtained by archiving the file system and database.

The file system archive includes the following paths:

  • /etc/rumble
  • /opt/rumble
  • /lib/systemd/system/rumble-console.service
  • /etc/systemd/system/multi-user.target.wants/rumble-console.service
  • /usr/bin/rumblectl

A sample file system backup command is:

# tar zcvf runzero-backup-fs.tar.gz /etc/rumble/ /opt/rumble/ \
/lib/systemd/system/rumble-console.service \
/etc/systemd/system/multi-user.target.wants/rumble-console.service \
/usr/bin/rumblectl

The PostgreSQL database must be backed up separately. A sample command is shown below:

# sudo su - postgres
$ pg_dumpall -f rumble.sql && gzip rumble.sql

runZero data restoration

To restore the runZero install, follow these steps.

  1. Stop any running runZero service:
# rumblectl stop
  1. Unpack the filesystem archive:
# tar -C / /path/to/runzero-backup-fs.tar.gz
  1. Restore the PostgreSQL database:
# sudo su - postgres
$ dropdb rumble; gzip -dc rumble.sql.gz | psql 
  1. Restart the runZero service:
sudo systemctl restart rumble-console

Support and debugging

The rumblectl command includes a debugging tool which can collect diagnostics from your server and assemble them into a zip file which you can send to support. If you are asked to do this, the command is:

rumblectl diagnostics run-script

Data is written to /opt/rumble/collector.

Alternatively you can save a copy of the script to the current directory so that you can examine it before running it:

rumblectl diagnostics write-script