Configuring for high-availability

Self-hosted installations of Rumble can be configured for high-availability. For this configuration, a load balancer is used to direct traffic to multiple console servers, which use a shared PostgreSQL cluster and storage backend. The following diagram illustrates an example architecture of a high-availability installation using AWS with two availability zones.

Self-Hosted HA Architecture

In this diagram, an application load balancer (ALB) is terminating TLS and pointing to a target group that consists of two Rumble servers, each in separate availability zones. The Rumble servers use a multi-availability-zone PostgreSQL RDS instance, also configured for the same two availability zones, and both servers point to the same set of S3 storage buckets.

Installation steps

These are the general steps for installing and configuring a high-availability instance of Rumble:

  1. Deploy an application load balancer.
  2. Install and configure the first server node.
  3. Install and configure subsequent server nodes.

Deploy a load balancer with TLS termination

In order to provide a high-availability interface to the Rumble cluster, an application load balancer should be used that receives TLS connections and forwards the HTTP requests to the available servers. This load balancer should be highly-available on its own. Any load balancer that can proxy large HTTP requests and websockets should be supported. The load balancer should add an X-Forwarded-For header to the HTTP requests sent onward to the Rumble servers. The load balancer should also support detection of unhealthy nodes in order to handle automatic failover. The /health endpoint on the Rumble servers can be used to implement health checks on the load balancer.

Prepare the first Rumble server node

The general steps for installing and configuring the first Rumble node are:

  1. Install the Rumble platform.
  2. Configure the console URL and TLS/XFF settings.
  3. Configure the database settings.
  4. Configure the shared storage settings.
  5. Verify your configuration.
  6. Create the initial user account.
  7. Add the Rumble server to the load balancer and verify the connection.

Install the Rumble platform

  1. Go to platform download page in the Rumble Console.
  2. Copy the URL to the platform installer from the download page. The download path for the installer is uniquely keyed to your license. Self-Hosted Platform Binary
  3. Download the install using Wget or cURL:
  • wget -O rumble-platform.bin https://https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/rumble-platform-vX.X.X-linux-amd64.bin
  • curl -o rumble-platform.bin https://https://console.runzero.com/download/platform-combined/<UNIQUE KEY>/62e41615/rumble-platform-vX.X.X-linux-amd64.bin
  1. Run the following command to make the installer file executable:
# chmod u+x rumble-platform.bin
  1. Run the install command with the manual database option:
# ./rumble-platform.bin install --manual-database

Configure the console URL and TLS/XFF

To configure your self-hosted installation for high-availability, the settings below should be configured in /etc/rumble/config, starting with the console URL and the TLS/XFF settings.

  • Set RUMBLE_CONSOLE variable to the hostname or IP address of the load balancer:
RUMBLE_CONSOLE=https://{IP ADDRESS OR HOSTNAME}:443
  • Disable TLS and configure the server to trust X-Forwarded-For headers with the following two lines:
TLS=false
RUMBLE_TRUST_XFF=true

Configure the database

The next step is to configure the database to be used by Rumble.

  • To do this, edit the DATABASE_URL value to match this format:
DATABASE_URL=postgres://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}?{DB_OPTIONS}

An example DATABASE_URL for AWS RDS will look like:

DATABASE_URL=postgres://rumble:[password]@rumbledb.rds-id.us-east-1.rds.amazonaws.com:5432/rumble?sslmode=require

To use a replica for read-only queries, specify DATABASE_REPLICA_URL with the same syntax. For RDS Aurora, specify the read-only endpoint as the replica URL. If no replicas are in use, only the DATABASE_URL is required, and this should point to the writer endpoint for RDS Aurora instances.

If a different type of PostgreSQL clustering is in use, make sure that the DATABASE_URL points to the highly-available endpoint for the active writer.

Configure shared storage

Next, configure the shared storage backend. This is used for storing asset data, raw scan data, and reports. Rumble supports both mounted NFS and object storage.

  • To use NFS, mount a read-write NFS share to /opt/rumble/storage and specify local storage mode with the following configuration options. The bucket names will be created under the NFS mount base directory.
RUMBLE_STORAGE_MODE=local
RUMBLE_STORAGE_PATH=/opt/rumble/storage
ASSET_BUCKET=rumble-assets
SCAN_BUCKET=rumble-scans
  • To use AWS S3 as the shared storage, either configure an IAM instance role with read-write access to two S3 buckets, or specify the AWS credentials in the configuration file.
    • To specify AWS credentials specifically for Rumble, use the following syntax:
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=[SECRET]
  • Specify that the S3 storage mechanism should be used and set the name of the two buckets:
AWS_REGION=<your-region>
RUMBLE_STORAGE_MODE=s3
ASSET_BUCKET=<company>-rumble-assets
SCAN_BUCKET=<company>-rumble-scans

If a non-AWS backend is used that is compatible with the S3 API, use the same AWS and bucket variables above but override AWS_REGION as appropriate. Reach out to Rumble support if you run into any issues with the endpoint configuration.

Verify the configuration

  1. After saving your configuration file, use this command to verify the self-hosted Rumble platform can connect to your database:
sudo rumblectl database verify
  1. Once your settings are configured and verified you can restart the self-hosted Rumble platform service:
sudo systemctl restart rumble-console

Create the initial user account

  • Use the rumblectl initial <your-email> command to create the initial user account.

Add the Rumble server to the load balancer and verify the connection

  • Point the load balancer to the new Rumble server’s IP on port 80 (unless modified).
  • Browse to the load balancer URL over HTTPS using the configured DNS name.
  • Verify that the user account configured previously can authenticate correctly.

Installing and configuring subsequent nodes

  • Download and install the Rumble platform binary on each node as above with the --manual-database option specified.
  • Copy the configuration file from the first node to /etc/rumble/config on each subsequent node.
  • After saving your configuration file, use this command to verify that the self-hosted Rumble platform can connect to your database:
sudo rumblectl database verify
  • Restart the platform service by running:
sudo systemctl restart rumble-console

Rumble nodes should be spread across multiple availability zones and added to the load balancer’s target group. Verify that requests are reaching each individual node by reviewing the syslog entries on that node.

AWS advanced configuration

AWS Secrets Manager can be used to retrieve configuration settings at startup. The Secrets Manager entries should match the key names of the configuration file. The secret name can be defined with:

AWS_SECRETS_MANAGER_KEY=rumble/production

To ensure that Rumble can access the Secrets Manager, ensure that an AWS_REGION is set in the configuration file, and that the instance has an IAM instance role with permission to read the Secrets Manager key or the AWS credentials have been specified directly in the configuration file. With a Secrets Manager configuration, only the AWS options need to be present in the configuration file and the rest of the settings will be loaded from the Secrets Manager key. This will require copying and defining various settings from the generated configuration file to the Secrets Manager key-value pairs.

Note that many settings need to be identical across the Rumble cluster for web requests and authentication to work correctly (including session secrets, DB encryption key, and the console URL).