Rudder security & hardening

Rudder provides secure defaults whenever possible, but depending on your use case you can add additional configurations to make your Rudder installation more secure.

Server

Server certificate

The Rudder root server accepts connections from both nodes and user browsers. If you want a different certificate for each, see next section (HTTPS).

The new certificate must have Digital Signature, Key Encipherment and Data Encipherment in its X509v3 Key Usage and UID=root in its Subject DN. If you cannot provide one, see next section (HTTPS).

If you have agents using port 5309 (CFEngine protocol) you must generate the new certificate with the same private kay as the old one, which can be found in /var/rudder/cfengine-community/ppkeys/localhost.priv on the server. If you can’t, see next section (HTTPS).

Provisioning a new certificate

To change the server certificate, you first need a new certificate. The new certificate MUST have Digital Signature, Key Encipherment and Data Encipherment in its X509v3 Key Usage.

If you changed the certificate private key, you must first do the following steps:

  • Get the new certificate’s public key hash with this command echo "sha256//$(openssl rsa -in private.key -outform der -pubout 2>/dev/null | openssl dgst -sha256 -binary | openssl enc -base64)"

  • Get the existing server key hash with the command rudder agent info on the server

  • Modify /opt/rudder/etc/rudder-web.properties and change rudder.server.certificate.additionalKeyHash= to contain both separated by a semicolon ;

  • Restart the server systemctl restart rudder-jetty

This allows the new certificate to be accepted by agent. You then need to make sure this is applied to all agents before continuing. Usually, waiting for 24h is enough, but do not forget nodes that might be down during this period.

HTTPS

  • By default, all HTTPS traffic is handled by the same virtual host in Apache httpd. To allow hardening the connection security, a first step is to split public Web/API from internal node-server communication. You can do so by commenting the default virtual host in /etc/apache2/sites-enabled/rudder.conf or /etc/httpd/conf.d/rudder.conf (depending on the distribution), and uncommenting the two separate configurations. You need to define a method to distinguish both configurations, in general with a different port (which allows applying different firewall rules too) or a specific domain for Web/API.

  • Once your virtual hosts are split, set up a valid certificate for the Web/API. This allows validating the server identity from Web browsers and API clients. You can use an internal PKI or a publicly trusted certificate, like with Let’s Encrypt. The certificate configuration is done in /etc/apache2/sites-enabled/rudder.conf or /etc/httpd/conf.d/rudder.conf depending on your distribution, in the Web/API virtual host. To configure the TLS settings of the Web/API virtual hosts depending on your requirements and platform, we recommend using Mozilla SSL configurator generator.

  • HTTP Strict Transport Security (HSTS) ensures the user’s browser will always use HTTPS to connect to your server. It is not enabled by default as it may conflict with other services served from the same domain (e.g. package repositories). If you only use HTTPS with your Rudder’s server domain, you can enable the HSTS header in /opt/rudder/etc/rudder-web.properties by modifying or adding the following properties (and restarting the rudder-jetty service to apply it):

rudder.server.hsts=true
rudder.server.hstsIncludeSubDomains=true

  • You may want to hide the Apache httpd version from the headers. It cannot be done inside Rudder’s configuration as it is a global httpd setting. To do so, you need to set the ServerTokens parameter to the Prod value (in /etc/httpd or /etc/apache depending on your distribution).

Authentication and user management

  • It is recommended to use an external authentication backend exposing an OpenID Connect or OAUTH2 interface with a second authentication factor (TOTP, WebAuthn, etc.), configured through the auth-backends plugins.

  • In case you use local Rudder users, your passwords must be hashed with bcrypt, which is now the only hash algorithm used when generating new passwords in the user management page. It may still be the case that users use other kind of unsalted hashed password (especially before Rudder 6.0), so they are still allowed but they will be deprecated soon in favor of bcrypt. You can check the documentation of /opt/rudder/etc/rudder-users.xml and of the user management page for the migration guide.

  • Give minimal privileges to Rudder users using roles. You can create custom roles to match your needs in /opt/rudder/etc/rudder-users.xml.

  • When a user have not logged in, by default for 180 days, the account is marked as disabled. You can change this value by modifying or adding the parameter rudder.users.cleanup.account.disableAfterLastLogin in /opt/rudder/etc/rudder-web.properties, for example if we want to disable a user after 100 days of inactivity we can set the parameter like this:

rudder.users.cleanup.account.disableAfterLastLogin=100d

  • A user can be deleted after a certain period of time. By default users are never deleted. You can define an automatic deletion by modifying or adding the parameter rudder.users.cleanup.account.deletedAfterLastLogin in /opt/rudder/etc/rudder-web.properties, for example if we want to delete a user after 365 days of inactivity we can set the parameter like this:

rudder.users.cleanup.account.deletedAfterLastLogin=365d

  • When an user is deleted (for example removed from rudder-user.xml file), then the user information, like status changes, are not purged immediately (typically to be able to do post-deletion accountability). You can modify the period of time (by default 30 days) after which the purge is performed by editing or adding the parameter rudder.users.cleanup.purgeDeletedAfter, like this:

rudder.users.cleanup.purgeDeletedAfter=60d

  • Give minimal privileges to Rudder users using roles (through the user-management plugin). You can create custom roles to match your needs in /opt/rudder/etc/rudder-users.xml.

  • After a configuration change on a user that might have a security impact, the user session will be immediately invalidated (when user password or roles change, or when user is disabled or deleted).

  • When upgrading from a previous version to 8.0, regenerate all your exiting API tokens to use the more secure hashed storage.

  • Give minimal privileges to your HTTP API tokens (though the api-authorizations plugin) and use a different token for each application or use-case.

Session login timeout

By default users are logged out after 30 minutes of user inactivity. The idle session expiration time can be configured in /opt/rudder/etc/rudder-web.properties by modifying or adding the following property (and restarting the rudder-jetty service to apply it):

rudder.auth.idle-timeout=15 minutes

Agent

  • If you don’t use the remote-run feature (using the Trigger agent button in node details or the nodes/apply HTTP API), you can totally disable the service listening on the network. This way, your Rudder agents will not expose anything on the network. To do so, you need to disable the rudder-cf-serverd sub-service, with a Rudder technique or the systemctl disable --now rudder-cf-serverd command.

    • Note: On Rudder servers and relays this service is necessary as it handles policy distribution, and hence cannot be disabled.

Network

Generalities

  • Even if all communications are encrypted with TLS 1.3, we recommend avoiding Rudder communications across public networks, and using a VPN in case you need access outside your private network.

  • Add firewall rules to limit access to Rudder ports to relevant systems.

Pre-establish trust

By default, Rudder agent and server/relay mutually authenticate (with mTLS) based on a Trust On First Use (TOFU) principle. The first inventory will provide a key that will be stored by the server on node acceptation, and the agent will pin the server certificate present in the first policies downloaded. The following steps allow configuring Rudder to perform certificate validation from the start by establishing mutual trust with pre-shared information during agent provisioning.

Provisioning an agent with pre-established server trust

Pre-configuring the server information requires you to know the server key hash to trust. Here the policy server refers to your node’s server, either a relay or a server.

On the policy server, get the key hashes with:

# rudder agent info
[...]
Key/Certificate
           Key hash: MD5=595221aa16c00dcec78ba1259d7708de
           Key hash: sha256//2cMrJbjcdh25hJkzFVlyKs62DXsaFmumbcFpQ6/ZguU=
[...]

On the agent, provide the hash to trust when configuring the policy-server:

# rudder agent policy-server -t sha256//2cMrJbjcdh25hJkzFVlyKs62DXsaFmumbcFpQ6/ZguU= mypolicyserver

You can check that the trust is established by running the following command:

[root@rudder ~]# rudder agent info
[...]
        Key pinning: full
[...]

Pre-provision a node on the server

In order to automate node acceptation on the server while checking the node’s identity, you can use the node creation API. You can make a call to this API in your node provisioning process, and provide the agentKey parameter (which should contain the content of /opt/rudder/etc/ssl/agent.cert on the node), and the accepted status.

This way, once the node sends its first inventory, if the node id and certificate match the pre-provisioned entry, the node inventory will be updated. Please note that you can also pre-define node properties using this API to ensure that the pre-provisioned node will immediately join in the correct groups

Using HTTPS port 443 only between Rudder server and nodes

By default, Linux nodes get their policies from the Rudder server by connecting to it on 5309/tcp, and send their reports back to the server by connecting to it on 443/tcp (https).

Windows nodes, when supported, always only use HTTPS to port 443/tcp.

If needed, it is also possible to configure the Linux nodes to communicate with the server using only HTTPS to port 443/tcp.

Limitations

Limiting Linux nodes to using 443/tcp comes with the following limitations:

  • Remote agent run trigger from the server’s GUI becomes impossible, as it relies on the rudder-cf-serverd service running on the node and using port 5309/tcp.

  • Recursive file copies from the server’s shared folder to the nodes become impossible. Only individual file copies with the file_from_shared_folder method works normally.

  • Relays synchronization will only work in the rsync mode which is not the default and requires additional configuration.

Linux Nodes configuration

If you wish to use Rudder to configure nodes to use HTTPS only, you first need to configure the nodes and only then turn off '5309/tcp' on the server.

  • Edit the agent configuration file (create it if absent): /opt/rudder/etc/agent.conf and insert the line:

https_only=true

That’s all that is needed on the nodes sides.

After having performed that change on a node, check that the Rudder agent if still working properly by issuing the command rudder agent run.

Server configuration

The server doesn’t need any additional configuration te be able to serve nodes using https only.

However, if we want to limit the server itself to accept only https and not use 5309/tcp at all, then the webapp’s configuration file, /opt/rudder/etc/rudder-web.properties needs to be modified to add the following parameter:

rudder.server.certificate.httpsOnly=true

This modification disables CFEngine-based file copies and disables the rudder-cf-serverd service everywhere (on both server and nodes).

  • The rudder-jetty service needs to be restarted after the change has been made.

  • The configuration change will be propagated to the nodes by the server, and will be persistent: rudder-cf-serverd will be stopped and disabled on the server and nodes. Should this need be to be reverted, it would have to be done manually on the nodes (or using a rudder ad-hoc technique).

Authentication with CA-signed certificates

If only the above changes are performed, the authentication system between the nodes and server will be performed in the usual way: TOFU (Trust On First Use) using certificates auto-generated on the server and nodes.

It is however possible to use a local CA to manually create https certificates for the Rudder server and each node.

For doing this, the additional following steps must be performed:

Certificates

On each of the node (including the server and relays), you need:

  • A private key. There is no specific requirement on the type (as long as it can be used with TLS 1.3), or size.

  • An X.509 certificate signed by a recognized CA. It must:

  • have digitalSignature as keyUsage.

  • have a DN Subject containing UID=<node id> where <node id> is the node’s ID in Rudder.

  • If it is not possible to have a certificate generated for a node with UID=<node id> in the DN Subject, it is then possible to put it in the Subject Alternative Name extension.

CA-signed certificates installation on the nodes

Install the generated certificate and key for the server’s agent:

  • Place the server’s agent private key into /opt/rudder/etc/ssl/agent-http.key

  • Place the server’s agent certificate into /opt/rudder/etc/ssl/agent-http.cert

  • If relevant, put the custom CA (with intermediate certificates if needed) into /var/rudder/lib/ssl/policy_server_ca.pem.

Edit the agent configuration file, `/opt/rudder/etc/agent.conf, and add the lines:

https_only=true
cert_validation=true

CA-signed certificates installation on the Rudder server and relays

You can use the same certificate for client-server communication using HTTPS protocol and the Rudder web interface. In this case, in addition to the UID, it must include the server’s hostname in the Subject Alternative Name extension.

If using a different certificate, it must have:

  • The server’s hostname in the Subject Alternative Name extension, and can hence be a standard web server certificate (compatible with Let’s Encrypt for example)

  • The digitalSignature and keyEncipherment keyUsage present.

Note: The hostname included in the certificate must be the one used by the agents to connect to the server (their policy server configuration).

Certificate and key installation

Install the generated certificate and key for the server’s agent:

  • Place the server’s agent private key into /opt/rudder/etc/ssl/agent-http.key

  • Place the server’s agent certificate into /opt/rudder/etc/ssl/agent-http.cert

If you use a separate certificate (and key) for the HTTP server:

  • Place the private key in opt/rudder/etc/ssl/server.key

  • Place the certificate in /opt/rudder/etc/ssl/server.cert

Configuration

Modify the webapp’s configuration file, /opt/rudder/etc/rudder-web.properties:

  • rudder.server.certificate.ca.path is used when provided, in place of the system store

  • rudder.server.certificate.httpsOnly disables CFEngine-based file copies. It disables the rudder-cf-serverd service everywhere (on nodes and server, see explanations in the previous section).

  • rudder.server.certificate.validation changes the validation mode of all HTTPS connections. It requires rudder.server.certificate.httpsOnly to be enabled.

The resulting /opt/rudder/etc/rudder-web.properties part will look like:

rudder.server.certificate.httpsOnly=true
rudder.server.certificate.validation=true
# If omitted, the system CA store is used
rudder.server.certificate.ca.path=/path/to/ca.pem

  • The rudder-jetty service needs to be restarted after the changes have been made.

  • The configuration changes will be propagated to the nodes by the server, and will be persistent on the nodes, to prevent downgrade attacks. If you want to change one of these parameters, you need to update it on all nodes.

Once rudder-jetty has been restarted, the agent must be ran on the Rudder server itself, to apply the changes: rudder agent run.

Troubleshooting

Services

To check the rudder-cf-serverd service is correctly stopped, run systemctl status rudder-cf-serverd.service. It should be stopped and disabled:

rudder-cf-serverd.service - CFEngine file server
	Loaded: loaded (/lib/systemd/system/rudder-cf-serverd.service; disabled; preset: enabled)
	Active: inactive (dead)
````

**Certificates and keys**

To check a node’s certificate:

* The command `openssl x509 -in agent1.pem -noout -subject` should return a line like `subject=UID=1ba6a1b9-e7d4-4588-a760-f6405ec3d445`. If the `UID` is not present, the node will not be able to download its policies from the server even if the certificate is valid.
* The command `openssl x509 -noout -ext subjectAltName -in certs/server.crt` for server certificates should return something like:

[source,ini]

X509v3 Subject Alternative Name: DNS:<policy server hostname>

To check the certificate chain from the node, you can use:

[source,ini]

openssl s_client -cert_chain /opt/rudder/etc/ssl/agent-http.cert -key /opt/rudder/etc/ssl/agent-http.key -verifyCAfile /var/rudder/lib/ssl/policy_server_ca.pem <SERVER HOSTNAME>:443 ``


← Uninstall Webapp administration →