Upgrade notes

Before upgrading a Rudder server, you should make a backup by following the backup procedure.

Webapp hardening

Migration of the rudder-users.xml file

As part of the web application hardening, the rudder-users.xml file has been moved from its previous location: /opt/rudder/etc/rudder-users.xml to: /opt/rudder/etc/webapp/rudder-users.xml This migration is handled automatically during the upgrade.

Hardening changes for the configuration-repository

All changes to the configuration-repository via Git must now be performed using the rudder-webapp user. To switch from the root user to the rudder-webapp user, use the command rudder server su.

Adding writable directories to systemd hardening

As part of the Rudder 9.2 release, the web application is now sandboxed using systemd directives.

If your configuration requires additional writable directories, you must override the default service configuration by running systemctl edit rudder-jetty, and adding the required directories to the ReadWritePaths directive.

For example:

[Service]
ReadWritePaths=/opt/rudder /var/rudder /var/log /etc/cron.d /etc/systemd/system /path/to/writable/directory

Tenants

Rudder parameters syntax deprecation

The syntax to access the properties is being made more consistent, with the deprecation of old variants, in favor of the ${rudder.properties[X]} syntax.

The following cases are deprecated and will stop working in directive parameters, breaking the generation. Usage in techniques is deprecated and will stop working in a future version.

  • ${rudder.param.X}

  • ${rudder.parameters[X]}

  • ${rudder_parameters.X}

Windows agent

Requirements

  • The 9.2+ policies now requires .NET standard 2.0 to be installed on the agents.

  • The Rudder event logs size is no longer forced to 50MB and now defaults to the Windows default size.

Moving from scheduled tasks to AgentD

Until 9.2.0, the Windows Rudder agent runs were triggered through Windows Scheduled Tasks. They are now triggered by a dedicated service named RudderAgentD. The legacy scheduled tasks are replaced only when both agent and policy server version are 9.2.0 or later, and the agent is not configured to run under an account other than SYSTEM.

Windows agents running under a custom account

If your agents run under an account other than SYSTEM, read this before upgrading

Agents configured with a custom account (with a non-empty C:\Program Files\Rudder\etc\scheduledTaskUser.conf) must be upgraded with the options SCHEDULEDTASKUSER and SCHEDULEDTASKPASSWORD passed to the installer to keep the legacy scheduled tasks.

The migration toward the new RudderAgentD service using a custom account requires that you:

  • have a 9.2+ policy server

  • upgrade the agent to 9.2+ using the options above

  • modify the service settings to make it run using your dedicated service account

  • remove the C:\Program Files\Rudder\etc\scheduledTaskUser.conf

  • start the service RudderAgentD

The next agent run should remove the legacy scheduled tasks. When migrated to RudderAgentD the agent will no longer require that you pass SCHEDULEDTASKUSER and SCHEDULEDTASKPASSWORD options to the installer when upgrading to future versions as the service settings are not managed by the agent, and your settings will be kept between upgrades.

AgentD configuration

The new service triggers the scheduled agent runs and inventories. It is configured by a TOML conf file in C:\Program Files\Rudder\etc\agentd.conf, managed by the agent. The service logs its events under the Applications and Services Logs/RudderAgentD Windows Event Viewer.

If the configuration file is malformed, the service will still start, using a default configuration that runs the agent every 5 minutes.

Changes to the installer

The graphical installer of the Windows agent now uses the Rudder colors and follows Microsoft Installer standards more closely:

  • Downgrading is no longer possible without uninstalling the newer version first

  • Re-running the installer of the currently installed version now performs an MSI repair

Compliance API

The compliance API endpoints are now integrated into each item’s type APIs for consistence.

Old endpoint New corresponding endpoint

/compliance/directives

/directives/compliance

/compliance/directives/{directiveId}

/directives/{directiveId}/compliance

/compliance/groups/{targetOrNodeGroupId}

/groups/{targetOrNodeGroupId}/compliance/global

/compliance/groups/{targetOrNodeGroupId}/target

/groups/{targetOrNodeGroupId}/compliance/targeted

/compliance/nodes

/nodes/compliance

/compliance/nodes/{nodeId}

/nodes/{nodeId}/compliance

/compliance/rules

/rules/compliance

/compliance/rules/{ruleId}

/rules/{ruleId}/compliance

The old endpoints are deprecated and will be removed in a future version.

Generation

Two main changes were made to the policy generation process:

  • A sanity check that was systematically done on the Linux policies is now disabled. It is useless in the majority of cases, and mostly indicated when developing "built-in" techniques. You can reenable it by removing the .disabled suffix of the /opt/rudder/etc/hooks.d/policy-generation-node-ready/11-linux-policies-check.disabled file on the server.

  • The internal templating engine is replaced by a more efficient implementation. You can switch back to the old implementation by setting rudder.policy.template.engine=string-template in the webapp configuration file. If this fixes something for you, please report it on the bug tracker as the deprecated backend is planned for removal.

Miscellaneous

  • The package methods will now install new dependencies when upgrading a package (by passing --with-new-pkgs).

  • The redhard_X conditions will now be defined on all RHEL-related distributions (Oracle, ROcky, Alma, CentOS).

  • The Rudder Security Configuration (rsc_01) benchmark which was deprecated since version 9.0.1 is now removed.

  • The inventory now runs ate the end of the agent run. This means the Trigger inventory method and other actions triggering an inventory (patch campaign, etc) now make an inventory in the same run agent they are called.

  • The rudder server debug command is deprecated. The rudder-cf-serverd logs now contains all relevant information.

  • A built-in TOTP MFA feature is added, and the auth-backends plugins moves is now a closed-source plugin.

  • The File from template method has a new sandboxing option, enabled by default, which prevents file system access. Only disable it on trusted templates.

Tenants

In 9.2, multi-tenancy (multi-tenants plugin) applies to all configuration objects and not only to nodes.

If you don’t use that feature or if all users and API have fulle access to all tenants, nothing changes for you. If you DO use multi-tenancy in previous version of Rudder on nodes, and as soon as one user or API account is restricted to some tenant, then you must read the following section and proceed to manual tagging of configuration objects. Missing to do so will result in tenant-retrained users loosing access to all configuration objects.

What changed

  • Up to 9.1, tenants only filtered nodes: every user saw and used all rules, directives, groups and techniques.

  • From 9.2, every configuration object carries a tenant tag, and a user only sees and changes the objects their tenants allow. This covers rules, directives, groups, techniques, global properties, their categories, campaigns, security benchmarks, event logs and change requests.

  • for nodes in 9.1, the tag attribute name was tenant and could only hold one value. In 9.2, the tag attribute name is security and it was extended to more cases (see below).

  • An object without a tag is visible to administrators only. This is the state of every existing object after the upgrade.

Tenant attribute values and meaning

Object with tag Who can sees it and uses it Who can changes it

none

administrators

administrators

open-ro

everybody

administrators

open-rw

everybody

anybody with write access

{"tenants":["zoneA","zoneB"]}

administrators, users of zoneA or zoneB

administrators, users with write access on zoneA or zoneB

Automatic changes on upgrade.

Some change aren’t linked to business decision and are automated during the migration to Rudder 9.2:

  • the root categories of rules, groups and techniques, the System groups category, the three special targets (All nodes, All nodes except policy servers, All policy servers) and the rudder global property are tagged open-ro,

  • techniques shipped by Rudder are declared open-ro in their metadata.xml, and the first boot copies that tag onto their active techniques and categories,

  • tags already set in 9.1 are kept,

  • event logs, change requests and campaign events recorded before 9.2 stay visible to administrators only.

Everything else must be tagged by hand. Rudder 9.2.0 has no UI for this and need manual action. The method depends on where the object is stored.

Manually setting tags to configuration objects.

Note: A user can only see an object if he can see its parent categories (included via open-ro visibility).

Techniques: file system

Technique editor techniques live in /var/rudder/configuration-repository/techniques/ncf_techniques/<id>/<version>/. Declare the tenant tag in their technique.yml:

security: open-ro
# or
security:
  tenants: [zoneA, zoneB]

A technique category declares it inside <xml> in its category.xml:

<security><open-ro/></security>
<!-- or -->
<security><tenants><tenant id="zoneA"/><tenant id="zoneB"/></tenants></security>

Then commit, and restart Rudder with the flag that makes it recompile every technique.yml:

cd /var/rudder/configuration-repository
git add techniques && git commit -m "Tag techniques with tenants"
su -s /bin/sh rudder-webapp -c "touch /var/rudder/lib/webapp/force_ncf_technique_update"
systemctl restart rudder-jetty

At every technique reload and at every boot, Rudder copies these file system tags onto the active techniques and their categories, so no LDAP change is needed for them. The file system value is the source of truth, so if LDAP contains another value, it will be overwritten.

Campaigns: file system

Campaigns (system updates, etc.) are JSON files in /var/rudder/configuration-repository/campaigns/. Add the tag in their info object:

{
  "info": {
    "id": "...",
    "name": "...",
    ...
    "security": {"tenants": ["zoneA"]}
  },
  ...
}

The tag is carried onto the rule and directive the campaign generates, so the campaign only applies to the nodes of its tenants. Then commit and restart:

cd /var/rudder/configuration-repository
git add campaigns && git commit -m "Tag campaigns with tenants"
systemctl restart rudder-jetty

Security benchmarks: API

Security benchmaks data are saved in several places, so for them, a minimal API was added to ease the migration. You will need to use an administrator API token:

curl -H "X-API-Token: ${TOKEN}" -H "Content-Type: application/json" -X POST \
  https://rudder.example.com/rudder/api/latest/securityBenchmarks/internal/<benchmarkId>/security \
  -d '{"security": {"tenants": ["zoneA"]}}'

"security": "open-ro" makes a benchmark visible to every tenant.

Other objects: LDAP

Rules, directives, groups, global properties, and rule and group categories carry the tag in the securityTag LDAP attribute. Stop the web application while editing it, so that the change is read at restart:

# LDAP password is in /opt/rudder/etc/rudder-passwords.conf
LDAP="-x -H ldap://localhost/ -D cn=manager,cn=rudder-configuration -W"

# list the user objects without a tag
ldapsearch ${LDAP} -LLL -b "ou=Rudder,cn=rudder-configuration" \
  '(&(|(objectClass=rule)(objectClass=ruleCategory)(objectClass=directive)(objectClass=nodeGroup)(objectClass=groupCategory)(objectClass=parameter))(!(securityTag=*))(!(isSystem=TRUE)))' dn

Then, you will need to decide which object can be seen on which tenant. Once that inventory is done, write an LDIF file with one block per object to tag:

dn: ruleId=32377fd7-02fd-43d0-aab7-28460a91347b,ou=Rules,ou=Rudder,cn=rudder-configuration
changetype: modify
replace: securityTag
securityTag: {"tenants":["zoneA"]}

dn: groupCategoryId=f0e1d2c3-...,groupCategoryId=GroupRoot,ou=Rudder,cn=rudder-configuration
changetype: modify
replace: securityTag
securityTag: open-ro

etc.

then apply it and restart:

ldapmodify ${LDAP} -f tags.ldif
systemctl start rudder-jetty

System objects (isSystem=TRUE) must stay untagged.

Tenants given by an identity provider

For the auth-backends OAuth2/OIDC and JWT providers, tenants.enabled (default false) tells whether the provider gives the tenants.

If the tenant feature is disabled in the mapping, then for consistency, users provided by OIDC get access to configuration objects: * rudder.auth.oauth2.provider.<idp>.tenants.enabled=false: the user’s tenants come from rudder-users.xml; a user absent from that file, or without a tenants attribute, gets . * rudder.auth.oauth2.jwt.provider.<idp>.tenants.enabled=false: every token gets , like an API account without a tenants attribute.

Therefor, if you use tenants, you must enable the tenants parameter on a server with restricted users: set tenants.enabled=true and a tenants.mapping on every provider.

Known limitations in 9.2.0

Both are planned to be fixed in 9.2.1.

  • Archive import (POST /archives/import) keeps the tags found in the archive, and archive export ignores tenants. A user with configuration write access can therefore read and change objects outside their tenants through archives. If your tenant isolation must hold against restricted users with configuration write access who are not administrator, wait for 9.2.1 before upgrading or restrict access to these endpoint in Apache reverse proxy configuration.

  • The web interface does not re-check a user’s tenants against the plugin after authentication. A tenant deleted in the plugin still gives access to its objects in the web interface, and a restricted user keeps their tenants when the plugin is disabled or its license expires. When you delete a tenant, also remove it from the users' configuration (rudder-users.xml or the identity provider mapping).

Plugins upgrade

If your server is connected to the Internet (directly or through a proxy), and you have configured your account in the setup wizard (or directly in /opt/rudder/etc/rudder-pkg/rudder-pkg.conf), the upgrade process will take care of upgrading to plugins to a compatible version.

If it is not the case, you will need to download the new ones from downloads.rudder.io. and install them following the usual installation procedure.

You can check the current state of plugins with:

rudder package list --all

Upgrade from Rudder 9.1 to 9.2

Upgrade from Rudder 9.1 is supported.

Upgrade from Rudder 9.0 or older

Direct upgrades from 9.0 versions and older are no longer supported on 9.2. If you are still running one of those, either on servers or nodes, please first upgrade to one of the supported versions and then upgrade to 9.2.


← on SLES on Debian/Ubuntu →