Maintenance procedures

Database maintenance

Rudder uses two backends to store information as of now: LDAP and SQL

To achieve this, OpenLDAP and PostgreSQL are installed with Rudder.

However, like every database, they require a small amount of maintenance to keep operating well. Thus, this chapter will introduce you to the basic maintenance procedure you might want to know about these particular database implementations.

Automatic PostgreSQL table maintenance

Rudder uses an automatic mechanism to automate the archival and pruning of the reports database.

By default, this system will:

  • Archive reports older that 3 days

  • Remove reports older than 15 days

  • Delete all historized compliance levels after 15 days

  • Delete logs older than twice the maximum agent run interval

Logs are extra information on agent runs, that are used for debugging purpose, and are not used in compliance status. It thus reduces the work overhead by only making Rudder handle relevant reports (fresh enough) and putting aside old ones.

This is configurable in /opt/rudder/etc/rudder-web.properties, by changing the following configuration elements:

  • rudder.batch.reportscleaner.archive.TTL: Set the maximum report age before archival (in days)

  • rudder.batch.reportscleaner.delete.TTL: Set the maximum report age before deletion (in days)

  • rudder.batch.reportscleaner.compliancelevels.delete.TTL : Set the maximum compliance age before removal (in days)

  • rudder.batch.reportsCleaner.deleteLogReport.TTL : Set the maximum retention of logs reports, (in runs number, using Nx notation, e.g. 2x for two runs, or in minutes)

The default values are OK for systems under moderate load, and should be adjusted in case of excessive database bloating (see next section).

The estimated disk space consumption, with a 5 minutes agent run frequency, is 500 to 900 kB per Directive, per day and per node, plus 150 kB per Directive per node per day for archived reports, plus 150 kB per Directive per node per day for compliance level, which equate to is roughly 5 to 7 MB per Directive per two weeks and per node.

Thus, 25 directives on 100 nodes, with the default reports retention policy, would take 13 to 18 GB, and 25 directives on 1000 nodes with a 1 hour agent execution period with the defaut reports retention policy would take 11 to 15 GB.

PostgreSQL database bloating

PostgreSQL database can grow over time, even if the number of nodes and directives remain the same. This is because even if the database is regularly cleaned by Rudder as requested, the physical storage backend does not reclaim space on the hard drive, resulting in a "fragmented" database.

This is often not an issue, as PostgreSQL handles this automatically, and new reports sent by the nodes to Rudder should fill the blanks in the database. This task is handled by the autovacuum process, which periodically cleans the storage regularly to prevent database bloating.

However, database can grow significantly, resulting in large disk usage, and slower performance, due to massive bloating (with database 3 or 4 times larger than necessary).

To cure (or prevent) this behaviour, you can use either pg_repack that repacks the tables online, or trigger vacuum full operations, which put an exclusive lock on tables, and will lock both the Rudder interface and the reporting system for quite a long time.

Reclaming space online, using pg_repack

It is necessary to install pg_repack, available on Postgresql.org repository. The procedure ballow shows how to install and configure it, using PostgreSQL repository.

First, you need to identify the Postgresql version on your Rudder Server Root

$ psql -U rudder -h localhost -c "select version();"
rudder=> select version();
               version
-----------------------------------------
 PostgreSQL 9.6.10 on x86_64-pc-linux-gnu
(1 row)

For the rest of the procedure, PG_VERSION refers to the PostgreSQL version, in format Major.Minor, and PG_SHORT_VERSION refers to the version in format MajorMinor. (note that version 10 and later don’t have minor versions). So, with PostgreSQL version 9.6.10, you will have to define

PG_VERSION="9.6"
PG_SHORT_VERSION="96"
Downloading and installing pg_repack on Debian
apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list
apt-get update
apt-get install postgresql-PG_VERSION-repack
Downloading and installing pg_repack on RHEL/CentOS
echo '[pg_repack]
name=pg_repack
baseurl=https://download.postgresql.org/pub/repos/yum/PGVERSION/redhat/rhel-$releasever-$arch/
gpgcheck=1
gpgkey=https://download.postgresql.org/pub/repos/yum/RPM-GPG-KEY-PGDG-PG_SHORT_VERSION' > /etc/yum.repos.d/pg_repack.repo


yum install pg_repackPG_SHORT_VERSION

Then pg_repack needs to be configured within postgresql, by editing postgresql.conf (typically in /etc/postgresql/PG_VERSION/main/postgresql.conf)

and setting the shared_preload_libraries

shared_preload_libraries = 'pg_repack'

Then restart postgresql, and set up the extension

$ service postgresql restart

$ su postgres
$ psql -d rudder

CREATE EXTENSION pg_repack;

ALTER FUNCTION repack.version() OWNER TO postgres;
ALTER FUNCTION repack.version_sql() OWNER TO postgres;
ALTER FUNCTION repack.array_accum(anyelement) OWNER TO postgres;
ALTER FUNCTION repack.oid2text(oid) OWNER TO postgres;
ALTER FUNCTION repack.get_index_columns(oid,text) OWNER TO postgres;
ALTER FUNCTION repack.get_order_by(oid,oid) OWNER TO postgres;
ALTER FUNCTION repack.get_create_index_type(oid,name) OWNER TO postgres;
ALTER FUNCTION repack.get_create_trigger(oid,oid) OWNER TO postgres;
ALTER FUNCTION repack.get_enable_trigger(oid) OWNER TO postgres;
ALTER FUNCTION repack.get_assign(oid,text) OWNER TO postgres;
ALTER FUNCTION repack.get_compare_pkey(oid,text) OWNER TO postgres;
ALTER FUNCTION repack.get_columns_for_create_as(oid) OWNER TO postgres;
ALTER FUNCTION repack.get_drop_columns(oid,text) OWNER TO postgres;
ALTER FUNCTION repack.get_storage_param(oid) OWNER TO postgres;
ALTER FUNCTION repack.get_alter_col_storage(oid) OWNER TO postgres;
ALTER FUNCTION repack.repack_indexdef(oid,oid,name,boolean) OWNER TO postgres;
ALTER FUNCTION repack.repack_trigger() OWNER TO postgres;
ALTER FUNCTION repack.conflicted_triggers(oid) OWNER TO postgres;
ALTER FUNCTION repack.disable_autovacuum(regclass) OWNER TO postgres;
ALTER FUNCTION repack.repack_apply(cstring,cstring,cstring,cstring,cstring,integer) OWNER TO postgres;
ALTER FUNCTION repack.repack_swap(oid) OWNER TO postgres;
ALTER FUNCTION repack.repack_drop(oid,integer) OWNER TO postgres;
ALTER FUNCTION repack.repack_index_swap(oid) OWNER TO postgres;
ALTER FUNCTION repack.get_table_and_inheritors(regclass) OWNER TO postgres;
ALTER DEFAULT PRIVILEGES IN SCHEMA repack GRANT INSERT ON TABLES TO PUBLIC;
ALTER DEFAULT PRIVILEGES IN SCHEMA repack GRANT USAGE, SELECT ON SEQUENCES TO PUBLIC;

pg_repack is now installed and configured, and you can start reclaming space, either manually, either via a cronjob (recommended solution)

The easiest solution is to create a cron file /etc/cron.d/repack_db

# Compress inline the database
PATH=/usr/bin

# 2:20: compress ruddersysevents
20 2 * * * postgres pg_repack -d rudder -t ruddersysevents -T 3600

# 3:40: compress ruddersysevents
40 3 * * * postgres pg_repack -d rudder -t archivedruddersysevents -T 3600

# 4:30: compress  nodecompliancelevels
30 2 * * * postgres pg_repack -d rudder -t nodecompliancelevels -T 3600

Reclaming space with locking, using VACUUM FULL

Manual vacuuming using the psql binary
# You can either use sudo to change owner to the postgres user, or use the rudder connection credentials.

# With sudo:
sudo -u postgres psql -d rudder

# With rudder credentials, it will ask the password in this case:
psql -u rudder -d rudder

# And then, when you are connected to the rudder database in the psql shell, trigger a vacuum:
rudder# VACUUM FULL rudder;
rudder# VACUUM FULL archivedruddersysevents;
rudder# VACUUM FULL nodecompliancelevels;

LDAP database reindexing

In some very rare case, you will encounter some LDAP database entries that are not indexed and used during searches. In that case, OpenLDAP will output warnings to notify you that they should be.

LDAP database reindexing
# Stop OpenLDAP
service rudder-slapd stop

# Reindex the databases
service rudder-slapd reindex

# Restart OpenLDAP
service rudder-slapd restart

Migration, backups and restores

It is advised to backup frequently your Rudder installation in case of a major outage.

These procedures will explain how to backup your Rudder installation.

Backup

This backup procedure will operate on principal Rudder data sources:

  • The LDAP database

  • The PostgreSQL database

  • The configuration-repository folder

  • Rudder configuration

It will also backup the application logs.

How to backup a Rudder installation
# First, backup the LDAP database:
/opt/rudder/sbin/slapcat -l /tmp/rudder-backup-$(date +%Y%m%d).ldif

# Second, the PostgreSQL database:
sudo -u postgres pg_dump rudder > /tmp/rudder-backup-$(date +%Y%m%d).sql

# Or without sudo, use the rudder application password:
pg_dump -Fc -U rudder rudder > /tmp/rudder-backup-$(date +%Y%m%d).sql

# Third, backup the configuration repository:
tar -C /var/rudder -zcf /tmp/rudder-backup-$(date +%Y%m%d).tar.gz configuration-repository/ cfengine-community/ppkeys/ packages/ plugin-resources/

# Then backup Rudder configuration
tar -C /opt/rudder /tmp/rudder-etc-backup-$(date +%Y%m%d).tar.gz etc/

# Finally, backup the logs:
tar -C /var/log -zcf /tmp/rudder-log-backup-$(date +%Y%m%d).tar.gz rudder/

# And put the backups wherever you want, here /root:
cp /tmp/rudder*-backup* /root

Restore

Of course, after a total machine crash, you will have your backups at hand, but what should you do with it?

Here is the restoration procedure:

How to restore a Rudder backup
# First, follow the standard installation procedure, this one assumes you have a working "blank"
# Rudder on the machine

# Disable Rudder agent
rudder agent disable

# Stop Rudder services
service rudder-server stop
service rudder-agent stop

# Drop the OpenLDAP database
rm -rf /var/rudder/ldap/openldap-data/*.mdb

# Import your backups

# Configuration repository
tar -C /var/rudder -zvxf /root/rudder-backup-XXXXXXXX.tar.gz

# LDAP backup
/opt/rudder/sbin/slapadd -l /root/rudder-backup-XXXXXXXX.ldif

# Start PostgreSQL
service postgresql start

# PostgreSQL backup
sudo -u postgres pg_restore -d rudder -c -C < /root/rudder-backup-XXXXXXXX.sql
# or
pg_restore -u rudder -d rudder -c -C -W < /root/rudder-backup-XXXXXXXX.sql

# Configuration backup
tar -C /opt/rudder -zxf /root/rudder-etc-backup-XXXXXXXX.tar.gz

# Logs backups
tar -C /var/log -zxf /root/rudder-log-backup-XXXXXXXX.tar.gz

# Enable Rudder agent
rudder agent enable

# And restart the machine or just Rudder:
service rudder-server restart
service rudder-agent restart

Then you need to trigger a full policy regeration in the status menu with the Regenerate all policies button.

Migration

To migrate a Rudder installation, just backup and restore your Rudder installation from one machine to another.

If your server address changed, you will also have to do the following on every node that is directly connected to it (managed nodes or relays):

  • Remove the server public key rm /var/rudder/cfengine-community/ppkeys/root-MD5=*.pub

  • Modify /var/rudder/cfengine-community/policy_server.dat with the new address, then you can force your nodes to send their inventory by running rudder agent inventory

Password management

You might want to change the default passwords used in Rudder’s managed daemons for evident security reasons.

Configuration of the postgres database password

You will have to adjust the postgres database and the rudder-web.properties file.

Here is a semi-automated procedure:

  • Generate a decently fair password. You can use an arbitrary one too.

PASS=`dd if=/dev/urandom count=128 bs=1 2>&1 | md5sum | cut -b-12`
  • Update the Postgres database user

su - postgres -c "psql -q -c \"ALTER USER blah WITH PASSWORD '$PASS'\""
  • Insert the password in the rudder-web.properties file

sed -i "s%^rudder.jdbc.password.*$%rudder.jdbc.password=$PASS%" /opt/rudder/etc/rudder-web.properties

Configuration of the OpenLDAP manager password

You will have to adjust the OpenLDAP and the rudder-web.properties file.

Here is a semi-automated procedure:

  • Generate a decently fair password. You can use an arbitrary one too.

PASS=`dd if=/dev/urandom count=128 bs=1 2>&1 | md5sum | cut -b-12`
  • Update the password in the slapd configuration

HASHPASS=`/opt/rudder/sbin/slappasswd -s $PASS`
sed -i "s%^rootpw.*$%rootpw          $HASHPASS%" /opt/rudder/etc/openldap/slapd.conf
  • Update the password in the rudder-web.properties file

sed -i "s%^ldap.authpw.*$%ldap.authpw=$PASS%" /opt/rudder/etc/rudder-web.properties

Configuration of the WebDAV access password

This time, the procedure is a bit more tricky, as you will have to update the Technique library as well as a configuration file.

Here is a semi-automated procedure:

  • Generate a decently fair password. You can use an arbitrary one too.

PASS=`dd if=/dev/urandom count=128 bs=1 2>&1 | md5sum | cut -b-12`
  • Update the password in the apache htaccess file

On some systems, especially SUSE ones, htpasswd is called as "htpasswd2"

htpasswd -b /opt/rudder/etc/htpasswd-webdav rudder $PASS
  • Update the password in Rudder’s system Techniques

cd /var/rudder/configuration-repository/techniques/system/common/1.0/
sed -i "s%^.*davpw.*$%   \"davpw\" string => \"$PASS\"\;%" site.st
git commit -m "Updated the rudder WebDAV access password" site.st
  • Update the Rudder Directives by either reloading them in the web interface (in the "Configuration Management/Techniques" tab) or restarting jetty (NOT recommended)

Password upgrade

This version of Rudder uses a central file to manage the passwords that will be used by the application: /opt/rudder/etc/rudder-passwords.conf.

In the package, this file is initialized with default values, and during postinstall it will be updated with randomly generated passwords.

On the majority of cases, this is fine, however you might want to adjust the passwords manually. This is possible, just be cautious when editing the file, as if you corrupt it Rudder will not be able to operate correctly anymore and will spit numerous errors in the program logs.

As of now, this file follows a simple syntax: ELEMENT:password

You are able to configure three passwords in it: The OpenLDAP one, the PostgreSQL one and the authenticated WebDAV one.

If you edit this file, Rudder will take care of applying the new passwords everywhere it is needed, however it will restart the application automatically when finished, so take care of notifying users of potential downtime before editing passwords.

Here is a sample command to regenerate the WebDAV password with a random password, that is portable on all supported systems. Just change the RUDDER_WEBDAV_PASSWORD to any password file statement corresponding to the password you want to change.

sed -i s/RUDDER_WEBDAV_PASSWORD.*/RUDDER_WEBDAV_PASSWORD:$(dd if=/dev/urandom count=128 bs=1 2>&1 | md5sum | cut -b-12)/ /opt/rudder/etc/rudder-passwords.conf

← Going further Database maintenance →