Advanced configuration management

The button Update policies on the top right of the screen, in the Status menu, allows you to force the regeneration of the policies. As changes in the inventory of the nodes are not automatically taken into account by Rudder, this feature can be useful after some changes impacting the inventory information.

The button Regenerate all policies on the top right of the screen, in the Status menu, allows you to force the regeneration of all policies. It will clear all internal caches, and force a complete computation of the policies. This is generally useful to make sure everything is correct after a problem on the central server.

Policy generation is a complex process and it is central to Rudder use. You will often want to know more about a policy generation result: what was the reason for a failure? What nodes where upated? Or even more specifics info like: where are info and debug log?

As explained in http://www.ncf.io/, ncf uses a structured directory tree composed of several layers of logic, from internal libraries to Techniques and user services. All the files and logic in these folders will be named "library" for simplicity

ncf directory structure exists in two root folders:

To share those folders to all nodes, Rudder makes a copy of these folders in two places:

Files in /var/rudder/ncf are synchronized automatically by the "rudder agent update" command when the agent runs on the server. So any modification done in files in these directories will be lost at the next synchronization.

Files under /var/rudder/share/ are updated during policy generation.

A node updates its ncf local library by copying the content of these two folders during its policy update phase.

Here we will explain how the Technique Editor integration to Rudder is done to transform ncf techniques into full fledged Rudder one. We will also get the big picture of the web flow and the resulting events triggered on Rudder server side.

Each action in the Technique Editor interface produces requests to an API defined over ncf.

All of the requests are authenticated thanks to a token passed in the JSESSIONID header. The token is generated when an authenticated user is connected to the Rudder interface (typically thanks to his browser).

That token is shared to the Technique Editor interface, which itself passes the JSESSIONID header to all requests.

If you have authentication issue, check that your Rudder session is not expired.

https://your.rudder.server/ncf-builder/#!?path=/var/rudder/configuration-repository/ncf

Get requests are triggered when accessing Technique editor.

The ncf API will parse all files in the parameter path by running "cf-promises -pjson" on all Techniques, checking that all Techniques are correctly formed.

The ncf API will also look to all Generic Methods description data to build the catalog of available Generic Methods.

The resulting information are sent back to the Technique Editor for displaying.

They are triggered when clicking on save/delete button.

The main difference with get requests is that hooks are launched before and after the action is made.

We will see all hooks behavior in the following dedicated hooks section.

On each POST request, pre- and post- hooks are executed by the Technique Editor. These hooks are used for the Rudder integration to help transform pure ncf Techniques into Rudder one.

As of March 2015, we have two post-hooks defined and no pre-hooks:

If you want to run post hooks by hand, you can use the following command:

You can use node properties almost everywhere in Rudder:

To use a property, simply use the variable node.properties with the variable call syntax.

Example with a property named 'datacenter':

In a mustache template, use:

The agent searches for optionnal properties files /var/rudder/local/properties.d/*.json, and will override existing properties.

As a result, if you have node properties defined server side as "sysctls_postgresql":{"kernel.shmall":"903330","kernel.shmmax":"3700041320"} and "vm":{"vm.dirty_ratio":"10"}

and a local property file /var/rudder/local/properties.d/postgresql_config.json as

The resulting properties will be:

"sysctls_postgresql":{"kernel.shmmax":"5368709120"} and "vm":{"vm.dirty_ratio":"10"}

sysctls_postgresql has been overriden by local property, and vm has been left untouched. Note that it is an override, as the semantic of merging is not deterministic with literal values, and it does not allow to unset values. If you need to merge, please refer to the next paragraph.

If you want to merge server defined properties with local defined properties, rather than override them, you will need to use the generic method variable_dict_merge_tolerant to define which variables you need to merge, and define the local variables in a different namespace than properties.

For instance, if you have defined in the node properties the following properties

"sysctls_postgresql":{"kernel.shmall":"903330","kernel.shmmax":"3700041320"}

and you wish to merge these values on a node with locally defined variable, to change the value of kernel.shmmax and set the value of kernel.shmmni, you can define the file /var/rudder/local/properties.d/postgresql_config.json with the following content

and use the generic method variable_dict_merge_tolerant to merge node.properties[sysctls_postgresql] and node.local_properties[sysctls_postgresql], and set the result in merged_properties.sysctls_postgresql (for instance): variable_dict_merge_tolerant("merged_properties", "sysctls_postgresql", "node.properties[sysctls_postgresql]", "node.local_properties[sysctls_postgresql]")

As a result, merged_properties.sysctls_postgresql will contain

"sysctls_postgresql": { "kernel.shmall":"903330", "kernel.shmmax":"5368709120", "kernel.shmmni":"4096" }

On the server, one or more properties files are written for each node in the /var/rudder/share/<uuid>/rules/cfengine-community/properties.d/ directory. This directory is then copied to each node by the agent with all other policy files.

In the agent, properties are made available in the node.<namespace> container that contains the values. Those values are read from /var/rudder/cfengine-community/inputs/properties/*.json. All files are taken in order and override the previous ones - the last one wins.

The agent searches for optional properties files /var/rudder/local/properties.d/*.json, and will define variables or override existing properties.

Each file must contain at least 2 levels of JSON content, the first level is the namespace level and the second level is the key level.

The namespace name must be an ASCII name that doesn’t start with and must match the following regex: [a-zA-Z0-9][a-zA-Z0-9]*

For example:

The merge is a first level merge done at the namespace level. This means that:

The result key is available in the node.<namespace> data variable. A usage example:

To get the original data (for debug only) there is the properties.property_<fileid> variable. A usage example:

If you upgraded to 3.1.14 (or a later 3.1.x version) or 3.2.7 (or a later 3.2.x version) from a previous Rudder version, this feature was disabled by default in order to mitigate any risk of undesired side effects on existing installations. You can enable it in the Administration/Settings page, using the Enable node properties expansion in Directives switch.

Rudder installations from 4.0.0 onwards have this feature enabled by default.

In any directive text field, you can access properties defined on nodes using the following syntax:

where:

If you upgraded from a version prior to 3.1.12 or 3.2.5, this feature is disabled by default in order to mitigate any risk of undesired side effects on existing installations. You can enable it in the Administration/Settings page, using the Enable script evaluation in Directives parameter.

Rudder installations from 4.0 onwards have this feature enabled by default.

All standard JavaScript methods are available, and a Rudder-specific library, prefixed with rudder. also provides some extra utilities. This library is documented below.

For example, to get the first 3 letters of each node’s hostname, you can write:

In a future version of Rudder, JavaScript evaluation will be supported in all fields in Directives, including non plain-text fields.

In the meantime, you can already test this functionality out by entering a JavaScript expression in any Directive field, prefixed by evaljs:. Please be aware that this is unsupported and untested, so do this at your own risk.

If you do encounter any issues, please get in touch or open a ticket - we’d love to hear about them!

There is currently no plan to extend this support to the fields in the Technique editor.

Hooks are organized in subdirectories. The root of the sub-directories is /opt/rudder/etc/hooks.d/

Each sub-directory has a name related to the event that will trigger the hooks execution. By default, a hook directory contains a template example and a Readme.txt file explaining the generalities about hooks and the specificities of that hook (parameters, etc).

Hooks must be executable. All executable files will be used as hooks EXCEPT if they end with one of the extensions listed in /opt/rudder/etc/rudder-web.properties for property *rudder.hooks.ignore-suffixes. A common convention is to use the .disabled suffixe to do so.

Non executable files will be ignored (which allows to put other files in these directories, like a readme, for example).

Hooks parameter are passed by environment variable. Rudder will fill dedicated environment variable for each hooks.

Hooks are executed sequentially, in lexical order. We encourage you to use the patter "NN-hookname", with NN a number like "05", "20", etc.

Hooks have normalized returned code. Return codes on hooks are interpreted as follow:

Available Hooked events: for now, all hooks are related to different steps of the policy generation process. In the future, more hooks will be supported like node acceptation.

In Rudder, Policies are generated from Directives, but several Directives based on the same Technique always lead to one Policy on the agent. For unique (non multi-instance) Technique, the one with the highest priority is selected. For multi-instance Technique, the different Directive values are merged into one Policy after having been sorted.

For Directive based on the same Technique, the sort order is based on the Priority value of the Directive. Between two Directive, the one with the highest Priority is the first:

If several Directives have the same Priority, the Rule name, and then the Directive name are used for sorting in alphanumeric order.

Rudder uses a best-effort method for ordering Policies, based on alphanumeric ordering of the corresponding Rule, then Directive name.

When several Directive were merged, Rudder choose the first (Rule name, Directive name) as the ordering value to use for the resulting Policy.