Node properties data sources

As explained in the chapter about node management, Nodes have properties that can be used to create groups or in techniques and directives parameters. These properties are key/value pairs, with values being a simple string or a well formed JSON document.

Rudder 4.1 introduces a new way to automatically import Node properties by defining data sources.

The following diagram explains the general working process of data source:

Data source description

As displayed, a data source provides a way for Rudder to query (when some conditions are met) a third party REST API to retrieve interesting properties for nodes and save them for a given Node property key.

Rudder compatibility and breaking changes

Rudder 4.3.7 needs version 4.3-1.4.1 or above

There was a Rudder internal API breaking change that needs to be repercuted here: - Plugin ticket: https://issues.rudder.io/issues/13933

Data source configuration screen

There are three main sets of properties to fill (by UI or via Rudder API) to configure a data source:

First set: data source description

The first set of properties allows to define an unique identifier for the data source, which will be used as the property key in node, along with a human readable name and description.

Second set: query configuration

The second set of properties allows to define how the third party REST API will be queried and the returned JSON response processed.

For now, we only support one query mode which is to do one HTTP query for each node. In the future, a mode where only one query is done to retrieve information for all nodes will be added.

For the query, you will define the HTTP method to use (GET or POST), what is the remote URL, if there are specific headers or query parameters to add.

In case a 404 error is returned, the corresponding node property is deleted (on that node). In case of a timeout or any other HTTP errors, this is considered a temporary problem, and the node property is left as is.

When a JSON document is returned, you can define a JSON path expression (cf https://github.com/jayway/JsonPath/) to select only a sub-part of the document as the actual data to use as a node property.

Finally, the resulting data is assigned to the node, using the key name defined in the data source configuration.

You can use Rudder variables expansion (${rudder.node.xxx}, ${rudder.parameter.xxx}, ${node.properties[xxx]}) in most of these configuration option: URL, headers, query parameters, JSON path. They will be replaced by their values for each node at the time the HTTP query is run.

Third set: query triggers

The last set of options allows to define when the data source should be queried.

For now, there are 3 available triggers:

  • a scheduled one, allowing to periodically do the update,

  • a trigger on policy generation, which allows to get a refresh of node properties before possibly using them in techniques or directives,

  • a trigger on node acceptation, so that a new node immediately get a working set of properties (for example to join the correct dynamic groups).

In addition to these configured triggers, data sources can be interactively refreshed with a call to a Rudder REST API or via the web interface.

Field description

In that part, we describe all fields.

Main parameters

Name

a user readable name. It can be updated after data source creation.

Key name

the node property key that will be set by that data source. It is also the data source unique identifier and can’t be updated ounce a data source is created.

Description

A human readable description.

Method

Choose between GET or POST HTTP request.

URL

The data source URL which must return JSON content. You can use Rudder variable expansion in that URL, for example: http://my.cmdb.com/rudder-endpoint/${rudder.node.id}

Headers (optional)

A set of headers, defined by the header name and the header value.

JSON path (optional)

An xpath-like path to only use a sub-part of the JSON content returned by the endpoint. By default, $. (which mean the whole content) is used. Documentation on json-path is available at: https://github.com/jayway/JsonPath/#getting-started.

Advanced options

Ignore SSL certificate validation

If checked, data source certificate validity won’t be verified. It is necessary to check that option if you use self-signed certificate.

HTTP request timeout

configure timeout for the HTTP request (default: 30s).

Data source update max duration

configure the maximum allowed time for all nodes to be updated.

Update triggers

Update periodically - scheduled

if checked, that data source we will be queried periodically with the period configured below the text.

Update when a policy generation starts

if checked, that data source will be queried at the begining of each policy generation.

Update when a new node is accepted

if checked, that data source will be queried each time a node is successfully accepted into Rudder.

What to do when a query for a Node returns a 404 error?

That part defines the behaviour to adopt if the data source endpoint returns an HTTP error 404 for a node.

You need to choose one behavior among:

  • Delete the node property corresponding to that data source (default behavior),

  • Do not change the node property corresponding to that data source,

  • Set the node property corresponding to the data source to a configured value. You have access to a field to fill the value, where JSON is accepted. If the field is let empty, the node property is deleted (ie equivalent to first option).


← Change validation Create Node API →