Testing

Write technique tests

NOTE: Not all methods can run successfully with this testing method. In particular all methods exchanging files with the Rudder server won't work.

The tests are placed in a tests folder in the technique directory. Each test case is defined by a YAML file in this folder.

The format of the test case file is:

• target (optional): Target platform, unix or windows. Defaults to unix.
• params (optional): Named parameters passed to the technique in as key-values.
• conditions (optional): Conditions to define before running the technique.
• policy_mode (optional): The mode to use for running the technique. audit or enforce, defaults to enforce.
• setup (optional): Steps to run sequentially to prepare the environment for the test.
• check: Test steps, run sequentially. If one of them fails the test will be considered as a failure.
• cleanup (optional): Steps to run sequentially after the test to clean the environment.

Supported steps:

Steps are usable in setup, check and cleanup stages:

• sh:
  • On Linux: executes commands in a shell (/usr/bin/sh), fails if the return code is not 0
  • On Windows: executes commands in powershell, encapsulated in a script, fails if the exit code is not 0 or if an exception was thrown

Example:

target: windows
params:
  param1: value1
  param2: true
conditions:
  - condition1
  - condition2
policy_mode: audit
setup:
  - sh: "touch /my/file"
check:
  # Linux target
  - sh: "test -f /my/file"
  # Windows target
  - sh: "if (Test-Path -Path C:\\my\\file) { exit 0 } else { exit 1 }"
cleanup:
  - sh: "rm -f /my/file"

The detailed test process is, for each *.yml file in the tests directory:

• Build the technique in standalone mode (i.e. equivalent to passing --standalone to the build command). This adds a prelude to the technique allowing it to run without a Rudder server.
• Run the setup commands sequentially. The commands are called in a shell.
• Run an agent locally, which will:
  • Load all methods from the library
  • Read the parameters values from the YAML test case
  • Define the conditions from the YAML test case
  • Run the technique with the given parameters
• Run the check commands sequentially. The commands are called in a shell.
• Run the cleanup commands sequentially, regardless of the result of the check commands.

The build stops when it encounters a command returning a non-zero return code.

Run technique tests

To run the test case1 from:

my_technique/
  ├── technique.yml
  ├── resources/
  └── tests/
        ├── case1.yml
        └── case1.conf.ref

Run the following:

# Run all defined tests
rudderc test --library /path/to/lib
# Run only the case1 test
rudderc test --library /path/to/lib case1

• When running this command outside a Rudder server, you need to pass it a library path (as it is required to run the agent).
  • It can be the tree directory in a clone of the ncf repository.
  • On Windows it can be the path to an agent installed locally or a decompressed msi.
• the optional argument allows filtering the tests, only run those containing the given string.

Test outputs

The test runner will parse the agent output and place it in JSON format into target/<case_name>.json. It is written before running check steps, so you can use it to assess reporting output (for example, using jq in a shell script).

For convenience, check steps have an environment variable REPORTS_FILE pointing to the current reports JSON.

The JSON file contains an array of entries:

[
  {
    "component": "Ensure correct ntp configuration",
    "key_value": "/tmp/rudderc_test_one",
    "event_type": "result_repaired",
    "msg": "Insert content into /tmp/rudderc_test_one was repaired",
    "report_id": "d86ce2e5-d5b6-45cc-87e8-c11cca71d907",
    "logs": []
  }
]