How to write metadata about tests with cmake or ctest

Content

If you use cmake as a tool for the configuration of your project, you may be using the testing framework ctest that comes along with it. This article describes how to obtain metadata with this setup.

Step-by-step guide

Register a new test

To define a new test, you have to tell cmake to enable testing by placing enable_testing() in your top-level CMakeLists.txt file. Then, you can register a new test with the function add_test.

Option a: Manually define tests

Let us assume that in some folder of a C++ project there is a file test.cc that is the main file containing the code to run some unit test , for instance.

The CMakeLists.txt file in that folder may look like this:

add_executable(<my_test> <test.cc>)                             # first argument is the executable, second the source file from which to compile
add_test(NAME <my_test_default_args> COMMAND <my_test>)         # calls executable "my_test" with default runtime arguments
add_test(NAME <my_test_args_1> COMMAND <my_test> <TEST_ARGS_1>) # specifies some runtime arguments
add_test(NAME <my_test_args_2> COMMAND <my_test> <TEST_ARGS_2>) # specifies different runtime arguments

First, the executable with the name ‘my_test’ is defined, specifying that test.cc is the associated source file. Then, three tests are registered using the cmake function add_test, which allows you to give each test a ‘NAME’ and specify a ‘COMMAND’ to be executed. All three tests use the same executable (my_test), but use different runtime arguments.

Option b: Use a function to define tests

In order to reduce the amount of code you have to write when registering new tests, it may be favourable to define a cmake function to handle this:

function(my_add_test)
    include(CMakeParseArguments)
    set(OPTIONS ONLY_COMPILE)
    set(SINGLEARGS NAME TARGET)
    set(MULTIARGS SOURCES COMMAND COMMAND_ARGS)
    cmake_parse_arguments(TEST "${OPTIONS}" "${SINGLEARGS}" "${MULTIARGS}" ${ARGN})

    if (NOT TEST_TARGET)
        if (NOT TEST_SOURCES)
            message(FATAL_ERROR "Either TARGET or SOURCES must be specified")
        endif ()

        add_executable(${TEST_NAME} ${TEST_SOURCES})
        set(TEST_TARGET ${TEST_NAME})
    endif ()

    if (NOT TEST_COMMAND)
        set(TEST_COMMAND ${TEST_NAME})
    endif ()

    if (NOT TEST_ONLY_COMPILE)
        add_test(NAME ${TEST_NAME} COMMAND ${TEST_COMMAND} ${TEST_COMMAND_ARGS})
    endif ()
endfunction(my_add_test)

Making use of the cmake’s cmake_parse_arguments, this function now accepts an option (ONLY_COMPILE) as well as single- and multi-valued arguments. A new test is defined by using an existing executable (if the user of the function defines a ‘TARGET’), or by creating a new executable using the ‘SOURCES’ specified. Moreover, you can specify the command and/or command arguments for test execution, while per default the test name is used as command without any runtime arguments. If the option ONLY_COMPILE is set, then add_test is not called, but it is only tested for successful compilation.

Assuming that you have included this function somewhere in your project, we can now declare our tests like this:

add_executable(<my_test_exe> <test.cc>)
my_add_test(NAME <my_test_default_args> TARGET <my_test>)
my_add_test(NAME <my_test_args_1> TARGET <my_test> <COMMAND_ARGS> <TEST_ARGS_1>)
my_add_test(NAME <my_test_args_2> TARGET <my_test> <COMMAND_ARGS> <TEST_ARGS_2>)
Comparison of options a and b

As you can see option b did not really reduce the amount of code necessary to define our tests when compared to option a, but you may already notice that this is because we are defining multiple tests on one executable. In the case that you only want to define a single test from one executble, the amount of code necessary is halved. For instance, the definition of the following test

add_executable(<my_other_test> <my_other_test.cc>)
add_test(NAME <my_other_test> COMMAND <my_other_test>)

reduces to

my_add_test(NAME <my_other_test> SOURCES <my_other_test.cc>)

Besides this, the additional layer introduced by my_add_test allows you to add options and other customization to your test definitions. This can be useful, as illustrated by the example given in the subsequent section.

Write metadata to a file

If all tests in your project use my_add_test, this now also gives you the possibility to gather and store metadata on all registered tests. As a simple example, you can write out a json file for each test, storing the name of the test and the associated executable (with the key “target”), by placing the following piece of code at the end of my_add_test:

file(MAKE_DIRECTORY "${CMAKE_BINARY_DIR}/TestMetaData")
file(WRITE "${CMAKE_BINARY_DIR}/TestMetaData/${TEST_NAME}.json"
           "{\n  \"name\": \"${TEST_NAME}\",\n  \"target\": \"${TEST_TARGET}\"\n}\n")

This creates a folder /TestMetaData in the top level of the project’s build tree, and stores the created json files therein.

See also