Usage

This document explains most features of Qst and shows how to use them. It is a complete walk-through and will take about 15 minutes.

Qst makes use of the QML language, a declarative language mixed with Javascript. The language is very intuitive, but you might want to have a look at the QML language reference or keep it open in a separate browser tab.

We will use the terms item and component a lot. An item refers to a data type whereas component usually refers to a specific instance of an item.

Running a simple test case

Let us now get our hands dirty and have a look at a trivial test case. Every test case is defined by a Testcase component and is characterized by at least 2 things:

  • a name that serves as an identifier,
  • a run() function which implements sequential test steps.

The name must be unique across the whole project and must be a plain string. The run() function may have arbitrary execution length and contain blocking and non-blocking function calls. The Testcase component is put into a file of an arbitrary name and in its simplest form it would look as follows:

simple-passing-test.qml
import qst 1.0

Testcase {
    name: "simple-passing-test"

    function run() {
        /* ... */
    }
}

The above test case does not do anything useful. It specifies a name and contains an empty run() function. This is equal to a test case without any failure, so it will simply pass. We will now execute simple-passing-test.qml by typing the following command in a terminal:

$ qst run --file simple-passing-test.qml
PASS, simple-passing-test,,,

Qst prints a line of comma-separated values containing the test result and the name of the test case. We will discuss the remaining commas in a second.

Now let’s make the test case fail. For that purpose, Qst offers a couple of helper services of which Qst with its verification functions Qst::compare() and Qst::verify() is the most important one:

simple-failing-test.qml
1
2
3
4
5
6
7
8
9
import qst 1.0

Testcase {
    name: "simple-failing-test"

    function run() {
        Qst.verify(false)
    }
}

We execute the test, using a slightly shorter command line than above:

$ qst run -f simple-failing-test.qml
FAIL, simple-failing-test, simple-failing-test, simple-failing-test.qml:7, verify() failed

Qst outputs test results in a comma-separated line with the following elements:

  • result of the test case,
  • name of the test case,
  • name of the current active component (on fail only),
  • file location of the failure (on fail only),
  • error message (on fail only)

We know the result and the name already from the simple-passing-test.qml example. The other elements are only shown when a test case does not pass. In our very simple example, the test case is the only active component. More complex test cases might form a component tree structure, hence the 3rd column. The current location is the caller position of a verification function. The message in the last column comes from the failing verification function and is either set by hand or automatically created.

Now we know how to set up a test case and how to run it. Still, it doesn’t do anything useful. We need to connect it to the outside world and fill it with life.

Adding probe components

Probes connect Qst to the environment, for instance to the file system and other programs on the computer or even to external hardware. Probes are not a special language construct, but pure QML components with properties, methods, signals and slots.

A versatile probe item is the ProcessProbe. It can invoke programs on the local computer and monitor their execution. A very simple test case that runs GNU Make on a Makefile in the current folder would look like this:

makefile-test-simple.qml
import qst 1.0

Testcase {

    name: "makefile-test-simple"

    ProcessProbe {
        id: make
        program : "/usr/bin/make"
        workingDirectory: path
        arguments: [
            "-f", path + "/Makefile",
            "all"
        ]
    }

    function run() {
        make.start()
        make.waitForFinished(10)
        Qst.compare(make.exitCode, 0,
            "Build did not succeed: " + make.readAllStandardError())
    }
}

This example demonstrates the following new elements:

  • It instantiates a nested component of the type ProcessProbe and assigns custom values to its properties.
  • The nested process probe is assigned an id attribute so that it can be referenced.
  • The global path context property is used which always points the physical directory of the current file.

Various other probe items exist. For a complete list, have a look at the reference documentation.

Extending and re-using components

The above test case makefile-test-simple.qml is not re-usable in this form because all parameters are hard-coded. Consider a project with a bunch of makefiles. It would be cumbersome to re-write the whole test case for each makefile. Instead, we can turn the test case into a re-usable component by defining additional properties:

MakefileTestcase.qml
import qst 1.0

Testcase {

    property string makefile : path + "/Makefile"
    property string target: "all"
    property int timeout : 10

    ProcessProbe {
        id: make
        program : (Qst.hostOs === "windows") ? "gmake.exe" : "make"
        workingDirectory: path
        arguments: [
            "-f", makefile,
            target
        ]
    }

    function run() {
        make.start()
        make.waitForFinished(timeout)
        Qst.compare(make.exitCode, 0,
            "Build did not succeed: " + make.readAllStandardError())
    }
}

When saving it to a file MakefileTestcase.qml, it is automatically available as component type MakefileTestcase in other qml files located in the same directory. Based upon MakefileTestcase.qml, we can now create multiple test case files in the following form:

test-app-build.qml
import qst 1.0

MakefileTestcase {
    name: "test-app-build"
    makefile: path + "/app.mak"
}
test-lib-build.qml
import qst 1.0

MakefileTestcase {
    name: "test-lib-build"
    makefile: path + "/lib.mak"
}

Later we will learn a way to define multiple test case in one document.

Custom properties are also helpful in large test cases. Instead of hard-coding parameters everywhere in-line, it is better to to put them upfront to make the test case more readable. This applies to all components in general and is common practise in QML.

Attaching signal handlers

In many probes and test cases we might observe onXXX: {...} constructs, for instance:

Testcase {
    /* ... */
    onCreated: {
        // prepare external resources
    }
}

Signal handlers are always written in the form on<CapitalizedSignal>. Signals and signal handlers are a core concept of the QML language and fall in 1 of 3 categories:

  1. explicitly defined signals, like ProcessProbe::finished(),
  2. implicitly defined property change signals,
  3. attached signals added by other components.

Signal handlers may contain arbitrary JavaScript code, but they have run-to- completion semantics and must never do blocking calls. You may follow above links to read more about this topic. Let us now have a look, how we can utilize signal handlers in a Qst project.

Our MakefileTestcase.qml file from above can only do one thing: invoke GNU make. But what if a test case is more complex and needs to invoke additional programs? In this case, it would be more benefitial to extend ProcessProbe instead of Testcase:

MakeProbe.qml
import qst 1.0

ProcessProbe {
    property string makefile : path + "/Makefile"
    property string target: "all"
    property int jobs: 1

    program : (Qst.hostOs === "windows") ? "gmake.exe" : "make"
    arguments: [
        "-f", makefile,
        "-j", jobs,
        target
    ]
    workingDirectory: path

    // Explicitly defined by ProcessProbe
    onFinished: {
        Qst.compare(exitCode, 0, readAllStandardError())
    }

    // Implicitly defined property change signal
    onJobsChanged: {
        Qst.verify(jobs <= 4, "The maximum number of jobs is 4.")
    }

    // Attached by Testcase
    Testcase.onFinished: {
        if (state === ProcessProbe.Running) {
            terminate()
            Qst.verify(false, "Make was still running.")
        }
    }

}

The MakeProbe.qml component can now be included even multiple times like this:

test-multi-build.qml
import qst 1.0

Testcase {
    name: "test-multi-build"

    MakeProbe {
        id: app
        name: "make-app"
        makefile: path + "/app.mak"
        jobs: 5
    }

    MakeProbe {
        id: lib
        name: "make-lib"
        makefile: path + "/lib.mak"
        jobs: 5
    }

    function run() {
        app.start()
        app.waitForFinished(20)
        lib.start()
        lib.waitForFinished(20)
    }

}

As we can see in MakeProbe.qml, implicit and explicit signal handlers must be defined in the scope of the signal’s owner component. For instance, onJobsChanged would not work outside MakeProbe. In cases where we need to handle a signal of a component that is not defined in the current component, we can use SignalProbe as shown in the ExtendedTestcase.qml example. It is also possible to use the Connections item from the QtQml package.

Using constraints for continuous evaluation

In the MakeProbe.qml example we have learned how signal handlers can be used for on-going verification. We do not have to think about them in the run() function, they work silently in the background.

Constraints follow the same principle, but are a bit more formalized and declarative. They make the test case fail immediately when they are violated. Constraints are usually connected to signals like the DurationConstraint or bound to properties like the ValueRangeConstraint.

Take the following test case as an example:

import qst 1.0

Testcase {
    property int responseTime: 0

    onResponseTimeChanged: {
        Qst.verify((responseTime >= 4) && (responseTime =< 8),
            "responseTime (" + responseTime
            + ") is not in the expected range.")
    }

    function run() {
        // ...
    }
}

The property responseTime could be validated by the help of an implicit property changed signal handler and Qst verification functions. But we could also use a constraint and improve the readability of the test case:

import qst 1.0

Testcase {
    property int responseTime: 0

    ValueRangeConstraint {
        value: responseTime
        minValue: 4
        maxValue: 8
    }

    function run() {
        // ...
    }
}

Structuring projects

Until now we have only discussed single test cases whereas a real project would contain many of them. Qst provides the Project item for this purpose. A Project component can reference all test cases and serves as the main file of the project:

Project {
    name: "referencing-project"
    references: [
        "testcase-file-1.qml",
        "testcase-file-2.qml",
        /* ... */
    ]
}

It is also possible to put multiple Testcase components into a the same file by enclosing them in a Project component. This is especially useful when parametrizing and instantiating a generic test case multiple times:

SpecialTestcase.qml
Testcase {
    property int speed

    function run() {
        Qst.info("Speed is set to " + speed)
    }
}
project.qml
Project {
    name: "inline-project"
    SpecialTestcase { name: "tc-1"; speed: 10 }
    SpecialTestcase { name: "tc-2"; speed: 42000 }
    /* ... */
}

The Project component is automatically attached to any referenced file and can be accessed as project context property:

project.qml
Project {
    property string host: "http://secr.et"

    references: [
        "test-case-1.qml",
        /* ... */
    ]
}
test-case-1.qml
Testcase {
    name: "test-case-1"

    function run() {
        Qst.info(name + " is using " + project.host)
    }
}

Although the project property is shared across the whole project, test cases are not supposed to write to the project’s properties.

Working with profiles

Qst projects may be developed and executed by multiple developers on different computers. One usual problem in such setups are differing installation paths, serial numbers, ports, etc. . Putting machine-dependent values into project files would only complicate version control and collaboration:

Testcase {
    property string port: "COM77"   // ttyUSB3 on another computer

    /* ... */
}

Profiles solve that problem by collecting machine-dependent properties in a JSON file that remains on the developer’s computer:

uart-testing.json on computer 1
{
    "port": "COM77",
}
uart-testing.json on computer 2
{
    "port": "ttyUSB3",
}

The profile name is determined by the file name and selected as command line option when executing a test project:

$ qst run --file mytest.qml --profile uart-testing

Profile values can be accessed from anywhere in the project as profile:

Testcase {
    property string port: profile.port
}

Qst tries to load the he selected profile first from the project directory and if it could not be found, it searches in the »profiles« folder inside the Qst configuration directory. Additional profile search paths can be specified with --profile-directory or -P respectively:

$ qst run --f mytest.qml -p uart-testing -P /path/to/profiles

The latter option might be given multiple times.

Storing temporary files

Qst uses a working directory to store intermediate files during test execution. The directory is automatically created in the current folder and by default it has the format .<project-name>-<profile-name>-<hash>. Each test case will have a sub-folder with its name in the project working directory.

Let us assume a test project that:

  1. builds a firmware image,
  2. downloads it to the hardware,
  3. executes various tests on the hardware.

The directory layout of this imagined example project would look like:

.myproject-myprofile-e413c0f7
├── testcase-build
│   ├── object-1.o
│   ├── object-2.o
│   └── firmware.hex
├── testcase-flash
├── testcase-some-feature
└── testcase-other-feature
    └── log.txt

For the run command, the working directory can be overridden by the --working-directory command-line option. When run is executed multiple times and the working directory already exists, each test case sub-folder is wiped out before the Testcase::created() signal is emitted.

Tagging test cases for data-driven tests

In MakefileTestcase.qml and SpecialTestcase.qml we have already seen two ways to re-use and parameterize Testcase components. This can be cumbersome for larger amount of data. It might also be annoying to give each test case a different name while only some input parameters change, but not the test case itself.

Especially for data-driven testing, Qst provides the Matrix item. A matrix spans a n-dimensional parameter space that is then applied to a one or more test cases. Each data sample is called a tag and the combination of a tag and a Testcase is called a job.

The following example project defines a matrix with two dimensions, each containing a parameter array of length 2:

matrix-project.qml
import qst 1.0

Project {
    Matrix {
        Dimension {
            animal: [
                "cat",
                "dog"
            ]
        }

        Dimension {
            does: [
                "moans",
                "bites"
            ]
        }

        testcases: [ "tagged-test" ]
    }

    Testcase {
        name: "tagged-test"
        property string animal
        property string does

        function run() {
            Qst.info("The " + animal + " " + does + ".")
        }
    }

    Testcase {
        name: "normal-test"

        function run() {}
    }
}

Thus, the matrix expands to 4 different tag combinations:

dog bites dog moans
cat bites cat moans

When executing above project, the command line output looks as follows:

$ qst run -f matrix-project.qml
PASS, normal-test,,,
INFO, tagged-test 1ms2r6i [ cat moans ], , /matrix-project.qml:27, The cat moans.
PASS, tagged-test 1ms2r6i [ cat moans ],,,
INFO, tagged-test 17tca19 [ dog bites ], , /matrix-project.qml:27, The dog bites.
PASS, tagged-test 17tca19 [ dog bites ],,,
INFO, tagged-test 0ni1i5d [ cat bites ], , /matrix-project.qml:27, The cat bites.
PASS, tagged-test 0ni1i5d [ cat bites ],,,
INFO, tagged-test 07cs7hy [ dog moans ], , /matrix-project.qml:27, The dog moans.
PASS, tagged-test 07cs7hy [ dog moans ],,,

Specifying dependencies between test cases

The default execution order of test cases is undefined in Qst. But in practise, a test case B might require the completion of another test case A. Such dependencies can be expressed with the Depends item as shown in the following code snippet:

simple-depends-depends.qml
import qst 1.0

Project {

    Testcase {
        name: "A"

        function run() {}
    }

    // Has to wait until A completes.
    Testcase {
        name: "B"

        Depends { name: "A" }

        function run() {}
    }
}

Of course the test cases A and B can be defined in different files and each test case can have multiple dependencies.

Sometimes it might also be desired to access data from a preceding test. By the help of the Exports item, a test case can specify which data is to be forward to dependent test cases. This is not possible otherwise because test cases are isolated from each other do not share any memory.