Usage¶
This document explains most features of Qst and shows how to use them. It is a complete walk-through and will take about 15 minutes.
- Running a simple test case
- Adding probe components
- Extending and re-using components
- Attaching signal handlers
- Using constraints for continuous evaluation
- Structuring projects
- Working with profiles
- Storing temporary files
- Tagging test cases for data-driven tests
- Specifying dependencies between test cases
Qst makes use of the QML language, a declarative language mixed with Javascript. The language is very intuitive, but you might want to have a look at the QML language reference or keep it open in a separate browser tab.
We will use the terms item and component a lot. An item refers to a data type whereas component usually refers to a specific instance of an item.
Running a simple test case¶
Let us now get our hands dirty and have a look at a trivial test case. Every
test case is defined by a Testcase
component and is characterized by at least
2 things:
The name must be unique across the whole project and must be a plain string. The
run()
function may have arbitrary execution length and contain blocking and
non-blocking function calls. The Testcase
component is put into a file of an
arbitrary name and in its simplest form it would look as follows:
import qst 1.0
Testcase {
name: "simple-passing-test"
function run() {
/* ... */
}
}
The above test case does not do anything useful. It specifies a name and
contains an empty run()
function. This is equal to a test case without any
failure, so it will simply pass. We will now execute simple-passing-test.qml
by typing the following command in a terminal:
$ qst run --file simple-passing-test.qml
PASS, simple-passing-test,,,
Qst prints a line of comma-separated values containing the test
result
and the name
of the test case. We will discuss the remaining commas in a
second.
Now let’s make the test case fail. For that purpose, Qst offers a couple of
helper services of which Qst
with its verification
functions Qst::compare()
and Qst::verify()
is the most
important one:
1 2 3 4 5 6 7 8 9 | import qst 1.0
Testcase {
name: "simple-failing-test"
function run() {
Qst.verify(false)
}
}
|
We execute the test, using a slightly shorter command line than above:
$ qst run -f simple-failing-test.qml
FAIL, simple-failing-test, simple-failing-test, simple-failing-test.qml:7, verify() failed
Qst outputs test results in a comma-separated line with the following elements:
result
of the test case,name
of the test case,- name of the current active component (on fail only),
- file location of the failure (on fail only),
- error message (on fail only)
We know the result
and the name
already from the simple-passing-test.qml example. The other
elements are only shown when a test case does not pass. In our very simple
example, the test case is the only active component. More complex test cases
might form a component tree structure, hence the 3rd column. The current
location is the caller position of a verification function. The message in
the last column comes from the failing verification function and is either set
by hand or automatically created.
Now we know how to set up a test case and how to run it. Still, it doesn’t do anything useful. We need to connect it to the outside world and fill it with life.
Adding probe components¶
Probes connect Qst to the environment, for instance to the file system and other programs on the computer or even to external hardware. Probes are not a special language construct, but pure QML components with properties, methods, signals and slots.
A versatile probe item is the ProcessProbe
. It can invoke programs
on the local computer and monitor their execution. A very simple test case that
runs GNU Make on a Makefile in the current folder would look like this:
import qst 1.0
Testcase {
name: "makefile-test-simple"
ProcessProbe {
id: make
program : "/usr/bin/make"
workingDirectory: path
arguments: [
"-f", path + "/Makefile",
"all"
]
}
function run() {
make.start()
make.waitForFinished(10)
Qst.compare(make.exitCode, 0,
"Build did not succeed: " + make.readAllStandardError())
}
}
This example demonstrates the following new elements:
- It instantiates a nested component of the type
ProcessProbe
and assigns custom values to its properties. - The nested process probe is assigned an id attribute so that it can be referenced.
- The global
path
context property is used which always points the physical directory of the current file.
Various other probe items exist. For a complete list, have a look at the reference documentation.
Extending and re-using components¶
The above test case makefile-test-simple.qml is not re-usable in this form because all parameters are hard-coded. Consider a project with a bunch of makefiles. It would be cumbersome to re-write the whole test case for each makefile. Instead, we can turn the test case into a re-usable component by defining additional properties:
import qst 1.0
Testcase {
property string makefile : path + "/Makefile"
property string target: "all"
property int timeout : 10
ProcessProbe {
id: make
program : (Qst.hostOs === "windows") ? "gmake.exe" : "make"
workingDirectory: path
arguments: [
"-f", makefile,
target
]
}
function run() {
make.start()
make.waitForFinished(timeout)
Qst.compare(make.exitCode, 0,
"Build did not succeed: " + make.readAllStandardError())
}
}
When saving it to a file MakefileTestcase.qml, it is automatically available
as component type MakefileTestcase
in other qml files located in the same
directory. Based upon MakefileTestcase.qml, we can now create multiple
test case files in the following form:
import qst 1.0
MakefileTestcase {
name: "test-app-build"
makefile: path + "/app.mak"
}
import qst 1.0
MakefileTestcase {
name: "test-lib-build"
makefile: path + "/lib.mak"
}
Later we will learn a way to define multiple test case in one document.
Custom properties are also helpful in large test cases. Instead of hard-coding parameters everywhere in-line, it is better to to put them upfront to make the test case more readable. This applies to all components in general and is common practise in QML.
Attaching signal handlers¶
In many probes and test cases we might observe onXXX: {...}
constructs, for
instance:
Testcase {
/* ... */
onCreated: {
// prepare external resources
}
}
Signal handlers are always written in the form on<CapitalizedSignal>. Signals and signal handlers are a core concept of the QML language and fall in 1 of 3 categories:
- explicitly defined signals,
like
ProcessProbe::finished()
, - implicitly defined property change signals,
- attached signals added by other components.
Signal handlers may contain arbitrary JavaScript code, but they have run-to- completion semantics and must never do blocking calls. You may follow above links to read more about this topic. Let us now have a look, how we can utilize signal handlers in a Qst project.
Our MakefileTestcase.qml file from above can only do one thing: invoke GNU
make. But what if a test case is more complex and needs to invoke additional
programs? In this case, it would be more benefitial to extend
ProcessProbe
instead of Testcase
:
import qst 1.0
ProcessProbe {
property string makefile : path + "/Makefile"
property string target: "all"
property int jobs: 1
program : (Qst.hostOs === "windows") ? "gmake.exe" : "make"
arguments: [
"-f", makefile,
"-j", jobs,
target
]
workingDirectory: path
// Explicitly defined by ProcessProbe
onFinished: {
Qst.compare(exitCode, 0, readAllStandardError())
}
// Implicitly defined property change signal
onJobsChanged: {
Qst.verify(jobs <= 4, "The maximum number of jobs is 4.")
}
// Attached by Testcase
Testcase.onFinished: {
if (state === ProcessProbe.Running) {
terminate()
Qst.verify(false, "Make was still running.")
}
}
}
The MakeProbe.qml component can now be included even multiple times like this:
import qst 1.0
Testcase {
name: "test-multi-build"
MakeProbe {
id: app
name: "make-app"
makefile: path + "/app.mak"
jobs: 5
}
MakeProbe {
id: lib
name: "make-lib"
makefile: path + "/lib.mak"
jobs: 5
}
function run() {
app.start()
app.waitForFinished(20)
lib.start()
lib.waitForFinished(20)
}
}
As we can see in MakeProbe.qml, implicit and explicit signal handlers must
be defined in the scope of the signal’s owner component. For instance,
onJobsChanged
would not work outside MakeProbe
. In cases where we
need to handle a signal of a component that is not defined in the current
component, we can use SignalProbe
as shown in the
ExtendedTestcase.qml example. It is also possible to use the
Connections item from the QtQml package.
Using constraints for continuous evaluation¶
In the MakeProbe.qml example we have learned how signal handlers can be
used for on-going verification. We do not have to think about them in the
run()
function, they work silently in the background.
Constraints follow the same principle, but are a bit more formalized and
declarative. They make the test case fail immediately when they are violated.
Constraints are usually connected to signals like the
DurationConstraint
or bound to properties like the
ValueRangeConstraint
.
Take the following test case as an example:
import qst 1.0
Testcase {
property int responseTime: 0
onResponseTimeChanged: {
Qst.verify((responseTime >= 4) && (responseTime =< 8),
"responseTime (" + responseTime
+ ") is not in the expected range.")
}
function run() {
// ...
}
}
The property responseTime
could be validated by the help of an implicit
property changed signal handler and Qst
verification functions. But
we could also use a constraint and improve the readability of the test case:
import qst 1.0
Testcase {
property int responseTime: 0
ValueRangeConstraint {
value: responseTime
minValue: 4
maxValue: 8
}
function run() {
// ...
}
}
Structuring projects¶
Until now we have only discussed single test cases whereas a real project would
contain many of them. Qst provides the Project
item for this purpose. A
Project
component can reference all test cases and serves as the main file of
the project:
Project {
name: "referencing-project"
references: [
"testcase-file-1.qml",
"testcase-file-2.qml",
/* ... */
]
}
It is also possible to put multiple Testcase
components into a the same
file by enclosing them in a Project
component. This is especially useful
when parametrizing and instantiating a generic test case multiple times:
Testcase {
property int speed
function run() {
Qst.info("Speed is set to " + speed)
}
}
Project {
name: "inline-project"
SpecialTestcase { name: "tc-1"; speed: 10 }
SpecialTestcase { name: "tc-2"; speed: 42000 }
/* ... */
}
The Project
component is automatically attached to any referenced file
and can be accessed as project
context property:
Project {
property string host: "http://secr.et"
references: [
"test-case-1.qml",
/* ... */
]
}
Testcase {
name: "test-case-1"
function run() {
Qst.info(name + " is using " + project.host)
}
}
Although the project
property is shared across the whole project,
test cases are not supposed to write to the project’s properties.
Working with profiles¶
Qst projects may be developed and executed by multiple developers on different computers. One usual problem in such setups are differing installation paths, serial numbers, ports, etc. . Putting machine-dependent values into project files would only complicate version control and collaboration:
Testcase {
property string port: "COM77" // ttyUSB3 on another computer
/* ... */
}
Profiles solve that problem by collecting machine-dependent properties in a JSON file that remains on the developer’s computer:
{
"port": "COM77",
}
{
"port": "ttyUSB3",
}
The profile name is determined by the file name and selected as command line option when executing a test project:
$ qst run --file mytest.qml --profile uart-testing
Profile values can be accessed from anywhere in the project as
profile
:
Testcase {
property string port: profile.port
}
Qst tries to load the he selected profile first from the project directory and
if it could not be found, it searches in the »profiles« folder inside the Qst
configuration directory. Additional profile search paths can be specified with
--profile-directory
or -P
respectively:
$ qst run --f mytest.qml -p uart-testing -P /path/to/profiles
The latter option might be given multiple times.
Storing temporary files¶
Qst uses a working directory to store intermediate files during test execution. The directory is automatically created in the current folder and by default it has the format .<project-name>-<profile-name>-<hash>. Each test case will have a sub-folder with its name in the project working directory.
Let us assume a test project that:
- builds a firmware image,
- downloads it to the hardware,
- executes various tests on the hardware.
The directory layout of this imagined example project would look like:
.myproject-myprofile-e413c0f7
├── testcase-build
│ ├── object-1.o
│ ├── object-2.o
│ └── firmware.hex
├── testcase-flash
├── testcase-some-feature
└── testcase-other-feature
└── log.txt
For the run
command, the working directory can be overridden by the
--working-directory
command-line option. When run
is
executed multiple times and the working directory already exists, each
test case sub-folder is wiped out before the Testcase::created()
signal is emitted.
Tagging test cases for data-driven tests¶
In MakefileTestcase.qml and SpecialTestcase.qml we have already seen
two ways to re-use and parameterize Testcase
components. This can be
cumbersome for larger amount of data. It might also be annoying to give each
test case a different name while only some input parameters change, but not the
test case itself.
Especially for data-driven testing, Qst provides the Matrix
item. A
matrix spans a n-dimensional parameter space that is then applied to a one or
more test cases. Each data sample is called a tag and the combination of a
tag and a Testcase
is called a job.
The following example project defines a matrix with two dimensions, each containing a parameter array of length 2:
import qst 1.0
Project {
Matrix {
Dimension {
animal: [
"cat",
"dog"
]
}
Dimension {
does: [
"moans",
"bites"
]
}
testcases: [ "tagged-test" ]
}
Testcase {
name: "tagged-test"
property string animal
property string does
function run() {
Qst.info("The " + animal + " " + does + ".")
}
}
Testcase {
name: "normal-test"
function run() {}
}
}
Thus, the matrix expands to 4 different tag combinations:
dog bites | dog moans |
cat bites | cat moans |
When executing above project, the command line output looks as follows:
$ qst run -f matrix-project.qml
PASS, normal-test,,,
INFO, tagged-test 1ms2r6i [ cat moans ], , /matrix-project.qml:27, The cat moans.
PASS, tagged-test 1ms2r6i [ cat moans ],,,
INFO, tagged-test 17tca19 [ dog bites ], , /matrix-project.qml:27, The dog bites.
PASS, tagged-test 17tca19 [ dog bites ],,,
INFO, tagged-test 0ni1i5d [ cat bites ], , /matrix-project.qml:27, The cat bites.
PASS, tagged-test 0ni1i5d [ cat bites ],,,
INFO, tagged-test 07cs7hy [ dog moans ], , /matrix-project.qml:27, The dog moans.
PASS, tagged-test 07cs7hy [ dog moans ],,,
Specifying dependencies between test cases¶
The default execution order of test cases is undefined in Qst. But in practise,
a test case B
might require the completion of another test case A
. Such
dependencies can be expressed with the Depends
item as shown in the
following code snippet:
import qst 1.0
Project {
Testcase {
name: "A"
function run() {}
}
// Has to wait until A completes.
Testcase {
name: "B"
Depends { name: "A" }
function run() {}
}
}
Of course the test cases A
and B
can be defined in different files and
each test case can have multiple dependencies.
Sometimes it might also be desired to access data from a preceding test. By the
help of the Exports
item, a test case can specify which data is to
be forward to dependent test cases. This is not possible otherwise because test
cases are isolated from each other do not share any memory.