1:Authors: Jason Lowe-Power 2 Sean Wilson 3 4This file explains how to use gem5's updated testing infrastructure. Running 5tests before submitting a patch is *incredibly important* so unexpected bugs 6don't creep into gem5. 7 8gem5's testing infrastructure has the following goals: 9 * Simple for *all* users to run 10 * Fast execution in the simple case 11 * High coverage of gem5 code 12 13# Running tests 14 15Below is the most common way the tests are run. This will run all of the 16"quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported 17platforms and use cases. When running these tests, you will likely want to us 18the option `-j <CPUs>` where `CPUs` is as large as you can make it. 19Additionally, it is often a good idea to run longer tests (e.g., linux boot) 20before submitting your patch. 21 22```shell 23cd tests 24./main.py run 25``` 26 27The above is the *minumum* you should run before posting a patch to 28https://gem5-review.googlesource.com 29 30## Specifying a subset of tests to run 31 32You can use the tag query interface to specify the exact tests you want to run. 33For instance, if you want to run only with `gem5.opt`, you can use 34 35```shell 36./main.py run --variant opt 37``` 38 39Or, if you want to just run X86 tests with the `gem5.opt` binary: 40 41```shell 42./main.py run --length quick --variant opt --isa X86 43``` 44 45 46To view all of the available tags, use 47 48```shell 49./main.py list --all-tags 50``` 51 52The output is split into tag *types* (e.g., isa, variant, length) and the 53tags for each type are listed after the type name. 54 55You can specify "or" between tags within the same type by using the tag flag 56multiple times. For instance, to run everything that is tagged "opt" or "fast" 57use 58 59```shell 60./main.py run --variant opt --variant fast 61``` 62 63You can also specify "and" between different types of tags by specifying more 64than one type on the command line. For instance, this will only run tests with 65both the "X86" and "opt" tags. 66 67```shell 68./main.py run --isa X86 --variant opt 69``` 70 71## Running tests in batch 72 73The testing infrastructure provides the two needed methods to run tests in 74batch. First, you can list all of the tests based on the same tags as above in 75a machine-readable format by passing the `-q` flag. This will list all of the 76*suites* that match the given tag(s). 77 78```shell 79./main.py list -q --suites 80SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt 81SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt 82SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt 83SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt 84SuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt 85SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt 86SuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt 87SuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt 88SuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt 89``` 90 91Next, you can run a single *suite* from the command line by passing the option 92`--uid`. For instance, 93 94```shell 95./main.py run --skip-build \ 96 --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt 97``` 98 99With this method, you can only run a *single* suite at a time. If you want to 100run more than one uid, you must call `./main.py` multiple times. 101 102Currently, you must specify `--skip-build` if you want to run a single suite or 103run in batch mode. Otherwise, you will build gem5 for all architectures. 104 105## Rerunning failed tests 106 107While developing software a common practice is to run tests, make a change, and 108assert that the tests still pass. If tests fail you'll likely want to 109rerun and fix those specific tests without running redundant ones. The testing 110infrastructure allows you to rerun tests which failed in the last execution by 111using the `rerun` command. 112 113```shell 114./main.py run 115# 116# Some tests fail... 117# 118 119# Rerun only the failed test suites (not the ones which passed). 120./main.py rerun 121``` 122 123# If something goes wrong 124 125The first step is to turn up the verbosity of the output using `-v`. This will 126allow you to see what tests are running and why a test is failing. 127 128If a test fails, the temporary directory where the gem5 output was saved is kept 129and the path to the directory is printed in the terminal. 130 131## Debugging the testing infrastructure 132 133Every command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will 134increase the verbosity level. If something isn't working correctly, you can 135start here. 136 137Most of the code for the testing infrastructure is in ext/testlib. This code 138contains the base code for tests, suites, fixtures, etc. The code in tests/gem5 139is *gem5-specific* code. For the most part, the code in tests/gem5 extends the 140structures in ext/testlib. 141 142## Common errors 143 144You may see a number of lines of output during test discovery that look like 145the following: 146 147```shell 148 Tried to load tests from ... but failed with an exception. 149 Tried to load tests from ... but failed with an exception. 150 ... 151``` 152 153The testing library searches all python files in the `tests/` directory. The 154test library executes each python file it finds searching for tests. It's okay 155if the file causes an exception. This means there are no tests in that file 156(e.g., it's not a new-style test). 157 158 159# Binary test applications 160 161The code for test binaries that are run in the gem5 guest during testing are 162found in `tests/test-progs`. 163There's one directory per test application. 164The source code is under the `source` directory. 165 166You may have a `bin` directory as well. 167The `bin` directory is automatically created when running the test case that 168uses the test binary. The binary is downloaded from the gem5 servers the first 169time it is referenced by a test. 170 171## Updating the test binaries 172 173The test infrastructure should check with the gem5 servers to ensure you have 174the latest binaries. However, if you believe your binaries are out of date, 175simply delete the `bin` directory and they will be re-downloaded to your local 176machine. 177 178## Building (new-style) test binaries 179 180In each `src/` directory under `tests/test-progs`, there is a Makefile. 181This Makefile downloads a docker image and builds the test binary for some ISA 182(e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make 183upload` it will upload the binaries to the gem5 server, if you have access to 184modify the binaries. *If you need to modify the binaries for updating a test or 185adding a new test and you don't have access to the gem5 server, contact a 186maintainer (see MAINTAINERS).* 187 188 189# Running Tests in Parallel 190 191Whimsy has support for parallel testing baked in. This system supports 192running multiple suites at the same time on the same computer. To run 193suites in parallel, supply the `-t <number-tests>` flag to the run command. 194 195For example, to run up to three test suites at the same time:: 196 197 ./main.py run --skip-build -t 3 198 199