TESTING.md revision 12883
112883Sjason@lowepower.com:Authors: Jason Lowe-Power 212883Sjason@lowepower.com Sean Wilson 312883Sjason@lowepower.com 412883Sjason@lowepower.comThis file explains how to use gem5's updated testing infrastructure. Running 512883Sjason@lowepower.comtests before submitting a patch is *incredibly important* so unexpected bugs 612883Sjason@lowepower.comdon't creep into gem5. 712883Sjason@lowepower.com 812883Sjason@lowepower.comgem5's testing infrastructure has the following goals: 912883Sjason@lowepower.com * Simple for *all* users to run 1012883Sjason@lowepower.com * Fast execution in the simple case 1112883Sjason@lowepower.com * High coverage of gem5 code 1212883Sjason@lowepower.com 1312883Sjason@lowepower.com# Running tests 1412883Sjason@lowepower.com 1512883Sjason@lowepower.comBelow is the most common way the tests are run. This will run all of the 1612883Sjason@lowepower.com"quick" tests for X86, ARM, and RISC-V. These tests make up our best-supported 1712883Sjason@lowepower.complatforms and use cases. When running these tests, you will likely want to us 1812883Sjason@lowepower.comthe option `-j <CPUs>` where `CPUs` is as large as you can make it. 1912883Sjason@lowepower.comAdditionally, it is often a good idea to run longer tests (e.g., linux boot) 2012883Sjason@lowepower.combefore submitting your patch. 2112883Sjason@lowepower.com 2212883Sjason@lowepower.com```shell 2312883Sjason@lowepower.comcd tests 2412883Sjason@lowepower.com./main.py run 2512883Sjason@lowepower.com``` 2612883Sjason@lowepower.com 2712883Sjason@lowepower.comThe above is the *minumum* you should run before posting a patch to 2812883Sjason@lowepower.comhttps://gem5-review.googlesource.com 2912883Sjason@lowepower.com 3012883Sjason@lowepower.com## Specifying a subset of tests to run 3112883Sjason@lowepower.com 3212883Sjason@lowepower.comYou can use the tag query interface to specify the exact tests you want to run. 3312883Sjason@lowepower.comFor instance, if you want to run only with `gem5.opt`, you can use 3412883Sjason@lowepower.com 3512883Sjason@lowepower.com```shell 3612883Sjason@lowepower.com./main.py run --variant opt 3712883Sjason@lowepower.com``` 3812883Sjason@lowepower.com 3912883Sjason@lowepower.comOr, if you want to just run X86 tests with the `gem5.opt` binary: 4012883Sjason@lowepower.com 4112883Sjason@lowepower.com```shell 4212883Sjason@lowepower.com./main.py run --length quick --variant opt --isa X86 4312883Sjason@lowepower.com``` 4412883Sjason@lowepower.com 4512883Sjason@lowepower.com 4612883Sjason@lowepower.comTo view all of the available tags, use 4712883Sjason@lowepower.com 4812883Sjason@lowepower.com```shell 4912883Sjason@lowepower.com./main.py list --all-tags 5012883Sjason@lowepower.com``` 5112883Sjason@lowepower.com 5212883Sjason@lowepower.comThe output is split into tag *types* (e.g., isa, variant, length) and the 5312883Sjason@lowepower.comtags for each type are listed after the type name. 5412883Sjason@lowepower.com 5512883Sjason@lowepower.comYou can specify "or" between tags within the same type by using the tag flag 5612883Sjason@lowepower.commultiple times. For instance, to run everything that is tagged "opt" or "fast" 5712883Sjason@lowepower.comuse 5812883Sjason@lowepower.com 5912883Sjason@lowepower.com```shell 6012883Sjason@lowepower.com./main.py run --variant opt --variant fast 6112883Sjason@lowepower.com``` 6212883Sjason@lowepower.com 6312883Sjason@lowepower.comYou can also specify "and" between different types of tags by specifying more 6412883Sjason@lowepower.comthan one type on the command line. For instance, this will only run tests with 6512883Sjason@lowepower.comboth the "X86" and "opt" tags. 6612883Sjason@lowepower.com 6712883Sjason@lowepower.com```shell 6812883Sjason@lowepower.com./main.py run --isa X86 --variant opt 6912883Sjason@lowepower.com``` 7012883Sjason@lowepower.com 7112883Sjason@lowepower.com## Running tests in batch 7212883Sjason@lowepower.com 7312883Sjason@lowepower.comThe testing infrastructure provides the two needed methods to run tests in 7412883Sjason@lowepower.combatch. First, you can list all of the tests based on the same tags as above in 7512883Sjason@lowepower.coma machine-readable format by passing the `-q` flag. This will list all of the 7612883Sjason@lowepower.com*suites* that match the given tag(s). 7712883Sjason@lowepower.com 7812883Sjason@lowepower.com```shell 7912883Sjason@lowepower.com./main.py list -q --suites 8012883Sjason@lowepower.comSuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-X86-opt 8112883Sjason@lowepower.comSuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-dynamic-X86-opt 8212883Sjason@lowepower.comSuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-X86-opt 8312883Sjason@lowepower.comSuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello64-static-ARM-opt 8412883Sjason@lowepower.comSuiteUID:tests/gem5/hello_se/test_hello_se.py:testhello32-static-ARM-opt 8512883Sjason@lowepower.comSuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt 8612883Sjason@lowepower.comSuiteUID:tests/gem5/test_build/test_build.py:build-X86-opt 8712883Sjason@lowepower.comSuiteUID:tests/gem5/test_build/test_build.py:build-RISCV-opt 8812883Sjason@lowepower.comSuiteUID:tests/gem5/test_build/test_build.py:build-ARM-opt 8912883Sjason@lowepower.com``` 9012883Sjason@lowepower.com 9112883Sjason@lowepower.comNext, you can run a single *suite* from the command line by passing the option 9212883Sjason@lowepower.com`--uid`. For instance, 9312883Sjason@lowepower.com 9412883Sjason@lowepower.com```shell 9512883Sjason@lowepower.com./main.py run --skip-build \ 9612883Sjason@lowepower.com --uid SuiteUID:tests/gem5/m5_util/test_exit.py:m5_exit_test-X86-opt 9712883Sjason@lowepower.com``` 9812883Sjason@lowepower.com 9912883Sjason@lowepower.comWith this method, you can only run a *single* suite at a time. If you want to 10012883Sjason@lowepower.comrun more than one uid, you must call `./main.py` multiple times. 10112883Sjason@lowepower.com 10212883Sjason@lowepower.comCurrently, you must specify `--skip-build` if you want to run a single suite or 10312883Sjason@lowepower.comrun in batch mode. Otherwise, you will build gem5 for all architectures. 10412883Sjason@lowepower.com 10512883Sjason@lowepower.com## Rerunning failed tests 10612883Sjason@lowepower.com 10712883Sjason@lowepower.comWhile developing software a common practice is to run tests, make a change, and 10812883Sjason@lowepower.comassert that the tests still pass. If tests fail you'll likely want to 10912883Sjason@lowepower.comrerun and fix those specific tests without running redundant ones. The testing 11012883Sjason@lowepower.cominfrastructure allows you to rerun tests which failed in the last execution by 11112883Sjason@lowepower.comusing the `rerun` command. 11212883Sjason@lowepower.com 11312883Sjason@lowepower.com```shell 11412883Sjason@lowepower.com./main.py run 11512883Sjason@lowepower.com# 11612883Sjason@lowepower.com# Some tests fail... 11712883Sjason@lowepower.com# 11812883Sjason@lowepower.com 11912883Sjason@lowepower.com# Rerun only the failed test suites (not the ones which passed). 12012883Sjason@lowepower.com./main.py rerun 12112883Sjason@lowepower.com``` 12212883Sjason@lowepower.com 12312883Sjason@lowepower.com# If something goes wrong 12412883Sjason@lowepower.com 12512883Sjason@lowepower.comThe first step is to turn up the verbosity of the output using `-v`. This will 12612883Sjason@lowepower.comallow you to see what tests are running and why a test is failing. 12712883Sjason@lowepower.com 12812883Sjason@lowepower.comIf a test fails, the temporary directory where the gem5 output was saved is kept 12912883Sjason@lowepower.comand the path to the directory is printed in the terminal. 13012883Sjason@lowepower.com 13112883Sjason@lowepower.com## Debugging the testing infrastructure 13212883Sjason@lowepower.com 13312883Sjason@lowepower.comEvery command takes an option for the verbosity. `-v`, `-vv`, `-vvv` will 13412883Sjason@lowepower.comincrease the verbosity level. If something isn't working correctly, you can 13512883Sjason@lowepower.comstart here. 13612883Sjason@lowepower.com 13712883Sjason@lowepower.comMost of the code for the testing infrastructure is in ext/testlib. This code 13812883Sjason@lowepower.comcontains the base code for tests, suites, fixtures, etc. The code in tests/gem5 13912883Sjason@lowepower.comis *gem5-specific* code. For the most part, the code in tests/gem5 extends the 14012883Sjason@lowepower.comstructures in ext/testlib. 14112883Sjason@lowepower.com 14212883Sjason@lowepower.com## Common errors 14312883Sjason@lowepower.com 14412883Sjason@lowepower.comYou may see a number of lines of output during test discovery that look like 14512883Sjason@lowepower.comthe following: 14612883Sjason@lowepower.com 14712883Sjason@lowepower.com```shell 14812883Sjason@lowepower.com Tried to load tests from ... but failed with an exception. 14912883Sjason@lowepower.com Tried to load tests from ... but failed with an exception. 15012883Sjason@lowepower.com ... 15112883Sjason@lowepower.com``` 15212883Sjason@lowepower.com 15312883Sjason@lowepower.comThe testing library searches all python files in the `tests/` directory. The 15412883Sjason@lowepower.comtest library executes each python file it finds searching for tests. It's okay 15512883Sjason@lowepower.comif the file causes an exception. This means there are no tests in that file 15612883Sjason@lowepower.com(e.g., it's not a new-style test). 15712883Sjason@lowepower.com 15812883Sjason@lowepower.com 15912883Sjason@lowepower.com# Binary test applications 16012883Sjason@lowepower.com 16112883Sjason@lowepower.comThe code for test binaries that are run in the gem5 guest during testing are 16212883Sjason@lowepower.comfound in `tests/test-progs`. 16312883Sjason@lowepower.comThere's one directory per test application. 16412883Sjason@lowepower.comThe source code is under the `source` directory. 16512883Sjason@lowepower.com 16612883Sjason@lowepower.comYou may have a `bin` directory as well. 16712883Sjason@lowepower.comThe `bin` directory is automatically created when running the test case that 16812883Sjason@lowepower.comuses the test binary. The binary is downloaded from the gem5 servers the first 16912883Sjason@lowepower.comtime it is referenced by a test. 17012883Sjason@lowepower.com 17112883Sjason@lowepower.com## Updating the test binaries 17212883Sjason@lowepower.com 17312883Sjason@lowepower.comThe test infrastructure should check with the gem5 servers to ensure you have 17412883Sjason@lowepower.comthe latest binaries. However, if you believe your binaries are out of date, 17512883Sjason@lowepower.comsimply delete the `bin` directory and they will be re-downloaded to your local 17612883Sjason@lowepower.commachine. 17712883Sjason@lowepower.com 17812883Sjason@lowepower.com## Building (new-style) test binaries 17912883Sjason@lowepower.com 18012883Sjason@lowepower.comIn each `src/` directory under `tests/test-progs`, there is a Makefile. 18112883Sjason@lowepower.comThis Makefile downloads a docker image and builds the test binary for some ISA 18212883Sjason@lowepower.com(e.g., Makefile.x86 builds the binary for x86). Additionally, if you run `make 18312883Sjason@lowepower.comupload` it will upload the binaries to the gem5 server, if you have access to 18412883Sjason@lowepower.commodify the binaries. *If you need to modify the binaries for updating a test or 18512883Sjason@lowepower.comadding a new test and you don't have access to the gem5 server, contact a 18612883Sjason@lowepower.commaintainer (see MAINTAINERS).* 18712883Sjason@lowepower.com 18812883Sjason@lowepower.com 18912883Sjason@lowepower.com# Running Tests in Parallel 19012883Sjason@lowepower.com 19112883Sjason@lowepower.comWhimsy has support for parallel testing baked in. This system supports 19212883Sjason@lowepower.comrunning multiple suites at the same time on the same computer. To run 19312883Sjason@lowepower.comsuites in parallel, supply the `-t <number-tests>` flag to the run command. 19412883Sjason@lowepower.com 19512883Sjason@lowepower.comFor example, to run up to three test suites at the same time:: 19612883Sjason@lowepower.com 19712883Sjason@lowepower.com ./main.py run --skip-build -t 3 19812883Sjason@lowepower.com 199