Running tests

You can run the entire test collection using x. But note that running the entire test collection is almost never what you want to do during local development because it takes a really long time. For local development, see the subsection after on how to run a subset of tests.

Running plain `./x test` will build the stage 1 compiler and then run the whole test suite. This not only include `tests/`, but also `library/`, `compiler/`, `src/tools/` package tests and more.

You usually only want to run a subset of the test suites (or even a smaller set of tests than that) which you expect will exercise your changes. PR CI exercises a subset of test collections, and merge queue CI will exercise all of the test collection.

./x test

The test results are cached and previously successful tests are ignored during testing. The stdout/stderr contents as well as a timestamp file for every test can be found under build/<target-triple>/test/ for the given <target-triple>. To force-rerun a test (e.g. in case the test runner fails to notice a change) you can use the --force-rerun CLI option.

Note on requirements of external dependencies

Some test suites may require external dependecies. This is especially true of debuginfo tests. Some debuginfo tests require a Python-enabled gdb. You can test if your gdb install supports Python by using the python command from within gdb. Once invoked you can type some Python code (e.g. print("hi")) followed by return and then CTRL+D to execute it. If you are building gdb from source, you will need to configure with --with-python=<path-to-python-binary>.

Running a subset of the test suites

When working on a specific PR, you will usually want to run a smaller set of tests. For example, a good "smoke test" that can be used after modifying rustc to see if things are generally working correctly would be to exercise the ui test suite (tests/ui):

./x test tests/ui

This will run the ui test suite. Of course, the choice of test suites is somewhat arbitrary, and may not suit the task you are doing. For example, if you are hacking on debuginfo, you may be better off with the debuginfo test suite:

./x test tests/debuginfo

If you only need to test a specific subdirectory of tests for any given test suite, you can pass that directory as a filter to ./x test:

./x test tests/ui/const-generics

Note for MSYS2

On MSYS2 the paths seem to be strange and ./x test neither recognizes tests/ui/const-generics nor tests\ui\const-generics. In that case, you can workaround it by using e.g. ./x test ui --test-args="tests/ui/const-generics".

Likewise, you can test a single file by passing its path:

./x test tests/ui/const-generics/const-test.rs

x doesn't support running a single tool test by passing its path yet. You'll have to use the --test-args argument as describled below.

./x test src/tools/miri --test-args tests/fail/uninit/padding-enum.rs

Run only the tidy script

./x test tidy

Run tests on the standard library

./x test --stage 0 library/std

Note that this only runs tests on std; if you want to test core or other crates, you have to specify those explicitly.

Run the tidy script and tests on the standard library

./x test --stage 0 tidy library/std

Run tests on the standard library using a stage 1 compiler

./x test --stage 1 library/std

By listing which test suites you want to run you avoid having to run tests for components you did not change at all.

Note that bors only runs the tests with the full stage 2 build; therefore, while the tests **usually** work fine with stage 1, there are some limitations.

Run all tests using a stage 2 compiler

./x test --stage 2
You almost never need to do this; CI will run these tests for you.

Run unit tests on the compiler/library

You may want to run unit tests on a specific file with following:

./x test compiler/rustc_data_structures/src/thin_vec/tests.rs

But unfortunately, it's impossible. You should invoke the following instead:

./x test compiler/rustc_data_structures/ --test-args thin_vec

Running an individual test

Another common thing that people want to do is to run an individual test, often the test they are trying to fix. As mentioned earlier, you may pass the full file path to achieve this, or alternatively one may invoke x with the --test-args option:

./x test tests/ui --test-args issue-1234

Under the hood, the test runner invokes the standard Rust test runner (the same one you get with #[test]), so this command would wind up filtering for tests that include "issue-1234" in the name. Thus, --test-args is a good way to run a collection of related tests.

Passing arguments to rustc when running tests

It can sometimes be useful to run some tests with specific compiler arguments, without using RUSTFLAGS (during development of unstable features, with -Z flags, for example).

This can be done with ./x test's --compiletest-rustc-args option, to pass additional arguments to the compiler when building the tests.

Editing and updating the reference files

If you have changed the compiler's output intentionally, or you are making a new test, you can pass --bless to the test subcommand. E.g. if some tests in tests/ui are failing, you can run

./x test tests/ui --bless

to automatically adjust the .stderr, .stdout or .fixed files of all tests. Of course you can also target just specific tests with the --test-args your_test_name flag, just like when running the tests.

Configuring test running

There are a few options for running tests:

  • config.toml has the rust.verbose-tests option. If false, each test will print a single dot (the default). If true, the name of every test will be printed. This is equivalent to the --quiet option in the Rust test harness
  • The environment variable RUST_TEST_THREADS can be set to the number of concurrent threads to use for testing.

Passing --pass $mode

Pass UI tests now have three modes, check-pass, build-pass and run-pass. When --pass $mode is passed, these tests will be forced to run under the given $mode unless the directive //@ ignore-pass exists in the test file. For example, you can run all the tests in tests/ui as check-pass:

./x test tests/ui --pass check

By passing --pass $mode, you can reduce the testing time. For each mode, please see Controlling pass/fail expectations.

Running tests with different "compare modes"

UI tests may have different output depending on certain "modes" that the compiler is in. For example, when using the Polonius mode, a test foo.rs will first look for expected output in foo.polonius.stderr, falling back to the usual foo.stderr if not found. The following will run the UI test suite in Polonius mode:

./x test tests/ui --compare-mode=polonius

See Compare modes for more details.

Running tests manually

Sometimes it's easier and faster to just run the test by hand. Most tests are just .rs files, so after creating a rustup toolchain, you can do something like:

rustc +stage1 tests/ui/issue-1234.rs

This is much faster, but doesn't always work. For example, some tests include directives that specify specific compiler flags, or which rely on other crates, and they may not run the same without those options.

Running run-make tests

Windows

Running the run-make test suite on Windows is a currently bit more involved. There are numerous prerequisites and environmental requirements:

  • Install msys2: https://www.msys2.org/
  • Specify MSYS2_PATH_TYPE=inherit in msys2.ini in the msys2 installation directory, run the following with MSYS2 MSYS:
    • pacman -Syuu
    • pacman -S make
    • pacman -S diffutils
    • pacman -S binutils
    • ./x test run-make (./x test tests/run-make doesn't work)

There is on-going work to not rely on Makefiles in the run-make test suite. Once this work is completed, you can run the entire run-make test suite on native Windows inside cmd or PowerShell without needing to install and use MSYS2. As of Oct 2024, it is already possible to run the vast majority of the run-make test suite outside of MSYS2, but there will be failures for the tests that still use Makefiles due to not finding make.

Running tests on a remote machine

Tests may be run on a remote machine (e.g. to test builds for a different architecture). This is done using remote-test-client on the build machine to send test programs to remote-test-server running on the remote machine. remote-test-server executes the test programs and sends the results back to the build machine. remote-test-server provides unauthenticated remote code execution so be careful where it is used.

To do this, first build remote-test-server for the remote machine, e.g. for RISC-V

./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu

The binary will be created at ./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server. Copy this over to the remote machine.

On the remote machine, run the remote-test-server with the --bind 0.0.0.0:12345 flag (and optionally -v for verbose output). Output should look like this:

$ ./remote-test-server -v --bind 0.0.0.0:12345
starting test server
listening on 0.0.0.0:12345!

Note that binding the server to 0.0.0.0 will allow all hosts able to reach your machine to execute arbitrary code on your machine. We strongly recommend either setting up a firewall to block external access to port 12345, or to use a more restrictive IP address when binding.

You can test if the remote-test-server is working by connecting to it and sending ping\n. It should reply pong:

$ nc $REMOTE_IP 12345
ping
pong

To run tests using the remote runner, set the TEST_DEVICE_ADDR environment variable then use x as usual. For example, to run ui tests for a RISC-V machine with the IP address 1.2.3.4 use

export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x test tests/ui --target riscv64gc-unknown-linux-gnu

If remote-test-server was run with the verbose flag, output on the test machine may look something like

[...]
run "/tmp/work/test1007/a"
run "/tmp/work/test1008/a"
run "/tmp/work/test1009/a"
run "/tmp/work/test1010/a"
run "/tmp/work/test1011/a"
run "/tmp/work/test1012/a"
run "/tmp/work/test1013/a"
run "/tmp/work/test1014/a"
run "/tmp/work/test1015/a"
run "/tmp/work/test1016/a"
run "/tmp/work/test1017/a"
run "/tmp/work/test1018/a"
[...]

Tests are built on the machine running x not on the remote machine. Tests which fail to build unexpectedly (or ui tests producing incorrect build output) may fail without ever running on the remote machine.

Testing on emulators

Some platforms are tested via an emulator for architectures that aren't readily available. For architectures where the standard library is well supported and the host operating system supports TCP/IP networking, see the above instructions for testing on a remote machine (in this case the remote machine is emulated).

There is also a set of tools for orchestrating running the tests within the emulator. Platforms such as arm-android and arm-unknown-linux-gnueabihf are set up to automatically run the tests under emulation on GitHub Actions. The following will take a look at how a target's tests are run under emulation.

The Docker image for armhf-gnu includes QEMU to emulate the ARM CPU architecture. Included in the Rust tree are the tools remote-test-client and remote-test-server which are programs for sending test programs and libraries to the emulator, and running the tests within the emulator, and reading the results. The Docker image is set up to launch remote-test-server and the build tools use remote-test-client to communicate with the server to coordinate running tests (see src/bootstrap/src/core/build_steps/test.rs).

TODO

  • Is there any support for using an iOS emulator?
  • It's also unclear to me how the wasm or asm.js tests are run.

Running rustc_codegen_gcc tests

First thing to know is that it only supports linux x86_64 at the moment. We will extend its support later on.

You need to update codegen-backends value in your config.toml file in the [rust] section and add "gcc" in the array:

codegen-backends = ["llvm", "gcc"]

Then you need to install libgccjit 12. For example with apt:

$ apt install libgccjit-12-dev

Now you can run the following command:

$ ./x test compiler/rustc_codegen_gcc/

If it cannot find the .so library (if you installed it with apt for example), you need to pass the library file path with LIBRARY_PATH:

$ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x test compiler/rustc_codegen_gcc/

If you encounter bugs or problems, don't hesitate to open issues on the rustc_codegen_gcc repository.