Compiletest
Introduction
compiletest
is the main test harness of the Rust test suite.
It allows test authors to organize large numbers of tests
(the Rust compiler has many thousands),
efficient test execution (parallel execution is supported),
and allows the test author to configure behavior and expected results of both
individual and groups of tests.
NOTE: For macOS users,
SIP
(System Integrity Protection) may consistently check the compiled binary by sending network requests to Apple, so you may get a huge performance degradation when running tests.You can resolve it by tweaking the following settings:
Privacy & Security -> Developer Tools -> Add Terminal (Or VsCode, etc.)
.
compiletest
may check test code for success, for runtime failure,
or for compile-time failure.
Tests are typically organized as a Rust source file with annotations in
comments before and/or within the test code.
These comments serve to direct compiletest
on if or how to run the test,
what behavior to expect, and more.
See header commands and the test suite documentation below
for more details on these annotations.
See the Adding new tests chapter for a tutorial on creating a new test, and the Running tests chapter on how to run the test suite.
Compiletest itself tries to avoid running tests when the artifacts
that are involved (mainly the compiler) haven't changed. You can use
x test --test-args --force-rerun
to rerun a test even when none of the
inputs have changed.
Test suites
All of the tests are in the tests
directory.
The tests are organized into "suites", with each suite in a separate subdirectory.
Each test suite behaves a little differently, with different compiler behavior
and different checks for correctness.
For example, the tests/incremental
directory contains tests for
incremental compilation.
The various suites are defined in src/tools/compiletest/src/common.rs
in
the pub enum Mode
declaration.
The following test suites are available, with links for more information:
ui
— tests that check the stdout/stderr from the compilation and/or running the resulting executableui-fulldeps
—ui
tests which require a linkable build ofrustc
(such as usingextern crate rustc_span;
or used as a plugin)pretty
— tests for pretty printingincremental
— tests incremental compilation behaviordebuginfo
— tests for debuginfo generation running debuggerscodegen
— tests for code generationcodegen-units
— tests for codegen unit partitioningassembly
— verifies assembly outputmir-opt
— tests for MIR generationrun-make
— general purpose tests using Rust programs (or Makefiles (legacy))run-pass-valgrind
— tests run with Valgrindcoverage
- tests for coverage instrumentationcoverage-run-rustdoc
- coverage tests that also run instrumented doctests- Rustdoc tests:
rustdoc
— tests for rustdoc, making sure that the generated files contain the expected documentation.rustdoc-gui
— tests for rustdoc's GUI using a web browser.rustdoc-js
— tests to ensure the rustdoc search is working as expected.rustdoc-js-std
— tests to ensure the rustdoc search is working as expected (run specifically on the std docs).rustdoc-json
— tests on the JSON output of rustdoc.rustdoc-ui
— tests on the terminal output of rustdoc.
Pretty-printer tests
The tests in tests/pretty
exercise the "pretty-printing" functionality of rustc
.
The -Z unpretty
CLI option for rustc
causes it to translate the input source
into various different formats, such as the Rust source after macro expansion.
The pretty-printer tests have several header commands described below. These commands can significantly change the behavior of the test, but the default behavior without any commands is to:
- Run
rustc -Zunpretty=normal
on the source file - Run
rustc -Zunpretty=normal
on the output of the previous step - The output of the previous two steps should be the same.
- Run
rustc -Zno-codegen
on the output to make sure that it can type check (this is similar to runningcargo check
)
If any of the commands above fail, then the test fails.
The header commands for pretty-printing tests are:
-
pretty-mode
specifies the mode pretty-print tests should run in (that is, the argument to-Zunpretty
). The default isnormal
if not specified. -
pretty-compare-only
causes a pretty test to only compare the pretty-printed output (stopping after step 3 from above). It will not try to compile the expanded output to type check it. This is needed for a pretty-mode that does not expand to valid Rust, or for other situations where the expanded output cannot be compiled. -
pretty-expanded
allows a pretty test to also check that the expanded output can be type checked. That is, after the steps above, it does two more steps:- Run
rustc -Zunpretty=expanded
on the original source - Run
rustc -Zno-codegen
on the expanded output to make sure that it can type check
This is needed because not all code can be compiled after being expanded. Pretty tests should specify this if they can. An example where this cannot be used is if the test includes
println!
. That macro expands to reference private internal functions of the standard library that cannot be called directly without thefmt_internals
feature gate.More history about this may be found in #23616.
- Run
-
pp-exact
is used to ensure a pretty-print test results in specific output. If specified without a value, then it means the pretty-print output should match the original source. If specified with a value, as in// pp-exact:foo.pp
, it will ensure that the pretty-printed output matches the contents of the given file. Otherwise, ifpp-exact
is not specified, then the pretty-printed output will be pretty-printed one more time, and the output of the two pretty-printing rounds will be compared to ensure that the pretty-printed output converges to a steady state.
Incremental tests
The tests in tests/incremental
exercise incremental compilation.
They use revision headers to tell compiletest to run the
compiler in a series of steps.
Compiletest starts with an empty directory with the -C incremental
flag, and
then runs the compiler for each revision, reusing the incremental results from
previous steps.
The revisions should start with:
rpass
— the test should compile and run successfullyrfail
— the test should compile successfully, but the executable should fail to runcfail
— the test should fail to compile
To make the revisions unique, you should add a suffix like rpass1
and rpass2
.
To simulate changing the source, compiletest also passes a --cfg
flag with
the current revision name.
For example, this will run twice, simulating changing a function:
//@ revisions: rpass1 rpass2
#[cfg(rpass1)]
fn foo() {
println!("one");
}
#[cfg(rpass2)]
fn foo() {
println!("two");
}
fn main() { foo(); }
cfail
tests support the forbid-output
header to specify that a certain
substring must not appear anywhere in the compiler output.
This can be useful to ensure certain errors do not appear, but this can be
fragile as error messages change over time, and a test may no longer be
checking the right thing but will still pass.
cfail
tests support the should-ice
header to specify that a test should
cause an Internal Compiler Error (ICE).
This is a highly specialized header to check that the incremental cache
continues to work after an ICE.
Debuginfo tests
The tests in tests/debuginfo
test debuginfo generation.
They build a program, launch a debugger, and issue commands to the debugger.
A single test can work with cdb, gdb, and lldb.
Most tests should have the // compile-flags: -g
header or something similar
to generate the appropriate debuginfo.
To set a breakpoint on a line, add a // #break
comment on the line.
The debuginfo tests consist of a series of debugger commands along with "check" lines which specify output that is expected from the debugger.
The commands are comments of the form // $DEBUGGER-command:$COMMAND
where
$DEBUGGER
is the debugger being used and $COMMAND
is the debugger command
to execute.
The debugger values can be:
cdb
gdb
gdbg
— GDB without Rust support (versions older than 7.11)gdbr
— GDB with Rust supportlldb
lldbg
— LLDB without Rust supportlldbr
— LLDB with Rust support (this no longer exists)
The command to check the output are of the form // $DEBUGGER-check:$OUTPUT
where $OUTPUT
is the output to expect.
For example, the following will build the test, start the debugger, set a breakpoint, launch the program, inspect a value, and check what the debugger prints:
//@ compile-flags: -g
//@ lldb-command: run
//@ lldb-command: print foo
//@ lldb-check: $0 = 123
fn main() {
let foo = 123;
b(); // #break
}
fn b() {}
The following header commands are available to disable a test based on the debugger currently being used:
min-cdb-version: 10.0.18317.1001
— ignores the test if the version of cdb is below the given versionmin-gdb-version: 8.2
— ignores the test if the version of gdb is below the given versionignore-gdb-version: 9.2
— ignores the test if the version of gdb is equal to the given versionignore-gdb-version: 7.11.90 - 8.0.9
— ignores the test if the version of gdb is in a range (inclusive)min-lldb-version: 310
— ignores the test if the version of lldb is below the given versionrust-lldb
— ignores the test if lldb is not contain the Rust plugin. NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored. This should probably be removed.
Codegen tests
The tests in tests/codegen
test LLVM code generation.
They compile the test with the --emit=llvm-ir
flag to emit LLVM IR.
They then run the LLVM FileCheck tool.
The test is annotated with various // CHECK
comments to check the generated code.
See the FileCheck documentation for a tutorial and more information.
See also the assembly tests for a similar set of tests.
Assembly tests
The tests in tests/assembly
test LLVM assembly output.
They compile the test with the --emit=asm
flag to emit a .s
file with the
assembly output.
They then run the LLVM FileCheck tool.
Each test should be annotated with the // assembly-output:
header
with a value of either emit-asm
or ptx-linker
to indicate
the type of assembly output.
Then, they should be annotated with various // CHECK
comments to check the
assembly output.
See the FileCheck documentation for a tutorial and more information.
See also the codegen tests for a similar set of tests.
Codegen-units tests
The tests in tests/codegen-units
test the
monomorphization collector and CGU partitioning.
These tests work by running rustc
with a flag to print the result of the
monomorphization collection pass, and then special annotations in the file are
used to compare against that.
Each test should be annotated with the // compile-flags:-Zprint-mono-items=VAL
header with the appropriate VAL to instruct rustc
to print the
monomorphization information.
Then, the test should be annotated with comments of the form //~ MONO_ITEM name
where name
is the monomorphized string printed by rustc like fn <u32 as Trait>::foo
.
To check for CGU partitioning, a comment of the form //~ MONO_ITEM name @@ cgu
where cgu
is a space separated list of the CGU names and the linkage
information in brackets.
For example: //~ MONO_ITEM static function::FOO @@ statics[Internal]
Mir-opt tests
The tests in tests/mir-opt
check parts of the generated MIR to make
sure it is generated correctly and is doing the expected optimizations.
Check out the MIR Optimizations chapter for more.
Compiletest will build the test with several flags to dump the MIR output and set a baseline for optimizations:
-Copt-level=1
-Zdump-mir=all
-Zmir-opt-level=4
-Zvalidate-mir
-Zdump-mir-exclude-pass-number
The test should be annotated with // EMIT_MIR
comments that specify files that
will contain the expected MIR output.
You can use x test --bless
to create the initial expected files.
There are several forms the EMIT_MIR
comment can take:
-
// EMIT_MIR $MIR_PATH.mir
— This will check that the given filename matches the exact output from the MIR dump. For example,my_test.main.SimplifyCfg-elaborate-drops.after.mir
will load that file from the test directory, and compare it against the dump from rustc.Checking the "after" file (which is after optimization) is useful if you are interested in the final state after an optimization. Some rare cases may want to use the "before" file for completeness.
-
// EMIT_MIR $MIR_PATH.diff
— where$MIR_PATH
is the filename of the MIR dump, such asmy_test_name.my_function.EarlyOtherwiseBranch
. Compiletest will diff the.before.mir
and.after.mir
files, and compare the diff output to the expected.diff
file from theEMIT_MIR
comment.This is useful if you want to see how an optimization changes the MIR.
-
// EMIT_MIR $MIR_PATH.dot
— When using specific flags that dump additional MIR data (e.g.-Z dump-mir-graphviz
to produce.dot
files), this will check that the output matches the given file.
By default 32 bit and 64 bit targets use the same dump files, which can be
problematic in the presence of pointers in constants or other bit width
dependent things. In that case you can add // EMIT_MIR_FOR_EACH_BIT_WIDTH
to
your test, causing separate files to be generated for 32bit and 64bit systems.
run-make
tests
NOTE: We are planning to migrate all existing Makefile-based
run-make
tests to Rust recipes. You should not be adding new Makefile-basedrun-make
tests.
The tests in tests/run-make
are general-purpose tests using Rust recipes,
which are small programs allowing arbitrary Rust code such as rustc
invocations, and is supported by a run_make_support
library. Using Rust
recipes provide the ultimate in flexibility.
These should be used as a last resort. If possible, you should use one of the other test suites.
If there is some minor feature missing which you need for your test,
consider extending compiletest to add a header command for what you need.
However, if running a bunch of commands is really what you need,
run-make
is here to the rescue!
Using Rust recipes
Each test should be in a separate directory with a rmake.rs
Rust program,
called the recipe. A recipe will be compiled and executed by compiletest
with the run_make_support
library linked in.
If you need new utilities or functionality, consider extending and improving
the run_make_support
library.
Compiletest directives like //@ only-<target>
or //@ ignore-<target>
are supported in
rmake.rs
, like in UI tests.
Two run-make
tests are ported over to Rust recipes as examples:
- https://github.com/rust-lang/rust/tree/master/tests/run-make/CURRENT_RUSTC_VERSION
- https://github.com/rust-lang/rust/tree/master/tests/run-make/a-b-a-linker-guard
Quickly check if rmake.rs
tests can be compiled
You can quickly check if rmake.rs
tests can be compiled without having to
build stage1 rustc by forcing rmake.rs
to be compiled with the stage0
compiler:
$ COMPILETEST_FORCE_STAGE0=1 x test --stage 0 tests/run-make/<test-name>
Of course, some tests will not successfully run in this way.
Using Makefiles (legacy)
NOTE: You should avoid writing new Makefile-based
run-make
tests.
Each test should be in a separate directory with a Makefile
indicating the
commands to run.
There is a tools.mk
Makefile which you can include which provides a bunch of
utilities to make it easier to run commands and compare outputs.
Take a look at some of the other tests for some examples on how to get started.
Valgrind tests
The tests in tests/run-pass-valgrind
are for use with Valgrind.
These are currently vestigial, as Valgrind is no longer used in CI.
These may be removed in the future.
Coverage tests
The tests in tests/coverage
are shared by multiple test modes that test
coverage instrumentation in different ways.
Running the coverage
test suite will automatically run each test in all of
the different coverage modes.
Each mode also has an alias to run the coverage tests in just that mode:
./x test coverage # runs all of tests/coverage in all coverage modes
./x test tests/coverage # same as above
./x test tests/coverage/if.rs # runs the specified test in all coverage modes
./x test coverage-map # runs all of tests/coverage in "coverage-map" mode only
./x test coverage-run # runs all of tests/coverage in "coverage-run" mode only
./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
In coverage-map
mode, these tests verify the mappings between source code
regions and coverage counters that are emitted by LLVM.
They compile the test with --emit=llvm-ir
,
then use a custom tool (src/tools/coverage-dump
)
to extract and pretty-print the coverage mappings embedded in the IR.
These tests don't require the profiler runtime, so they run in PR CI jobs and
are easy to run/bless locally.
These coverage map tests can be sensitive to changes in MIR lowering or MIR optimizations, producing mappings that are different but produce identical coverage reports.
As a rule of thumb, any PR that doesn't change coverage-specific
code should feel free to re-bless the coverage-map
tests as necessary,
without worrying about the actual changes, as long as the coverage-run
tests
still pass.
In coverage-run
mode, these tests perform an end-to-end test of coverage reporting.
They compile a test program with coverage instrumentation, run that program to
produce raw coverage data, and then use LLVM tools to process that data into a
human-readable code coverage report.
Instrumented binaries need to be linked against the LLVM profiler runtime,
so coverage-run
tests are automatically skipped
unless the profiler runtime is enabled in config.toml
:
# config.toml
[build]
profiler = true
This also means that they typically don't run in PR CI jobs, though they do run as part of the full set of CI jobs used for merging.
The tests in tests/coverage-run-rustdoc
also run instrumented doctests and
include them in the coverage report. This avoids having to build rustdoc when
only running the main coverage
suite.
Building auxiliary crates
It is common that some tests require additional auxiliary crates to be compiled. There are multiple headers to assist with that:
aux-build
aux-crate
aux-bin
aux-codegen-backend
aux-build
will build a separate crate from the named source file.
The source file should be in a directory called auxiliary
beside the test file.
//@ aux-build: my-helper.rs
extern crate my_helper;
// ... You can use my_helper.
The aux crate will be built as a dylib if possible (unless on a platform that
does not support them, or the no-prefer-dynamic
header is specified in the
aux file).
The -L
flag is used to find the extern crates.
aux-crate
is very similar to aux-build
; however, it uses the --extern
flag to link to the extern crate.
That allows you to specify the additional syntax of the --extern
flag, such
as renaming a dependency.
For example, // aux-crate:foo=bar.rs
will compile auxiliary/bar.rs
and
make it available under then name foo
within the test.
This is similar to how Cargo does dependency renaming.
aux-bin
is similar to aux-build
but will build a binary instead of a
library. The binary will be available in auxiliary/bin
relative to the working
directory of the test.
aux-codegen-backend
is similar to aux-build
, but will then pass the compiled
dylib to -Zcodegen-backend
when building the main file. This will only work
for tests in tests/ui-fulldeps
, since it requires the use of compiler crates.
Auxiliary proc-macro
If you want a proc-macro dependency, then there currently is some ceremony
needed.
Place the proc-macro itself in a file like auxiliary/my-proc-macro.rs
with the following structure:
//@ force-host
//@ no-prefer-dynamic
#![crate_type = "proc-macro"]
extern crate proc_macro;
use proc_macro::TokenStream;
#[proc_macro]
pub fn foo(input: TokenStream) -> TokenStream {
"".parse().unwrap()
}
The force-host
is needed because proc-macros are loaded in the host
compiler, and no-prefer-dynamic
is needed to tell compiletest to not use
prefer-dynamic
which is not compatible with proc-macros.
The #![crate_type]
attribute is needed to specify the correct crate-type.
Then in your test, you can build with aux-build
:
//@ aux-build: my-proc-macro.rs
extern crate my_proc_macro;
fn main() {
my_proc_macro::foo!();
}
Revisions
Revisions allow a single test file to be used for multiple tests. This is done by adding a special header at the top of the file:
//@ revisions: foo bar baz
This will result in the test being compiled (and tested) three times,
once with --cfg foo
, once with --cfg bar
, and once with --cfg baz
.
You can therefore use #[cfg(foo)]
etc within the test to tweak
each of these results.
You can also customize headers and expected error messages to a particular
revision. To do this, add [revision-name]
after the //
comment, like so:
// A flag to pass in only for cfg `foo`:
//@[foo]compile-flags: -Z verbose-internals
#[cfg(foo)]
fn test_foo() {
let x: usize = 32_u32; //[foo]~ ERROR mismatched types
}
Multiple revisions can be specified in a comma-separated list, such as
//[foo,bar,baz]~^
.
In test suites that use the LLVM FileCheck tool, the current revision name is also registered as an additional prefix for FileCheck directives:
//@ revisions: NORMAL COVERAGE
//@ [COVERAGE] compile-flags: -Cinstrument-coverage
//@ [COVERAGE] needs-profiler-support
// COVERAGE: @__llvm_coverage_mapping
// NORMAL-NOT: @__llvm_coverage_mapping
// CHECK: main
fn main() {}
Note that not all headers have meaning when customized to a revision.
For example, the ignore-test
header (and all "ignore" headers)
currently only apply to the test as a whole, not to particular
revisions. The only headers that are intended to really work when
customized to a revision are error patterns and compiler flags.
Following is classes of tests that support revisions:
- UI
- assembly
- codegen
- coverage
- debuginfo
- rustdoc UI tests
- incremental (these are special in that they inherently cannot be run in parallel)
Ignoring unused revision names
Normally, revision names mentioned in other headers and error annotations must
correspond to an actual revision declared in a revisions
header. This is
enforced by an ./x test tidy
check.
If a revision name needs to be temporarily removed from the revision list for
some reason, the above check can be suppressed by adding the revision name to
an //@ unused-revision-names:
header instead.
Specifying an unused name of *
(i.e. //@ unused-revision-names: *
) will
permit any unused revision name to be mentioned.
Compare modes
Compiletest can be run in different modes, called compare modes, which can be used to compare the behavior of all tests with different compiler flags enabled. This can help highlight what differences might appear with certain flags, and check for any problems that might arise.
To run the tests in a different mode, you need to pass the --compare-mode
CLI flag:
./x test tests/ui --compare-mode=chalk
The possible compare modes are:
polonius
— Runs with Polonius with-Zpolonius
.chalk
— Runs with Chalk with-Zchalk
.split-dwarf
— Runs with unpacked split-DWARF with-Csplit-debuginfo=unpacked
.split-dwarf-single
— Runs with packed split-DWARF with-Csplit-debuginfo=packed
.
See UI compare modes for more information about how UI tests support different output for different modes.
In CI, compare modes are only used in one Linux builder, and only with the following settings:
tests/debuginfo
: Usessplit-dwarf
mode. This helps ensure that none of the debuginfo tests are affected when enabling split-DWARF.
Note that compare modes are separate to revisions.
All revisions are tested when running ./x test tests/ui
, however
compare-modes must be manually run individually via the --compare-mode
flag.