Compiletest directives

Directives are special comments that tell compiletest how to build and interpret a test. They may also appear in rmake.rs run-make tests.

They are normally put after the short comment that explains the point of this test. Compiletest test suites use //@ to signal that a comment is a directive. For example, this test uses the //@ compile-flags command to specify a custom flag to give to rustc when the test is compiled:

// Test the behavior of `0 - 1` when overflow checks are disabled.

//@ compile-flags: -C overflow-checks=off

fn main() {
    let x = 0 - 1;
    ...
}

Directives can be standalone (like //@ run-pass) or take a value (like //@ compile-flags: -C overflow-checks=off).

Directives are written one directive per line: you cannot write multiple directives on the same line. For example, if you write //@ only-x86 only-windows then only-windows is interpreted as a comment, not a separate directive.

Listing of compiletest directives

The following is a list of compiletest directives. Directives are linked to sections that describe the command in more detail if available. This list may not be exhaustive. Directives can generally be found by browsing the TestProps structure found in directives.rs from the compiletest source.

Assembly

Auxiliary builds

See Building auxiliary crates

Controlling outcome expectations

See Controlling pass/fail expectations.

Controlling output snapshots and normalizations

See Normalization, Output comparison and Rustfix tests for more details.

Controlling when tests are run

These directives are used to ignore the test in some situations, which means the test won't be compiled or run.

• ignore-X where X is a target detail or other criteria on which to ignore the test (see below)
• only-X is like ignore-X, but will only run the test on that target or stage
• ignore-auxiliary is intended for files that participate in one or more other main test files but that compiletest should not try to build the file itself. Please backlink to which main test is actually using the auxiliary file.
• ignore-test always ignores the test. This can be used to temporarily disable a test if it is currently not working, but you want to keep it in tree to re-enable it later.

Some examples of X in ignore-X or only-X:

• A full target triple: aarch64-apple-ios
• Architecture: aarch64, arm, mips, wasm32, x86_64, x86, ...
• OS: android, emscripten, freebsd, ios, linux, macos, windows, ...
• Environment (fourth word of the target triple): gnu, msvc, musl
• WASM: wasm32-bare matches wasm32-unknown-unknown. emscripten also matches that target as well as the emscripten targets.
• Pointer width: 32bit, 64bit
• Endianness: endian-big
• Stage: stage1, stage2
• Binary format: elf
• Channel: stable, beta
• When cross compiling: cross-compile
• When remote testing is used: remote
• When particular debuggers are being tested: cdb, gdb, lldb
• When particular debugger versions are matched: ignore-gdb-version
• Specific compare modes: compare-mode-polonius, compare-mode-chalk, compare-mode-split-dwarf, compare-mode-split-dwarf-single
• The two different test modes used by coverage tests: ignore-coverage-map, ignore-coverage-run
• When testing a dist toolchain: dist
  • This needs to be enabled with COMPILETEST_ENABLE_DIST_TESTS=1
• The rustc_abi of the target: e.g. rustc_abi-x86_64-sse2

The following directives will check rustc build settings and target settings:

• needs-asm-support — ignores if the host architecture doesn't have stable support for asm!. For tests that cross-compile to explicit targets via --target, use needs-llvm-components instead to ensure the appropriate backend is available.
• needs-profiler-runtime — ignores the test if the profiler runtime was not enabled for the target (build.profiler = true in rustc's bootstrap.toml)
• needs-sanitizer-support — ignores if the sanitizer support was not enabled for the target (sanitizers = true in rustc's bootstrap.toml)
• needs-sanitizer-{address,hwaddress,leak,memory,thread} — ignores if the corresponding sanitizer is not enabled for the target (AddressSanitizer, hardware-assisted AddressSanitizer, LeakSanitizer, MemorySanitizer or ThreadSanitizer respectively)
• needs-run-enabled — ignores if it is a test that gets executed, and running has been disabled. Running tests can be disabled with the x test --run=never flag, or running on fuchsia.
• needs-unwind — ignores if the target does not support unwinding
• needs-rust-lld — ignores if the rust lld support is not enabled (rust.lld = true in bootstrap.toml)
• needs-threads — ignores if the target does not have threading support
• needs-subprocess — ignores if the target does not have subprocess support
• needs-symlink — ignores if the target does not support symlinks. This can be the case on Windows if the developer did not enable privileged symlink permissions.
• ignore-std-debug-assertions — ignores if std was built with debug assertions.
• needs-std-debug-assertions — ignores if std was not built with debug assertions.
• ignore-rustc-debug-assertions — ignores if rustc was built with debug assertions.
• needs-rustc-debug-assertions — ignores if rustc was not built with debug assertions.
• needs-target-has-atomic — ignores if target does not have support for all specified atomic widths, e.g. the test with //@ needs-target-has-atomic: 8, 16, ptr will only run if it supports the comma-separated list of atomic widths.
• needs-dynamic-linking — ignores if target does not support dynamic linking (which is orthogonal to it being unable to create dylib and cdylib crate types)
• needs-crate-type — ignores if target platform does not support one or more of the comma-delimited list of specified crate types. For example, //@ needs-crate-type: cdylib, proc-macro will cause the test to be ignored on wasm32-unknown-unknown target because the target does not support the proc-macro crate type.
• needs-target-std — ignores if target platform does not have std support.
• ignore-backends — ignores the listed backends, separated by whitespace characters.
• needs-backends — only runs the test if current codegen backend is listed.

The following directives will check LLVM support:

• exact-llvm-major-version: 19 — ignores if the llvm major version does not match the specified llvm major version.
• min-llvm-version: 13.0 — ignored if the LLVM version is less than the given value
• min-system-llvm-version: 12.0 — ignored if using a system LLVM and its version is less than the given value
• max-llvm-major-version: 19 — ignored if the LLVM major version is higher than the given major version
• ignore-llvm-version: 9.0 — ignores a specific LLVM version
• ignore-llvm-version: 7.0 - 9.9.9 — ignores LLVM versions in a range (inclusive)
• needs-llvm-components: powerpc — ignores if the specific LLVM component was not built. Note: The test will fail on CI (when COMPILETEST_REQUIRE_ALL_LLVM_COMPONENTS is set) if the component does not exist.
• needs-forced-clang-based-tests — test is ignored unless the environment variable RUSTBUILD_FORCE_CLANG_BASED_TESTS is set, which enables building clang alongside LLVM
  • This is only set in two CI jobs (x86_64-gnu-debug and aarch64-gnu-debug), which only runs a subset of run-make tests. Other tests with this directive will not run at all, which is usually not what you want.

See also Debuginfo tests for directives for ignoring debuggers.

Affecting how tests are built

The edition directive

The //@ edition directive can take an exact edition, a bounded range of editions, or a left-bounded half-open range of editions. This affects which edition is used by ./x test to run the test.

For example:

• A test with the //@ edition: 2018 directive will only run under the 2018 edition.
• A test with the //@ edition: 2015..2021 directive can be run under the 2015, 2018, and 2021 editions. However, CI will only run the test with the lowest edition in the range (which is 2015 in this example).
• A test with the //@ edition: 2018.. directive will run under 2018 edition or greater. However, CI will only run the test with the lowest edition in the range (which is 2018 in this example).

You can also force ./x test to use a specific edition by passing the -- --edition= argument. However, tests with the //@ edition directive will clamp the value passed to the argument. For example, if we run ./x test -- --edition=2015:

• A test with the //@ edition: 2018 will run with the 2018 edition.
• A test with the //@ edition: 2015..2021 will be run with the 2015 edition.
• A test with the //@ edition: 2018.. will run with the 2018 edition.

Rustdoc

Test-suite-specific directives

The test suites rustdoc, rustdoc-js/rustdoc-js-std and rustdoc-json each feature an additional set of directives whose basic syntax resembles the one of compiletest directives but which are ultimately read and checked by separate tools. For more information, please read their respective chapters as linked above.

Pretty printing

See Pretty-printer.

Misc directives

• no-auto-check-cfg — disable auto check-cfg (only for --check-cfg tests)
• revisions — compile multiple times -forbid-output — incremental cfail rejects output pattern
• should-ice — incremental cfail should ICE
• reference — an annotation linking to a rule in the reference
• disable-gdb-pretty-printers — disable gdb pretty printers for debuginfo tests

Tool-specific directives

The following directives affect how certain command-line tools are invoked, in test suites that use those tools:

• filecheck-flags adds extra flags when running LLVM's FileCheck tool.
  • Used by codegen tests, assembly tests, and MIR-opt tests.
• llvm-cov-flags adds extra flags when running LLVM's llvm-cov tool.
  • Used by coverage tests in coverage-run mode.

Tidy specific directives

The following directives control how the tidy script verifies tests.

• ignore-tidy-target-specific-tests disables checking that the appropriate LLVM component is required (via a needs-llvm-components directive) when a test is compiled for a specific target (via the --target flag in a compile-flag directive).
• unused-revision-names - suppress tidy checks for mentioning unknown revision names.

Substitutions

Directive values support substituting a few variables which will be replaced with their corresponding value. For example, if you need to pass a compiler flag with a path to a specific file, something like the following could work:

//@ compile-flags: --remap-path-prefix={{src-base}}=/the/src

Where the sentinel {{src-base}} will be replaced with the appropriate path described below:

• {{cwd}}: The directory where compiletest is run from. This may not be the root of the checkout, so you should avoid using it where possible.
  • Examples: /path/to/rust, /path/to/build/root
• {{src-base}}: The directory where the test is defined. This is equivalent to $DIR for output normalization.
  • Example: /path/to/rust/tests/ui/error-codes
• {{build-base}}: The base directory where the test's output goes. This is equivalent to $TEST_BUILD_DIR for output normalization.
  • Example: /path/to/rust/build/x86_64-unknown-linux-gnu/test/ui
• {{rust-src-base}}: The sysroot directory where libstd/libcore/... are located
• {{sysroot-base}}: Path of the sysroot directory used to build the test.
  • Mainly intended for ui-fulldeps tests that run the compiler via API.
• {{target-linker}}: Linker that would be passed to -Clinker for this test, or blank if no linker override is active.
  • Mainly intended for ui-fulldeps tests that run the compiler via API.
• {{target}}: The target the test is compiling for
  • Example: x86_64-unknown-linux-gnu

See tests/ui/argfile/commandline-argfile.rs for an example of a test that uses this substitution.

Adding a directive

One would add a new directive if there is a need to define some test property or behavior on an individual, test-by-test basis. A directive property serves as the directive's backing store (holds the command's current value) at runtime.

To add a new directive property:

1. Look for the pub struct TestProps declaration in src/tools/compiletest/src/directives.rs and add the new public property to the end of the declaration.
2. Look for the impl TestProps implementation block immediately following the struct declaration and initialize the new property to its default value.

Adding a new directive parser

When compiletest encounters a test file, it parses the file a line at a time by calling every parser defined in the Config struct's implementation block, also in src/tools/compiletest/src/directives.rs (note that the Config struct's declaration block is found in src/tools/compiletest/src/common.rs). TestProps's load_from() method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (//@) directive such as //@ must-compile-successfully or //@ failure-status. Whitespace after the comment marker is optional.

Parsers will override a given directive property's default value merely by being specified in the test file as a directive or by having a parameter value specified in the test file, depending on the directive.

Parsers defined in impl Config are typically named parse_<directive-name> (note kebab-case <directive-command> transformed to snake-case <directive_command>). impl Config also defines several 'low-level' parsers which make it simple to parse common patterns like simple presence or not (parse_name_directive()), directive:parameter(s) (parse_name_value_directive()), optional parsing only if a particular cfg attribute is defined (has_cfg_prefix()) and many more. The low-level parsers are found near the end of the impl Config block; be sure to look through them and their associated parsers immediately above to see how they are used to avoid writing additional parsing code unnecessarily.

As a concrete example, here is the implementation for the parse_failure_status() parser, in src/tools/compiletest/src/directives.rs:

@@ -232,6 +232,7 @@ pub struct TestProps {
     // customized normalization rules
     pub normalize_stdout: Vec<(String, String)>,
     pub normalize_stderr: Vec<(String, String)>,
+    pub failure_status: i32,
 }

 impl TestProps {
@@ -260,6 +261,7 @@ impl TestProps {
             run_pass: false,
             normalize_stdout: vec![],
             normalize_stderr: vec![],
+            failure_status: 101,
         }
     }

@@ -383,6 +385,10 @@ impl TestProps {
             if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") {
                 self.normalize_stderr.push(rule);
             }
+
+            if let Some(code) = config.parse_failure_status(ln) {
+                self.failure_status = code;
+            }
         });

         for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] {
@@ -488,6 +494,13 @@ impl Config {
         self.parse_name_directive(line, "pretty-compare-only")
     }

+    fn parse_failure_status(&self, line: &str) -> Option<i32> {
+        match self.parse_name_value_directive(line, "failure-status") {
+            Some(code) => code.trim().parse::<i32>().ok(),
+            _ => None,
+        }
+    }

Implementing the behavior change

When a test invokes a particular directive, it is expected that some behavior will change as a result. What behavior, obviously, will depend on the purpose of the directive. In the case of failure-status, the behavior that changes is that compiletest expects the failure code defined by the directive invoked in the test, rather than the default value.

Although specific to failure-status (as every directive will have a different implementation in order to invoke behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement failure-status, the check_correct_failure_status() function found in the TestCx implementation block, located in src/tools/compiletest/src/runtest.rs, was modified as per below:

@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
     }

     fn check_correct_failure_status(&self, proc_res: &ProcRes) {
-        // The value the Rust runtime returns on failure
-        const RUST_ERR: i32 = 101;
-        if proc_res.status.code() != Some(RUST_ERR) {
+        let expected_status = Some(self.props.failure_status);
+        let received_status = proc_res.status.code();
+
+        if expected_status != received_status {
             self.fatal_proc_rec(
-                &format!("failure produced the wrong error: {}", proc_res.status),
+                &format!("Error: expected failure status ({:?}) but received status {:?}.",
+                         expected_status,
+                         received_status),
                 proc_res,
             );
         }
@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> {
         );

         let proc_res = self.exec_compiled_test();
-
         if !proc_res.status.success() {
             self.fatal_proc_rec("test run failed!", &proc_res);
         }
@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> {
                 expected,
                 actual
             );
-            panic!();
         }
     }

Note the use of self.props.failure_status to access the directive property. In tests which do not specify the failure status directive, self.props.failure_status will evaluate to the default value of 101 at the time of this writing. But for a test which specifies a directive of, for example, //@ failure-status: 1, self.props.failure_status will evaluate to 1, as parse_failure_status() will have overridden the TestProps default value, for that test specifically.

1. please see the Auxiliary proc-macro section in the compiletest chapter for specifics. ↩

2. presently this has a weird quirk where the test binary's stdout and stderr gets concatenated and then error-patterns are matched on this combined output, which is ??? slightly questionable to say the least. ↩