Compiletest directives

• Listing of compiletest directives
  • Assembly
  • Auxiliary builds
  • Controlling outcome expectations
  • Controlling output snapshots and normalizations
  • Controlling when tests are run
  • Affecting how tests are built
  • Rustdoc
  • Pretty printing
    • Misc directives
  • Tool-specific directives
• Substitutions
• Adding a directive
  • Adding a new directive parser
  • Implementing the behavior change

FIXME(jieyouxu) completely revise this chapter.

Directives are special comments that tell compiletest how to build and interpret a test. They must appear before the Rust source in the test. They may also appear in rmake.rs or legacy Makefiles for run-make tests.

They are normally put after the short comment that explains the point of this test. Compiletest test suites use //@ to signal that a comment is a directive. For example, this test uses the //@ compile-flags command to specify a custom flag to give to rustc when the test is compiled:

// Test the behavior of `0 - 1` when overflow checks are disabled.

//@ compile-flags: -C overflow-checks=off

fn main() {
    let x = 0 - 1;
    ...
}

Directives can be standalone (like //@ run-pass) or take a value (like //@ compile-flags: -C overflow-checks=off).

Directives are written one directive per line: you cannot write multiple directives on the same line. For example, if you write //@ only-x86 only-windows then only-windows is interpreted as a comment, not a separate directive.

Listing of compiletest directives

The following is a list of compiletest directives. Directives are linked to sections that describe the command in more detail if available. This list may not be exhaustive. Directives can generally be found by browsing the TestProps structure found in header.rs from the compiletest source.

Assembly

Auxiliary builds

Controlling outcome expectations

See Controlling pass/fail expectations.

Controlling output snapshots and normalizations

See Normalization, Output comparison and Rustfix tests for more details.

Controlling when tests are run

These directives are used to ignore the test in some situations, which means the test won't be compiled or run.

• ignore-X where X is a target detail or stage will ignore the test accordingly (see below)
• only-X is like ignore-X, but will only run the test on that target or stage
• ignore-test always ignores the test. This can be used to temporarily disable a test if it is currently not working, but you want to keep it in tree to re-enable it later.

Some examples of X in ignore-X or only-X:

• A full target triple: aarch64-apple-ios
• Architecture: aarch64, arm, mips, wasm32, x86_64, x86, ...
• OS: android, emscripten, freebsd, ios, linux, macos, windows, ...
• Environment (fourth word of the target triple): gnu, msvc, musl
• WASM: wasm32-bare matches wasm32-unknown-unknown. emscripten also matches that target as well as the emscripten targets.
• Pointer width: 32bit, 64bit
• Endianness: endian-big
• Stage: stage0, stage1, stage2
• Channel: stable, beta
• When cross compiling: cross-compile
• When remote testing is used: remote
• When debug-assertions are enabled: debug
• When particular debuggers are being tested: cdb, gdb, lldb
• When particular debugger versions are matched: ignore-gdb-version
• Specific compare modes: compare-mode-polonius, compare-mode-chalk, compare-mode-split-dwarf, compare-mode-split-dwarf-single
• The two different test modes used by coverage tests: ignore-coverage-map, ignore-coverage-run

The following directives will check rustc build settings and target settings:

• needs-asm-support — ignores if it is running on a target that doesn't have stable support for asm!
• needs-profiler-runtime — ignores the test if the profiler runtime was not enabled for the target (build.profiler = true in rustc's config.toml)
• needs-sanitizer-support — ignores if the sanitizer support was not enabled for the target (sanitizers = true in rustc's config.toml)
• needs-sanitizer-{address,hwaddress,leak,memory,thread} — ignores if the corresponding sanitizer is not enabled for the target (AddressSanitizer, hardware-assisted AddressSanitizer, LeakSanitizer, MemorySanitizer or ThreadSanitizer respectively)
• needs-run-enabled — ignores if it is a test that gets executed, and running has been disabled. Running tests can be disabled with the x test --run=never flag, or running on fuchsia.
• needs-unwind — ignores if the target does not support unwinding
• needs-rust-lld — ignores if the rust lld support is not enabled (rust.lld = true in config.toml)
• needs-threads — ignores if the target does not have threading support
• needs-symlink — ignores if the target does not support symlinks. This can be the case on Windows if the developer did not enable privileged symlink permissions.

The following directives will check LLVM support:

• no-system-llvm — ignores if the system llvm is used
• min-llvm-version: 13.0 — ignored if the LLVM version is less than the given value
• min-system-llvm-version: 12.0 — ignored if using a system LLVM and its version is less than the given value
• ignore-llvm-version: 9.0 — ignores a specific LLVM version
• ignore-llvm-version: 7.0 - 9.9.9 — ignores LLVM versions in a range (inclusive)
• needs-llvm-components: powerpc — ignores if the specific LLVM component was not built. Note: The test will fail on CI (when COMPILETEST_REQUIRE_ALL_LLVM_COMPONENTS is set) if the component does not exist.
• needs-forced-clang-based-tests — test is ignored unless the environment variable RUSTBUILD_FORCE_CLANG_BASED_TESTS is set, which enables building clang alongside LLVM
  • This is only set in one CI job (x86_64-gnu-debug), which only runs a tiny subset of run-make tests. Other tests with this directive will not run at all, which is usually not what you want.

See also Debuginfo tests for directives for ignoring debuggers.

Affecting how tests are built

Rustdoc

FIXME(rustdoc): what does check-test-line-numbers-match do?

Asked in https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/What.20is.20the.20.60check-test-line-numbers-match.60.20directive.3F.

Pretty printing

See Pretty-printer.

Misc directives

• no-auto-check-cfg — disable auto check-cfg (only for --check-cfg tests)
• revisions — compile multiple times
• unused-revision-names - suppress tidy checks for mentioning unknown revision names -forbid-output — incremental cfail rejects output pattern
• should-ice — incremental cfail should ICE

Tool-specific directives

The following directives affect how certain command-line tools are invoked, in test suites that use those tools:

• filecheck-flags adds extra flags when running LLVM's FileCheck tool.
  • Used by codegen tests, assembly tests, and MIR-opt tests.
• llvm-cov-flags adds extra flags when running LLVM's llvm-cov tool.
  • Used by coverage tests in coverage-run mode.

Substitutions

Directive values support substituting a few variables which will be replaced with their corresponding value. For example, if you need to pass a compiler flag with a path to a specific file, something like the following could work:

//@ compile-flags: --remap-path-prefix={{src-base}}=/the/src

Where the sentinel {{src-base}} will be replaced with the appropriate path described below:

• {{cwd}}: The directory where compiletest is run from. This may not be the root of the checkout, so you should avoid using it where possible.
  • Examples: /path/to/rust, /path/to/build/root
• {{src-base}}: The directory where the test is defined. This is equivalent to $DIR for output normalization.
  • Example: /path/to/rust/tests/ui/error-codes
• {{build-base}}: The base directory where the test's output goes. This is equivalent to $TEST_BUILD_DIR for output normalization.
  • Example: /path/to/rust/build/x86_64-unknown-linux-gnu/test/ui
• {{rust-src-base}}: The sysroot directory where libstd/libcore/... are located
• {{sysroot-base}}: Path of the sysroot directory used to build the test.
  • Mainly intended for ui-fulldeps tests that run the compiler via API.
• {{target-linker}}: Linker that would be passed to -Clinker for this test, or blank if no linker override is active.
  • Mainly intended for ui-fulldeps tests that run the compiler via API.
• {{target}}: The target the test is compiling for
  • Example: x86_64-unknown-linux-gnu

See tests/ui/commandline-argfile.rs for an example of a test that uses this substitution.

Adding a directive

One would add a new directive if there is a need to define some test property or behavior on an individual, test-by-test basis. A directive property serves as the directive's backing store (holds the command's current value) at runtime.

To add a new directive property:

1. Look for the pub struct TestProps declaration in src/tools/compiletest/src/header.rs and add the new public property to the end of the declaration.
2. Look for the impl TestProps implementation block immediately following the struct declaration and initialize the new property to its default value.

Adding a new directive parser

When compiletest encounters a test file, it parses the file a line at a time by calling every parser defined in the Config struct's implementation block, also in src/tools/compiletest/src/header.rs (note that the Config struct's declaration block is found in src/tools/compiletest/src/common.rs). TestProps's load_from() method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (//@) directive such as //@ must-compile-successfully or //@ failure-status. Whitespace after the comment marker is optional.

Parsers will override a given directive property's default value merely by being specified in the test file as a directive or by having a parameter value specified in the test file, depending on the directive.

Parsers defined in impl Config are typically named parse_<directive-name> (note kebab-case <directive-command> transformed to snake-case <directive_command>). impl Config also defines several 'low-level' parsers which make it simple to parse common patterns like simple presence or not (parse_name_directive()), directive:parameter(s) (parse_name_value_directive()), optional parsing only if a particular cfg attribute is defined (has_cfg_prefix()) and many more. The low-level parsers are found near the end of the impl Config block; be sure to look through them and their associated parsers immediately above to see how they are used to avoid writing additional parsing code unnecessarily.

As a concrete example, here is the implementation for the parse_failure_status() parser, in src/tools/compiletest/src/header.rs:

@@ -232,6 +232,7 @@ pub struct TestProps {
     // customized normalization rules
     pub normalize_stdout: Vec<(String, String)>,
     pub normalize_stderr: Vec<(String, String)>,
+    pub failure_status: i32,
 }

 impl TestProps {
@@ -260,6 +261,7 @@ impl TestProps {
             run_pass: false,
             normalize_stdout: vec![],
             normalize_stderr: vec![],
+            failure_status: 101,
         }
     }

@@ -383,6 +385,10 @@ impl TestProps {
             if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") {
                 self.normalize_stderr.push(rule);
             }
+
+            if let Some(code) = config.parse_failure_status(ln) {
+                self.failure_status = code;
+            }
         });

         for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] {
@@ -488,6 +494,13 @@ impl Config {
         self.parse_name_directive(line, "pretty-compare-only")
     }

+    fn parse_failure_status(&self, line: &str) -> Option<i32> {
+        match self.parse_name_value_directive(line, "failure-status") {
+            Some(code) => code.trim().parse::<i32>().ok(),
+            _ => None,
+        }
+    }

Implementing the behavior change

When a test invokes a particular directive, it is expected that some behavior will change as a result. What behavior, obviously, will depend on the purpose of the directive. In the case of failure-status, the behavior that changes is that compiletest expects the failure code defined by the directive invoked in the test, rather than the default value.

Although specific to failure-status (as every directive will have a different implementation in order to invoke behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement failure-status, the check_correct_failure_status() function found in the TestCx implementation block, located in src/tools/compiletest/src/runtest.rs, was modified as per below:

@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
     }

     fn check_correct_failure_status(&self, proc_res: &ProcRes) {
-        // The value the Rust runtime returns on failure
-        const RUST_ERR: i32 = 101;
-        if proc_res.status.code() != Some(RUST_ERR) {
+        let expected_status = Some(self.props.failure_status);
+        let received_status = proc_res.status.code();
+
+        if expected_status != received_status {
             self.fatal_proc_rec(
-                &format!("failure produced the wrong error: {}", proc_res.status),
+                &format!("Error: expected failure status ({:?}) but received status {:?}.",
+                         expected_status,
+                         received_status),
                 proc_res,
             );
         }
@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> {
         );

         let proc_res = self.exec_compiled_test();
-
         if !proc_res.status.success() {
             self.fatal_proc_rec("test run failed!", &proc_res);
         }
@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> {
                 expected,
                 actual
             );
-            panic!();
         }
     }

Note the use of self.props.failure_status to access the directive property. In tests which do not specify the failure status directive, self.props.failure_status will evaluate to the default value of 101 at the time of this writing. But for a test which specifies a directive of, for example, //@ failure-status: 1, self.props.failure_status will evaluate to 1, as parse_failure_status() will have overridden the TestProps default value, for that test specifically.