Profile Guided Optimization

rustc supports doing profile-guided optimization (PGO). This chapter describes what PGO is and how the support for it is implemented in rustc.

What Is Profiled-Guided Optimization?

The basic concept of PGO is to collect data about the typical execution of a program (e.g. which branches it is likely to take) and then use this data to inform optimizations such as inlining, machine-code layout, register allocation, etc.

There are different ways of collecting data about a program's execution. One is to run the program inside a profiler (such as perf) and another is to create an instrumented binary, that is, a binary that has data collection built into it, and run that. The latter usually provides more accurate data.

How is PGO implemented in rustc?

rustc current PGO implementation relies entirely on LLVM. LLVM actually supports multiple forms of PGO:

  • Sampling-based PGO where an external profiling tool like perf is used to collect data about a program's execution.
  • GCOV-based profiling, where code coverage infrastructure is used to collect profiling information.
  • Front-end based instrumentation, where the compiler front-end (e.g. Clang) inserts instrumentation intrinsics into the LLVM IR it generates (but see the 1"Note").
  • IR-level instrumentation, where LLVM inserts the instrumentation intrinsics itself during optimization passes.

rustc supports only the last approach, IR-level instrumentation, mainly because it is almost exclusively implemented in LLVM and needs little maintenance on the Rust side. Fortunately, it is also the most modern approach, yielding the best results.

So, we are dealing with an instrumentation-based approach, i.e. profiling data is generated by a specially instrumented version of the program that's being optimized. Instrumentation-based PGO has two components: a compile-time component and run-time component, and one needs to understand the overall workflow to see how they interact.

1

Note: rustc now supports front-end-based coverage instrumentation, via the experimental option -C instrument-coverage, but using these coverage results for PGO has not been attempted at this time.

Overall Workflow

Generating a PGO-optimized program involves the following four steps:

  1. Compile the program with instrumentation enabled (e.g. rustc -C profile-generate main.rs)
  2. Run the instrumented program (e.g. ./main) which generates a default-<id>.profraw file
  3. Convert the .profraw file into a .profdata file using LLVM's llvm-profdata tool.
  4. Compile the program again, this time making use of the profiling data (e.g. rustc -C profile-use=merged.profdata main.rs)

Compile-Time Aspects

Depending on which step in the above workflow we are in, two different things can happen at compile time:

Create Binaries with Instrumentation

As mentioned above, the profiling instrumentation is added by LLVM. rustc instructs LLVM to do so by setting the appropriate flags when creating LLVM PassManagers:

	// `PMBR` is an `LLVMPassManagerBuilderRef`
    unwrap(PMBR)->EnablePGOInstrGen = true;
    // Instrumented binaries have a default output path for the `.profraw` file
    // hard-coded into them:
    unwrap(PMBR)->PGOInstrGen = PGOGenPath;

rustc also has to make sure that some of the symbols from LLVM's profiling runtime are not removed by marking the with the right export level.

Compile Binaries Where Optimizations Make Use Of Profiling Data

In the final step of the workflow described above, the program is compiled again, with the compiler using the gathered profiling data in order to drive optimization decisions. rustc again leaves most of the work to LLVM here, basically just telling the LLVM PassManagerBuilder where the profiling data can be found:

	unwrap(PMBR)->PGOInstrUse = PGOUsePath;

LLVM does the rest (e.g. setting branch weights, marking functions with cold or inlinehint, etc).

Runtime Aspects

Instrumentation-based approaches always also have a runtime component, i.e. once we have an instrumented program, that program needs to be run in order to generate profiling data, and collecting and persisting this profiling data needs some infrastructure in place.

In the case of LLVM, these runtime components are implemented in compiler-rt and statically linked into any instrumented binaries. The rustc version of this can be found in library/profiler_builtins which basically packs the C code from compiler-rt into a Rust crate.

In order for profiler_builtins to be built, profiler = true must be set in rustc's config.toml.

Testing PGO

Since the PGO workflow spans multiple compiler invocations most testing happens in run-make tests (the relevant tests have pgo in their name). There is also a codegen test that checks that some expected instrumentation artifacts show up in LLVM IR.

Additional Information

Clang's documentation contains a good overview on PGO in LLVM.