Compilation Benchmark

The Compilation benchmark evaluates the ability of a computer system to build real-world software projects using production toolchains.

Unlike synthetic compiler benchmarks that repeatedly compile small source files or isolated functions, the Scalionix Compilation benchmark measures complete software builds consisting of thousands of source files, multiple compilation units, dependency resolution, optimization, linking, filesystem interaction, and executable generation.

Compilation is one of the most demanding workloads encountered by software developers and continuous integration systems. Modern compilers exercise nearly every subsystem of a computer including processor execution units, memory hierarchy, filesystem performance, parallel scheduling, and operating system services.

The benchmark therefore represents one of the closest approximations to everyday software engineering workloads.

Objectives

The Compilation benchmark has five primary objectives.

• Measure complete project build performance.
• Evaluate compiler scalability.
• Measure build system efficiency.
• Measure filesystem interaction during compilation.
• Measure practical software development performance.

Unlike traditional CPU benchmarks, the objective is not to measure isolated compiler execution but the performance of the complete software build pipeline.

Why Compilation?

Compilation is one of the most common high-performance workloads executed by development machines.

Typical environments include:

• Software development
• Continuous Integration
• Continuous Delivery
• Build servers
• Release engineering
• Package generation
• Compiler development
• Large-scale source repositories

Modern compilation stresses multiple hardware subsystems simultaneously.

            Processor execution
                    │
                    ▼
                 Parser
                    │
                    ▼
                Optimizer
                    │
                    ▼
            Code generation
                    │
                    ▼
                Filesystem
                    │
                    ▼
                  Linker
                    │
                    ▼
          Executable generation

Because nearly every subsystem participates during compilation, build performance is an excellent indicator of overall workstation capability.

Benchmark Philosophy

The benchmark intentionally measures complete production projects rather than synthetic compiler stress tests.

Every benchmark fixture represents an actual software project that can be built using the standard build process recommended by its maintainers.

Projects are compiled exactly as a developer would build them on a production workstation.

No benchmark-specific modifications are introduced.

Each benchmark request performs a completely clean build.

Compiler caches are disabled.

Previous build artifacts are removed.

The benchmark therefore measures real compilation work instead of incremental rebuild performance.

Typical Compilation Pipeline

         Extract Fixture
                │
                ▼
      Prepare Build Directory
                │
                ▼
      Remove Previous Objects
                │
                ▼
        Invoke Build System
                │
                ▼
     Compile Source Files
                │
                ▼
         Link Executable
                │
                ▼
      Verify Build Success
                │
                ▼
      Measure Build Duration

Benchmark Fixtures

The benchmark uses deterministic software projects called fixtures.

Every fixture is embedded inside the benchmark executable.

No network access is required during benchmark execution.

This guarantees:

• Deterministic execution
• Reproducible results
• Offline compatibility
• Identical benchmark input on every machine

Each fixture is distributed as a compressed archive and extracted into an isolated working directory before compilation begins.

Current Fixtures

The current benchmark distribution includes multiple software projects representing different compilation characteristics.

Rust — Typst

Typst is a modern document processing system written entirely in Rust.

The project consists of a large number of crates, extensive generic programming, procedural macros, and heavy optimization.

Compilation characteristics:

• Very large dependency graph
• Extensive LLVM optimization
• Significant memory usage
• Long compilation time

Scenario

compilation.rust.typst.release

C++ — Scalionix Stress Project

The C++ fixture is a deterministic stress project specifically designed to generate substantial compilation workload.

The project contains hundreds of independent translation units that can be compiled in parallel before final linking.

Compilation characteristics:

• Many independent source files
• Intensive parallel compilation
• Heavy optimizer usage
• Significant linker workload

Scenario

compilation.cpp.scalionix.gcc.release

Additional fixtures may be introduced in future benchmark versions without affecting the scoring model.

Platform Support

Compilation workloads execute natively using the operating system toolchains.

Current supported environments include:

Linux

• Rust
• GCC

macOS

• Rust
• Apple Clang

Windows compilation workloads are currently not executed.

This decision avoids introducing unnecessary complexity caused by multiple compiler environments while maintaining deterministic benchmark behavior across supported platforms.

Build Environment

The benchmark automatically verifies that the required compilation environment is available before execution.

Required components include:

• Rust Toolchain
• Cargo
• C++ Compiler
• Standard C++ Library
• Platform Build Tools

The benchmark provides a dedicated setup command capable of validating and preparing the compilation environment.

This ensures that benchmark execution remains reproducible across supported operating systems.

Build Execution

Each benchmark request performs multiple measured compilation iterations.

The workflow is intentionally deterministic.

For every iteration the benchmark performs:

            Extract fixture
                    │
                    ▼
            Clean previous build
                    │
                    ▼
            Compile project
                    │
                    ▼
            Verify successful build
                    │
                    ▼
            Record build duration

Only successful compilations participate in benchmark scoring.

Compilation failures invalidate the corresponding benchmark request.

Best Iteration Selection

Multiple measured iterations are executed for every fixture.

Rather than averaging execution times, the benchmark selects the fastest successful build.

This approach minimizes the influence of temporary operating system activity, background processes, filesystem maintenance, and scheduler interruptions.

The selected iteration therefore represents the maximum build capability of the evaluated system under stable operating conditions.

Worker Configuration

Unlike other benchmark categories, Compilation evaluates only the native build parallelism supported by the compiler.

The benchmark does not execute multiple benchmark workers simultaneously.

Instead, the compiler itself fully utilizes the available processor resources.

This approach reflects practical software development where a single build process already schedules parallel compilation internally.

Consequently, Compilation is treated as a Multi-Thread Only benchmark category.

Verification

Every compilation workload performs strict validation.

Verification includes:

• Successful extraction
• Successful compilation
• Successful linking
• Executable generation
• Zero compiler errors

Only successful builds contribute to benchmark scoring.

Measured Metrics

Primary benchmark metrics include:

• Total build duration
• Builds per second
• Executable verification
• Build profile
• Compiler
• Language
• Iteration timing

Unlike throughput-oriented benchmarks, Compilation converts build duration into Builds Per Second before score normalization.

This allows Compilation to integrate naturally into the common scoring model used by every benchmark category.

Score Calculation

Compilation participates in the Compute Score.

Unlike the remaining benchmark categories, Compilation does not calculate a Single-thread Score.

The compiler already executes using its native parallel scheduling model.

Consequently:

Single-thread Score

Not applicable.

Displayed as zero.

Multi-thread Score

Calculated from the measured build throughput.

Scaling Score

Fixed at the baseline reference value because parallel scheduling is entirely controlled by the compiler itself.

The final Compilation score therefore equals the normalized Multi-thread Score.

Hardware Characteristics Measured

Compilation simultaneously exercises nearly every major hardware subsystem.

Processor Architecture

Parser execution, optimization, code generation, and linking.

Memory Capacity

Large software projects require substantial memory during optimization.

Memory Bandwidth

Compiler frontends continuously stream source code and intermediate representations.

Filesystem Performance

Compilation creates and reads thousands of temporary files.

Process Scheduling

Modern compilers execute numerous concurrent compilation jobs.

Linker Performance

Final executable generation stresses both CPU and storage subsystems.

For this reason, Compilation is widely regarded as one of the best indicators of practical workstation performance.

Practical Interpretation

High Compilation benchmark scores generally indicate:

• Excellent software development performance
• Efficient processor architecture
• Strong filesystem performance
• Fast memory subsystem
• Efficient compiler scheduling

Systems achieving high Compilation scores are typically well suited for software engineering, continuous integration, large-scale builds, and compiler-intensive workloads.

Design Summary

The Compilation benchmark evaluates complete production software builds using deterministic embedded projects executed entirely offline.

By compiling real-world applications with native toolchains, the benchmark provides a realistic evaluation of workstation performance under one of the most demanding software engineering workloads.

Unlike synthetic compiler benchmarks, the resulting score reflects the practical ability of a system to build complex software projects efficiently and reproducibly.