Reference Calibration

The Scalionix System Benchmark uses a deterministic reference calibration system to transform raw benchmark measurements into normalized performance scores.

Rather than comparing benchmark results against other benchmark executions, every workload is evaluated against predefined reference values.

This approach guarantees that benchmark scores remain deterministic, reproducible, and comparable across different hardware platforms, operating systems, benchmark executions, and future benchmark releases.

Reference calibration represents one of the fundamental design principles of the Scalionix System Benchmark.

Purpose

Reference calibration has several primary objectives.

• Normalize heterogeneous benchmark measurements.
• Produce stable benchmark scores.
• Eliminate dependence on historical benchmark data.
• Enable deterministic score generation.
• Preserve long-term compatibility.

Without deterministic references, benchmark scores would continuously change as newer hardware becomes available.

Instead, every workload is evaluated relative to a fixed calibration baseline.

Reference Philosophy

Every benchmark workload has exactly one reference.

References describe the expected performance of a predefined benchmark platform for a specific workload configuration.

Examples include:

            JSON serialization
                    │
                    ▼
            64 KiB dataset
                    │
                    ▼
              100 objects
                    │
                    ▼
            Reference throughput

            Storage random read
                    │
                    ▼
              4 KiB blocks
                    │
                    ▼
              1 GiB dataset
                    │
                    ▼
           Reference throughput

               Compilation
                    │
                    ▼
              Typst Release
                    │
                    ▼
         Reference build duration

Rather than comparing benchmark executions with each other, every benchmark execution compares itself against these deterministic reference values.

Reference Platform

All benchmark references are calibrated using a predefined reference workstation.

Current reference platform:

Processor

AMD Ryzen 9 7950X3D

Memory

64 GB DDR5-6400

Storage

Samsung 9100 Pro PCIe Gen5 NVMe SSD

Operating System

Linux

This hardware configuration was selected because it represents a modern high-performance desktop workstation while remaining broadly accessible.

Future benchmark releases may introduce additional calibration platforms through new scoring versions.

Why Approximately 10,000 Points?

Benchmark scores are intentionally normalized around approximately ten thousand points.

This decision was made for several reasons.

Human Readability

Values around ten thousand are easy to interpret.

Room for Future Hardware

Future processors may naturally exceed the reference score without requiring score redesign.

Regression Visibility

Small performance changes remain immediately visible.

Long-Term Stability

The score range remains meaningful across multiple hardware generations.

Ten thousand should therefore be interpreted as a calibration target rather than an absolute upper limit.

Benchmark scores are expected to exceed this value as hardware evolves.

Reference Dimensions

Every reference describes a unique workload.

Current reference dimensions include:

• Benchmark category
• Scenario
• Workload variant
• Dataset size

Additional dimensions may be introduced in future scoring versions whenever benchmark complexity increases.

Only workloads matching all required dimensions participate in score calculation.

Dataset Factors

Many benchmark scenarios execute multiple dataset sizes.

Rather than defining completely independent reference values for every dataset, references are calibrated using deterministic scaling factors.

Examples include:

            64 KiB
               │
               ▼
            Reference Factor
            
            256 KiB
               │
               ▼
            Reference Factor
            
            1 MiB
               │
               ▼
            Reference Factor

This approach significantly reduces reference duplication while preserving deterministic score calculation.

Workload Variant Factors

Some benchmark scenarios execute multiple workload variants.

Examples include:

• Compression algorithms
• Storage block sizes
• Storage durability modes
• Compilation fixtures

Rather than duplicating reference tables, workload-specific calibration factors are applied.

This simplifies future benchmark maintenance while preserving scoring consistency.

Worker Configuration Calibration

Many benchmark workloads execute using multiple worker configurations.

The benchmark evaluates every supported worker count.

Examples include:

• 1
• 2
• 4
• 8
• 16
• 32

Only the highest measured throughput contributes to the Multi-thread Score.

Worker calibration therefore evaluates scalability rather than absolute processor utilization.

Category Calibration

Every benchmark category maintains an independent reference table.

Examples include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Independent calibration allows categories to evolve separately without affecting the remainder of the scoring system.

Calibration Updates

Reference calibration intentionally evolves slowly.

References are updated only when required.

Typical reasons include:

• Major benchmark redesign
• New benchmark categories
• Revised benchmark methodology
• New calibration platform

Minor benchmark implementation improvements do not automatically require recalibration.

This preserves historical benchmark stability.

Scoring Version

Every reference table belongs to a specific scoring version.

The scoring version defines:

• Reference values
• Dataset factors
• Workload factors
• Category aggregation
• Normalization behavior

Historical benchmark reports always record the scoring version used during benchmark execution.

This guarantees reproducibility.

Calibration Independence

Reference calibration is independent of benchmark execution.

Benchmark execution measures performance.

Reference calibration interprets performance.

Separating these responsibilities provides several advantages.

• Benchmark implementation may evolve.
• Scoring may evolve independently.
• Historical benchmark reports remain valid.
• Score recalculation becomes possible.

This separation represents one of the core architectural decisions of the benchmark.

Practical Maintenance

When introducing a new benchmark category, the recommended workflow is:

            Implement Benchmark
                    │
                    ▼
            Execute Benchmark on Reference Platform
                    │
                    ▼
            Collect Benchmark Measurements
                    │
                    ▼
            Create Reference Entries
                    │
                    ▼
            Verify Diagnostics
                    │
                    ▼
            Publish New Scoring Version

Following this workflow guarantees that every benchmark category participates consistently in the scoring engine.

Long-Term Evolution

The reference system has been designed to support long-term benchmark evolution.

Future benchmark releases may introduce:

• Additional workload dimensions
• Additional benchmark categories
• Revised calibration platforms
• Alternative reference hardware

Existing benchmark reports remain compatible because scoring versions preserve historical reference tables.

Design Summary

The Reference Calibration system transforms benchmark measurements into deterministic, normalized scores using predefined workload references rather than historical benchmark comparisons.

By combining reference platforms, workload dimensions, dataset scaling, workload variants, scoring versioning, and independent category calibration, the benchmark achieves stable, reproducible, and future-proof score generation while preserving long-term compatibility across benchmark releases and hardware generations.