Benchmark Comparison Report
The Benchmark Comparison Report provides a deterministic comparison between two benchmark executions.
Rather than comparing raw benchmark measurements, the comparison engine evaluates normalized benchmark scores generated by the scoring system.
This approach allows benchmark results produced on different hardware platforms, at different points in time, and under different operating systems to be compared consistently.
The Benchmark Comparison Report is intended for hardware upgrades, regression analysis, benchmark validation, operating system comparisons, and long-term performance tracking.
Purpose
The comparison engine has several primary objectives.
- Compare two benchmark executions.
- Detect performance regressions.
- Quantify hardware improvements.
- Validate benchmark stability.
- Compare operating systems.
- Compare benchmark revisions.
The comparison engine is completely deterministic.
Given identical benchmark reports, the generated comparison will always produce identical results.
Input Reports
Benchmark comparison operates exclusively on Global Score Reports.
Two reports participate in every comparison.
Baseline Report
Represents the original benchmark execution.
Candidate Report
Represents the benchmark execution being evaluated.
Both reports must be generated by compatible scoring versions before comparison begins.
Comparison Pipeline
The comparison process follows a deterministic pipeline.
Baseline Report
│
▼
Compatibility Check
▲
│
Candidate Report
│
▼
Category Comparison
│
▼
Score Comparison
│
▼
Percentage Difference
│
▼
Comparison Report
Every comparison follows exactly the same sequence.
Compatibility Verification
Before scores are compared, the benchmark verifies report compatibility.
Verification includes:
- Scoring version
- Benchmark format
- Worker configuration
- Benchmark categories
- Report integrity
If incompatible reports are detected, comparison is terminated with a descriptive diagnostic message.
This prevents misleading benchmark comparisons.
Compute Score Comparison
The comparison engine evaluates Compute Scores from both reports.
For every Compute Score comparison the report records:
- Baseline score
- Candidate score
- Absolute difference
- Percentage difference
Positive percentage values indicate performance improvement.
Negative percentage values indicate reduced performance.
Storage Score Comparison
Storage Scores are compared independently from Compute Scores.
The comparison follows exactly the same methodology.
Recorded values include:
- Baseline storage score
- Candidate storage score
- Absolute difference
- Percentage difference
Because Storage Score represents an independent performance domain, its interpretation remains independent from Compute Score.
Category Comparison
Every benchmark category is compared individually.
Current categories include:
- JSON
- Hashing
- Encryption
- Compression
- Collections
- Concurrency
- Compilation
- Storage
Each category comparison records:
- Baseline score
- Candidate score
- Percentage difference
This allows benchmark users to identify precisely which workload domain improved or regressed.
Percentage Difference
Percentage differences are calculated relative to the Baseline Report.
Positive values indicate improvement.
Examples:
+1.24%
Negative values indicate regression.
Examples:
-3.61%
Zero indicates identical benchmark performance.
Percentage differences provide the most intuitive interpretation of benchmark evolution.
Practical Applications
The comparison engine supports numerous practical workflows.
Hardware Upgrade
Evaluate processor, memory, or storage upgrades.
Operating System Comparison
Compare Linux, macOS, and Windows benchmark behavior.
Compiler Evaluation
Measure performance changes resulting from different compiler versions.
Driver Validation
Evaluate storage or chipset driver updates.
Regression Testing
Detect unexpected benchmark regressions.
Continuous Integration
Automatically compare benchmark results across software revisions.
Because benchmark comparison operates entirely on normalized scores, comparisons remain meaningful regardless of the underlying benchmark implementation.
Example Comparison
A typical comparison summary appears similar to the following.
Compute Score
Baseline:
10234.81
Candidate:
10562.17
Difference:
+327.36
Improvement:
+3.20%
Category comparisons follow the same structure.
This presentation provides an immediate understanding of benchmark evolution.
Comparison Independence
The comparison engine does not require benchmark execution.
Only previously generated benchmark reports are needed.
This allows historical benchmark archives to be compared years after the original benchmark execution without repeating any workloads.
Consequently, benchmark comparison remains extremely fast regardless of the size or duration of the original benchmark.
Long-Term Compatibility
The comparison engine has been designed for long-term compatibility.
As benchmark categories evolve, comparison continues to operate on all categories supported by both benchmark reports.
New benchmark categories introduced in future scoring versions do not invalidate historical benchmark archives.
Compatibility is maintained through explicit scoring versioning and deterministic report structures.
Design Summary
The Benchmark Comparison Report provides a deterministic, reproducible, and version-aware comparison between two benchmark executions.
By comparing normalized benchmark scores rather than raw benchmark measurements, the comparison engine enables meaningful long-term performance tracking across hardware upgrades, operating system changes, compiler revisions, benchmark improvements, and future scoring revisions.
Together with the Full Benchmark Report, Scoring Report, and Global Score Report, the Benchmark Comparison Report completes the reporting subsystem of the Scalionix System Benchmark.