Benchmark Execution Model

Scalionix System Benchmark is built around a hierarchical execution model.

Rather than treating benchmark execution as a collection of unrelated tests, the framework organizes every workload into a strictly defined hierarchy.

Each level of the hierarchy has a single responsibility and represents one abstraction level of benchmark execution.

This design simplifies implementation, report generation, score calculation, and future extensibility.

The complete hierarchy is illustrated below.

Benchmark
    ├── Category
    │      ├── Scenario
    │      │      ├── Request
    │      │      │      ├── Iteration
    │      │      │      └── Result
    │      │      └── ...
    │      └── ...
    └── ...

Each component is described below.

Benchmark

A benchmark represents one complete execution of the Scalionix System Benchmark.

Every benchmark execution produces:

• One benchmark report
• One hardware snapshot
• One scoring report
• One benchmark identifier
• One execution summary

The benchmark is the highest execution unit within the framework.

A benchmark may execute one category or multiple categories during the same run.

Examples include:

Complete Benchmark

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Single Category Execution

• Storage only

Selected Categories

• Compilation
• Storage
• Compression

Regardless of which categories are executed, the framework always treats the execution as one benchmark.

Benchmark Category

A benchmark category measures one major subsystem of the machine.

Categories are intentionally independent.

Each category has:

• Execution generator
• Benchmark tasks
• Scoring adapter
• Reference definitions
• Diagnostics

Current benchmark categories are:

JSON

Measures structured data processing.

Hashing

Measures cryptographic hashing throughput.

Encryption

Measures authenticated encryption throughput.

Compression

Measures lossless compression performance.

Collections

Measures container manipulation and memory behavior.

Concurrency

Measures synchronization and parallel execution efficiency.

Compilation

Measures software build throughput.

Storage

Measures filesystem and storage subsystem behavior.

Categories do not share workload implementations.

Each category produces exactly one category score.

Benchmark Scenario

A scenario represents one logical benchmark operation.

Scenarios are what users usually recognize as benchmark names.

Examples include:

• json.serialize.memory
• json.deserialize.file
• hash.sha256.chunked
• hash.blake3.one_shot
• compression.zstd.compress
• storage.random.read
• storage.sync.write_fsync
• compilation.rust.typst.release
• compilation.cpp.scalionix.gcc.release

A scenario defines:

• Benchmark algorithm
• Execution logic
• Measured metric
• Verification method
• Scoring references

Scenarios never execute directly.

Instead, scenarios generate benchmark requests.

Benchmark Request

A request represents one executable benchmark configuration.

Every request belongs to exactly one scenario.

A request combines:

• Scenario
• Dataset
• Worker configuration
• Algorithm parameters
• Execution configuration

For example:

Scenario:
storage.random.read

Dataset:
1 GiB

Block size:
4096 bytes

Workers:
16

Iterations:
3

This combination produces one benchmark request.

Each request executes independently from every other request.

Benchmark Workload

Within the scoring system, requests that represent the same logical workload are grouped together into workloads.

A workload combines benchmark results that differ only by worker configuration.

Example:

storage.random.read

Dataset:
1 GiB

Block:
4096 bytes

Workers:
[ 1 / 2 / 4 / 8 / 16 ]
[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

These requests belong to one workload.

The workload allows the scoring engine to evaluate:

• Single-thread performance
• Best parallel performance
• Scalability

instead of treating every worker configuration as an independent benchmark.

This design significantly reduces score fragmentation.

Worker Configuration

Most benchmark categories evaluate scalability by executing the same workload using multiple worker counts.

Worker configurations may include values such as:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The exact matrix depends on the detected hardware.

Worker configurations allow the benchmark to measure:

• Single-thread efficiency
• Parallel throughput
• Scalability
• Synchronization overhead
• Scheduling efficiency

Compilation currently represents the only exception.

Because software compilation is naturally executed using parallel build systems, the benchmark measures only the representative multi-thread configuration based on the number of physical processor cores.

Benchmark Iteration

Every request executes multiple iterations.

Iterations serve two purposes.

First, they reduce the influence of temporary operating system activity.

Second, they provide statistical confidence that the measured workload is stable.

Iterations are divided into two groups.

Warm-up Iterations

Warm-up iterations prepare the execution environment.

They are excluded from benchmark reports.

Measured Iterations

Measured iterations contribute to benchmark statistics.

Each iteration records:

• Execution duration
• Throughput
• Verification status
• Execution diagnostics

The benchmark currently selects the best verified iteration as the representative result.

Verification

Verification ensures benchmark correctness.

Every category implements workload-specific verification logic.

Examples include:

Serialization

Generated output must deserialize successfully.

Encryption

Decrypted output must match original input.

Compression

Decompressed output must equal original data.

Compilation

Build must complete successfully.

Storage

Filesystem operations must complete successfully.

Verification failures invalidate benchmark results regardless of execution speed.

This guarantees that benchmark scores represent successful computation rather than incomplete execution.

Benchmark Metrics

Different benchmark categories naturally measure different performance metrics.

Examples include:

Bytes per second

Used for sequential throughput benchmarks.

Operations per second

Used for metadata and synchronization workloads.

Files per second

Used for filesystem benchmarks.

Objects per second

Used for collection workloads.

Builds per second

Used for compilation.

Although categories measure different native metrics, every benchmark adapter converts these values into a unified scoring model.

This abstraction allows completely different benchmark domains to participate in the same scoring engine.

Benchmark Result

Every executed request produces one benchmark result.

A benchmark result contains:

• Execution metadata
• Benchmark configuration
• Measured metrics
• Verification status
• Representative iteration
• Hardware-independent diagnostics

Results remain immutable after benchmark execution.

No scoring information is stored within benchmark results.

This separation allows scoring algorithms to evolve independently from benchmark execution.

Benchmark Report

After every request completes successfully, the benchmark produces a complete execution report.

The report contains every measured request regardless of its contribution to the final score.

This includes:

• Successful requests
• Failed requests
• Verification failures
• Skipped requests
• Execution timings
• Hardware snapshot
• Configuration metadata

Benchmark reports intentionally preserve significantly more information than is required for score calculation.

This allows future scoring versions to reuse existing benchmark reports without repeating benchmark execution.

Execution Philosophy

The execution model separates measurement from evaluation.

Benchmark execution answers one question:

“What actually happened during execution?”

The scoring engine answers a different question:

“How good was the measured performance compared to the calibrated reference platform?”

Keeping these responsibilities separate provides several important advantages.

• Benchmark implementations remain deterministic.
• Scoring algorithms may evolve independently.
• Historical benchmark reports remain valid.
• New scoring models can be introduced without modifying benchmark execution.

This separation between execution and evaluation forms one of the core architectural principles of Scalionix System Benchmark.