Introduction

Scalionix System Benchmark is a cross-platform native benchmarking framework designed to measure real-world system performance across modern desktop and workstation hardware.

Unlike traditional synthetic benchmarks that focus on isolated processor instructions or theoretical hardware limits, Scalionix evaluates complete software workloads representative of modern development and production environments. Every benchmark category executes actual application logic, filesystem operations, data processing pipelines, or software compilation tasks that closely resemble workloads encountered during everyday software engineering.

The benchmark has been designed from the beginning with three primary objectives:

Cross-platform consistency

Every benchmark category is implemented using identical algorithms and execution models across Linux, macOS, and Windows whenever the underlying operating system allows equivalent behavior. This enables meaningful comparison between operating systems while minimizing platform-specific bias.

Deterministic execution

Benchmark inputs are generated using deterministic datasets, predefined execution matrices, and repeatable workload configurations. Every execution of the same benchmark produces equivalent computational work regardless of machine architecture.

Long-term comparability

The scoring model is reference-based rather than machine-relative. Scores are calculated against calibrated reference workloads instead of comparing only against the current execution. This allows benchmark results produced months or years apart to remain directly comparable.

The framework is intended primarily for evaluating developer workstations, engineering laptops, high-performance desktop systems, and professional workstations. Although the benchmark can execute on lower-end hardware, the workload selection and scoring methodology are optimized for modern multi-core processors, NVMe storage devices, and high-bandwidth memory subsystems.

Scalionix System Benchmark does not attempt to measure gaming performance, graphics rendering performance, GPU compute throughput, or network bandwidth. Instead, the benchmark focuses exclusively on CPU execution, memory hierarchy efficiency, storage subsystem behavior, software compilation throughput, and related operating system services that directly affect software development and backend infrastructure workloads.

A complete benchmark execution consists of multiple independent categories. Each category measures a different subsystem of the machine and produces its own normalized score. Categories are intentionally isolated from one another so that improvements in one hardware subsystem do not artificially inflate the measured performance of another.

The benchmark currently evaluates the following major workload domains:

• JSON serialization and deserialization

  Measures structured data processing performance using realistic object graphs and varying dataset sizes.

• Cryptographic hashing

  Measures throughput of modern cryptographic hash algorithms using sequential memory workloads.

• Encryption

  Measures authenticated encryption and decryption throughput using production-grade cryptographic algorithms.

• Compression

  Measures lossless data compression and decompression performance across multiple data types and compression ratios.

• Collections

  Measures common in-memory container operations, allocation behavior, lookup performance, insertion throughput, iteration efficiency, and removal workloads.

• Concurrency

  Measures synchronization primitives, parallel execution scalability, work distribution efficiency, and thread coordination performance.

• Compilation

  Measures full production software builds using representative Rust and C++ projects executed with clean build environments and disabled compiler caches.

• Storage

  Measures sequential I/O, random I/O, metadata operations, directory traversal, synchronization durability, and small-file workloads using real filesystem operations.

These categories collectively represent the types of workloads commonly executed by software engineers, backend infrastructure, continuous integration systems, build servers, and development workstations.

The benchmark intentionally avoids synthetic microbenchmarks whenever equivalent production workloads can be executed instead. This design philosophy ensures that benchmark scores reflect practical system performance rather than isolated hardware characteristics.

The framework has also been designed with future extensibility in mind. New benchmark categories can be introduced without modifying the scoring model or affecting existing benchmark results. Every category remains versioned independently through the scoring reference system, allowing the benchmark to evolve while preserving historical comparability.

Throughout this documentation, the terms benchmark, category, scenario, workload, request, iteration, and score have precise technical meanings. These concepts form the foundation of the execution pipeline and are explained in the following chapters.

Design Goals

The Scalionix System Benchmark was designed around a set of architectural principles intended to provide consistent, repeatable, and meaningful performance measurements across modern computer systems.

Rather than maximizing benchmark score variability or producing synthetic peak numbers, the framework focuses on measuring workloads that resemble those executed by real software development environments, backend infrastructure, and production systems.

Every design decision within the benchmark follows one or more of the principles described below.

Real-World Workloads

The benchmark intentionally avoids artificial instruction loops whose only purpose is to maximize processor utilization.

Instead, every category performs complete application workloads that involve realistic algorithms, data structures, memory access patterns, filesystem interaction, synchronization primitives, or software compilation.

Examples include:

• JSON serialization and deserialization
• Cryptographic hashing
• Authenticated encryption
• Lossless compression
• Container manipulation
• Thread synchronization
• Storage subsystem operations
• Complete software compilation

This approach produces benchmark results that more closely resemble practical machine performance during everyday engineering work.

Cross-Platform Consistency

One of the primary objectives of the framework is to produce comparable benchmark results across different operating systems.

Whenever possible, identical algorithms and workload definitions are executed on:

• Linux
• macOS
• Windows

Although operating systems inevitably differ in scheduler behavior, filesystem implementation, memory management, and system APIs, benchmark scenarios attempt to minimize these differences by performing equivalent computational work.

Platform-specific implementations are introduced only when operating system interfaces fundamentally differ.

Examples include:

• Filesystem synchronization APIs
• Storage device discovery
• Hardware information collection
• Compiler toolchain detection

Whenever platform-specific implementations are necessary, they are designed to produce equivalent benchmark semantics rather than identical implementation details.

Deterministic Execution

Benchmark repeatability is considered significantly more important than achieving the highest possible throughput.

Every workload executes deterministic input generated from predefined benchmark datasets.

The benchmark avoids using:

• Random execution order
• Adaptive optimization
• Dynamically changing datasets
• Workload mutation between iterations

Whenever random values are required, deterministic seeds are used to guarantee reproducible benchmark inputs.

This ensures that identical benchmark configurations perform identical computational work across different executions.

Reference-Based Scoring

Scalionix does not compare benchmark executions against other machines available at runtime.

Instead, every workload is evaluated against a calibrated reference throughput.

Each benchmark scenario has predefined reference values representing expected throughput for a calibrated reference platform.

Individual workload scores are calculated by comparing measured throughput against these reference values.

This design provides several advantages:

• Historical benchmark reports remain comparable
• Benchmark scores remain stable over time
• Results are independent of currently available machines
• New benchmark executions can be compared with reports generated years earlier

The reference calibration process is described in a dedicated chapter later in this document.

Independent Benchmark Categories

Each benchmark category measures one subsystem of the machine independently.

Examples include:

• Serialization
• Hashing
• Encryption
• Storage
• Compilation

The execution of one category does not affect the scoring logic of another category.

This separation allows individual benchmark categories to evolve independently without requiring changes to unrelated benchmark logic.

It also allows users to execute only selected benchmark categories when evaluating a particular subsystem.

Representative Scaling

Modern software rarely executes on a single processor core.

For this reason, benchmark workloads evaluate both computational performance and scalability across multiple worker configurations.

Most compute categories execute using a predefined worker matrix ranging from a single worker up to the maximum supported configuration.

This allows the benchmark to measure:

• Single-thread efficiency
• Parallel throughput
• Scalability
• Scheduler efficiency
• Synchronization overhead

The Compilation benchmark intentionally differs from this model.

Because production software compilation is almost exclusively performed using parallel build systems, compilation workloads execute only the representative multi-thread configuration using the machine’s physical processor core count.

This design reflects practical software engineering workflows rather than synthetic processor measurements.

Hardware Independence

The benchmark does not assume specific processor vendors, storage devices, or memory configurations.

Supported hardware includes:

• x86-64 processors
• ARM64 processors
• Desktop systems
• Laptops
• Workstations

Hardware characteristics are detected dynamically during benchmark initialization and recorded within the benchmark report.

Scoring remains independent of hardware manufacturer.

The benchmark evaluates measured performance rather than hardware specifications.

Extensibility

The benchmark architecture has been designed to allow new benchmark categories to be introduced without affecting existing implementations.

Each category defines:

• Execution model
• Request generator
• Benchmark tasks
• Scoring adapter
• Reference calibration
• Report serialization

As a result, new benchmark domains can be integrated with minimal impact on the remaining system.

Future benchmark categories may include additional serialization formats, networking workloads, message processing, or other production-oriented performance measurements while preserving compatibility with existing benchmark reports.

Long-Term Stability

Benchmark software is expected to evolve over time.

To preserve historical comparability, the framework introduces explicit versioning of benchmark scoring.

Every generated benchmark report contains:

• Benchmark version
• Scoring version
• Benchmark configuration
• Workload definitions
• Hardware snapshot
• Execution metadata

Changes to scoring algorithms or reference values are introduced through versioned scoring models rather than modifying historical benchmark reports.

This approach ensures that benchmark results generated using previous scoring models remain valid and reproducible.

Engineering-Oriented Philosophy

Scalionix System Benchmark was created primarily for software engineers.

The benchmark is intended to answer practical questions such as:

• Which workstation compiles software faster?
• Which processor scales better under parallel workloads?
• Which storage subsystem handles development workloads more efficiently?
• How does one machine compare against another using identical software tasks?

Rather than attempting to maximize synthetic benchmark numbers, the framework emphasizes meaningful engineering metrics that directly influence software development productivity.

Every benchmark category exists because it represents a workload commonly encountered in modern software engineering environments.

The benchmark therefore measures not only raw computational performance but also the practical efficiency of complete systems executing realistic software workloads.

Benchmark Execution Model

Scalionix System Benchmark is built around a hierarchical execution model.

Rather than treating benchmark execution as a collection of unrelated tests, the framework organizes every workload into a strictly defined hierarchy.

Each level of the hierarchy has a single responsibility and represents one abstraction level of benchmark execution.

This design simplifies implementation, report generation, score calculation, and future extensibility.

The complete hierarchy is illustrated below.

Benchmark
    ├── Category
    │      ├── Scenario
    │      │      ├── Request
    │      │      │      ├── Iteration
    │      │      │      └── Result
    │      │      └── ...
    │      └── ...
    └── ...

Each component is described below.

Benchmark

A benchmark represents one complete execution of the Scalionix System Benchmark.

Every benchmark execution produces:

• One benchmark report
• One hardware snapshot
• One scoring report
• One benchmark identifier
• One execution summary

The benchmark is the highest execution unit within the framework.

A benchmark may execute one category or multiple categories during the same run.

Examples include:

Complete Benchmark

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Single Category Execution

• Storage only

Selected Categories

• Compilation
• Storage
• Compression

Regardless of which categories are executed, the framework always treats the execution as one benchmark.

Benchmark Category

A benchmark category measures one major subsystem of the machine.

Categories are intentionally independent.

Each category has:

• Execution generator
• Benchmark tasks
• Scoring adapter
• Reference definitions
• Diagnostics

Current benchmark categories are:

JSON

Measures structured data processing.

Hashing

Measures cryptographic hashing throughput.

Encryption

Measures authenticated encryption throughput.

Compression

Measures lossless compression performance.

Collections

Measures container manipulation and memory behavior.

Concurrency

Measures synchronization and parallel execution efficiency.

Compilation

Measures software build throughput.

Storage

Measures filesystem and storage subsystem behavior.

Categories do not share workload implementations.

Each category produces exactly one category score.

Benchmark Scenario

A scenario represents one logical benchmark operation.

Scenarios are what users usually recognize as benchmark names.

Examples include:

• json.serialize.memory
• json.deserialize.file
• hash.sha256.chunked
• hash.blake3.one_shot
• compression.zstd.compress
• storage.random.read
• storage.sync.write_fsync
• compilation.rust.typst.release
• compilation.cpp.scalionix.gcc.release

A scenario defines:

• Benchmark algorithm
• Execution logic
• Measured metric
• Verification method
• Scoring references

Scenarios never execute directly.

Instead, scenarios generate benchmark requests.

Benchmark Request

A request represents one executable benchmark configuration.

Every request belongs to exactly one scenario.

A request combines:

• Scenario
• Dataset
• Worker configuration
• Algorithm parameters
• Execution configuration

For example:

Scenario:
storage.random.read

Dataset:
1 GiB

Block size:
4096 bytes

Workers:
16

Iterations:
3

This combination produces one benchmark request.

Each request executes independently from every other request.

Benchmark Workload

Within the scoring system, requests that represent the same logical workload are grouped together into workloads.

A workload combines benchmark results that differ only by worker configuration.

Example:

storage.random.read

Dataset:
1 GiB

Block:
4096 bytes

Workers:
[ 1 / 2 / 4 / 8 / 16 ]
[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

These requests belong to one workload.

The workload allows the scoring engine to evaluate:

• Single-thread performance
• Best parallel performance
• Scalability

instead of treating every worker configuration as an independent benchmark.

This design significantly reduces score fragmentation.

Worker Configuration

Most benchmark categories evaluate scalability by executing the same workload using multiple worker counts.

Worker configurations may include values such as:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The exact matrix depends on the detected hardware.

Worker configurations allow the benchmark to measure:

• Single-thread efficiency
• Parallel throughput
• Scalability
• Synchronization overhead
• Scheduling efficiency

Compilation currently represents the only exception.

Because software compilation is naturally executed using parallel build systems, the benchmark measures only the representative multi-thread configuration based on the number of physical processor cores.

Benchmark Iteration

Every request executes multiple iterations.

Iterations serve two purposes.

First, they reduce the influence of temporary operating system activity.

Second, they provide statistical confidence that the measured workload is stable.

Iterations are divided into two groups.

Warm-up Iterations

Warm-up iterations prepare the execution environment.

They are excluded from benchmark reports.

Measured Iterations

Measured iterations contribute to benchmark statistics.

Each iteration records:

• Execution duration
• Throughput
• Verification status
• Execution diagnostics

The benchmark currently selects the best verified iteration as the representative result.

Verification

Verification ensures benchmark correctness.

Every category implements workload-specific verification logic.

Examples include:

Serialization

Generated output must deserialize successfully.

Encryption

Decrypted output must match original input.

Compression

Decompressed output must equal original data.

Compilation

Build must complete successfully.

Storage

Filesystem operations must complete successfully.

Verification failures invalidate benchmark results regardless of execution speed.

This guarantees that benchmark scores represent successful computation rather than incomplete execution.

Benchmark Metrics

Different benchmark categories naturally measure different performance metrics.

Examples include:

Bytes per second

Used for sequential throughput benchmarks.

Operations per second

Used for metadata and synchronization workloads.

Files per second

Used for filesystem benchmarks.

Objects per second

Used for collection workloads.

Builds per second

Used for compilation.

Although categories measure different native metrics, every benchmark adapter converts these values into a unified scoring model.

This abstraction allows completely different benchmark domains to participate in the same scoring engine.

Benchmark Result

Every executed request produces one benchmark result.

A benchmark result contains:

• Execution metadata
• Benchmark configuration
• Measured metrics
• Verification status
• Representative iteration
• Hardware-independent diagnostics

Results remain immutable after benchmark execution.

No scoring information is stored within benchmark results.

This separation allows scoring algorithms to evolve independently from benchmark execution.

Benchmark Report

After every request completes successfully, the benchmark produces a complete execution report.

The report contains every measured request regardless of its contribution to the final score.

This includes:

• Successful requests
• Failed requests
• Verification failures
• Skipped requests
• Execution timings
• Hardware snapshot
• Configuration metadata

Benchmark reports intentionally preserve significantly more information than is required for score calculation.

This allows future scoring versions to reuse existing benchmark reports without repeating benchmark execution.

Execution Philosophy

The execution model separates measurement from evaluation.

Benchmark execution answers one question:

“What actually happened during execution?”

The scoring engine answers a different question:

“How good was the measured performance compared to the calibrated reference platform?”

Keeping these responsibilities separate provides several important advantages.

• Benchmark implementations remain deterministic.
• Scoring algorithms may evolve independently.
• Historical benchmark reports remain valid.
• New scoring models can be introduced without modifying benchmark execution.

This separation between execution and evaluation forms one of the core architectural principles of Scalionix System Benchmark.

Hardware Detection and System Snapshot

Every benchmark execution begins by collecting a complete hardware snapshot of the system.

The hardware snapshot represents the execution environment in which benchmark results were produced. It serves as a permanent description of the tested machine and allows benchmark reports generated on different systems or at different points in time to be compared with confidence.

Hardware detection is performed before the first benchmark request is executed and remains completely independent from benchmark scoring.

The collected hardware information never modifies benchmark scores.

Instead, it provides the contextual information required to correctly interpret benchmark results.

Design Objectives

The hardware snapshot subsystem was designed with several objectives.

• Produce reproducible benchmark reports.
• Identify the exact hardware configuration used during execution.
• Detect significant hardware changes between benchmark runs.
• Allow comparison between machines without relying on user-provided information.
• Provide platform-independent hardware descriptions.

Every benchmark report therefore contains both performance measurements and the environment in which those measurements were produced.

Snapshot Collection

Hardware information is collected once during benchmark initialization.

The snapshot remains immutable throughout benchmark execution.

Collecting hardware information only once provides several advantages.

• No benchmark measurements are affected by repeated hardware queries.
• Benchmark execution remains deterministic.
• Operating system overhead is minimized.
• Every benchmark category shares the same hardware information.

The hardware snapshot is embedded into every generated benchmark report.

Processor Information

The CPU section describes the processor responsible for executing benchmark workloads.

Collected information includes:

• Processor model
• Architecture
• Vendor identification
• Physical processor core count
• Logical processor count
• Processor frequency (when available)
• Operating system identifier

Whenever possible, information is obtained directly from the operating system rather than inferred from benchmark execution.

The benchmark distinguishes between physical processor cores and logical processors because these values directly influence benchmark execution matrices.

Physical cores determine the representative worker configuration for the Compilation benchmark.

Logical processors define the upper bound of worker configurations for scalable benchmark categories.

Memory Information

The memory snapshot records both operating system memory statistics and installed hardware characteristics.

Operating system information includes:

• Total installed memory
• Currently available memory
• Memory usage during benchmark initialization

Whenever supported by the operating system, additional hardware information is collected from firmware interfaces.

Examples include:

• Memory technology
• DDR generation
• Configured frequency
• Rated frequency
• Module manufacturer
• Module part number
• Module capacity
• Configured voltage
• ECC capability
• Number of installed modules

This information provides significantly more detail than operating system memory statistics alone.

Storage Information

Storage detection identifies the physical device used for benchmark execution.

The benchmark intentionally records the storage device containing the benchmark working directory rather than enumerating every storage device present in the system.

Collected information may include:

• Physical storage model
• Firmware revision
• Device type
• Storage technology
• Filesystem
• Device identifier
• Mount point
• Benchmark working directory

Examples include:

• NVMe SSD
• PCIe Gen4 SSD
• PCIe Gen5 SSD
• SATA SSD
• SATA HDD
• External storage

Platform-specific implementations obtain this information using native operating system facilities.

Operating System Information

The benchmark records the execution platform independently from hardware information.

Collected information includes:

• Operating system
• Platform architecture
• Hostname
• System identifier
• Kernel information (when available)

This information allows benchmark reports produced on different operating systems to be compared correctly.

Benchmark Storage Location

Storage benchmark results depend on the storage device containing benchmark data.

For this reason, the benchmark explicitly records the location used during execution.

The benchmark storage snapshot identifies:

• Requested benchmark directory
• Resolved filesystem path
• Physical storage device
• Filesystem mount point

This information is particularly important for Storage benchmark interpretation because benchmark throughput depends on the characteristics of the underlying storage device.

Platform-Specific Collection

Hardware information is collected using platform-specific implementations.

Linux

Linux provides extensive hardware information through operating system interfaces, firmware tables, and storage subsystem utilities.

Examples include:

• SMBIOS
• procfs
• sysfs
• Storage subsystem metadata

macOS

macOS hardware information is collected using native system interfaces.

Examples include:

• sysctl
• IOKit
• Disk Arbitration

Windows

Windows hardware detection uses native operating system facilities to collect equivalent information.

Examples include:

• Win32 API
• Windows Management Instrumentation (WMI)

Although the implementations differ between operating systems, the resulting hardware snapshot follows a unified data model.

Hardware Fingerprint

Every hardware snapshot produces a deterministic fingerprint.

The fingerprint uniquely identifies the detected hardware configuration while remaining independent from benchmark results.

The fingerprint is intended to detect meaningful hardware changes, including:

• CPU replacement
• Memory upgrades
• Storage replacement
• Significant platform modifications

Minor operating system updates do not affect the hardware fingerprint.

The fingerprint is stored within benchmark reports and may be used by future benchmark comparison tools to determine whether two benchmark reports originate from the same physical machine.

Hardware Snapshot and Scoring

Hardware information is intentionally excluded from the scoring model.

Benchmark scores are calculated exclusively from measured workload performance.

Examples:

• Installing additional memory does not directly increase the benchmark score.
• Replacing the storage device does not change the score until benchmark execution measures improved storage performance.
• Upgrading the processor changes benchmark scores only because computational performance changes, not because the processor model differs.

This distinction is fundamental to the benchmark design.

The hardware snapshot describes the execution environment.

The benchmark score describes the measured performance.

The two concepts remain intentionally independent.

Report Integration

The hardware snapshot becomes a permanent component of every benchmark report.

Once the benchmark completes, the snapshot accompanies every measured result, allowing benchmark reports to remain self-contained.

A benchmark report therefore contains all information necessary to answer the following questions:

• Which machine executed the benchmark?
• Which operating system was used?
• Which processor, memory, and storage subsystem were present?
• Which storage device was benchmarked?
• Which benchmark configuration was executed?
• Which benchmark scores were produced?

No external database or user-provided hardware description is required to interpret benchmark results.

Design Philosophy

The hardware snapshot subsystem was designed to maximize reproducibility rather than hardware inventory completeness.

Only information relevant to benchmark interpretation is collected.

The benchmark deliberately avoids collecting unnecessary system information that does not contribute to performance analysis.

This approach keeps benchmark reports compact while ensuring they remain sufficiently detailed for long-term comparison, calibration, and historical benchmarking.

The hardware snapshot therefore serves as the foundation upon which every benchmark report is built, providing the contextual information required to correctly interpret performance measurements across different machines, operating systems, and hardware generations.

JSON Benchmark

The JSON benchmark evaluates the performance of structured data processing using complete serialization and deserialization workloads.

JSON remains one of the most widely used data interchange formats in modern software systems. Nearly every backend service, REST API, message broker, monitoring platform, and configuration system relies on JSON as a primary serialization format.

Unlike synthetic benchmarks that measure parser speed using isolated loops, the Scalionix JSON benchmark measures complete application workflows including object creation, serialization, deserialization, validation, memory allocation, filesystem interaction, and end-to-end processing.

The benchmark therefore represents realistic software workloads rather than isolated library performance.

Objectives

The JSON benchmark has four primary objectives.

• Measure serialization throughput.
• Measure deserialization throughput.
• Measure complete JSON processing pipelines.
• Measure filesystem interaction when processing JSON documents.

These workloads represent some of the most common operations executed by backend applications, microservices, API gateways, monitoring systems, and distributed services.

Why JSON?

JSON was selected because it remains the dominant structured data format used by modern software.

Typical production workloads include:

• REST APIs
• Configuration files
• Monitoring data
• Telemetry
• Logging
• Event processing
• Message queues
• Service discovery

Most software systems spend a significant amount of CPU time converting in-memory structures into JSON and reconstructing them back into native objects.

Measuring these workloads therefore provides a meaningful representation of practical application performance.

Benchmark Philosophy

The benchmark intentionally avoids measuring parser microbenchmarks.

Instead, every scenario executes complete application logic.

Typical benchmark execution consists of:

            Object generation
                    │
                    ▼
              Serialization
                    │
                    ▼
       Optional filesystem write
                    │
                    ▼
         Optional filesystem read
                    │
                    ▼
             Deserialization
                    │
                    ▼
           Object verification
                    │
                    ▼
          Performance measurement

The benchmark therefore measures the complete cost of structured data processing rather than individual library functions.

Benchmark Scenarios

The JSON benchmark currently consists of six independent scenarios.

JSON Serialization (Memory)

Measures serialization of generated objects directly into memory buffers.

Scenario

json.serialize.memory

JSON Serialization (File)

Measures serialization followed by writing JSON documents to persistent storage.

Scenario

json.serialize.file

JSON Deserialization (Memory)

Measures reconstruction of objects from in-memory JSON documents.

Scenario

json.deserialize.memory

JSON Deserialization (File)

Measures complete filesystem read followed by object reconstruction.

Scenario

json.deserialize.file

JSON Round Trip (Memory)

Measures complete in-memory processing.

Pipeline

            Generate objects
                    │
                    ▼
                Serialize
                    │
                    ▼
               Deserialize
                    │
                    ▼
                 Verify

Scenario

json.round_trip.memory

JSON Round Trip (File)

Measures complete persistent storage workflow.

Pipeline

            Generate objects
                    │
                    ▼
                Serialize
                    │
                    ▼
               Write file
                    │
                    ▼
                Read file
                    │
                    ▼
               Deserialize
                    │
                    ▼
                 Verify

Scenario

json.round_trip.file

Dataset Generation

Every benchmark dataset is generated deterministically.

The benchmark does not rely on static JSON files.

Instead, object graphs are generated programmatically before benchmark execution.

Dataset generation ensures:

• Identical benchmark input
• Deterministic execution
• Reproducible reports
• Platform independence

Generated datasets include nested objects, arrays, strings, numeric values, boolean values, and realistic field distributions representative of production APIs.

Dataset Sizes

The benchmark evaluates multiple dataset sizes.

Larger datasets exercise memory bandwidth and cache hierarchy.

Smaller datasets emphasize parser overhead and allocation efficiency.

Every dataset size represents the amount of serialized JSON data processed by the benchmark.

Dataset sizes remain deterministic across every supported platform.

Object Counts

In addition to total dataset size, workloads vary the number of serialized objects.

Changing the object count modifies several characteristics simultaneously.

• Allocation frequency
• Parser complexity
• Container growth
• Object traversal
• Memory locality

This allows the benchmark to evaluate parser efficiency under different object graph densities.

Worker Configuration

The JSON benchmark evaluates scalability using multiple worker configurations.

Representative worker counts include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The exact worker matrix depends on the detected hardware configuration.

Each worker processes an independent dataset.

No synchronization is required between workers.

This allows the benchmark to evaluate pure scalability of structured data processing.

Verification

Every JSON benchmark performs strict correctness verification.

Verification includes:

• Successful serialization
• Successful parsing
• Object reconstruction
• Structural validation
• Equality verification

Benchmark results are accepted only when reconstructed objects match the original generated dataset.

Serialization failures immediately invalidate benchmark execution.

Measured Metrics

The benchmark records multiple execution metrics.

Primary metrics include:

• Execution duration
• Bytes processed
• Throughput
• Worker configuration
• Iteration timing

The scoring adapter converts these measurements into normalized throughput expressed as bytes processed per second.

Score Calculation

JSON workloads participate fully in the compute scoring model.

Every workload contributes:

Single-thread Score

Measures processor efficiency using one execution worker.

Multi-thread Score

Measures maximum throughput achieved across worker configurations.

Scaling Score

Measures parallel scalability relative to the single-thread execution.

The final workload score is calculated using the weighted scoring model described later in this documentation.

Scenario scores are aggregated into the JSON category score.

Practical Interpretation

High JSON scores typically indicate:

• Efficient CPU architecture
• Strong branch prediction
• High memory bandwidth
• Efficient cache hierarchy
• Optimized allocator behavior

Because JSON processing is memory intensive rather than purely computational, benchmark results often reflect both processor performance and memory subsystem efficiency.

For this reason, the JSON benchmark represents one of the best indicators of real-world backend application performance.

Design Summary

The JSON benchmark represents complete structured data processing rather than isolated parser performance.

By combining deterministic datasets, multiple execution scenarios, filesystem interaction, verification, and scalable worker configurations, the benchmark provides a realistic evaluation of one of the most common workloads executed by modern software systems.

Unlike synthetic parser benchmarks, the resulting score reflects the practical efficiency of the entire JSON processing pipeline and therefore serves as a reliable indicator of structured data processing performance in production environments.

Hashing Benchmark

The Hashing benchmark evaluates the throughput of modern cryptographic hash algorithms using realistic data processing workloads.

Cryptographic hashing represents one of the most fundamental building blocks of modern software systems. Hash functions are extensively used for data integrity verification, content addressing, software distribution, version control systems, databases, backup systems, deduplication engines, blockchain technologies, authentication mechanisms, and countless backend services.

Unlike encryption algorithms, cryptographic hash functions are designed to produce deterministic fixed-size digests that uniquely represent arbitrary amounts of input data.

The benchmark measures the ability of a system to process large volumes of data using production-grade hashing algorithms under realistic execution conditions.

Objectives

The Hashing benchmark has four primary objectives.

• Measure sequential hashing throughput.
• Compare algorithm implementation efficiency.
• Evaluate memory subsystem performance during streaming workloads.
• Measure scalability across multiple execution workers.

The benchmark intentionally focuses on sustained throughput rather than latency, as throughput represents the dominant performance characteristic in most production hashing workloads.

Why Hashing?

Hashing operations are executed continuously in modern computing environments.

Typical applications include:

• File integrity verification
• Package managers
• Software repositories
• Git object storage
• Artifact distribution
• Backup verification
• Object storage
• Container registries
• Deduplication systems
• Content-addressable storage
• Digital signatures
• Password processing

Large-scale backend infrastructure frequently spends significant CPU time computing cryptographic hashes.

Consequently, hashing throughput serves as an excellent indicator of sustained computational performance.

Benchmark Philosophy

The benchmark evaluates complete hashing operations rather than isolated compression functions.

Every workload performs continuous processing of deterministic datasets while measuring sustained throughput.

The benchmark intentionally avoids:

• Tiny synthetic buffers
• Repeated hashing of identical cache-resident data
• Unrealistic benchmark loops

Instead, workloads process datasets large enough to exceed processor cache capacity whenever appropriate.

This produces measurements representative of practical software systems.

Supported Algorithms

The benchmark currently evaluates multiple production-grade cryptographic hash functions.

BLAKE3

BLAKE3 is a modern cryptographic hash algorithm optimized for extremely high throughput and efficient parallel execution.

Its tree-based design allows exceptional scalability on multi-core processors.

Scenarios

hash.blake3.chunked
hash.blake3.one_shot

SHA-256

SHA-256 remains one of the most widely deployed cryptographic hash algorithms in modern software infrastructure.

It is commonly used by:

• TLS
• Software distribution
• Package verification
• Container images
• Blockchain systems

Scenarios

hash.sha256.chunked
hash.sha256.one_shot

SHA-512

SHA-512 belongs to the SHA-2 family and frequently achieves higher throughput than SHA-256 on modern 64-bit processors.

It is commonly used in:

• Backup systems
• Large-scale storage
• Authentication
• Enterprise software

Scenarios

hash.sha512.chunked
hash.sha512.one_shot

Chunked vs One-Shot Processing

The benchmark intentionally measures two different processing models.

One-Shot

Entire datasets are hashed using a single hashing operation.

This represents workloads where complete data is already available in memory.

Typical examples include:

• Memory buffers
• Download verification
• Configuration validation
• API payloads

Chunked

Datasets are processed incrementally using multiple update operations.

This represents streaming workloads commonly encountered in production systems.

Examples include:

• File processing
• Network streams
• Backup software
• Archive creation
• Object storage

Evaluating both execution models provides a more complete representation of practical hashing performance.

Dataset Sizes

Multiple dataset sizes are evaluated.

Small datasets emphasize algorithm overhead.

Large datasets emphasize sustained throughput, cache hierarchy, and memory bandwidth.

Dataset sizes remain deterministic across every supported operating system.

Worker Configuration

Hashing workloads evaluate parallel scalability using multiple worker configurations.

Each worker processes an independent dataset.

No synchronization occurs between workers during hashing.

This execution model measures pure computational scalability.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The actual execution matrix depends on the detected hardware.

Verification

Every hashing workload verifies digest correctness.

Verification ensures that every produced digest exactly matches the expected reference value.

Benchmark results are accepted only when hash verification succeeds.

Incorrect digests immediately invalidate benchmark execution.

This guarantees that benchmark throughput always represents successful cryptographic computation.

Measured Metrics

Primary benchmark metrics include:

• Processed bytes
• Execution duration
• Throughput
• Worker configuration
• Iteration timing

The scoring adapter converts measured throughput into normalized bytes processed per second.

Score Calculation

Hashing workloads fully participate in the compute scoring model.

Each workload contributes:

Single-thread Score

Measures processor efficiency using one execution worker.

Multi-thread Score

Measures maximum throughput achieved across all worker configurations.

Scaling Score

Measures parallel scalability relative to the single-thread execution.

Scenario scores are aggregated into the Hashing category score.

Hardware Characteristics Measured

Although hashing is primarily CPU intensive, measured performance depends on multiple hardware characteristics.

Processor Architecture

Instruction throughput directly affects hashing performance.

Cache Hierarchy

Efficient cache utilization improves sustained throughput.

Memory Bandwidth

Streaming workloads frequently become memory bandwidth limited.

SIMD Capabilities

Modern hash implementations leverage SIMD instructions extensively.

Parallel Execution

Tree-based algorithms such as BLAKE3 demonstrate exceptional multi-core scalability.

Consequently, hashing performance often reflects the combined efficiency of the processor, cache subsystem, compiler optimizations, and runtime implementation.

Practical Interpretation

High Hashing benchmark scores generally indicate:

• Efficient processor architecture
• Strong SIMD performance
• High sustained memory bandwidth
• Efficient compiler optimizations
• Excellent multi-core scalability

Systems achieving strong hashing performance typically perform well in many streaming data processing workloads encountered by backend infrastructure and software development tools.

Design Summary

The Hashing benchmark evaluates production-grade cryptographic hashing using realistic streaming workloads rather than synthetic algorithm microbenchmarks.

By measuring multiple algorithms, execution models, dataset sizes, and worker configurations, the benchmark provides a comprehensive evaluation of sustained hashing performance representative of modern software engineering environments.

The resulting category score reflects the practical computational efficiency of cryptographic hash processing across a broad range of real-world applications.

Encryption Benchmark

The Encryption benchmark evaluates the throughput of modern authenticated encryption algorithms using realistic production workloads.

Encryption has become a fundamental component of virtually every modern software system. Secure communication protocols, distributed storage systems, cloud infrastructure, authentication services, VPN technologies, databases, and backup platforms all rely on authenticated encryption to guarantee confidentiality and integrity of data.

Unlike traditional encryption benchmarks that measure only raw cipher throughput, the Scalionix Encryption benchmark evaluates complete authenticated encryption pipelines, including key initialization, nonce generation, authenticated encryption, authenticated decryption, integrity verification, and complete round-trip processing.

This approach provides measurements that more accurately reflect practical software engineering workloads.

Objectives

The Encryption benchmark has four primary objectives.

• Measure authenticated encryption throughput.
• Measure authenticated decryption throughput.
• Measure complete encryption pipelines.
• Evaluate scalability across multiple processor cores.

Unlike synthetic benchmarks, the objective is not to maximize cipher speed, but to measure practical performance during realistic software execution.

Why Encryption?

Modern software encrypts data continuously.

Examples include:

• HTTPS traffic
• TLS connections
• REST APIs
• gRPC communication
• VPN tunnels
• Secure backups
• Database encryption
• Object storage
• Cloud synchronization
• Software update distribution
• Secret management
• Authentication systems

Because authenticated encryption is now ubiquitous, encryption throughput has become a meaningful indicator of practical CPU performance.

Benchmark Philosophy

Every benchmark scenario performs complete authenticated encryption rather than isolated block cipher operations.

Each workload includes all operations required by production software.

Typical execution consists of:

       Generate deterministic input
                    │
                    ▼
            Initialize cipher
                    │
                    ▼
             Generate nonce
                    │
                    ▼
                 Encrypt
                    │
                    ▼
       Generate authentication tag
                    │
                    ▼
            Optionally decrypt
                    │
                    ▼
            Verify original data
                    │
                    ▼
            Measure throughput

The benchmark therefore evaluates practical cryptographic workloads rather than theoretical algorithm performance.

Supported Algorithms

The current benchmark evaluates three modern authenticated encryption algorithms.

AES-128-GCM

AES-128-GCM represents one of the most widely deployed authenticated encryption algorithms in modern infrastructure.

It is commonly used by:

• TLS
• HTTPS
• VPNs
• Enterprise software
• Cloud infrastructure

Scenarios

encryption.aes128_gcm.encrypt
encryption.aes128_gcm.decrypt
encryption.aes128_gcm.round_trip

AES-256-GCM

AES-256-GCM provides increased cryptographic security while remaining one of the most common authenticated encryption algorithms in enterprise environments.

Scenarios

encryption.aes256_gcm.encrypt
encryption.aes256_gcm.decrypt
encryption.aes256_gcm.round_trip

ChaCha20-Poly1305

ChaCha20-Poly1305 is a modern authenticated stream cipher optimized for software implementations.

Unlike AES, ChaCha20 does not require dedicated processor instructions to achieve excellent performance.

It is widely deployed in:

• TLS
• WireGuard
• SSH
• Mobile devices
• Cloud-native software

Scenarios

encryption.chacha20_poly1305.encrypt
encryption.chacha20_poly1305.decrypt
encryption.chacha20_poly1305.round_trip

Encryption, Decryption and Round Trip

Each algorithm is evaluated using three independent execution scenarios.

Encryption

Measures authenticated encryption throughput.

Pipeline

            Generate plaintext
                    │
                    ▼
                 Encrypt
                    │
                    ▼
            Authentication tag
                    │
                    ▼
                 Measure

Decryption

Measures authenticated decryption throughput.

Pipeline

              Encrypted data
                    │
                    ▼
        Verify authentication tag
                    │
                    ▼
                 Decrypt
                    │
                    ▼
                 Measure

Round Trip

Measures complete secure processing.

Pipeline

            Generate plaintext
                    │
                    ▼
                 Encrypt
                    │
                    ▼
                 Decrypt
                    │
                    ▼
            Verify equality
                    │
                    ▼
                 Measure

The Round Trip scenario most closely represents complete application behavior.

Dataset Sizes

Encryption workloads execute multiple deterministic dataset sizes.

Smaller datasets emphasize initialization overhead.

Larger datasets emphasize sustained throughput.

Dataset sizes remain identical across all supported platforms.

Worker Configuration

Encryption benchmarks evaluate scalability using multiple worker configurations.

Each worker encrypts an independent dataset.

Workers never share encryption state.

This execution model reflects production services where multiple independent encryption operations execute concurrently.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

Verification

Every encryption workload performs strict correctness verification.

Verification includes:

• Successful encryption
• Successful authentication
• Successful decryption
• Equality verification

Benchmark execution is considered valid only when decrypted data exactly matches the original plaintext.

Authentication failures immediately invalidate benchmark results.

Measured Metrics

Primary metrics include:

• Processed bytes
• Execution duration
• Throughput
• Worker configuration
• Iteration timing

The scoring adapter converts throughput into normalized bytes processed per second.

Score Calculation

Encryption workloads participate fully in the Compute Score.

Each workload contributes:

Single-thread Score

Measures processor efficiency using a single execution worker.

Multi-thread Score

Measures maximum sustained throughput.

Scaling Score

Measures parallel scalability.

Scenario scores are aggregated into the Encryption category score.

Hardware Characteristics Measured

Encryption performance reflects multiple processor characteristics.

Processor Architecture

Modern CPUs provide dedicated cryptographic instructions.

Memory Bandwidth

Large streaming datasets stress the memory subsystem.

SIMD Capabilities

Vector instructions significantly improve throughput.

Instruction Scheduling

Efficient instruction scheduling improves sustained cryptographic performance.

Parallel Execution

Independent encryption streams scale efficiently across processor cores.

Because authenticated encryption combines arithmetic, memory access, and cryptographic primitives, the resulting benchmark score reflects overall computational efficiency rather than raw processor frequency.

Practical Interpretation

High Encryption scores generally indicate:

• Strong processor architecture
• Efficient cryptographic instruction support
• High sustained throughput
• Efficient runtime implementation
• Excellent multi-core scalability

Systems with strong encryption performance typically perform well in backend services, secure communication platforms, storage infrastructure, and cloud-native applications.

Design Summary

The Encryption benchmark evaluates complete authenticated encryption pipelines using production-grade algorithms and deterministic workloads.

Rather than measuring isolated cryptographic primitives, the benchmark measures realistic encryption workflows representative of modern software systems.

The resulting category score therefore reflects practical cryptographic performance encountered in everyday engineering environments.

Compression Benchmark

The Compression benchmark evaluates the performance of modern lossless compression algorithms using realistic datasets that represent different classes of production data.

Lossless compression is an essential component of contemporary software systems. Storage engines, databases, distributed file systems, backup platforms, package managers, container registries, messaging systems, and software distribution services all rely on compression to reduce storage requirements and network bandwidth while preserving complete data integrity.

Unlike traditional compression benchmarks that repeatedly compress a single file, the Scalionix Compression benchmark evaluates multiple data characteristics using deterministic datasets specifically designed to represent real-world software workloads.

By combining different compression algorithms, dataset sizes, entropy distributions, and execution scenarios, the benchmark provides a comprehensive evaluation of practical compression performance.

Objectives

The Compression benchmark has five primary objectives.

• Measure compression throughput.
• Measure decompression throughput.
• Evaluate different compression algorithms.
• Measure behavior across different dataset characteristics.
• Evaluate scalability across multiple execution workers.

The benchmark intentionally focuses on production-oriented compression workloads rather than artificial compression ratios.

Why Compression?

Modern software compresses data continuously.

Typical examples include:

• Database storage engines
• Backup software
• Software repositories
• Artifact registries
• Object storage
• Distributed file systems
• Log aggregation
• Telemetry systems
• Message brokers
• Software package managers
• Cloud storage

Because compression executes in numerous infrastructure components, measuring compression throughput provides valuable insight into sustained processor performance under realistic workloads.

Benchmark Philosophy

Compression performance depends heavily on the characteristics of the processed data.

Measuring only one input dataset would therefore provide a distorted view of practical performance.

Instead, the benchmark evaluates multiple deterministic dataset classes representing common production scenarios.

Each workload executes complete compression pipelines including:

            Dataset preparation
                    │
                    ▼
               Compression
                    │
                    ▼
          Optional decompression
                    │
                    ▼
           Output verification
                    │
                    ▼
          Performance measurement

This approach measures complete software behavior rather than isolated library functions.

Typical Compression Pipeline

                 Dataset
                    │
                    ▼
               Compression
                    │
                    ▼
           Compressed Output
                    │
                    ▼
              Decompression
                    │
                    ▼
              Verification
                    │
                    ▼
          Throughput Measurement

Supported Algorithms

The benchmark currently evaluates modern lossless compression algorithms commonly used by production software.

Algorithms have been selected because they represent different design goals.

Examples include:

• Maximum throughput
• Balanced compression
• High compression ratio
• General-purpose compression

Each algorithm executes identical benchmark datasets, allowing direct comparison of implementation efficiency under equivalent workloads.

Compression Scenarios

Each compression algorithm is evaluated using independent execution scenarios.

Compression

Measures raw compression throughput.

Scenario

compression.<algorithm>.compress

Decompression

Measures decompression throughput.

Scenario

compression.<algorithm>.decompress

Round Trip

Measures complete application workflow.

Pipeline

Generate dataset
        │
        ▼
    Compress
        │
        ▼
    Decompress
        │
        ▼
     Verify
        │
        ▼
     Measure

Scenario

compression.<algorithm>.round_trip

Round Trip workloads most closely represent practical application behavior because both compression and decompression contribute to measured performance.

Dataset Types

One of the defining characteristics of the Compression benchmark is the use of multiple deterministic dataset classes.

Each dataset represents a different entropy profile commonly encountered in production environments.

Structured JSON

Represents highly repetitive structured documents similar to REST API payloads, configuration files, telemetry data, and monitoring systems.

Typical characteristics include:

• High repetition
• Predictable field names
• Nested object hierarchies
• Excellent compressibility

Natural Text

Represents human-readable documents.

Examples include:

• Documentation
• Log files
• Source code
• Configuration files
• Documentation repositories

Natural language typically compresses well while exhibiting more variation than structured JSON.

Structured Binary

Represents deterministic binary formats.

Examples include:

• Serialized objects
• Binary protocols
• Executable metadata
• Compact storage formats

Binary datasets generally contain less repetition than structured text and therefore provide different compression behavior.

Mixed Entropy

Represents heterogeneous production workloads.

This dataset intentionally combines:

• Structured data
• Textual content
• Binary sections
• High-entropy regions

Mixed entropy datasets most closely resemble practical storage workloads encountered by backup software, archives, container images, and software repositories.

Why Multiple Dataset Types?

Compression algorithms perform differently depending on input characteristics.

An algorithm that performs exceptionally well on structured JSON may perform significantly worse on high-entropy binary data.

Using multiple deterministic datasets prevents benchmark results from being overly optimized toward one particular compression pattern.

This design provides a much more representative evaluation of practical compression performance.

Dataset Sizes

Compression workloads execute multiple deterministic dataset sizes.

Smaller datasets emphasize algorithm initialization overhead.

Larger datasets emphasize sustained throughput, memory hierarchy efficiency, and streaming behavior.

Dataset sizes remain identical across every supported platform.

Worker Configuration

Compression workloads evaluate scalability using multiple worker configurations.

Each worker compresses an independent dataset.

Workers never share compression state.

This execution model closely resembles production software such as backup systems, package builders, artifact repositories, and parallel archive generation.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The exact worker matrix depends on the detected hardware configuration.

Verification

Every compression workload performs strict correctness verification.

Verification includes:

• Successful compression
• Successful decompression
• Output integrity
• Byte-for-byte equality verification

Benchmark execution is accepted only when decompressed output exactly matches the original dataset.

Verification failures invalidate benchmark results regardless of measured throughput.

Measured Metrics

Primary benchmark metrics include:

• Compressed bytes
• Original bytes
• Execution duration
• Throughput
• Worker configuration
• Compression ratio

Although compression ratio is recorded for diagnostic purposes, benchmark scoring is based exclusively on processing throughput.

This decision ensures that benchmark scores represent computational efficiency rather than algorithm-specific compression characteristics.

Score Calculation

Compression workloads fully participate in the Compute Score.

Each workload contributes:

Single-thread Score

Measures processor efficiency using one execution worker.

Multi-thread Score

Measures maximum sustained throughput.

Scaling Score

Measures parallel scalability across worker configurations.

Scenario scores are aggregated into the Compression category score.

Hardware Characteristics Measured

Compression workloads exercise multiple hardware subsystems simultaneously.

Processor Architecture

Compression algorithms execute large numbers of arithmetic and logical operations.

Cache Hierarchy

Dictionary-based compression heavily benefits from efficient cache utilization.

Memory Bandwidth

Streaming datasets require sustained memory throughput.

Branch Prediction

Compression algorithms frequently contain complex decision trees whose performance depends on processor branch prediction efficiency.

Parallel Execution

Independent compression tasks scale efficiently across processor cores.

As a result, the Compression benchmark reflects the combined efficiency of processor architecture, cache hierarchy, memory subsystem, compiler optimizations, and runtime implementation.

Practical Interpretation

High Compression scores generally indicate:

• Efficient processor architecture
• Strong memory subsystem
• Excellent cache utilization
• Efficient branch prediction
• Good multi-core scalability

Systems with strong compression performance typically perform well in storage infrastructure, backup platforms, software distribution systems, and archive generation workloads.

Design Summary

The Compression benchmark evaluates complete lossless compression workflows using deterministic datasets representative of practical software engineering environments.

By combining multiple algorithms, dataset characteristics, execution scenarios, and worker configurations, the benchmark provides a realistic evaluation of compression performance across a broad range of production workloads.

Unlike synthetic compression benchmarks, the resulting category score reflects practical computational efficiency rather than isolated algorithm behavior.

Collections Benchmark

The Collections benchmark evaluates the performance of fundamental in-memory container operations that form the backbone of modern software systems.

Virtually every application continuously manipulates dynamic collections of objects. Web servers process request tables, databases maintain indexes, schedulers manage task queues, compilers build symbol tables, operating systems track resources, and distributed systems maintain routing structures.

Although these operations appear simple, they collectively account for a significant portion of CPU execution time in many production environments.

Unlike synthetic memory benchmarks that repeatedly allocate identical buffers, the Scalionix Collections benchmark evaluates complete container lifecycles including allocation, insertion, lookup, iteration, update, removal, sorting, and memory reclamation.

The resulting measurements closely resemble workloads encountered by production software.

Objectives

The Collections benchmark has six primary objectives.

• Measure allocation performance.
• Measure insertion throughput.
• Measure lookup efficiency.
• Measure iteration performance.
• Measure update operations.
• Measure removal and cleanup performance.

Rather than evaluating one isolated operation, the benchmark measures complete collection lifecycles.

Why Collections?

Collections are one of the most frequently used abstractions in software engineering.

Examples include:

• Routing tables
• HTTP header maps
• Session stores
• Symbol tables
• Object registries
• Job schedulers
• Resource tracking
• Database indexes
• Cache implementations
• Runtime object graphs

Because these operations dominate many CPU profiles, measuring collection performance provides valuable insight into practical application efficiency.

Benchmark Philosophy

The benchmark intentionally avoids measuring individual library functions.

Instead, every workload performs realistic sequences of container operations.

Typical execution consists of:

            Generate Objects
                    │
                    ▼
           Allocate Collection
                    │
                    ▼
               Insert Data
                    │
                    ▼
                 Lookup
                    │
                    ▼
                Iteration
                    │
                    ▼
            Optional Modification
                    │
                    ▼
                 Removal
                    │
                    ▼
               Verification
                    │
                    ▼
          Performance Measurement

This execution model reflects the complete lifecycle of collections within production applications.

Typical Collection Pipeline

            Generate Objects
                    │
                    ▼
           Allocate Collection
                    │
                    ▼
               Insert Data
                    │
                    ▼
            Lookup Operations
                    │
                    ▼
           Iterate Collection
                    │
                    ▼
          Update / Modify Data
                    │
                    ▼
             Remove Objects
                    │
                    ▼
              Verify State
                    │
                    ▼
          Throughput Measurement

Benchmark Scenarios

The Collections benchmark evaluates multiple categories of container operations.

Insertion

Measures sustained insertion throughput into initially empty collections.

Scenario examples

collections.insert.vector
collections.insert.hash_map
collections.insert.btree_map

Lookup

Measures repeated access to previously inserted elements.

Scenario examples

collections.lookup.hash_map
collections.lookup.btree_map
collections.lookup.vector

Iteration

Measures sequential traversal of populated containers.

Scenario examples

collections.iteration.vector
collections.iteration.hash_map

Removal

Measures deletion throughput.

Scenario examples

collections.remove.hash_map
collections.remove.vector

Sorting

Measures ordering of deterministic datasets.

Scenario examples

collections.sort.vector
collections.sort.stable

Mixed Operations

Measures practical workloads consisting of insertion, lookup, update, and deletion executed together.

This scenario most closely resembles production software.

Dataset Characteristics

Every workload operates on deterministic datasets.

Objects are generated programmatically before benchmark execution.

Datasets contain realistic field distributions instead of synthetic integer sequences.

This ensures that measured performance reflects practical object layouts encountered by modern software.

Collection Sizes

Multiple collection sizes are evaluated.

Smaller collections emphasize allocator overhead.

Larger collections emphasize:

• Cache hierarchy
• Memory bandwidth
• Branch prediction
• Pointer chasing
• Allocator scalability

Dataset sizes remain deterministic across every supported platform.

Worker Configuration

Collections benchmark scalability is evaluated using multiple worker configurations.

Each worker operates on an independent collection.

Workers never share containers.

This execution model isolates computational scalability from synchronization overhead.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

Verification

Every workload verifies collection correctness.

Verification includes:

• Expected element count
• Successful lookup
• Successful iteration
• Ordering validation
• Successful removal

Verification failures invalidate benchmark execution.

This guarantees that benchmark scores always represent correct container behavior.

Measured Metrics

Primary benchmark metrics include:

• Operations per second
• Execution duration
• Processed objects
• Worker configuration
• Iteration timing

The scoring adapter converts measured operations into normalized throughput expressed as operations per second.

Score Calculation

Collections workloads fully participate in the Compute Score.

Every workload contributes:

Single-thread Score

Measures processor efficiency during sequential container operations.

Multi-thread Score

Measures maximum sustained throughput across all worker configurations.

Scaling Score

Measures parallel scalability of independent collection workloads.

Scenario scores are aggregated into the Collections category score.

Hardware Characteristics Measured

Collection performance depends on several hardware characteristics.

Processor Architecture

Instruction execution affects arithmetic and pointer manipulation.

Cache Hierarchy

Most collection operations are cache sensitive.

Memory Bandwidth

Large collections generate continuous memory traffic.

Branch Prediction

Tree traversal and hash lookups frequently depend on branch prediction accuracy.

Memory Allocator

Container growth frequently triggers dynamic allocation.

Consequently, collection performance reflects the combined behavior of the processor, cache hierarchy, memory subsystem, allocator implementation, and compiler optimizations.

Practical Interpretation

High Collections benchmark scores generally indicate:

• Efficient cache hierarchy
• Strong branch prediction
• Fast allocator implementation
• Excellent memory bandwidth
• Efficient pointer traversal

These characteristics directly influence the performance of compilers, databases, schedulers, web services, and numerous backend applications.

Design Summary

The Collections benchmark evaluates complete container lifecycles using deterministic datasets representative of modern software systems.

Rather than measuring isolated library calls, the benchmark measures realistic object management workloads that continuously occur inside production applications.

The resulting category score therefore reflects practical memory and container performance encountered during everyday software development.

Concurrency Benchmark

The Concurrency benchmark evaluates the efficiency of the operating system, processor architecture, runtime libraries, and synchronization primitives during parallel execution.

Modern software rarely executes as a single sequential process. Web servers, databases, distributed systems, compilers, game engines, messaging platforms, storage systems, and cloud-native applications continuously coordinate hundreds or thousands of concurrent execution units.

The performance of these systems depends not only on raw processor speed, but also on the efficiency of synchronization mechanisms, scheduler behavior, memory consistency, cache coherency, and inter-thread communication.

Unlike synthetic thread creation benchmarks, the Scalionix Concurrency benchmark measures realistic synchronization workloads representative of production software.

Objectives

The Concurrency benchmark has five primary objectives.

• Measure synchronization overhead.
• Evaluate parallel execution efficiency.
• Measure operating system scheduling behavior.
• Evaluate runtime synchronization primitives.
• Measure scalability across increasing numbers of execution workers.

The benchmark intentionally focuses on practical synchronization workloads rather than theoretical thread creation performance.

Why Concurrency?

Concurrency is fundamental to modern computing.

Typical production workloads include:

• Web servers
• Databases
• Task schedulers
• Message queues
• Build systems
• Logging frameworks
• Distributed storage
• Game engines
• Runtime schedulers
• Background workers

Every one of these systems spends part of its execution coordinating concurrent tasks.

Consequently, synchronization efficiency directly affects practical application performance.

Benchmark Philosophy

The benchmark evaluates synchronization primitives during realistic execution rather than artificial contention loops.

Each workload performs useful computational work while coordinating execution between multiple workers.

Typical execution consists of:

             Create workers
                    │
                    ▼
            Execute workload
                    │
                    ▼
               Synchronize
                    │
                    ▼
             Exchange data
                    │
                    ▼
            Complete execution
                    │
                    ▼
            Measure throughput

The benchmark therefore evaluates practical parallel software behavior rather than isolated operating system calls.

Typical Concurrency Pipeline

           Create Workers
                 │
                 ▼
        Execute Workload
                 │
                 ▼
        Synchronization Phase
                 │
                 ▼
       Shared State Access
                 │
                 ▼
        Worker Completion
                 │
                 ▼
      Throughput Measurement

Benchmark Scenarios

The benchmark evaluates multiple synchronization mechanisms commonly used by production software.

Representative workloads include:

Mutex Synchronization

Measures exclusive access performance using mutual exclusion primitives.

Read-Write Locks

Measures concurrent reader and exclusive writer behavior.

Atomic Operations

Measures lock-free synchronization using atomic variables.

Channels

Measures producer-consumer communication.

Barrier Synchronization

Measures coordinated execution across multiple workers.

Work Queue Processing

Measures dynamic task scheduling between concurrent workers.

Every scenario executes deterministic workloads while measuring synchronization efficiency.

Workload Characteristics

Each workload has been designed to represent common software engineering patterns rather than synthetic contention.

Workloads include combinations of:

• Shared state
• Independent computation
• Synchronization
• Message passing
• Task distribution

This approach produces measurements representative of practical concurrent software.

Worker Configuration

Unlike other benchmark categories, Concurrency intentionally emphasizes scaling.

The benchmark executes increasing worker counts to evaluate scheduler efficiency and synchronization overhead.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

The exact worker matrix depends on the detected processor topology.

Verification

Every concurrency workload verifies execution correctness.

Verification includes:

• Completed work
• Synchronization correctness
• Absence of lost updates
• Deterministic final state

Any synchronization failure immediately invalidates benchmark execution.

Measured Metrics

Primary benchmark metrics include:

• Operations per second
• Synchronization latency
• Worker configuration
• Execution duration
• Completed operations

The scoring adapter converts these measurements into normalized operations per second.

Score Calculation

Concurrency workloads fully participate in the Compute Score.

Every workload contributes:

Single-thread Score

Measures sequential execution without synchronization overhead.

Multi-thread Score

Measures maximum throughput under concurrent execution.

Scaling Score

Measures how efficiently synchronization scales as additional workers are introduced.

Scenario scores are aggregated into the Concurrency category score.

Hardware Characteristics Measured

Concurrency workloads exercise several processor and operating system components simultaneously.

Processor Architecture

Instruction execution and cache coherency.

Memory Subsystem

Shared memory traffic generated by concurrent workers.

Cache Coherency

Synchronization frequently requires cache line ownership transfers between processor cores.

Operating System Scheduler

Thread scheduling directly affects execution efficiency.

Runtime Library

Synchronization primitives implemented by the language runtime significantly influence measured throughput.

Consequently, Concurrency represents one of the best indicators of real-world multi-threaded application performance.

Practical Interpretation

High Concurrency benchmark scores generally indicate:

• Efficient processor cache coherency
• Low synchronization overhead
• Strong operating system scheduler performance
• Efficient runtime implementation
• Excellent parallel scalability

These characteristics directly benefit backend servers, distributed systems, compilers, messaging platforms, and cloud-native infrastructure.

Design Summary

The Concurrency benchmark evaluates practical synchronization behavior using deterministic workloads representative of modern multi-threaded software.

Rather than measuring isolated synchronization primitives, the benchmark measures complete concurrent execution patterns that continuously occur inside production systems.

The resulting category score therefore reflects the practical ability of a computer system to execute highly parallel software efficiently while maintaining deterministic synchronization behavior.

Compilation Benchmark

The Compilation benchmark evaluates the ability of a computer system to build real-world software projects using production toolchains.

Unlike synthetic compiler benchmarks that repeatedly compile small source files or isolated functions, the Scalionix Compilation benchmark measures complete software builds consisting of thousands of source files, multiple compilation units, dependency resolution, optimization, linking, filesystem interaction, and executable generation.

Compilation is one of the most demanding workloads encountered by software developers and continuous integration systems. Modern compilers exercise nearly every subsystem of a computer including processor execution units, memory hierarchy, filesystem performance, parallel scheduling, and operating system services.

The benchmark therefore represents one of the closest approximations to everyday software engineering workloads.

Objectives

The Compilation benchmark has five primary objectives.

• Measure complete project build performance.
• Evaluate compiler scalability.
• Measure build system efficiency.
• Measure filesystem interaction during compilation.
• Measure practical software development performance.

Unlike traditional CPU benchmarks, the objective is not to measure isolated compiler execution but the performance of the complete software build pipeline.

Why Compilation?

Compilation is one of the most common high-performance workloads executed by development machines.

Typical environments include:

• Software development
• Continuous Integration
• Continuous Delivery
• Build servers
• Release engineering
• Package generation
• Compiler development
• Large-scale source repositories

Modern compilation stresses multiple hardware subsystems simultaneously.

            Processor execution
                    │
                    ▼
                 Parser
                    │
                    ▼
                Optimizer
                    │
                    ▼
            Code generation
                    │
                    ▼
                Filesystem
                    │
                    ▼
                  Linker
                    │
                    ▼
          Executable generation

Because nearly every subsystem participates during compilation, build performance is an excellent indicator of overall workstation capability.

Benchmark Philosophy

The benchmark intentionally measures complete production projects rather than synthetic compiler stress tests.

Every benchmark fixture represents an actual software project that can be built using the standard build process recommended by its maintainers.

Projects are compiled exactly as a developer would build them on a production workstation.

No benchmark-specific modifications are introduced.

Each benchmark request performs a completely clean build.

Compiler caches are disabled.

Previous build artifacts are removed.

The benchmark therefore measures real compilation work instead of incremental rebuild performance.

Typical Compilation Pipeline

         Extract Fixture
                │
                ▼
      Prepare Build Directory
                │
                ▼
      Remove Previous Objects
                │
                ▼
        Invoke Build System
                │
                ▼
     Compile Source Files
                │
                ▼
         Link Executable
                │
                ▼
      Verify Build Success
                │
                ▼
      Measure Build Duration

Benchmark Fixtures

The benchmark uses deterministic software projects called fixtures.

Every fixture is embedded inside the benchmark executable.

No network access is required during benchmark execution.

This guarantees:

• Deterministic execution
• Reproducible results
• Offline compatibility
• Identical benchmark input on every machine

Each fixture is distributed as a compressed archive and extracted into an isolated working directory before compilation begins.

Current Fixtures

The current benchmark distribution includes multiple software projects representing different compilation characteristics.

Rust — Typst

Typst is a modern document processing system written entirely in Rust.

The project consists of a large number of crates, extensive generic programming, procedural macros, and heavy optimization.

Compilation characteristics:

• Very large dependency graph
• Extensive LLVM optimization
• Significant memory usage
• Long compilation time

Scenario

compilation.rust.typst.release

C++ — Scalionix Stress Project

The C++ fixture is a deterministic stress project specifically designed to generate substantial compilation workload.

The project contains hundreds of independent translation units that can be compiled in parallel before final linking.

Compilation characteristics:

• Many independent source files
• Intensive parallel compilation
• Heavy optimizer usage
• Significant linker workload

Scenario

compilation.cpp.scalionix.gcc.release

Additional fixtures may be introduced in future benchmark versions without affecting the scoring model.

Platform Support

Compilation workloads execute natively using the operating system toolchains.

Current supported environments include:

Linux

• Rust
• GCC

macOS

• Rust
• Apple Clang

Windows compilation workloads are currently not executed.

This decision avoids introducing unnecessary complexity caused by multiple compiler environments while maintaining deterministic benchmark behavior across supported platforms.

Build Environment

The benchmark automatically verifies that the required compilation environment is available before execution.

Required components include:

• Rust Toolchain
• Cargo
• C++ Compiler
• Standard C++ Library
• Platform Build Tools

The benchmark provides a dedicated setup command capable of validating and preparing the compilation environment.

This ensures that benchmark execution remains reproducible across supported operating systems.

Build Execution

Each benchmark request performs multiple measured compilation iterations.

The workflow is intentionally deterministic.

For every iteration the benchmark performs:

            Extract fixture
                    │
                    ▼
            Clean previous build
                    │
                    ▼
            Compile project
                    │
                    ▼
            Verify successful build
                    │
                    ▼
            Record build duration

Only successful compilations participate in benchmark scoring.

Compilation failures invalidate the corresponding benchmark request.

Best Iteration Selection

Multiple measured iterations are executed for every fixture.

Rather than averaging execution times, the benchmark selects the fastest successful build.

This approach minimizes the influence of temporary operating system activity, background processes, filesystem maintenance, and scheduler interruptions.

The selected iteration therefore represents the maximum build capability of the evaluated system under stable operating conditions.

Worker Configuration

Unlike other benchmark categories, Compilation evaluates only the native build parallelism supported by the compiler.

The benchmark does not execute multiple benchmark workers simultaneously.

Instead, the compiler itself fully utilizes the available processor resources.

This approach reflects practical software development where a single build process already schedules parallel compilation internally.

Consequently, Compilation is treated as a Multi-Thread Only benchmark category.

Verification

Every compilation workload performs strict validation.

Verification includes:

• Successful extraction
• Successful compilation
• Successful linking
• Executable generation
• Zero compiler errors

Only successful builds contribute to benchmark scoring.

Measured Metrics

Primary benchmark metrics include:

• Total build duration
• Builds per second
• Executable verification
• Build profile
• Compiler
• Language
• Iteration timing

Unlike throughput-oriented benchmarks, Compilation converts build duration into Builds Per Second before score normalization.

This allows Compilation to integrate naturally into the common scoring model used by every benchmark category.

Score Calculation

Compilation participates in the Compute Score.

Unlike the remaining benchmark categories, Compilation does not calculate a Single-thread Score.

The compiler already executes using its native parallel scheduling model.

Consequently:

Single-thread Score

Not applicable.

Displayed as zero.

Multi-thread Score

Calculated from the measured build throughput.

Scaling Score

Fixed at the baseline reference value because parallel scheduling is entirely controlled by the compiler itself.

The final Compilation score therefore equals the normalized Multi-thread Score.

Hardware Characteristics Measured

Compilation simultaneously exercises nearly every major hardware subsystem.

Processor Architecture

Parser execution, optimization, code generation, and linking.

Memory Capacity

Large software projects require substantial memory during optimization.

Memory Bandwidth

Compiler frontends continuously stream source code and intermediate representations.

Filesystem Performance

Compilation creates and reads thousands of temporary files.

Process Scheduling

Modern compilers execute numerous concurrent compilation jobs.

Linker Performance

Final executable generation stresses both CPU and storage subsystems.

For this reason, Compilation is widely regarded as one of the best indicators of practical workstation performance.

Practical Interpretation

High Compilation benchmark scores generally indicate:

• Excellent software development performance
• Efficient processor architecture
• Strong filesystem performance
• Fast memory subsystem
• Efficient compiler scheduling

Systems achieving high Compilation scores are typically well suited for software engineering, continuous integration, large-scale builds, and compiler-intensive workloads.

Design Summary

The Compilation benchmark evaluates complete production software builds using deterministic embedded projects executed entirely offline.

By compiling real-world applications with native toolchains, the benchmark provides a realistic evaluation of workstation performance under one of the most demanding software engineering workloads.

Unlike synthetic compiler benchmarks, the resulting score reflects the practical ability of a system to build complex software projects efficiently and reproducibly.

Storage Benchmark

The Storage benchmark evaluates the practical performance of the complete storage subsystem rather than the theoretical bandwidth of a storage device.

Modern software interacts with persistent storage in many different ways. Some applications continuously stream large files, while others perform millions of metadata operations, create thousands of small files, execute random reads, synchronize write buffers, or recursively traverse complex directory trees.

Each of these workloads stresses a different part of the storage stack.

Unlike traditional storage benchmarks that primarily report sequential bandwidth or synthetic IOPS, the Scalionix Storage benchmark evaluates a diverse collection of realistic filesystem workloads representative of modern software engineering, backend infrastructure, databases, package managers, build systems, and distributed storage platforms.

The resulting Storage Score therefore represents the practical efficiency of the complete storage subsystem rather than the maximum performance of the storage hardware alone.

Objectives

The Storage benchmark has six primary objectives.

• Measure sustained sequential throughput.
• Measure random access performance.
• Measure filesystem metadata operations.
• Measure small file performance.
• Measure durability synchronization overhead.
• Measure filesystem traversal performance.

Together these workloads provide a comprehensive evaluation of practical storage behavior encountered during everyday software execution.

Why Storage?

Persistent storage remains one of the most important components of every computer system.

Nearly every application continuously performs storage operations.

Typical examples include:

• Software compilation
• Package managers
• Source code repositories
• Databases
• Object storage
• Backup software
• Artifact repositories
• Logging systems
• Telemetry collection
• Software installation

Although storage devices continue to increase in bandwidth, application performance frequently depends more on filesystem behavior than raw transfer speed.

Consequently, measuring only sequential bandwidth provides an incomplete representation of practical storage performance.

Benchmark Philosophy

The benchmark intentionally evaluates complete filesystem operations instead of isolated device benchmarks.

Every workload interacts with the operating system filesystem exactly as production software would.

Typical execution consists of:

         Create benchmark dataset
                    │
                    ▼
           Filesystem operation
                    │
                    ▼
       Operating system interaction
                    │
                    ▼
        Filesystem synchronization
                    │
                    ▼
              Verification
                    │
                    ▼
         Performance measurement

This execution model allows the benchmark to measure the combined behavior of:

• Storage hardware
• Filesystem implementation
• Operating system
• Kernel page cache
• Storage drivers
• Scheduler

rather than measuring only the storage controller.

Typical Storage Pipeline

              Prepare Dataset
                    │
                    ▼
       Execute Filesystem Operation
                    │
                    ▼
          Operating System Layer
                    │
                    ▼
             Storage Subsystem
                    │
                    ▼
           Verify Correctness
                    │
                    ▼
          Throughput Measurement

Storage Workload Categories

The Storage benchmark consists of six independent workload groups.

Each group measures a different aspect of storage subsystem behavior.

Sequential I/O

Large continuous file transfers.

Random I/O

Small block access across large datasets.

Small Files

Creation, reading, and deletion of thousands of files.

Metadata

Filesystem metadata operations.

Synchronization

Durability and persistence operations.

Directory Traversal

Recursive filesystem enumeration.

Each workload category contributes independently to the final Storage Score.

Sequential I/O

Sequential workloads measure continuous streaming performance.

Scenarios

storage.sequential.read
storage.sequential.write

The benchmark evaluates multiple dataset sizes together with multiple block sizes.

Sequential workloads primarily measure:

• Sustained storage throughput
• Filesystem buffering
• Page cache efficiency
• Storage controller performance

Unlike synthetic benchmarks, verification is performed after every operation.

Random I/O

Random access workloads evaluate storage behavior when reading or writing independent blocks distributed across the dataset.

Scenarios

storage.random.read
storage.random.write

Multiple block sizes are evaluated.

Typical configurations include:

• 4 KiB
• 16 KiB
• 64 KiB

Random workloads primarily measure:

• Random access latency
• Storage controller scheduling
• Flash translation layer efficiency
• Filesystem allocation behavior

The resulting measurements are expressed as operations per second.

Small File Operations

Many real-world applications manipulate enormous numbers of very small files.

Examples include:

• Source code repositories
• Package managers
• Build systems
• Software installation
• Configuration management

The benchmark evaluates three scenarios.

storage.small_files.create
storage.small_files.read
storage.small_files.delete

Multiple file counts and file sizes are evaluated.

These workloads stress:

• Inode allocation
• Directory management
• Metadata caching
• Filesystem allocation algorithms

rather than storage bandwidth.

Metadata Operations

Metadata operations are among the most frequently executed filesystem requests.

The benchmark currently evaluates:

storage.metadata.stat
storage.metadata.rename

These workloads exercise filesystem metadata handling independently from file contents.

Typical applications include:

• Build systems
• Package managers
• Compilers
• Source control systems
• Backup software

Metadata benchmarks frequently reveal filesystem bottlenecks invisible to bandwidth-oriented benchmarks.

Directory Traversal

Directory traversal evaluates recursive filesystem enumeration.

Scenario

storage.directory.walk

The benchmark recursively visits deterministic directory trees while collecting filesystem metadata.

Typical production examples include:

• Compiler dependency discovery
• Backup indexing
• Antivirus scanning
• Filesystem synchronization
• Source repository indexing

Unlike other Storage scenarios, Directory Walk executes only as a single-threaded workload.

Filesystem traversal is inherently limited by directory hierarchy traversal and operating system behavior.

Consequently, the benchmark evaluates absolute traversal efficiency rather than artificial parallel execution.

Synchronization Workloads

Durability guarantees are essential for many software systems.

The benchmark therefore evaluates explicit synchronization operations.

Scenarios

storage.sync.fsync
storage.sync.write_fsync

Multiple block sizes are evaluated together with different durability modes.

Synchronization workloads primarily measure:

• Flush latency
• Filesystem durability overhead
• Storage controller behavior
• Operating system synchronization cost

These scenarios are particularly relevant for databases, transactional systems, and persistent logging.

Worker Configuration

Most Storage workloads evaluate multiple worker configurations.

Representative worker counts include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

Workers operate on independent datasets whenever possible.

Directory Walk is the only exception.

Because recursive filesystem traversal is fundamentally sequential, it executes exclusively using a single worker.

The benchmark intentionally avoids artificial parallelization that would not represent practical application behavior.

Verification

Every Storage workload performs correctness verification.

Verification includes:

• Successful file creation
• Successful reads
• Successful writes
• Metadata validation
• Filesystem consistency
• Directory traversal completion

Benchmark results are accepted only when all filesystem operations complete successfully.

Measured Metrics

Different workload groups naturally produce different performance metrics.

Sequential Workloads

• Bytes per second

Random Workloads

• Operations per second

Small File Workloads

• Files per second

Metadata Workloads

• Operations per second

Synchronization Workloads

• Durability operations per second

Directory Traversal

• Directory operations per second

The scoring adapter normalizes all metrics into the common scoring model.

Score Calculation

Every Storage workload produces three normalized values.

Single-thread Score

Measures filesystem performance using one execution worker.

Multi-thread Score

Measures maximum throughput achieved across all evaluated worker configurations.

Scaling Score

Measures parallel scalability relative to the single-thread execution.

Directory Walk represents the only exception.

Because the workload executes exclusively using a single worker, its Multi-thread Score is intentionally equal to its Single-thread Score.

Its Scaling Score is fixed at the baseline reference value.

This allows the scenario to integrate naturally into the common scoring model while preserving realistic execution behavior.

Scenario scores are aggregated into the Storage category score.

Hardware Characteristics Measured

The Storage benchmark simultaneously exercises multiple layers of the storage stack rather than measuring the storage device in isolation.

Unlike compute-oriented benchmarks, storage performance depends on the interaction between hardware, operating system services, filesystem implementation, and storage controller behavior.

Consequently, Storage Score reflects the combined efficiency of the complete storage subsystem.

The benchmark measures several hardware characteristics simultaneously.

Storage Device

Sequential throughput, random access latency, internal parallelism, controller efficiency, flash translation layer behavior, and media performance directly influence benchmark results.

Filesystem

Filesystem allocation algorithms, metadata organization, journaling implementation, directory indexing, and caching strategies significantly affect practical storage performance.

Operating System

Kernel buffering, page cache management, asynchronous writeback, request scheduling, and synchronization mechanisms contribute to observed throughput.

Memory Subsystem

Large sequential transfers rely heavily on sustained memory bandwidth, while metadata workloads frequently benefit from cache efficiency.

Processor

Although Storage is not intended as a CPU benchmark, filesystem operations still require substantial processor resources for address translation, metadata processing, allocation management, checksum generation, and synchronization.

Storage Driver

Driver implementation, queue management, interrupt handling, and command submission latency contribute directly to measured throughput.

Because all of these components participate simultaneously, Storage Score represents the practical performance experienced by real applications rather than theoretical device specifications.

Why Storage Is Evaluated Separately

One of the primary architectural decisions of Scalionix System Benchmark is the separation of Compute Score and Storage Score.

This decision was made intentionally after evaluating how modern hardware evolves.

Processor performance continues to improve substantially with each generation.

Increasing core counts, wider execution engines, larger caches, improved branch prediction, higher memory bandwidth, and architectural enhancements allow compute-oriented workloads to scale significantly over time.

Storage hardware follows a fundamentally different evolution.

Although modern NVMe devices continue to improve, their maximum practical performance is constrained by several physical limits.

These include:

• PCI Express bandwidth
• NAND flash characteristics
• Controller architecture
• Storage protocols
• Filesystem overhead

As a result, storage performance does not scale proportionally with processor performance.

For example, replacing a 16-core desktop processor with a 64-core workstation processor may dramatically increase Compute Score.

The same system may produce only a relatively small improvement in Storage Score if both systems use comparable high-end NVMe devices.

Including Storage directly in the Compute Score would therefore distort the interpretation of overall system performance.

A workstation with substantially greater computational capability could appear artificially limited by storage hardware that has already approached the practical limits of current desktop technology.

For this reason, Storage is evaluated independently.

This allows both performance domains to evolve naturally without influencing each other.

• Compute Score represents computational capability.
• Storage Score represents filesystem and storage capability.

Both values remain equally important while preserving meaningful comparisons across different hardware generations.

Storage Reference Calibration

Storage references are calibrated independently from Compute references.

Every Storage workload defines deterministic reference throughput representing the target performance for a specific workload configuration.

Reference values are normalized to approximately ten thousand points on the benchmark reference platform.

Unlike Compute categories, Storage references are intentionally expected to evolve much more slowly.

Modern flagship NVMe devices already approach the practical bandwidth limits available on consumer desktop platforms.

Consequently, future recalibration of Storage references is expected to occur far less frequently than Compute reference updates.

This approach preserves long-term benchmark stability while allowing gradual adaptation as storage technology evolves.

Practical Interpretation

Storage Score should be interpreted independently from Compute Score.

High Storage scores generally indicate:

• Fast storage hardware
• Efficient filesystem implementation
• Low synchronization overhead
• Strong metadata performance
• Excellent small-file behavior
• Efficient operating system storage stack

Applications that frequently benefit from high Storage Score include:

• Software compilation
• Backup software
• Package managers
• Artifact repositories
• Databases
• Source code repositories
• Virtualization platforms
• Container runtimes
• Continuous integration systems

Because Storage Score reflects practical filesystem behavior rather than theoretical bandwidth, it often provides a much better indicator of real application performance than synthetic storage benchmarks.

Design Summary

The Storage benchmark evaluates the practical behavior of modern storage systems using deterministic workloads representative of real software engineering environments.

Rather than measuring only sequential bandwidth or synthetic IOPS, the benchmark evaluates multiple independent filesystem workloads including sequential transfers, random access, metadata operations, directory traversal, synchronization, and small file processing.

The resulting Storage Score therefore represents the complete efficiency of the storage subsystem experienced by real applications.

Unlike Compute Score, Storage Score is intentionally evaluated as an independent performance domain.

This separation ensures that future processor generations may continue to increase computational capability without artificially influencing storage evaluation.

Together, Compute Score and Storage Score provide a balanced representation of modern workstation performance while preserving meaningful comparisons across different hardware architectures, operating systems, and storage technologies.

Scoring System

The primary purpose of the Scalionix scoring system is to convert heterogeneous benchmark measurements into a single normalized performance model while preserving the characteristics of each individual workload.

Different benchmark categories naturally produce different measurement units.

For example:

• JSON measures bytes processed per second.
• Hashing measures bytes processed per second.
• Encryption measures bytes processed per second.
• Compression measures bytes processed per second.
• Collections measure operations per second.
• Concurrency measures synchronization operations per second.
• Compilation measures completed software builds per second.
• Storage measures bytes per second, files per second, or filesystem operations per second depending on the workload.

Because these measurements cannot be compared directly, every workload is normalized against deterministic reference values.

The resulting normalized scores allow workloads with fundamentally different units to participate in a common scoring model while preserving their relative performance.

Design Goals

The scoring system has several primary design goals.

• Platform independence.
• Deterministic score generation.
• Stable long-term comparisons.
• Hardware-independent normalization.
• Future extensibility.
• Resistance to measurement noise.

Every benchmark category follows exactly the same scoring pipeline regardless of the underlying workload.

Scoring Pipeline

Every benchmark workload passes through the same normalization process.

            Raw Benchmark Result
                     │
                     ▼
             Reference Matching
                     │
                     ▼
            Throughput Normalization
                     │
                     ▼
             Single / Multi Score
                     │
                     ▼
             Scaling Score
                     │
                     ▼
             Weighted Combination
                     │
                     ▼
             Workload Score
                     │
                     ▼
             Scenario Score
                     │
                     ▼
             Category Score
                     │
                     ▼
             Compute / Storage Score

Every stage performs a specific transformation while preserving deterministic benchmark behavior.

Reference Matching

Every workload produced by the benchmark is matched against exactly one predefined reference entry.

A reference uniquely identifies a benchmark workload using multiple dimensions.

These dimensions include:

• Benchmark category
• Scenario
• Dataset size
• Workload variant

Additional dimensions may be introduced in future scoring versions without affecting existing benchmark reports.

Only successfully matched workloads participate in score calculation.

Unmatched workloads are reported separately through benchmark diagnostics.

Normalization

After a matching reference has been located, the measured throughput is normalized against the reference throughput.

The benchmark supports two normalization directions.

Higher Is Better

Used by workloads where larger throughput indicates better performance.

Examples include:

• Serialization
• Hashing
• Encryption
• Compilation
• Storage throughput

Lower Is Better

Used by workloads where lower execution time indicates better performance.

Although the current benchmark primarily uses throughput-based measurements, the scoring engine supports both normalization models.

Reference Platform

All benchmark references are calibrated against a predefined reference workstation.

The current reference platform consists of:

Processor

AMD Ryzen 9 7950X3D

Memory

64 GB Kingston Fury Renegade DDR5-6400 CL32

Storage

Samsung 9100 Pro 1 TB PCIe Gen5 NVMe SSD

Reference values are calibrated so that the reference workstation produces approximately ten thousand points across benchmark categories.

This calibration simplifies long-term interpretation while preserving proportional score scaling.

Future reference platforms may be introduced through new scoring versions without invalidating historical benchmark reports.

Workload Scores

Every successfully executed workload produces up to three normalized scores.

Single-thread Score

Measures workload performance using one execution worker.

Multi-thread Score

Measures the highest throughput achieved across all evaluated worker configurations.

Scaling Score

Measures parallel execution efficiency relative to single-thread execution.

These three values describe different aspects of workload behavior.

Some benchmark categories intentionally omit one or more of these values when they are not applicable.

Compilation, for example, evaluates only native compiler parallelism and therefore contributes exclusively through its Multi-thread Score.

Weighted Combination

Workload scores are combined using weighted geometric averaging.

The default scoring weights are:

Single-thread

35%

Multi-thread

60%

Scaling

5%

This weighting intentionally prioritizes sustained production throughput while still rewarding efficient single-thread execution and parallel scalability.

Using a geometric mean instead of an arithmetic mean prevents one exceptionally high metric from completely masking weaknesses in another metric.

As a result, balanced hardware configurations receive more representative overall scores.

Scenario Scores

Many benchmark scenarios execute multiple workload configurations.

Examples include:

• Multiple dataset sizes
• Different block sizes
• Multiple object counts
• Compression variants
• Durability modes

All workloads belonging to the same scenario are aggregated into a single Scenario Score.

This aggregation produces stable scenario-level results while reducing sensitivity to individual workload variance.

Category Scores

Scenario scores are aggregated into Category Scores.

Each category therefore produces one deterministic normalized score representing the practical performance of that benchmark domain.

Category scores also include detailed diagnostics describing:

• Executed workloads
• Matched references
• Rejected workloads
• Missing references
• Successfully scored workloads

These diagnostics simplify benchmark validation and future reference calibration.

Compute Score

All compute-oriented benchmark categories contribute to the Compute Score.

These currently include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation

Each category contributes using predefined category weights.

The resulting Compute Score represents the computational capability of the evaluated system independently from storage performance.

Storage Score

Storage workloads contribute exclusively to the Storage Score.

Storage intentionally remains independent from Compute because storage technology scales fundamentally differently from processor performance.

This separation preserves meaningful long-term comparisons across hardware generations.

The rationale behind this architectural decision was discussed in Chapter 6.

Scoring Versioning

Every benchmark report includes a scoring version.

Scoring versions define:

• Reference values
• Workload calibration
• Category aggregation
• Normalization behavior

Historical benchmark reports remain fully reproducible because the scoring version is stored together with every generated benchmark report.

Future scoring revisions therefore do not invalidate historical benchmark data.

Deterministic Results

The scoring system has been designed to produce deterministic results.

Given identical benchmark execution and identical scoring references, the generated scores will always be identical.

This property is essential for:

• Historical comparisons
• Regression analysis
• Hardware validation
• Public benchmark rankings
• Long-term performance tracking

Design Summary

The Scalionix scoring system transforms heterogeneous benchmark measurements into a unified performance model through deterministic reference normalization.

By combining workload normalization, scenario aggregation, category scoring, weighted geometric averaging, and versioned reference calibration, the scoring engine produces stable, reproducible, and meaningful benchmark scores across a wide variety of hardware platforms.

The resulting Compute Score and Storage Score provide independent yet complementary views of modern system performance, allowing accurate comparison of computational capability and storage subsystem efficiency without conflating fundamentally different performance domains.

Reference Calibration

The Scalionix System Benchmark uses a deterministic reference calibration system to transform raw benchmark measurements into normalized performance scores.

Rather than comparing benchmark results against other benchmark executions, every workload is evaluated against predefined reference values.

This approach guarantees that benchmark scores remain deterministic, reproducible, and comparable across different hardware platforms, operating systems, benchmark executions, and future benchmark releases.

Reference calibration represents one of the fundamental design principles of the Scalionix System Benchmark.

Purpose

Reference calibration has several primary objectives.

• Normalize heterogeneous benchmark measurements.
• Produce stable benchmark scores.
• Eliminate dependence on historical benchmark data.
• Enable deterministic score generation.
• Preserve long-term compatibility.

Without deterministic references, benchmark scores would continuously change as newer hardware becomes available.

Instead, every workload is evaluated relative to a fixed calibration baseline.

Reference Philosophy

Every benchmark workload has exactly one reference.

References describe the expected performance of a predefined benchmark platform for a specific workload configuration.

Examples include:

            JSON serialization
                    │
                    ▼
            64 KiB dataset
                    │
                    ▼
              100 objects
                    │
                    ▼
            Reference throughput

            Storage random read
                    │
                    ▼
              4 KiB blocks
                    │
                    ▼
              1 GiB dataset
                    │
                    ▼
           Reference throughput

               Compilation
                    │
                    ▼
              Typst Release
                    │
                    ▼
         Reference build duration

Rather than comparing benchmark executions with each other, every benchmark execution compares itself against these deterministic reference values.

Reference Platform

All benchmark references are calibrated using a predefined reference workstation.

Current reference platform:

Processor

AMD Ryzen 9 7950X3D

Memory

64 GB DDR5-6400

Storage

Samsung 9100 Pro PCIe Gen5 NVMe SSD

Operating System

Linux

This hardware configuration was selected because it represents a modern high-performance desktop workstation while remaining broadly accessible.

Future benchmark releases may introduce additional calibration platforms through new scoring versions.

Why Approximately 10,000 Points?

Benchmark scores are intentionally normalized around approximately ten thousand points.

This decision was made for several reasons.

Human Readability

Values around ten thousand are easy to interpret.

Room for Future Hardware

Future processors may naturally exceed the reference score without requiring score redesign.

Regression Visibility

Small performance changes remain immediately visible.

Long-Term Stability

The score range remains meaningful across multiple hardware generations.

Ten thousand should therefore be interpreted as a calibration target rather than an absolute upper limit.

Benchmark scores are expected to exceed this value as hardware evolves.

Reference Dimensions

Every reference describes a unique workload.

Current reference dimensions include:

• Benchmark category
• Scenario
• Workload variant
• Dataset size

Additional dimensions may be introduced in future scoring versions whenever benchmark complexity increases.

Only workloads matching all required dimensions participate in score calculation.

Dataset Factors

Many benchmark scenarios execute multiple dataset sizes.

Rather than defining completely independent reference values for every dataset, references are calibrated using deterministic scaling factors.

Examples include:

            64 KiB
               │
               ▼
            Reference Factor
            
            256 KiB
               │
               ▼
            Reference Factor
            
            1 MiB
               │
               ▼
            Reference Factor

This approach significantly reduces reference duplication while preserving deterministic score calculation.

Workload Variant Factors

Some benchmark scenarios execute multiple workload variants.

Examples include:

• Compression algorithms
• Storage block sizes
• Storage durability modes
• Compilation fixtures

Rather than duplicating reference tables, workload-specific calibration factors are applied.

This simplifies future benchmark maintenance while preserving scoring consistency.

Worker Configuration Calibration

Many benchmark workloads execute using multiple worker configurations.

The benchmark evaluates every supported worker count.

Examples include:

• 1
• 2
• 4
• 8
• 16
• 32

Only the highest measured throughput contributes to the Multi-thread Score.

Worker calibration therefore evaluates scalability rather than absolute processor utilization.

Category Calibration

Every benchmark category maintains an independent reference table.

Examples include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Independent calibration allows categories to evolve separately without affecting the remainder of the scoring system.

Calibration Updates

Reference calibration intentionally evolves slowly.

References are updated only when required.

Typical reasons include:

• Major benchmark redesign
• New benchmark categories
• Revised benchmark methodology
• New calibration platform

Minor benchmark implementation improvements do not automatically require recalibration.

This preserves historical benchmark stability.

Scoring Version

Every reference table belongs to a specific scoring version.

The scoring version defines:

• Reference values
• Dataset factors
• Workload factors
• Category aggregation
• Normalization behavior

Historical benchmark reports always record the scoring version used during benchmark execution.

This guarantees reproducibility.

Calibration Independence

Reference calibration is independent of benchmark execution.

Benchmark execution measures performance.

Reference calibration interprets performance.

Separating these responsibilities provides several advantages.

• Benchmark implementation may evolve.
• Scoring may evolve independently.
• Historical benchmark reports remain valid.
• Score recalculation becomes possible.

This separation represents one of the core architectural decisions of the benchmark.

Practical Maintenance

When introducing a new benchmark category, the recommended workflow is:

            Implement Benchmark
                    │
                    ▼
            Execute Benchmark on Reference Platform
                    │
                    ▼
            Collect Benchmark Measurements
                    │
                    ▼
            Create Reference Entries
                    │
                    ▼
            Verify Diagnostics
                    │
                    ▼
            Publish New Scoring Version

Following this workflow guarantees that every benchmark category participates consistently in the scoring engine.

Long-Term Evolution

The reference system has been designed to support long-term benchmark evolution.

Future benchmark releases may introduce:

• Additional workload dimensions
• Additional benchmark categories
• Revised calibration platforms
• Alternative reference hardware

Existing benchmark reports remain compatible because scoring versions preserve historical reference tables.

Design Summary

The Reference Calibration system transforms benchmark measurements into deterministic, normalized scores using predefined workload references rather than historical benchmark comparisons.

By combining reference platforms, workload dimensions, dataset scaling, workload variants, scoring versioning, and independent category calibration, the benchmark achieves stable, reproducible, and future-proof score generation while preserving long-term compatibility across benchmark releases and hardware generations.

Overview

Full Benchmark Report

The Full Benchmark Report represents the complete output of benchmark execution.

It contains every successfully executed benchmark request together with all measured performance metrics, execution metadata, verification results, benchmark configuration, hardware information, and execution statistics.

Unlike the remaining report formats, the Full Benchmark Report intentionally preserves all benchmark execution details.

This report serves as the authoritative source from which every other benchmark report is generated.

For this reason, the Full Benchmark Report should always be preserved after benchmark execution.

Purpose

The Full Benchmark Report has several primary purposes.

• Preserve complete benchmark execution.
• Allow future score recalculation.
• Enable historical comparison.
• Support benchmark diagnostics.
• Provide reproducible benchmark data.

Unlike compact score reports, the Full Benchmark Report contains enough information to regenerate scoring results using future scoring versions.

Report Generation

The report is generated immediately after benchmark execution completes successfully.

Every executed benchmark request contributes one or more benchmark records.

Failed benchmark requests are also recorded together with failure diagnostics.

The report therefore represents the exact execution state of the benchmark.

Report Structure

The report consists of several logical sections.

General Information

Contains benchmark identifiers and execution metadata.

Hardware Snapshot

Describes the hardware configuration used during benchmark execution.

Benchmark Configuration

Contains benchmark parameters supplied during execution.

Category Results

Contains raw execution results for every benchmark category.

Execution Statistics

Contains execution counters and timing information.

Failures

Contains benchmark failures together with diagnostic information.

Together these sections completely describe one benchmark execution.

General Information

The report begins with general benchmark metadata.

Typical information includes:

• Report identifier
• User identifier
• Benchmark version
• Scoring version
• Execution timestamp
• Operating system
• Architecture
• Benchmark duration

These values uniquely identify the benchmark execution.

Hardware Snapshot

The complete detected hardware configuration is embedded directly inside the report.

The hardware snapshot contains:

Processor

• Processor model
• Physical cores
• Logical processors

Memory

• Installed capacity
• Available memory
• Memory modules
• Manufacturer
• Frequency
• ECC information

Storage

• Storage devices
• Firmware versions
• Benchmark storage device

Operating System

• Operating system
• Architecture
• Hostname

Hardware Fingerprint

• Deterministic hardware identifier

Embedding the hardware snapshot directly into the report guarantees long-term reproducibility.

Benchmark Configuration

The report records every benchmark parameter used during execution.

Examples include:

• Executed benchmark categories
• Worker configurations
• Benchmark directory
• Report identifier
• User identifier
• Measured iterations
• Warmup iterations
• Compilation environment

Recording benchmark configuration allows historical benchmark executions to be reproduced exactly.

Benchmark Results

The largest section of the report contains raw benchmark execution results.

Every benchmark category records its own execution format.

Examples include:

JSON

• Throughput
• Object counts
• Dataset sizes

Hashing

• Processed bytes
• Algorithm
• Throughput

Encryption

• Processed bytes
• Algorithm
• Verification

Compilation

• Build duration
• Compiler
• Build profile
• Build verification

Storage

• Throughput
• Operations
• Block sizes
• Filesystem configuration

Raw benchmark results remain intentionally category-specific.

No normalization is performed inside the Full Benchmark Report.

Verification Results

Every benchmark execution records verification status.

Verification includes:

• Execution success
• Validation status
• Verification failures
• Diagnostic messages

Only verified benchmark executions participate in score generation.

Failures remain preserved for diagnostic purposes.

Execution Statistics

Execution statistics summarize benchmark progress.

Typical statistics include:

• Planned requests
• Executed requests
• Successful requests
• Failed requests
• Planned scenarios
• Successful scenarios
• Failed scenarios
• Total benchmark duration

These values provide a quick overview of benchmark execution quality.

Timing Information

Execution timing is recorded both globally and per benchmark category.

Examples include:

• Total execution duration
• JSON benchmark duration
• Hashing duration
• Encryption duration
• Compilation duration
• Storage duration

Recording execution timing allows benchmark users to identify which benchmark categories dominate overall execution time.

Failure Information

If benchmark execution encounters failures, diagnostic information is preserved.

Typical failure information includes:

• Benchmark category
• Scenario
• Request parameters
• Failure reason
• Verification status
• Execution directory

Failures never prevent successful benchmark results from being preserved.

Instead, they become part of the benchmark history for later investigation.

Relationship to Other Reports

The Full Benchmark Report acts as the source for every remaining benchmark report.

            Full Benchmark Report
                     │
                     ├────────► Scoring Report
                     │
                     ├────────► Global Score Report
                     │
                     └────────► Benchmark Comparison

No additional benchmark execution is required to generate these reports.

Every subsequent report is derived exclusively from the Full Benchmark Report.

Long-Term Compatibility

The Full Benchmark Report has been designed for long-term compatibility.

Future benchmark releases may introduce:

• Additional benchmark categories
• Additional workload parameters
• New scoring versions
• New diagnostics

Historical reports remain valid because benchmark execution data is preserved independently from score calculation.

This design allows benchmark scores to be recalculated in the future without repeating benchmark execution.

Design Summary

The Full Benchmark Report represents the complete historical record of one benchmark execution.

It preserves benchmark configuration, hardware information, execution results, diagnostics, verification status, timing information, and execution statistics in a deterministic and versioned format.

Because every remaining benchmark report is derived from this document, the Full Benchmark Report serves as the authoritative source of benchmark information and should always be preserved for future analysis, comparison, and score recalculation.

Scoring Report

The Scoring Report contains the complete output of the benchmark scoring engine.

Unlike the Full Benchmark Report, which preserves raw execution results, the Scoring Report contains normalized benchmark scores generated from those execution results.

This report represents the complete scoring state of a benchmark execution.

Every workload score, scenario score, category score, normalization result, reference match, and scoring diagnostic is preserved.

The Scoring Report serves as the primary document for benchmark analysis and score validation.

Purpose

The Scoring Report has several primary objectives.

• Convert raw benchmark measurements into normalized scores.
• Preserve workload-level scoring information.
• Record benchmark diagnostics.
• Preserve reference matching information.
• Enable future score validation.

Unlike the Global Score Report, this report contains every intermediate scoring result.

Report Generation

The report is generated immediately after benchmark execution completes.

The scoring engine processes every successfully verified benchmark workload.

Each workload is matched against its corresponding reference before normalized scores are calculated.

Once all workloads have been processed, the scoring engine generates:

• Workload scores
• Scenario scores
• Category scores
• Compute Score
• Storage Score
• Scoring diagnostics

The resulting document completely describes how every benchmark score was calculated.

Report Structure

The report consists of several logical sections.

General Information

Contains benchmark metadata.

Compute Score

Overall compute performance.

Storage Score

Independent storage performance.

Benchmark Categories

Normalized scores for every benchmark category.

Scenario Scores

Aggregated scenario-level scores.

Workload Scores

Individual normalized benchmark workloads.

Diagnostics

Reference matching and scoring statistics.

Together these sections describe the complete scoring process.

Compute Score

The Compute Score represents the aggregated performance of all compute-oriented benchmark categories.

Current compute categories include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation

Every category contributes according to its configured category weight.

Category aggregation is performed using the weighted geometric mean described in Chapter 7.

The Compute Score intentionally excludes every Storage workload.

Storage Score

Storage workloads are aggregated independently from Compute.

Storage Score includes:

• Sequential I/O
• Random I/O
• Small Files
• Metadata
• Directory Walk
• Synchronization

The Storage Score represents the overall practical efficiency of the storage subsystem.

Its calculation follows the same normalization rules as compute workloads while remaining completely independent from Compute Score.

Category Scores

Each benchmark category produces one normalized category score.

A category contains one or more benchmark scenarios.

Examples include:

            JSON
             │
             ▼
            Serialization
             │
             ▼
            Workloads

            Storage
             │
             ▼
            Sequential Read
             │
             ▼
            Workloads

            Compilation
             │
             ▼
            Typst Release
             │
             ▼
            Workload

Category scores therefore represent aggregated performance across all benchmark scenarios belonging to the same benchmark domain.

Scenario Scores

Every benchmark scenario contains one or more benchmark workloads.

Examples include:

JSON Serialization

• 64 KiB
• 256 KiB
• 1 MiB
• Multiple object counts

Storage Random Read

• Multiple dataset sizes
• Multiple block sizes

Compilation Typst

• Production build

Scenario aggregation combines these workload scores into one representative Scenario Score.

This reduces benchmark noise while preserving workload diversity.

Workload Scores

Workloads represent the smallest scoring unit.

Every workload records:

• Benchmark parameters
• Matched reference
• Measured throughput
• Normalized throughput
• Single-thread Score
• Multi-thread Score
• Scaling Score
• Combined Score
• Best worker configuration

These values represent the complete normalization result for one benchmark workload.

Diagnostics

Every benchmark category records detailed diagnostics.

Diagnostics include:

Input Samples

Number of benchmark measurements received.

Workload Groups

Number of workload groups created before scoring.

Matched References

Successfully matched reference entries.

Missing References

Workloads that could not be matched.

Successfully Scored Workloads

Workloads contributing to the final score.

Rejected Workloads

Measurements rejected by the scoring engine.

Missing Reference Keys

Complete identifiers describing unmatched workloads.

Rejected Workload Keys

Identifiers of workloads intentionally rejected.

These diagnostics greatly simplify benchmark maintenance and future reference calibration.

Reference Matching

Before a workload can be scored, the scoring engine must locate an appropriate reference.

Matching currently considers multiple workload dimensions.

Typical matching dimensions include:

• Benchmark category
• Scenario name
• Workload variant
• Dataset size

Additional dimensions may be introduced in future scoring versions without affecting historical benchmark reports.

Workloads without matching references are excluded from score calculation while remaining visible through diagnostics.

Worker Analysis

For workloads supporting multiple execution workers, the scoring engine records the best-performing worker configuration.

Examples include:

[16w → 9]

indicating that all nine evaluated workloads achieved their maximum throughput using sixteen execution workers.

This information provides valuable insight into scalability characteristics of the evaluated hardware.

Category Diagnostics

Every category maintains completely independent diagnostics.

This allows benchmark developers to identify problems affecting one category without influencing benchmark analysis elsewhere.

Examples include:

• Missing references affecting only Storage.
• Rejected workloads affecting only Compression.
• Verification failures affecting only Compilation.

Independent diagnostics significantly simplify benchmark evolution as additional benchmark categories are introduced.

Relationship to Other Reports

The Scoring Report is generated directly from the Full Benchmark Report.

The relationship between reports is illustrated below.

            Full Benchmark Report
                       │
                       ▼
                 Scoring Engine
                       │
                       ▼
                 Scoring Report
                       │
                       ├────────► Global Score Report
                       │
                       └────────► Benchmark Comparison

Unlike the Full Benchmark Report, the Scoring Report contains only normalized benchmark information.

Practical Applications

The Scoring Report is intended primarily for:

• Benchmark developers
• Hardware reviewers
• Benchmark validation
• Reference calibration
• Performance analysis
• Regression testing
• Future scoring revisions

It represents the most detailed description of benchmark scoring available within the Scalionix System Benchmark.

Long-Term Compatibility

The report has been designed for long-term compatibility.

Future benchmark releases may introduce:

• Additional benchmark categories
• Additional workload parameters
• New diagnostics
• Revised scoring models

Because every report includes explicit scoring version information, historical benchmark reports remain reproducible.

Design Summary

The Scoring Report represents the complete output of the benchmark normalization engine.

It records workload scores, scenario scores, category scores, Compute Score, Storage Score, diagnostics, reference matching, and benchmark aggregation in a deterministic and versioned format.

Together with the Full Benchmark Report, it provides a complete description of benchmark execution and score generation, allowing future score recalculation, regression analysis, benchmark validation, and long-term performance tracking.

Global Score Report

The Global Score Report is the public representation of a benchmark execution.

Unlike the Full Benchmark Report and the Scoring Report, which contain extensive execution details, the Global Score Report intentionally contains only the information required for benchmark comparison, historical tracking, and future submission to the Scalionix Global Ranking Service.

Its primary purpose is to provide a compact, deterministic, and versioned description of benchmark performance while remaining independent from benchmark implementation details.

The Global Score Report is expected to become the standard exchange format for benchmark results.

Purpose

The Global Score Report has several primary objectives.

• Represent benchmark performance using a compact format.
• Enable public benchmark comparison.
• Support historical benchmark tracking.
• Serve as the upload format for the Global Ranking Service.
• Preserve benchmark compatibility across future scoring versions.

Unlike the Full Benchmark Report, this report intentionally excludes raw benchmark measurements.

Design Philosophy

The report has been designed with portability as the primary objective.

A benchmark result should be easily transferable between users without requiring gigabytes of benchmark history.

Consequently, the Global Score Report contains only normalized benchmark information.

The report should remain sufficiently small to:

• Attach to issue reports
• Publish on GitHub
• Store inside version control
• Upload to online ranking services
• Archive for long-term historical comparison

Compactness and long-term compatibility were considered fundamental design requirements.

Report Structure

The report consists of several logical sections.

Benchmark Identity

Identifies the benchmark execution.

Benchmark Summary

Provides execution statistics.

Hardware Summary

Describes the benchmarked machine.

Compute Score

Overall computational performance.

Storage Score

Overall storage subsystem performance.

Category Scores

Normalized benchmark category scores.

Scoring Metadata

Contains scoring version and compatibility information.

Each section has been intentionally kept compact while preserving enough information for meaningful benchmark comparison.

Benchmark Identity

Every report contains benchmark identity information.

Typical fields include:

• Report identifier
• User identifier
• Benchmark version
• Scoring version
• Execution timestamp

These values uniquely identify one benchmark execution.

Hardware Summary

Rather than storing the complete hardware snapshot, the Global Score Report stores only the information required for hardware identification.

Typical information includes:

Processor

• Processor model
• Physical core count
• Logical processor count

Memory

• Installed capacity

Storage

• Benchmark storage device

Operating System

• Operating system
• Architecture

Hardware Fingerprint

• Deterministic hardware identifier

This compact summary allows future ranking services to group comparable systems while minimizing report size.

Benchmark Summary

The report contains a concise summary of benchmark execution.

Typical information includes:

• Executed benchmark count
• Successful scenarios
• Failed scenarios
• Total benchmark duration
• Benchmark completion status

This allows ranking services to validate benchmark completeness before accepting uploaded scores.

Compute Score

The Compute Score represents the aggregated computational capability of the evaluated system.

It is calculated exclusively from compute-oriented benchmark categories.

Current contributors include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation

The Compute Score intentionally excludes every Storage workload.

Storage Score

The Storage Score represents the practical performance of the storage subsystem.

It includes:

• Sequential I/O
• Random I/O
• Metadata
• Small Files
• Synchronization
• Directory Traversal

Storage Score is intentionally reported independently from Compute Score.

This separation preserves meaningful comparisons across future processor generations while accurately representing storage performance.

Category Scores

Every benchmark category contributes an independent normalized score.

Typical category entries include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Publishing category scores allows users to compare specific workload domains rather than relying exclusively on a single overall score.

Scoring Metadata

The report records all information required for score compatibility.

This includes:

• Scoring version
• Benchmark version
• Category count
• Reference compatibility

Scoring metadata allows future ranking services to reject incompatible benchmark reports while preserving historical benchmark archives.

Intended Usage

The Global Score Report is intended for:

• Public benchmark sharing
• Hardware comparison
• Historical performance tracking
• Online benchmark ranking
• Workstation validation
• Continuous integration performance monitoring

It intentionally excludes benchmark implementation details that are unnecessary for public comparison.

Relationship to Other Reports

The Global Score Report represents the final stage of benchmark processing.

The report hierarchy is illustrated below.

      Full Benchmark Report
                │
                ▼
         Scoring Report
                │
                ▼
       Global Score Report
                │
                ▼
   Global Ranking Submission

Unlike previous reports, the Global Score Report is specifically designed for external distribution.

Public Ranking Compatibility

The report has been designed specifically for future integration with the Scalionix Global Ranking Service.

Expected server-side operations include:

• Report validation
• Scoring version verification
• Hardware classification
• Duplicate detection
• Public leaderboard generation
• Historical benchmark storage

Because the report is deterministic and versioned, ranking services may reliably compare benchmark results generated years apart while preserving compatibility.

Long-Term Stability

The Global Score Report is expected to remain stable across benchmark releases.

Future benchmark versions may introduce:

• Additional benchmark categories
• Revised reference values
• New scoring versions
• Extended hardware summaries

Existing reports remain valid because every benchmark execution explicitly records its scoring version.

This guarantees that historical benchmark results never become ambiguous.

Design Summary

The Global Score Report represents the canonical public benchmark format produced by the Scalionix System Benchmark.

It combines compactness, deterministic scoring, hardware identification, benchmark metadata, and long-term compatibility into a single portable document suitable for public sharing, historical tracking, and future submission to the Scalionix Global Ranking Service.

Unlike the Full Benchmark Report and the Scoring Report, the Global Score Report intentionally contains only the information necessary for meaningful benchmark comparison, making it the preferred format for benchmark exchange between users and automated systems.

Hardware Snapshot

Every benchmark execution includes a complete Hardware Snapshot describing the system on which the benchmark was executed.

Unlike many benchmark tools that record only the processor model, the Scalionix System Benchmark captures detailed information about the processor, memory subsystem, storage devices, operating system, and benchmark storage location.

The Hardware Snapshot provides the contextual information necessary to correctly interpret benchmark results and compare different hardware configurations.

It also serves as the foundation for future hardware classification and public benchmark ranking.

Purpose

The Hardware Snapshot has several primary objectives.

• Preserve complete hardware information.
• Associate benchmark results with the executed machine.
• Enable hardware comparison.
• Support benchmark reproducibility.
• Generate deterministic hardware fingerprints.

Unlike benchmark scores, which may evolve with future scoring versions, the Hardware Snapshot describes the physical execution environment.

Snapshot Generation

Hardware detection is performed before benchmark execution begins.

The benchmark inspects the operating system together with available hardware interfaces to collect system information.

Operating-system-specific collectors are used whenever additional information can be obtained beyond standard runtime APIs.

The resulting snapshot is embedded into every generated report.

Snapshot Structure

The Hardware Snapshot consists of several logical sections.

General System Information

Processor

Memory

Storage Devices

Benchmark Storage

Operating System

Hardware Fingerprint

Identity Confidence

Each section contributes to a complete description of the benchmark platform.

General System Information

General system information identifies the machine on which the benchmark executed.

Typical fields include:

• Collection timestamp
• Hostname
• Operating system
• Processor architecture

This information uniquely identifies the execution environment independently from benchmark results.

Processor Information

The benchmark records processor characteristics required for benchmark interpretation.

Typical processor information includes:

• Processor model
• Vendor
• Physical core count
• Logical processor count
• Processor architecture

These values are used throughout benchmark execution when generating benchmark matrices and worker configurations.

Future benchmark releases may extend processor detection with additional information such as cache hierarchy and processor frequencies.

Memory Information

The benchmark records both runtime memory information and physical memory configuration whenever available.

Typical information includes:

Runtime Information

• Installed memory
• Available memory

Physical Module Information

• Module count
• Manufacturer
• Part number
• Memory technology
• Configured frequency
• Rated frequency
• Operating voltage
• ECC capability
• Form factor

Whenever supported by the operating system, memory information is collected directly from firmware tables instead of relying solely on runtime APIs.

This approach provides significantly more detailed hardware identification.

Storage Device Information

The benchmark identifies storage devices participating in benchmark execution.

Typical information includes:

• Storage model
• Firmware version
• Storage technology
• Operating system device identifier
• Filesystem mount point

Current storage technologies include:

• NVMe
• SATA SSD
• HDD

Additional storage technologies may be introduced in future benchmark releases.

Benchmark Storage

One storage device receives special treatment.

The benchmark explicitly identifies the storage location where benchmark datasets are created.

Typical information includes:

• Requested benchmark path
• Resolved filesystem path
• Storage device
• Mount point

This distinction is important because benchmark execution may occur on a storage device different from the operating system installation.

The recorded benchmark storage location therefore represents the device whose performance contributes to the Storage Score.

Operating System Information

The Hardware Snapshot records the execution platform.

Typical information includes:

• Operating system
• System architecture
• Kernel information

Future benchmark releases may record additional operating-system-specific information as required.

Hardware Fingerprint

The benchmark generates a deterministic hardware fingerprint.

The fingerprint is derived from stable hardware characteristics rather than temporary runtime values.

Its purpose is not to uniquely identify a person or a computer.

Instead, it provides a stable identifier representing the benchmark hardware configuration.

Typical applications include:

• Historical benchmark tracking
• Duplicate detection
• Workstation comparison
• Future online ranking services

Minor runtime changes do not affect the generated fingerprint.

Identity Confidence

Not every operating system exposes identical hardware information.

For this reason, the benchmark records an Identity Confidence value.

Identity Confidence represents how reliably the benchmark could identify the underlying hardware configuration.

Higher confidence indicates that more hardware characteristics were successfully detected.

Lower confidence may occur on operating systems with limited firmware access or restricted hardware permissions.

This value assists future ranking services when comparing hardware fingerprints.

Operating System Differences

Hardware detection varies between operating systems.

Linux

Provides the most detailed hardware information through firmware interfaces and device metadata.

macOS

Provides extensive processor and storage information using native system APIs.

Windows

Provides hardware information through Windows system interfaces while respecting operating system security restrictions.

Whenever platform-specific information is unavailable, benchmark execution continues using the best information available.

Benchmark Independence

Hardware detection operates independently from benchmark execution.

The Hardware Snapshot is generated regardless of which benchmark categories are executed.

Consequently, even partial benchmark executions preserve complete hardware information.

This design ensures consistent benchmark reports across different benchmark configurations.

Practical Applications

The Hardware Snapshot supports numerous practical use cases.

• Benchmark comparison
• Workstation inventory
• Regression analysis
• Benchmark reproducibility
• Hardware validation
• Online ranking
• Historical benchmark archives

Because hardware information is stored together with benchmark scores, benchmark reports remain meaningful even many years after benchmark execution.

Design Summary

The Hardware Snapshot represents a complete description of the benchmark execution platform.

By combining processor information, memory configuration, storage devices, operating system details, benchmark storage location, deterministic hardware fingerprints, and identity confidence into a single structured document, the benchmark preserves all information necessary for accurate interpretation, comparison, and long-term archival of benchmark results.

The Hardware Snapshot therefore serves as the foundation for reliable hardware identification throughout the Scalionix System Benchmark ecosystem.

Diagnostics

The Scalionix System Benchmark records detailed diagnostic information during score generation.

Unlike benchmark scores, which describe benchmark performance, diagnostics describe the behavior of the scoring engine itself.

Diagnostics are intended primarily for benchmark developers, benchmark maintainers, reference calibration, regression analysis, and future benchmark evolution.

They provide complete visibility into how benchmark workloads were processed, matched, scored, rejected, or excluded from score calculation.

Diagnostics never influence benchmark scores directly.

Instead, they explain how benchmark scores were produced.

Purpose

The Diagnostics subsystem has several primary objectives.

• Validate benchmark execution.
• Verify reference completeness.
• Detect missing benchmark references.
• Detect rejected benchmark workloads.
• Simplify future benchmark maintenance.
• Support reference calibration.

Rather than silently ignoring inconsistencies, the benchmark records every important scoring decision.

Diagnostic Generation

Diagnostics are generated automatically during score calculation.

For every benchmark category the scoring engine records:

• Received benchmark samples
• Created workload groups
• Matched references
• Missing references
• Rejected workloads
• Successfully scored workloads

The resulting diagnostic information becomes part of the Scoring Report.

Category Isolation

Diagnostics are collected independently for every benchmark category.

For example:

            JSON
             │
             ▼
            Independent diagnostics
            
            Hashing
             │
             ▼
            Independent diagnostics
            
            Storage
             │
             ▼
            Independent diagnostics
            
            Compilation
             │
             ▼
            Independent diagnostics

Problems affecting one benchmark category never influence diagnostics generated for another category.

This greatly simplifies benchmark maintenance.

Input Samples

Input Samples represent the number of benchmark measurements received by the scoring engine.

Every successfully executed benchmark request contributes one or more input samples.

Input Samples therefore describe the amount of raw benchmark information available before workload grouping begins.

Unexpectedly low sample counts frequently indicate incomplete benchmark execution.

Workload Groups

Multiple benchmark measurements are frequently combined into logical workload groups before scoring.

Examples include:

            Multiple dataset sizes
                      │
                      ▼
                One scenario
            
            Multiple object counts
                      │
                      ▼
               One workload family
            
            Multiple block sizes
                      │
                      ▼
             One storage workload

Workload Groups therefore represent the logical benchmark units processed by the scoring engine.

Matched References

Every workload must be matched against exactly one scoring reference.

Matched References records the total number of successful matches.

A successful match indicates that the scoring engine found a deterministic reference describing the executed workload.

Only successfully matched workloads participate in score calculation.

Missing References

If no matching reference exists, the workload cannot be scored.

Such workloads are recorded as Missing References.

Typical causes include:

• Newly introduced benchmark scenarios
• Modified dataset sizes
• Changed workload variants
• Incomplete reference tables

Missing references never terminate benchmark execution.

Instead, they are reported so that benchmark developers may update reference calibration.

Successfully Scored Workloads

Successfully Scored Workloads records the number of benchmark workloads contributing to the final score.

Only workloads satisfying all of the following conditions participate:

• Benchmark executed successfully
• Verification passed
• Reference successfully matched
• Workload accepted by the scoring engine

This value therefore represents the effective benchmark size after scoring validation.

Rejected Workloads

Some benchmark workloads may be intentionally rejected.

Typical reasons include:

• Invalid benchmark measurements
• Incomplete workload data
• Unsupported benchmark configuration
• Failed verification

Rejected workloads remain visible through diagnostics while being excluded from score calculation.

Missing Reference Keys

For every missing reference the scoring engine records the complete workload identifier.

Typical identifiers contain:

• Scenario name
• Dataset size
• Workload variant

These identifiers uniquely describe the missing workload.

Benchmark developers can therefore generate new reference entries directly from diagnostic output.

This significantly simplifies future benchmark expansion.

Rejected Workload Keys

Rejected benchmark workloads are also recorded individually.

Each rejected workload preserves its complete benchmark identifier.

This allows benchmark maintainers to investigate why a workload was rejected without repeating benchmark execution.

Diagnostic Interpretation

Healthy benchmark execution typically produces diagnostics similar to the following.

Matched References

Equal to the number of workload groups.

Missing References

Zero.

Rejected Workloads

Zero.

Successfully Scored Workloads

Equal to the number of workload groups.

Such diagnostics indicate complete benchmark coverage.

Conversely, increasing numbers of missing references usually indicate that benchmark scenarios have evolved faster than scoring references.

Practical Applications

Diagnostics support numerous maintenance activities.

• Reference calibration
• Benchmark validation
• Regression testing
• Scoring verification
• Benchmark development
• Future benchmark categories

Because diagnostics describe the internal behavior of the scoring engine, they are especially valuable during benchmark evolution.

Long-Term Maintenance

The Diagnostics subsystem has been designed specifically to simplify long-term benchmark maintenance.

As new benchmark categories are introduced, diagnostics immediately identify:

• Workloads requiring references
• Rejected benchmark configurations
• Incomplete benchmark coverage

This allows the benchmark to evolve incrementally without sacrificing scoring reliability.

Design Summary

The Diagnostics subsystem provides complete visibility into the internal operation of the scoring engine.

By recording workload grouping, reference matching, workload acceptance, rejected benchmark measurements, and scoring completeness, diagnostics ensure that every benchmark score can be validated, reproduced, and maintained throughout the lifetime of the Scalionix System Benchmark.

Unlike benchmark scores, diagnostics describe the quality of benchmark processing itself, making them an essential tool for long-term benchmark evolution and reference maintenance.

Benchmark Comparison Report

The Benchmark Comparison Report provides a deterministic comparison between two benchmark executions.

Rather than comparing raw benchmark measurements, the comparison engine evaluates normalized benchmark scores generated by the scoring system.

This approach allows benchmark results produced on different hardware platforms, at different points in time, and under different operating systems to be compared consistently.

The Benchmark Comparison Report is intended for hardware upgrades, regression analysis, benchmark validation, operating system comparisons, and long-term performance tracking.

Purpose

The comparison engine has several primary objectives.

• Compare two benchmark executions.
• Detect performance regressions.
• Quantify hardware improvements.
• Validate benchmark stability.
• Compare operating systems.
• Compare benchmark revisions.

The comparison engine is completely deterministic.

Given identical benchmark reports, the generated comparison will always produce identical results.

Input Reports

Benchmark comparison operates exclusively on Global Score Reports.

Two reports participate in every comparison.

Baseline Report

Represents the original benchmark execution.

Candidate Report

Represents the benchmark execution being evaluated.

Both reports must be generated by compatible scoring versions before comparison begins.

Comparison Pipeline

The comparison process follows a deterministic pipeline.

      Baseline Report
             │
             ▼
     Compatibility Check
             ▲
             │
     Candidate Report
             │
             ▼
     Category Comparison
             │
             ▼
     Score Comparison
             │
             ▼
     Percentage Difference
             │
             ▼
     Comparison Report

Every comparison follows exactly the same sequence.

Compatibility Verification

Before scores are compared, the benchmark verifies report compatibility.

Verification includes:

• Scoring version
• Benchmark format
• Worker configuration
• Benchmark categories
• Report integrity

If incompatible reports are detected, comparison is terminated with a descriptive diagnostic message.

This prevents misleading benchmark comparisons.

Compute Score Comparison

The comparison engine evaluates Compute Scores from both reports.

For every Compute Score comparison the report records:

• Baseline score
• Candidate score
• Absolute difference
• Percentage difference

Positive percentage values indicate performance improvement.

Negative percentage values indicate reduced performance.

Storage Score Comparison

Storage Scores are compared independently from Compute Scores.

The comparison follows exactly the same methodology.

Recorded values include:

• Baseline storage score
• Candidate storage score
• Absolute difference
• Percentage difference

Because Storage Score represents an independent performance domain, its interpretation remains independent from Compute Score.

Category Comparison

Every benchmark category is compared individually.

Current categories include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

Each category comparison records:

• Baseline score
• Candidate score
• Percentage difference

This allows benchmark users to identify precisely which workload domain improved or regressed.

Percentage Difference

Percentage differences are calculated relative to the Baseline Report.

Positive values indicate improvement.

Examples:

+1.24%

Negative values indicate regression.

Examples:

-3.61%

Zero indicates identical benchmark performance.

Percentage differences provide the most intuitive interpretation of benchmark evolution.

Practical Applications

The comparison engine supports numerous practical workflows.

Hardware Upgrade

Evaluate processor, memory, or storage upgrades.

Operating System Comparison

Compare Linux, macOS, and Windows benchmark behavior.

Compiler Evaluation

Measure performance changes resulting from different compiler versions.

Driver Validation

Evaluate storage or chipset driver updates.

Regression Testing

Detect unexpected benchmark regressions.

Continuous Integration

Automatically compare benchmark results across software revisions.

Because benchmark comparison operates entirely on normalized scores, comparisons remain meaningful regardless of the underlying benchmark implementation.

Example Comparison

A typical comparison summary appears similar to the following.

Compute Score

Baseline:
 10234.81

Candidate:
 10562.17

Difference:
 +327.36

Improvement:
 +3.20%

Category comparisons follow the same structure.

This presentation provides an immediate understanding of benchmark evolution.

Comparison Independence

The comparison engine does not require benchmark execution.

Only previously generated benchmark reports are needed.

This allows historical benchmark archives to be compared years after the original benchmark execution without repeating any workloads.

Consequently, benchmark comparison remains extremely fast regardless of the size or duration of the original benchmark.

Long-Term Compatibility

The comparison engine has been designed for long-term compatibility.

As benchmark categories evolve, comparison continues to operate on all categories supported by both benchmark reports.

New benchmark categories introduced in future scoring versions do not invalidate historical benchmark archives.

Compatibility is maintained through explicit scoring versioning and deterministic report structures.

Design Summary

The Benchmark Comparison Report provides a deterministic, reproducible, and version-aware comparison between two benchmark executions.

By comparing normalized benchmark scores rather than raw benchmark measurements, the comparison engine enables meaningful long-term performance tracking across hardware upgrades, operating system changes, compiler revisions, benchmark improvements, and future scoring revisions.

Together with the Full Benchmark Report, Scoring Report, and Global Score Report, the Benchmark Comparison Report completes the reporting subsystem of the Scalionix System Benchmark.

Command Line Interface

The Scalionix System Benchmark is executed entirely through its command-line interface.

The CLI has been designed with several primary objectives.

• Simple execution.
• Predictable behavior.
• Deterministic benchmark configuration.
• Complete automation.
• Easy integration with CI/CD systems.

Every benchmark execution is fully controlled through command-line flags.

No interactive input is required.

This design allows the benchmark to be executed manually, integrated into automated performance testing pipelines, or incorporated into continuous integration environments.

Basic Execution

Executing the complete benchmark requires only a single command.

scalionix-system-benchmark

The benchmark automatically performs the following steps.

• Detect hardware.
• Verify benchmark environment.
• Create benchmark workspace.
• Execute benchmark categories.
• Calculate normalized scores.
• Generate benchmark reports.
• Print execution summary.

When no additional flags are supplied, the benchmark executes using its default configuration.

Execution Pipeline

Every benchmark execution follows the same deterministic workflow.

Command Line
      │
      ▼
Parse Arguments
      │
      ▼
Validate Configuration
      │
      ▼
Detect Hardware
      │
      ▼
Prepare Benchmark
      │
      ▼
Execute Categories
      │
      ▼
Score Calculation
      │
      ▼
Generate Reports
      │
      ▼
Print Summary

Each stage is executed exactly once.

Any fatal error immediately terminates execution with a descriptive diagnostic message.

Command-Line Philosophy

Every command-line flag modifies only one aspect of benchmark execution.

Flags are intentionally independent whenever possible.

This design provides several advantages.

• Predictable execution.
• Simple automation.
• Easy scripting.
• Deterministic behavior.

Benchmark execution never depends on hidden configuration files.

Everything affecting benchmark execution is explicitly visible through the command line.

Output Files

Successful benchmark execution generates multiple output files.

Typical outputs include:

• Full Benchmark Report
• Scoring Report
• Global Score Report

Depending on execution mode, additional reports may also be generated.

The benchmark prints the location of every generated report after execution completes.

Exit Status

The benchmark returns standard operating system exit codes.

Exit code 0

Benchmark completed successfully.

Non-zero exit codes

Benchmark execution failed.

Typical failure reasons include:

• Invalid command-line arguments
• Missing benchmark environment
• Benchmark verification failures
• Filesystem errors
• Unsupported operating system configuration

This behavior allows the benchmark to integrate naturally with shell scripts and automated build systems.

Logging

Benchmark execution continuously reports progress to standard output.

Progress messages include:

• Benchmark category
• Executed scenario
• Worker configuration
• Execution progress
• Benchmark summary

Errors are reported immediately together with sufficient diagnostic information to identify the underlying problem.

Deterministic Execution

Given identical hardware, identical benchmark configuration, and identical benchmark version, the command-line interface always produces identical execution behavior.

No benchmark configuration is inferred implicitly.

Every execution parameter is either supplied explicitly or generated deterministically by the benchmark.

Design Summary

The Scalionix command-line interface provides a deterministic and automation-friendly interface for benchmark execution.

By exposing every important benchmark parameter through explicit command-line flags while avoiding hidden configuration, the CLI remains predictable, reproducible, and suitable for both interactive use and fully automated benchmark environments.

Available Command-Line Flags

The Scalionix System Benchmark exposes all configurable behavior through command-line flags.

Every flag controls a specific aspect of benchmark execution.

Unless explicitly stated otherwise, command-line flags are independent and may be combined freely.

The following sections describe every supported flag in detail.

--output

Purpose

Specifies the directory where benchmark reports will be written.

Syntax

--output <directory>

Example

scalionix-system-benchmark \
    --output ./benchmarks

Description

If the directory does not exist, it is created automatically.

All generated benchmark reports are written into this directory.

Typical generated files include:

• Full Benchmark Report
• Scoring Report
• Global Score Report

Default Value

Current working directory.

Notes

Changing the output directory does not affect benchmark execution.

Only report locations are modified.

--benchmark-work-dir

Purpose

Specifies the working directory used for temporary benchmark data.

Syntax

--benchmark-work-dir <directory>

Example

scalionix-system-benchmark \
    --benchmark-work-dir /mnt/nvme0/benchmark-work

Description

Benchmark datasets are created inside this directory.

Storage benchmarks execute directly against this location.

Compilation fixtures are extracted here before compilation begins.

The benchmark automatically creates and removes temporary benchmark files when appropriate.

Default Value

Platform-specific temporary benchmark directory.

Notes

The selected working directory directly influences Storage benchmark results because it determines which storage device is evaluated.

--report-id

Purpose

Assigns a custom identifier to the benchmark execution.

Syntax

--report-id <identifier>

Example

scalionix-system-benchmark \
    --report-id workstation-7950x3d-linux

Description

The Report ID is stored inside every generated report.

Typical Report IDs include:

• workstation-001
• build-server
• laptop-arm64
• benchmark-2026-08

Default Value

Automatically generated UUID Version 7.

Notes

Report IDs are intended to identify benchmark executions rather than hardware.

--user-id

Purpose

Associates benchmark reports with a specific user.

Syntax

--user-id <identifier>

Example

scalionix-system-benchmark \
    --user-id john-doe

Description

The User ID becomes part of every generated report.

Future Global Ranking services use this value to associate benchmark results with benchmark owners.

Default Value

Platform-specific default user identifier.

Notes

Changing the User ID never influences benchmark scores.

It serves only as report metadata.

--check-hardware

Purpose

Detects hardware and prints the complete Hardware Snapshot.

Syntax

--check-hardware

Example

scalionix-system-benchmark \
    --check-hardware

Description

The benchmark detects hardware exactly as it would before normal execution.

The Hardware Snapshot is printed to standard output.

Benchmark execution does not begin.

Typical information includes:

• Processor
• Memory
• Storage devices
• Benchmark storage
• Operating system
• Hardware fingerprint

Exit Status

Success after printing hardware information.

Notes

Useful for validating hardware detection before executing the benchmark.

--setup-compilation-environment

Purpose

Verifies the software environment required by the Compilation benchmark.

Syntax

--setup-compilation-environment

Example

scalionix-system-benchmark \
    --setup-compilation-environment

Description

The benchmark verifies the availability of all required compilation tools.

Current verification includes:

Rust

• cargo
• rustc
• rustup

C++

Linux

• g++
• make
• build-essential (or equivalent)

macOS

• clang++
• Xcode Command Line Tools

The benchmark reports:

• Installed tools
• Detected versions
• Missing components
• Installation recommendations

Benchmark execution does not begin.

Exit Status

Success if the environment satisfies benchmark requirements.

Notes

This command verifies the environment.

It does not automatically install missing software.

--compare-score

Purpose

Compares two previously generated Global Score Reports.

Syntax

--compare-score

Required Companion Flags

• --baseline
• --candidate

Example

scalionix-system-benchmark \
    --compare-score \
    --baseline old-score.json \
    --candidate new-score.json

Description

The benchmark loads both reports.

Compatibility is verified.

Category scores are compared.

Compute and Storage Scores are compared.

Percentage differences are calculated.

Benchmark execution is skipped.

Exit Status

Success after comparison completes.

--baseline

Purpose

Specifies the baseline Global Score Report.

Syntax

--baseline <file>

Example

--baseline benchmark-old.json

Description

The specified report becomes the reference against which the candidate report is evaluated.

This flag has meaning only when used together with:

• --compare-score

--candidate

Purpose

Specifies the benchmark report being evaluated.

Syntax

--candidate <file>

Example

--candidate benchmark-new.json

Description

The candidate report is compared against the baseline report.

This flag has meaning only when used together with:

• --compare-score

Flag Combination Rules

Certain command-line flags are intended to be used together.

Hardware Detection

scalionix-system-benchmark \
    --check-hardware

Compilation Environment Verification

scalionix-system-benchmark \
    --setup-compilation-environment

Benchmark Comparison

scalionix-system-benchmark \
    --compare-score \
    --baseline score-a.json \
    --candidate score-b.json

Custom Output Directory

scalionix-system-benchmark \
    --output ./benchmarks

Benchmark Execution

scalionix-system-benchmark \
    --report-id workstation-linux \
    --user-id john \
    --output ./benchmarks

The benchmark validates incompatible flag combinations before execution begins.

Design Summary

The command-line interface has been designed to expose every important benchmark capability through explicit, deterministic, and automation-friendly flags.

Rather than relying on hidden configuration files or interactive prompts, every benchmark parameter is fully controlled through the command line.

This design ensures predictable execution, straightforward scripting, seamless integration with continuous integration systems, and long-term reproducibility of benchmark results.

Typical Usage Examples

This chapter demonstrates the most common workflows when using the Scalionix System Benchmark.

Rather than describing individual command-line flags, the following examples illustrate complete benchmark workflows from initial installation to benchmark comparison.

These examples are intended as practical starting points for everyday benchmark usage.

First-Time Installation

Before executing the benchmark for the first time, verify that the compilation environment is correctly installed.

scalionix-system-benchmark \
    --setup-compilation-environment

The benchmark verifies all required software components.

Typical verification includes:

• Rust toolchain
• Cargo
• Rustup
• C++ compiler
• Build tools
• Operating system support

If any required component is missing, installation recommendations are displayed.

No benchmark execution begins.

Verify Hardware Detection

Before executing the benchmark, hardware detection may be verified independently.

scalionix-system-benchmark \
    --check-hardware

Typical output includes:

• Processor
• Physical cores
• Logical processors
• Installed memory
• Storage devices
• Benchmark storage device
• Operating system
• Hardware fingerprint

This command is useful for validating benchmark configuration before long benchmark executions.

Execute Complete Benchmark

Executing the complete benchmark requires only a single command.

scalionix-system-benchmark

The benchmark automatically performs:

• Hardware detection
• Benchmark preparation
• Benchmark execution
• Score calculation
• Report generation
• Benchmark summary

Typical execution time depends on hardware performance and benchmark configuration.

Store Reports in a Custom Directory

Benchmark reports may be written to a custom directory.

scalionix-system-benchmark \
    --output ./benchmarks

All generated reports are written to the specified directory.

Existing benchmark execution remains unchanged.

Assign Report Identity

Benchmark executions may be assigned explicit identifiers.

scalionix-system-benchmark \
    --report-id workstation-linux \
    --user-id john-doe

Both identifiers become part of every generated report.

This simplifies historical benchmark tracking.

Execute Benchmark on a Specific Storage Device

Storage benchmarks evaluate the filesystem containing the benchmark working directory.

To benchmark another storage device, specify a working directory located on that device.

scalionix-system-benchmark \
    --benchmark-work-dir /mnt/nvme1/scalionix-benchmark

Storage Score now reflects the selected storage device.

This workflow is commonly used when comparing multiple SSDs installed in the same workstation.

Compare Two Benchmark Reports

Previously generated benchmark reports may be compared without executing the benchmark again.

scalionix-system-benchmark \
    --compare-score \
    --baseline workstation-before.json \
    --candidate workstation-after.json

The comparison engine verifies report compatibility before calculating:

• Compute Score difference
• Storage Score difference
• Category differences
• Percentage improvements

Benchmark execution is skipped entirely.

Hardware Upgrade Validation

A common workflow consists of executing the benchmark before and after upgrading hardware.

Step 1

Execute the benchmark before the upgrade.

scalionix-system-benchmark \
    --report-id before-upgrade

Step 2

Upgrade hardware.

Examples include:

• Processor
• Memory
• Storage

Step 3

Execute the benchmark again.

scalionix-system-benchmark \
    --report-id after-upgrade

Step 4

Compare both reports.

scalionix-system-benchmark \
    --compare-score \
    --baseline before-upgrade-global-score.json \
    --candidate after-upgrade-global-score.json

This workflow provides an objective measurement of hardware improvements.

Operating System Comparison

The benchmark may be executed on multiple operating systems installed on the same hardware.

Typical workflow:

Linux
  │
  ▼
Generate Global Score Report
  │
  ▼
Windows
  │
  ▼
Generate Global Score Report
  │
  ▼
Compare Reports

This allows operating system performance to be evaluated independently from hardware differences.

Benchmark Archival

Generated reports should be preserved after benchmark execution.

Recommended files include:

• Full Benchmark Report
• Scoring Report
• Global Score Report

Keeping these reports allows future benchmark comparison without repeating benchmark execution.

Historical benchmark archives become increasingly valuable as hardware evolves.

Continuous Integration

The benchmark is designed for automated execution.

Typical CI workflow:

Checkout Source
       │
       ▼
Build Benchmark
       │
       ▼
Verify Environment
       │
       ▼
Execute Benchmark
       │
       ▼
Generate Reports
       │
       ▼
Archive Reports
       │
       ▼
Compare Against Previous Run

This workflow enables automatic detection of performance regressions.

Public Benchmark Submission

The recommended workflow for public benchmark submission is:

Execute Benchmark
        │
        ▼
Verify Benchmark Completion
        │
        ▼
Review Generated Reports
        │
        ▼
Preserve Full Benchmark Report
        │
        ▼
Submit Global Score Report

The Global Score Report contains all information required for public benchmark comparison while remaining compact and portable.

Recommended Workflow

The following workflow is recommended for most users.

Verify Compilation Environment
             │
             ▼
      Check Hardware
             │
             ▼
     Execute Benchmark
             │
             ▼
    Review Benchmark Summary
             │
             ▼
     Preserve Generated Reports
             │
             ▼
Compare With Previous Results
             │
             ▼
 Upload Global Score Report

Following this workflow ensures reproducible benchmark execution while preserving all information required for future comparison and analysis.

Design Summary

The Scalionix System Benchmark has been designed around simple, deterministic workflows requiring minimal user interaction.

Whether executed interactively, integrated into automated infrastructure, or used for long-term hardware validation, the benchmark follows the same predictable execution model while producing identical report structures and scoring behavior.

Global Ranking System

Benchmark Design Principles

The Scalionix System Benchmark has been designed around a small set of fundamental architectural principles.

Every component of the benchmark, from workload execution to score generation and report creation, follows these principles.

The objective is not merely to measure hardware performance, but to provide a benchmarking platform that remains reliable, reproducible, and maintainable for many years.

Deterministic Execution

Every benchmark execution is deterministic.

Given identical:

• Hardware
• Operating system
• Benchmark version
• Scoring version
• Benchmark configuration

the benchmark always executes the same workloads in the same order using the same execution rules.

No benchmark behavior depends on random scheduling, historical benchmark results, online services, or external databases.

Deterministic execution represents the foundation of reproducible benchmarking.

Offline Operation

The benchmark is fully self-contained.

Internet connectivity is never required during benchmark execution.

All benchmark assets required for execution are distributed together with the benchmark itself.

Compilation fixtures, benchmark datasets, and scoring references are embedded into the executable whenever practical.

As a result, benchmark execution remains reproducible regardless of network availability.

Platform Independence

The benchmark has been designed to behave consistently across supported operating systems.

Platform-specific implementations are isolated behind common interfaces.

Whenever platform-specific behavior is required, benchmark semantics remain identical.

Users should obtain comparable benchmark results regardless of whether the benchmark is executed on Linux, macOS, or Windows.

Independent Benchmark Categories

Every benchmark category is completely independent.

Categories do not share measurements, references, or scoring logic.

Current categories include:

• JSON
• Hashing
• Encryption
• Compression
• Collections
• Concurrency
• Compilation
• Storage

This architecture allows categories to evolve independently without affecting benchmark stability.

Separation of Execution and Scoring

Benchmark execution and score calculation are independent subsystems.

Benchmark execution produces measurements.

The scoring engine interprets those measurements.

This separation provides several important advantages.

• Benchmark implementations may evolve.
• Scoring algorithms may improve.
• Historical benchmark reports remain valid.
• Score recalculation becomes possible.

Execution therefore never depends on scoring.

Versioned Architecture

Every important benchmark component is versioned.

Examples include:

• Benchmark format
• Scoring version
• Benchmark reports
• References

Versioning guarantees long-term compatibility while allowing future benchmark evolution.

Deterministic References

Benchmark scores are always calculated relative to predefined references.

The benchmark never derives scores from previously submitted benchmark executions.

Consequently:

• Benchmark rankings never modify historical scores.
• New hardware never invalidates existing results.
• Benchmark scores remain stable.

Reference-driven scoring is one of the defining characteristics of the Scalionix System Benchmark.

Complete Transparency

Every benchmark decision is visible.

Examples include:

• Benchmark configuration
• Hardware detection
• Workload grouping
• Reference matching
• Diagnostics
• Score calculation

No hidden weighting or undocumented adjustments are performed.

Transparency greatly simplifies benchmark validation and community review.

Extensibility

The benchmark has been designed for continuous evolution.

New benchmark categories may be introduced without redesigning the benchmark architecture.

Similarly, new workload variants, dataset dimensions, hardware collectors, report formats, and scoring models may be added incrementally.

Backward compatibility remains a primary design objective.

Long-Term Maintainability

The benchmark architecture favors maintainability over short-term optimization.

Benchmark modules remain isolated.

Reference tables remain versioned.

Reports remain deterministic.

Diagnostics remain comprehensive.

This philosophy allows the benchmark to continue evolving over many hardware generations without architectural redesign.

Open Architecture

The benchmark has been designed as an open system.

Every benchmark report is human-readable.

Every scoring rule is documented.

Every benchmark category is reproducible.

Future contributors should be able to understand, validate, and extend the benchmark without requiring undocumented knowledge.

Design Summary

The Scalionix System Benchmark has been designed around deterministic execution, independent benchmark categories, reference-driven scoring, complete transparency, versioned reports, and long-term maintainability.

These principles guide every architectural decision and ensure that benchmark results remain reliable, reproducible, and meaningful across future benchmark releases and hardware generations.

Future Roadmap

The Scalionix System Benchmark has been designed as a continuously evolving benchmarking platform.

While the current benchmark already covers a broad range of practical workloads, additional benchmark categories and ecosystem components are planned for future releases.

This chapter summarizes the currently envisioned development roadmap.

Additional Benchmark Categories

Future benchmark releases may introduce additional workload categories.

Examples currently under consideration include:

MessagePack

Evaluation of binary serialization performance.

Network Benchmark

• HTTP/2 request throughput
• HTTP/3 (QUIC) request throughput
• REST API performance
• WebSocket performance

Filesystem Extensions

• Additional storage durability modes
• Large directory traversal
• Mixed read/write workloads

Database Benchmarks

• SQLite
• LMDB
• RocksDB

Additional benchmark categories will be introduced only when they provide meaningful real-world performance information.

Expanded Compilation Benchmark

Future versions of the Compilation benchmark may include additional real-world projects.

Potential additions include:

• Larger Rust projects
• Additional C++ projects
• Optional Go projects

Compilation fixtures will continue to prioritize practical software projects over synthetic build workloads.

Hardware Support

Future hardware support may include:

• Additional ARM platforms
• ARM servers
• Threadripper reference systems
• EPYC servers
• Apple Silicon reference calibration

Additional reference platforms may become available through future scoring versions.

Global Ranking Service

The primary long-term objective is the public Global Ranking Service.

Planned capabilities include:

• Online benchmark submission
• Public leaderboards
• Hardware comparison
• Historical benchmark tracking
• Benchmark search
• Hardware classification
• Public benchmark profiles

The Global Ranking Service represents the natural extension of the benchmark ecosystem.

Continuous Benchmark Evolution

The benchmark will continue evolving as hardware evolves.

Future development priorities include:

• New workload categories
• Improved hardware detection
• Additional diagnostics
• Improved benchmark reporting
• Extended scoring analysis

Compatibility with historical benchmark reports will remain a primary objective.

Community Contributions

The benchmark architecture encourages community contributions.

Areas particularly suitable for contribution include:

• New benchmark categories
• Additional operating-system support
• Improved hardware collectors
• Benchmark validation
• Documentation improvements

All contributions should preserve deterministic benchmark behavior and long-term compatibility.

Long-Term Vision

The long-term objective of the Scalionix System Benchmark is to become a comprehensive, deterministic, open, and reproducible benchmarking platform suitable for professional hardware evaluation.

Rather than focusing on a single synthetic score, the benchmark aims to provide detailed insight into real-world workloads while maintaining a transparent scoring model and fully documented benchmark architecture.

The accompanying Global Ranking System is intended to complement this objective by enabling meaningful comparison of hardware platforms without compromising reproducibility or historical consistency.

Final Remarks

The Scalionix System Benchmark is the result of a design philosophy centered on determinism, transparency, reproducibility, and long-term maintainability.

Every benchmark category, scoring rule, report format, and architectural component has been designed to remain understandable, extensible, and stable across future benchmark generations.

As new hardware, operating systems, compilers, and workload types emerge, the benchmark is expected to evolve while preserving the compatibility and reproducibility of historical benchmark results.

The ultimate goal is not simply to measure performance, but to establish a trustworthy and open benchmarking ecosystem capable of serving developers, hardware enthusiasts, researchers, and organizations for many years to come.