Diagnostics

The Scalionix System Benchmark records detailed diagnostic information during score generation.

Unlike benchmark scores, which describe benchmark performance, diagnostics describe the behavior of the scoring engine itself.

Diagnostics are intended primarily for benchmark developers, benchmark maintainers, reference calibration, regression analysis, and future benchmark evolution.

They provide complete visibility into how benchmark workloads were processed, matched, scored, rejected, or excluded from score calculation.

Diagnostics never influence benchmark scores directly.

Instead, they explain how benchmark scores were produced.

Purpose

The Diagnostics subsystem has several primary objectives.

• Validate benchmark execution.
• Verify reference completeness.
• Detect missing benchmark references.
• Detect rejected benchmark workloads.
• Simplify future benchmark maintenance.
• Support reference calibration.

Rather than silently ignoring inconsistencies, the benchmark records every important scoring decision.

Diagnostic Generation

Diagnostics are generated automatically during score calculation.

For every benchmark category the scoring engine records:

• Received benchmark samples
• Created workload groups
• Matched references
• Missing references
• Rejected workloads
• Successfully scored workloads

The resulting diagnostic information becomes part of the Scoring Report.

Category Isolation

Diagnostics are collected independently for every benchmark category.

For example:

            JSON
             │
             ▼
            Independent diagnostics
            
            Hashing
             │
             ▼
            Independent diagnostics
            
            Storage
             │
             ▼
            Independent diagnostics
            
            Compilation
             │
             ▼
            Independent diagnostics

Problems affecting one benchmark category never influence diagnostics generated for another category.

This greatly simplifies benchmark maintenance.

Input Samples

Input Samples represent the number of benchmark measurements received by the scoring engine.

Every successfully executed benchmark request contributes one or more input samples.

Input Samples therefore describe the amount of raw benchmark information available before workload grouping begins.

Unexpectedly low sample counts frequently indicate incomplete benchmark execution.

Workload Groups

Multiple benchmark measurements are frequently combined into logical workload groups before scoring.

Examples include:

            Multiple dataset sizes
                      │
                      ▼
                One scenario
            
            Multiple object counts
                      │
                      ▼
               One workload family
            
            Multiple block sizes
                      │
                      ▼
             One storage workload

Workload Groups therefore represent the logical benchmark units processed by the scoring engine.

Matched References

Every workload must be matched against exactly one scoring reference.

Matched References records the total number of successful matches.

A successful match indicates that the scoring engine found a deterministic reference describing the executed workload.

Only successfully matched workloads participate in score calculation.

Missing References

If no matching reference exists, the workload cannot be scored.

Such workloads are recorded as Missing References.

Typical causes include:

• Newly introduced benchmark scenarios
• Modified dataset sizes
• Changed workload variants
• Incomplete reference tables

Missing references never terminate benchmark execution.

Instead, they are reported so that benchmark developers may update reference calibration.

Successfully Scored Workloads

Successfully Scored Workloads records the number of benchmark workloads contributing to the final score.

Only workloads satisfying all of the following conditions participate:

• Benchmark executed successfully
• Verification passed
• Reference successfully matched
• Workload accepted by the scoring engine

This value therefore represents the effective benchmark size after scoring validation.

Rejected Workloads

Some benchmark workloads may be intentionally rejected.

Typical reasons include:

• Invalid benchmark measurements
• Incomplete workload data
• Unsupported benchmark configuration
• Failed verification

Rejected workloads remain visible through diagnostics while being excluded from score calculation.

Missing Reference Keys

For every missing reference the scoring engine records the complete workload identifier.

Typical identifiers contain:

• Scenario name
• Dataset size
• Workload variant

These identifiers uniquely describe the missing workload.

Benchmark developers can therefore generate new reference entries directly from diagnostic output.

This significantly simplifies future benchmark expansion.

Rejected Workload Keys

Rejected benchmark workloads are also recorded individually.

Each rejected workload preserves its complete benchmark identifier.

This allows benchmark maintainers to investigate why a workload was rejected without repeating benchmark execution.

Diagnostic Interpretation

Healthy benchmark execution typically produces diagnostics similar to the following.

Matched References

Equal to the number of workload groups.

Missing References

Zero.

Rejected Workloads

Zero.

Successfully Scored Workloads

Equal to the number of workload groups.

Such diagnostics indicate complete benchmark coverage.

Conversely, increasing numbers of missing references usually indicate that benchmark scenarios have evolved faster than scoring references.

Practical Applications

Diagnostics support numerous maintenance activities.

• Reference calibration
• Benchmark validation
• Regression testing
• Scoring verification
• Benchmark development
• Future benchmark categories

Because diagnostics describe the internal behavior of the scoring engine, they are especially valuable during benchmark evolution.

Long-Term Maintenance

The Diagnostics subsystem has been designed specifically to simplify long-term benchmark maintenance.

As new benchmark categories are introduced, diagnostics immediately identify:

• Workloads requiring references
• Rejected benchmark configurations
• Incomplete benchmark coverage

This allows the benchmark to evolve incrementally without sacrificing scoring reliability.

Design Summary

The Diagnostics subsystem provides complete visibility into the internal operation of the scoring engine.

By recording workload grouping, reference matching, workload acceptance, rejected benchmark measurements, and scoring completeness, diagnostics ensure that every benchmark score can be validated, reproduced, and maintained throughout the lifetime of the Scalionix System Benchmark.

Unlike benchmark scores, diagnostics describe the quality of benchmark processing itself, making them an essential tool for long-term benchmark evolution and reference maintenance.