Collections Benchmark

The Collections benchmark evaluates the performance of fundamental in-memory container operations that form the backbone of modern software systems.

Virtually every application continuously manipulates dynamic collections of objects. Web servers process request tables, databases maintain indexes, schedulers manage task queues, compilers build symbol tables, operating systems track resources, and distributed systems maintain routing structures.

Although these operations appear simple, they collectively account for a significant portion of CPU execution time in many production environments.

Unlike synthetic memory benchmarks that repeatedly allocate identical buffers, the Scalionix Collections benchmark evaluates complete container lifecycles including allocation, insertion, lookup, iteration, update, removal, sorting, and memory reclamation.

The resulting measurements closely resemble workloads encountered by production software.

Objectives

The Collections benchmark has six primary objectives.

• Measure allocation performance.
• Measure insertion throughput.
• Measure lookup efficiency.
• Measure iteration performance.
• Measure update operations.
• Measure removal and cleanup performance.

Rather than evaluating one isolated operation, the benchmark measures complete collection lifecycles.

Why Collections?

Collections are one of the most frequently used abstractions in software engineering.

Examples include:

• Routing tables
• HTTP header maps
• Session stores
• Symbol tables
• Object registries
• Job schedulers
• Resource tracking
• Database indexes
• Cache implementations
• Runtime object graphs

Because these operations dominate many CPU profiles, measuring collection performance provides valuable insight into practical application efficiency.

Benchmark Philosophy

The benchmark intentionally avoids measuring individual library functions.

Instead, every workload performs realistic sequences of container operations.

Typical execution consists of:

            Generate Objects
                    │
                    ▼
           Allocate Collection
                    │
                    ▼
               Insert Data
                    │
                    ▼
                 Lookup
                    │
                    ▼
                Iteration
                    │
                    ▼
            Optional Modification
                    │
                    ▼
                 Removal
                    │
                    ▼
               Verification
                    │
                    ▼
          Performance Measurement

This execution model reflects the complete lifecycle of collections within production applications.

Typical Collection Pipeline

            Generate Objects
                    │
                    ▼
           Allocate Collection
                    │
                    ▼
               Insert Data
                    │
                    ▼
            Lookup Operations
                    │
                    ▼
           Iterate Collection
                    │
                    ▼
          Update / Modify Data
                    │
                    ▼
             Remove Objects
                    │
                    ▼
              Verify State
                    │
                    ▼
          Throughput Measurement

Benchmark Scenarios

The Collections benchmark evaluates multiple categories of container operations.

Insertion

Measures sustained insertion throughput into initially empty collections.

Scenario examples

collections.insert.vector
collections.insert.hash_map
collections.insert.btree_map

Lookup

Measures repeated access to previously inserted elements.

Scenario examples

collections.lookup.hash_map
collections.lookup.btree_map
collections.lookup.vector

Iteration

Measures sequential traversal of populated containers.

Scenario examples

collections.iteration.vector
collections.iteration.hash_map

Removal

Measures deletion throughput.

Scenario examples

collections.remove.hash_map
collections.remove.vector

Sorting

Measures ordering of deterministic datasets.

Scenario examples

collections.sort.vector
collections.sort.stable

Mixed Operations

Measures practical workloads consisting of insertion, lookup, update, and deletion executed together.

This scenario most closely resembles production software.

Dataset Characteristics

Every workload operates on deterministic datasets.

Objects are generated programmatically before benchmark execution.

Datasets contain realistic field distributions instead of synthetic integer sequences.

This ensures that measured performance reflects practical object layouts encountered by modern software.

Collection Sizes

Multiple collection sizes are evaluated.

Smaller collections emphasize allocator overhead.

Larger collections emphasize:

• Cache hierarchy
• Memory bandwidth
• Branch prediction
• Pointer chasing
• Allocator scalability

Dataset sizes remain deterministic across every supported platform.

Worker Configuration

Collections benchmark scalability is evaluated using multiple worker configurations.

Each worker operates on an independent collection.

Workers never share containers.

This execution model isolates computational scalability from synchronization overhead.

Representative worker configurations include:

[ 1 / 2 / 4 / 6 / 8 / 12 / 16 / 20 / 24 / 32 ]

Verification

Every workload verifies collection correctness.

Verification includes:

• Expected element count
• Successful lookup
• Successful iteration
• Ordering validation
• Successful removal

Verification failures invalidate benchmark execution.

This guarantees that benchmark scores always represent correct container behavior.

Measured Metrics

Primary benchmark metrics include:

• Operations per second
• Execution duration
• Processed objects
• Worker configuration
• Iteration timing

The scoring adapter converts measured operations into normalized throughput expressed as operations per second.

Score Calculation

Collections workloads fully participate in the Compute Score.

Every workload contributes:

Single-thread Score

Measures processor efficiency during sequential container operations.

Multi-thread Score

Measures maximum sustained throughput across all worker configurations.

Scaling Score

Measures parallel scalability of independent collection workloads.

Scenario scores are aggregated into the Collections category score.

Hardware Characteristics Measured

Collection performance depends on several hardware characteristics.

Processor Architecture

Instruction execution affects arithmetic and pointer manipulation.

Cache Hierarchy

Most collection operations are cache sensitive.

Memory Bandwidth

Large collections generate continuous memory traffic.

Branch Prediction

Tree traversal and hash lookups frequently depend on branch prediction accuracy.

Memory Allocator

Container growth frequently triggers dynamic allocation.

Consequently, collection performance reflects the combined behavior of the processor, cache hierarchy, memory subsystem, allocator implementation, and compiler optimizations.

Practical Interpretation

High Collections benchmark scores generally indicate:

• Efficient cache hierarchy
• Strong branch prediction
• Fast allocator implementation
• Excellent memory bandwidth
• Efficient pointer traversal

These characteristics directly influence the performance of compilers, databases, schedulers, web services, and numerous backend applications.

Design Summary

The Collections benchmark evaluates complete container lifecycles using deterministic datasets representative of modern software systems.

Rather than measuring isolated library calls, the benchmark measures realistic object management workloads that continuously occur inside production applications.

The resulting category score therefore reflects practical memory and container performance encountered during everyday software development.