A quantum circuit can be small, fast on a simulator, and still perform poorly on hardware. This guide presents a repeatable benchmarking workflow for measuring circuit size, depth, two-qubit operations, transpilation changes, simulator runtime, shot variance, fidelity, and hardware error indicators. Use it to compare circuit versions, SDK configurations, simulators, and cloud backends without treating one metric as the whole story.
Overview
Quantum circuit benchmarking is the process of recording how a circuit is constructed, transformed, executed, and measured. The purpose is not simply to find the circuit with the fewest gates. A useful benchmark connects resource cost with result quality.
For a practical comparison, separate your measurements into four groups:
- Structure: qubit count, total gate count, circuit depth, and the number of two-qubit gates.
- Compilation: changes introduced by transpilation, including added gates, altered depth, and mapped qubits.
- Execution: simulator or hardware runtime, queue time when relevant, shot count, and repetitions.
- Quality: fidelity against a reference result, error rates, output distribution, and statistical variation.
These measurements answer different questions. Depth estimates how many sequential layers must be executed. Two-qubit gates often deserve separate attention because they can be more sensitive to hardware constraints than single-qubit operations. Runtime describes execution cost, but a faster run is not automatically better if it produces a less reliable result. Fidelity describes agreement with a chosen reference, while error rates help explain why that agreement changes.
Start with an ideal or classically tractable reference whenever possible. Then run the same logical circuit through each compiler configuration, simulator, or backend you want to compare. Keep the circuit, input data, shot count, optimization settings, SDK version, and backend configuration fixed unless the variable being tested is one of those inputs.
How to compare options
Benchmarking becomes unreliable when the comparison changes several variables at once. Define the test before collecting numbers.
1. Establish the reference circuit
Save the original circuit before transpilation. Record its logical qubit count, operations, measurements, parameters, and intended output. If the circuit is part of an algorithm such as Grover's search, VQE, QAOA, or a variational classifier, also record the problem instance and parameter values. A benchmark should compare equivalent workloads, not merely circuits with similar names.
2. Choose a reference result
For small circuits, an ideal statevector or exact probability distribution can provide a useful reference. For larger circuits, use a trusted simulator configuration, a noiseless execution, analytical expectations, or a high-quality baseline. State clearly which reference you selected. Fidelity is meaningful only in relation to a defined target.
3. Fix execution conditions
Use the same number of shots when comparing sampling results. Record whether measurements were obtained from a statevector, a shot-based simulator, a noisy simulator, or physical hardware. On cloud systems, distinguish device execution time from queue or job-wait time when the platform exposes those values. Queue conditions can vary and may not reflect circuit efficiency.
4. Compile separately for each target
A circuit optimized for one device topology may not be efficient on another. Transpilation can add routing operations, decompose gates into a different basis, and change circuit depth. Preserve both the original and compiled circuits so you can explain the difference rather than reporting only the final number.
5. Repeat measurements where randomness matters
Shot-based results fluctuate. Run enough repeated trials to observe whether a change is consistent, and report the shot count and random seed when supported. Do not compare one noisy sample from one circuit with a heavily repeated sample from another.
Feature-by-feature breakdown
Circuit depth
Circuit depth is the number of sequential layers required when operations that act on separate qubits can run in parallel. It is a useful proxy for exposure to time-dependent noise, but it does not capture every cost. Two circuits with the same depth can have different gate types, connectivity demands, or numbers of idle qubits.
Record depth before and after transpilation. The difference shows how much the target architecture and compiler altered the workload. Also record depth after optimization at each setting you test; a higher optimization level may reduce depth while increasing compilation time or changing the gate decomposition.
Gate count and two-qubit operations
Total gate count provides a broad size measure. Break it into single-qubit and two-qubit counts, and include the basis-gate counts after compilation. The two-qubit count is especially useful when comparing hardware targets because connectivity constraints may require additional routing operations.
Do not assume that minimizing total gates always minimizes error. Gate duration, calibration quality, parallel scheduling, and the particular gate family can also affect execution quality. Treat gate counts as explanatory metrics rather than a complete performance score.
Transpilation overhead
Transpilation overhead is the change between the logical circuit and the circuit submitted to a target. Calculate simple ratios or differences such as:
- compiled depth minus logical depth;
- compiled two-qubit gates minus logical two-qubit gates;
- compiled total gates divided by logical total gates.
These figures help identify whether a circuit's abstract design fits the target architecture. A circuit with modest logical depth may become expensive after qubit mapping and routing.
Runtime and reproducibility
Measure simulator runtime separately from hardware job latency whenever possible. For simulators, record system resources, simulation method, precision, noise model, and whether results were sampled or calculated exactly. For hardware, record submission time, execution time if available, shot count, and backend identifier. Avoid presenting a single runtime as universal: local hardware, cloud load, SDK overhead, and job batching can all change the result.
Fidelity and output quality
Fidelity measures how closely an observed state or distribution agrees with a reference. The appropriate calculation depends on whether you compare statevectors, probability distributions, or sampled bitstrings. If your workflow uses a custom similarity score instead, name it and define it.
For measured distributions, also inspect important bitstrings, success probability, total variation distance, or another clearly defined distribution metric. A single aggregate score can hide a failure on the output that matters to the application.
Error indicators
Use backend-reported error indicators as context, not as a guarantee of application performance. Depending on the platform, you may encounter per-gate error estimates, readout error information, coherence-related values, or calibration timestamps. Record the units, definition, and time at which each value was retrieved. Hardware indicators can change, and their meaning may differ across providers.
Shot variance
Sampling noise is part of the measurement. For a bitstring with estimated probability p from N shots, the approximate standard deviation of the estimate is often described by the binomial expression √(p(1−p)/N), provided the measurement is treated as a binary event. This is a planning aid, not a substitute for repeated experiments or uncertainty analysis across a full distribution.
Benchmark worksheet
| Field | Record |
|---|---|
| Workload | Circuit name, algorithm, problem instance, parameters |
| Environment | SDK and version, language version, simulator or backend |
| Compilation | Optimization setting, basis gates, layout, routing method |
| Logical structure | Qubits, total gates, two-qubit gates, depth |
| Compiled structure | Total gates, two-qubit gates, depth, added routing operations |
| Execution | Shots, runtime, queue time if available, repetitions, seed |
| Quality | Reference definition, fidelity or distance, success rate, variance |
| Hardware context | Backend identifier, calibration time, reported error indicators |
Best fit by scenario
Comparing circuit optimizations: Keep the backend and execution settings fixed. Compare logical and compiled depth, two-qubit count, runtime, and quality. A change is useful only if its resource reduction does not create an unacceptable quality loss.
Choosing a quantum simulator: Use the same circuit, shot count, noise assumptions, and accuracy target. Report simulator runtime with the machine configuration and distinguish exact state calculation from sampled execution. For a broader tool comparison, see the guide to quantum simulators and cloud backends.
Moving from simulation to hardware: First compare the compiled circuit with the simulator's logical version. Then measure output quality against the same reference and inspect whether routing or readout effects dominate. The discussion in quantum error correction explained provides useful context for separating physical noise from higher-level reliability goals.
Evaluating an algorithm prototype: Benchmark a small, repeatable instance before scaling. For variational workflows, record the number of circuit evaluations as well as the cost of each circuit. A circuit may be shallow but still expensive if an optimizer calls it many times; this matters in quantum machine learning and hybrid algorithms.
Learning the workflow: Begin with a simple circuit and inspect it before and after compilation. A beginner-friendly Qiskit circuit tutorial can provide the construction baseline, while an SDK comparison can help when the same benchmark must be expressed across Qiskit, Cirq, and PennyLane.
When to revisit
Revisit a benchmark whenever the inputs that determine its result change. That includes an SDK upgrade, a compiler or transpiler configuration change, a new simulator version, a different noise model, a backend migration, a new calibration period, or a change in shot count. Also rerun it after modifying qubit layout, gate decomposition, optimization settings, or the algorithm's parameterized ansatz.
Keep benchmark records under version control. Store the source circuit, compiled circuit, environment details, raw counts, reference result, and analysis script together. Use stable workload names and avoid overwriting previous runs. A small comparison table is more useful when every row can be reproduced.
For your next benchmark, start with one representative circuit and create two runs: a noiseless reference and a target execution. Fill in the worksheet, compare logical versus compiled depth, separate one- and two-qubit operations, and evaluate the output against a stated reference. Only then add more backends or optimization settings. This staged approach makes it easier to identify whether a change improved the circuit, the compilation process, or merely the measurement conditions.