Skip to content

Benchmarks and Parity

Use these workflows to guard runtime, numerical stability, and external parity.

1) Standard Benchmark Suite

PYTHONPATH=build/python python3 benchmarks/benchmark_runner.py \
  --output-dir benchmarks/out

For switching-heavy converter cases:

PYTHONPATH=build/python python3 benchmarks/benchmark_runner.py \
  --only buck_switching boost_switching_complex interleaved_buck_3ph buck_mosfet_nonlinear \
  --output-dir benchmarks/out_converters

For AC sweep focused coverage:

PYTHONPATH=build/python python3 benchmarks/benchmark_runner.py \
  --only ac_rc_lowpass ac_control_workflow_expected_failure \
  --output-dir benchmarks/out_ac

For averaged-converter paired coverage:

PYTHONPATH=build/python python3 benchmarks/benchmark_runner.py \
  --only buck_switching_paired buck_averaged_mvp buck_averaged_expected_failure \
  --output-dir benchmarks/phase14_averaged_artifacts/benchmarks

2) Solver/Integrator Validation Matrix

PYTHONPATH=build/python python3 benchmarks/validation_matrix.py \
  --output-dir benchmarks/matrix

This matrix is useful to detect solver regressions across fixed/variable timestep modes.

3) External Parity

ngspice

PYTHONPATH=build/python python3 benchmarks/benchmark_ngspice.py \
  --backend ngspice \
  --output-dir benchmarks/ngspice_out

LTspice

PYTHONPATH=build/python python3 benchmarks/benchmark_ngspice.py \
  --backend ltspice \
  --ltspice-exe "/Applications/LTspice.app/Contents/MacOS/LTspice" \
  --output-dir benchmarks/ltspice_out

4) Tiered Stress Suite

PYTHONPATH=build/python python3 benchmarks/stress_suite.py \
  --output-dir benchmarks/stress_out

Electro-thermal stress variant:

PYTHONPATH=build/python python3 benchmarks/stress_suite.py \
  --benchmarks benchmarks/electrothermal_benchmarks.yaml \
  --catalog benchmarks/electrothermal_stress_catalog.yaml \
  --output-dir benchmarks/stress_out_electrothermal

5) GUI-to-Backend Parity Gate

PYTHONPATH=build/python pytest -q python/tests/test_gui_component_parity.py
PYTHONPATH=build/python pytest -q python/tests/test_runtime_bindings.py
./build-test/core/pulsim_simulation_tests "[v1][yaml][gui-parity]"

6) KPI Regression Gate

python3 benchmarks/kpi_gate.py \
  --bench-results benchmarks/out/results.json \
  --stress-summary benchmarks/stress_out/stress_summary.json \
  --report-out benchmarks/out/kpi_gate_report.json \
  --print-report

Baseline and threshold files:

  • benchmarks/kpi_baselines/modular_runtime_phase13_2026-03-07/kpi_baseline.json
  • benchmarks/kpi_thresholds.yaml
  • benchmarks/kpi_thresholds_ac.yaml (AC-focused KPI gate)
  • benchmarks/kpi_baselines/averaged_converter_phase14_2026-03-07/kpi_baseline.json (averaged-focused gate)
  • benchmarks/kpi_thresholds_averaged.yaml (averaged-focused KPI gate)

Averaged-focused gate command:

python3 benchmarks/kpi_gate.py \
  --baseline benchmarks/kpi_baselines/averaged_converter_phase14_2026-03-07/kpi_baseline.json \
  --bench-results benchmarks/phase14_averaged_artifacts/benchmarks/results.json \
  --thresholds benchmarks/kpi_thresholds_averaged.yaml \
  --report-out benchmarks/phase14_averaged_artifacts/reports/kpi_gate_averaged.json \
  --print-report

Artifact Contract

These files are the recommended CI contract:

  • benchmark: results.csv, results.json, summary.json
  • parity: parity_results.csv, parity_results.json, parity_summary.json
  • stress: stress_results.csv, stress_results.json, stress_summary.json

Primary hybrid/electro-thermal KPI keys emitted in benchmark outputs:

  • state_space_primary_ratio
  • dae_fallback_ratio
  • loss_energy_balance_error
  • thermal_peak_temperature_delta
  • component_coverage_rate
  • component_coverage_gap
  • component_loss_summary_consistency_error
  • component_thermal_summary_consistency_error
  • runtime_module_order_crc32
  • runtime_module_count_match
  • output_reallocation_total
  • ac_sweep_mag_error
  • ac_sweep_phase_error
  • ac_runtime_p95
  • averaged_pair_case_count
  • averaged_pair_fidelity_error
  • averaged_pair_runtime_speedup_min