math-eval: Reproducible Mathematical Reasoning Generation and Evaluation
Abstract. math-eval is a software workflow for reproducible evaluation of mathematical reasoning systems. It accepts canonical JSONL records, generates model outputs through vLLM or an OpenAI-compatible API, preserves raw sample artifacts, and replays parsing and scoring independently. This separation lets an evaluation change its parser or metrics without repeating model inference. The implementation records configuration, prompt, environment, and artifact hashes; supports resumable runs and deterministic data partitions; and validates shard merges before replay. Its default math-v5.2-dual contract reports boxed-only strict results as formal scores and retains whole-response soft results for diagnosis when no complete box is present. Mathematical equivalence is evaluated with Math-Verify 0.9.0. Canonical datasets are maintained separately by math-vault, preserving their provenance and licensing records.
Workflow
Generation writes raw shards, while replay reads the frozen artifacts to produce verdicts and aggregate metrics. This makes the generation and scoring stages independently repeatable.
Reproducibility properties
Each run stores manifest data with configuration and prompt snapshots, hashes, and environment information. Interrupted runs can resume using the same run identifier. Independent workers can process deterministic partitions; the merge step rejects missing, duplicate, incomplete, hash-inconsistent, revision-inconsistent, or environment-inconsistent shards.
Evaluation semantics
The default parser uses the last complete \boxed{} expression for the strict score. If no complete box exists, soft evaluation can inspect the nonempty full response for diagnostic purposes only. A five-second hard timeout during prediction normalization is recorded as a parse error, preventing one pathological expression from blocking replay.
Data relationship and documented use
math-vault maintains traceable snapshots of public mathematical reasoning datasets and their provenance; its canonical JSONL is a direct input to math-eval. The OPD paper documents a use of the software and dataset artifacts. It is a use case, not an endorsement.
How to cite
Software documentation report: Ge, X. (2026). math-eval: Reproducible Mathematical Reasoning Generation and Evaluation. Software documentation report, published 31 August 2026.
Related software artifact: Ge, X. (2026). math-eval: Reproducible Mathematical Reasoning Generation and Evaluation (Version v0.1.0). Zenodo. https://doi.org/10.5281/zenodo.21411208