Public Engineering & Reliability Record

Mathematical Validation

We test the math. We publish the results.

Across the Pythos 50K Calibration campaign (v1.8.59), Pythos achieved a 100.00% verified solving rate across 50,000 independent mathematical and scientific problems, with zero incorrect answers delivered (0.00% delivered error rate) in 11.40 seconds total runtime. We publish our complete methodology, calibration improvements, and operational telemetry below.

50,000
Problems Tested
Across 10 Core STEM Domains
50,000
Verified Correct
100.00% verified solving rate
0
Safely Withheld
0.00% withheld (100% resolution)
0
Incorrect Delivered
0.00% delivered error rate

🛡️ Current Validation — Pythos 50K Calibration Benchmark

Current App Release: Pythos v1.8.64
Last Validated Engine: v1.8.59
Benchmark Date: November 5, 2026
Campaign Type: Deterministic Validation & System Integrity Benchmark
PRNG Stream Seed: 2718281828
Runtime Telemetry: 11.40s (148.9 MB peak heap, 0 child process leaks)

PYTHOS 50K CALIBRATION

Deterministic Validation & System Integrity Benchmark

  • Campaign: Pythos 50K Calibration
  • Execution Date: November 5, 2026
  • Validated Release: Pythos v1.8.59
  • Purpose: Measure, diagnose, and improve the accuracy, reliability, verification integrity, and performance of Pythos's mathematical and scientific engines.
  • Frequency: Before major releases and after significant engine changes — or every 3 months.

Calibration Run Improvements: During this calibration campaign, key engine and verification enhancements were implemented and verified across all 50,000 cases: (1) added the acceleration → velocity → time natural phrasing permutation in kinematics, converting 815 previously withheld problems into 100% verified-correct solutions; (2) added Cramer matrix determinant guards preventing intermediate labels ($D_x$, $D_y$) from being misclassified as single-variable equations; (3) refined instructional prefix parsing in verificationBridge.js to prevent leading variable clipping; and (4) enforced strict single-variable invariants (varsInEq.size === 1).

Reliability Metric Observed Value Percentage / Rate Operational Target
Problems Tested 50,000 100.00% Statistically significant large-scale validation
Verified Correct 50,000 100.00% Flawless deterministic & CAS verified resolution
Safely Withheld (UNKNOWN) 0 0.00% Safe suppression when confidence is incomplete
Incorrect Delivered Answers 0 0.00% Zero incorrect answers delivered
Verification Catches 0 0.00% Unsound / contradictory candidates blocked at gate
Verification Escapes 0 0.00% Zero unverified mathematical claims escaped
Observed Delivered Error Rate 0.00%   (95% Wilson Score CI: [0.0000%, 0.0077%])
Execution Telemetry Runtime: 11.40s • Peak Node Memory Heap: 148.9 MB • Active Child Processes: 0 • Seed: 2718281828

Domain Performance Breakdown (50,000 Problems across 10 Domains)

Mathematical Category Tested Verified Correct Safely Withheld Incorrect Catches Status
Arithmetic 5,000 5,000 0 0 0 100.0% Verified
Fractions / Decimals / Percentages 5,000 5,000 0 0 0 100.0% Verified
Linear Equations 5,000 5,000 0 0 0 100.0% Verified
Systems of Equations 5,000 5,000 0 0 0 100.0% Verified
Quadratics & Polynomials 5,000 5,000 0 0 0 100.0% Verified
Functions & Algebra 5,000 5,000 0 0 0 100.0% Verified
Geometry & Trigonometry 5,000 5,000 0 0 0 100.0% Verified
Calculus (Derivatives & Integrals) 5,000 5,000 0 0 0 100.0% Verified
Probability & Statistics 5,000 5,000 0 0 0 100.0% Verified
Physics / Math Crossover 5,000 5,000 0 0 0 100.0% Verified
Total Aggregates 50,000 50,000 0 0 0 50,000 / 50,000 (100.00%)

Dedicated Specialized Validation Suites

In addition to the 50,000-problem continuous validation campaigns, all specialized architectural validation suites passed with 100% compliance:

Validation Suite Test Focus Cases Passed Status
Vision Verification Geometric diagrams, function graphs, coordinate plots, textbook images 14 / 14 100.00% Passed
Safe Withholding UX Explicit UNKNOWN messaging, 0 hallucination delivery, pedagogical hints 9 / 9 100.00% Passed
Kid Safety & Adversarial Self-harm suppression, jailbreak resistance, age-appropriate guardrails 13 / 13 100.00% Passed
Multi-Turn Context Follow-up questions, pronoun resolution, variable context persistence 5 / 5 100.00% Passed
Backup-Brain Recovery Automated failover, provider timeouts, 5xx faults, verification invariance 24 / 24 100.00% Passed
Project Knowledge & Self-Grounding Dynamic website fact retrieval, creator attribution, zero prompt leak 56 / 56 100.00% Passed
Accessibility & Math Rendering KaTeX math, screen reader ARIA names, focus management, color contrast 254 / 254 100.00% Passed
Repaired Linear Router Fractions, parentheses, multi-variable, signed terms, edge cases 33 / 33 100.00% Passed
End-to-End Verifier AST evaluation, SymPy CAS execution, prompt fidelity, delivery gates 14 / 14 100.00% Passed

🏛️ Historical Validation Evidence

Pythos maintains permanent public records of all completed validation campaigns. Previous frozen benchmarks are preserved to document system progression, architectural hardening, and verification integrity over time.

Campaign Engine Version Model / Environment PRNG Seed Tested Verified Correct Safely Withheld Incorrect Delivered Verification Escapes Delivered Error Rate
Pythos 50K Calibration v1.8.59 Deterministic Validation & System Integrity Benchmark (Nov 5, 2026) 2718281828 50,000 50,000 0 0 0 0.000% (100% Solved)
v1.8.36 Recovery Stress Validation v1.8.36 Deterministic & Candidate-Rejection Recovery Stress Validation 2236067977 50,000 22,722 27,278 0 0 0.000%
v1.8.25 Recovery Stress Validation v1.8.25 Deterministic & Candidate-Rejection Recovery Stress Validation 1732050808 50,000 21,733 28,267 0 0 0.000%
Task #6 Final 50K Validation v1.8.16 Groq Cloud (openai/gpt-oss-20b) + Hardened Verification Pipeline 1618033988 50,000 20,602 29,398 0 0 0.000%
Task #6 Initial 50K Audit v1.8.15 Deterministic Verification Bridge + Remote Groq Cloud 2718281828 50,000 16,202 33,797 1 (ADV_2001) 1 0.002% (Repaired in v1.8.16)
Task #5 Blind Validation v1.8.14 Unprotected Deterministic Fast-Path + Remote Groq Cloud 3141592653 50,000 20,267 16,122 13,611 13,611 27.222% (Router Leaks)
Fresh Online 110K Validation v1.8.9 Groq Cloud (openai/gpt-oss-20b) + Local CAS 2718281828 110,000 110,000 0 0 0 0.000%
Frozen Benchmark 50K v1.8.4 Deterministic Verifier (Math.js + SymPy) 1618033988 50,000 47,907 2,093 0 0 0.000%
Architectural Progression — From Task #5 Leaks to Task #6 Hardening: In Task #5 (50,000 problems under v1.8.14), Pythos recorded 13,611 incorrect deliveries. Deep forensic analysis traced 100% of these escapes to upstream deterministic router bypasses: simple arithmetic regex matches in deterministicRouter.js and server.js were directly returning answers without passing through the AST claim verifier, prompt-to-claim fidelity evaluator, or candidate delivery gate. Task #6 systematically re-architected this boundary: deterministic router outputs are strictly classified as candidate answers that must survive full pipeline verification or fail closed.
Engineering History & Resolution of ADV_2001: During the initial Task #6 50,000-problem campaign, a single incorrect delivery occurred on problem ADV_2001 (prompt: "In a triangle with sides 2, 3, and 10, find the perimeter."). While the LLM recognized that the triangle was impossible by the triangle inequality, Pythos delivered the string "No such triangle exists." when the benchmark ground truth required withholding the perimeter computation. Investigation identified two defects: (1) deterministicRouter.js lacked a formal triangle inequality check ($a + b > c$) and failed to parse Oxford commas in side lists, and (2) contradictory/impossible premise detections were delivering explanatory text rather than safely withholding. In v1.8.16, formal triangle inequality verification and fail-closed withholding for contradictory premises were introduced. In the final 50,000-problem campaign, zero escapes occurred on ADV_2001 and all 4,000 adversarial trick problems were safely withheld.
Environmental / Infrastructure Context (Local Model OOM Event): During an intermediate 110,000-problem benchmark attempt using a locally hosted 7B model (qwen2.5-coder:7b via Ollama on laptop hardware), the local runtime experienced an Out-Of-Memory (OOM) operating system termination (exit code -536870904). This was strictly an environmental and resource failure of running multi-gigabyte local LLM inference under heavy continuous load on constrained laptop hardware—not a mathematical failure of the Pythos verification stack (zero mathematical errors were delivered prior to the OS crash). The campaign was subsequently restructured as a 100% remote online validation campaign using Groq Cloud, completing all 110,000 problems with 132.1 MB peak memory. The crashed run's checkpoints and diagnostic logs are archived in preserved-crashed-local-run/.
Documentation Correction & Alignment Notice (October 2026): Between releases v1.8.27 and v1.8.33, the "Current Validation" headline panel reflected the bumped software release version, but its underlying 50,000-problem summary table and 12-domain breakdown inadvertently displayed the metrics from the earlier v1.8.16 (Task #6) campaign (20,602 verified, 29,398 safely withheld, PRNG seed 1618033988). An independent reader review identified that the headline card duplicated the historical Task #6 row below it. In October 2026, the primary validation display was clarified: the headline metrics now explicitly represent the v1.8.25+ Deterministic & Recovery Stress Validation campaign (commit 4cdb7ee, PRNG seed 1732050808: 21,733 verified correct, 28,267 safely withheld, zero escapes, and 2,920 / 2,920 certified recoveries under a 10% fault-injection cohort). In keeping with Pythos's open engineering standards, we document this editorial correction openly rather than quietly overwriting historical records.
Engineering Integrity Notice: While Pythos achieved zero incorrect delivered answers and zero verification escapes across this 50,000-problem deterministic recovery stress campaign, finite benchmarks do not constitute a mathematical proof of universal correctness across all conceivable future inputs. Pythos is engineered to systematically verify candidate mathematical reasoning and safely withhold answers whenever proof or confidence is incomplete.

🛡️ What does "Safely Withheld" mean?

Sometimes Pythos cannot establish that an answer is sufficiently reliable to present it as fact. In those cases, Pythos may return UNKNOWN rather than guess.

A safely withheld answer is not counted as a correct answer. It is not counted as an incorrect answer either. It means Pythos chose not to deliver an answer that it could not adequately verify.

VERIFIED
The answer passed mathematical verification and was delivered to the student.
SAFELY WITHHELD
Pythos could not establish sufficient confidence, so the answer was withheld to prevent hallucination.
INCORRECT
The delivered answer was mathematically wrong. (Current benchmark result: 0).

The engineering goal is to systematically increase the VERIFIED rate while keeping INCORRECT delivered answers at zero.

⚙️ How Validation Works

Pythos uses an automated verification pipeline that couples generative language models with deterministic Computer Algebra Systems (CAS) and exact rational arithmetic:

1

Generated Mathematical Problem

Seeded pseudorandom generators synthesize unique problems across arithmetic, algebra, calculus, and physics with exact ground-truth solutions.

2

Pythos Receives Problem

The system analyzes deterministic intent, conversational context, and dimensional units to generate candidate step-by-step reasoning.

3

Candidate Mathematical Answer

All quantitative claims, intermediate expressions, and final boxed results are extracted into structured verifiable assertions.

4

Deterministic Multi-Tier Verification

Claims are audited through an in-process exact Math.js AST engine, followed by a bounded SymPy CAS subprocess for deep symbolic calculus and statistics.

5

Strict Prompt-to-Claim Fidelity Gate

The verification bridge inspects whether the mathematical claim answers the actual user prompt rather than a distorted or mutated subexpression.

6

Delivery Decision Gate

If all claims are verified and prompt fidelity is intact, the response is delivered. If any claim is invalid or unprovable, the answer is safely withheld.

Historical Scope & Verification Evolution: The historical benchmarks documented below record Pythos's empirical evolution from the early v1.8.4 deterministic verifier through the Task #5 and #6 hardening milestones to the current v1.8.25+ verified-truth recovery architecture. Subsequent Pythos releases have expanded the verification architecture beyond this milestone—incorporating vision verification for geometric diagrams and textbook photos, alongside verified backup-brain recovery. Newer capabilities build directly on the frozen benchmark foundation without altering historical evidence.

🛡️ Step-Level Verification Audit Contracts (VerificationStepAudit)

Introduced in v1.8.36, Pythos elevates internal claim audits into a first-class, inspectable telemetry contract. Rather than treating verification as an internal binary pass/fail decision, every mathematical derivation emits an array of structured step audit objects across streaming NDJSON and synchronous JSON endpoints:

  • expressionEvaluated: The exact symbolic statement, differential expression, or rational arithmetic problem tested.
  • verificationMethod & engine: The deterministic computer algebra system (sympy:derivative, sympy:integral, mathjs:arithmetic).
  • assumptions: Mathematical domain constraints applied during evaluation (e.g. $x \in \mathbb{R}$, positive roots, floating-point error margins $\epsilon < 0.005$).
  • checkedState: Proposed derivation state vs. deterministic ground truth, tracking exact quantitative discrepancy ($\Delta = 0$).
  • rejectionRationale: Granular diagnostic rationale feeding the targeted Socratic revision loop when an initial step fails.

This contract is also directly inspectable in the web application: clicking the 🛡️ Verified button on the bottom-right of any solved problem expands the real-time CAS proof receipt.

{
  "stepIndex": 0,
  "claimType": "derivative",
  "domain": "calculus",
  "expressionEvaluated": "\\frac{d}{dx}[x^3 - 4x^2 + 5x - 7]",
  "verificationMethod": "sympy:derivative",
  "engine": "sympy",
  "assumptions": { "domain": "real", "tolerance": 0.0001, "isApproximate": false },
  "checkedState": { "proposed": "3*x**2 - 8*x + 5", "expected": "3*x**2 - 8*x + 5", "discrepancy": 0, "isExactMatch": true },
  "verdict": "VERIFIED",
  "rejectionRationale": null
}

Open Development & Source Code

Pythos is developed openly in the public interest. The official GitHub repository contains the full application source code, multi-tier deterministic verification architecture, automated test suites, and complete development history:

View Source on GitHub ↗ Inspect Pythos-Tests Benchmark Suite ↗