A Fortran 77 ray-tracing kernel from the mid-1980s still runs in some seismic tomography pipelines. Its test suite checks travel times against a reference Earth model. The tests pass. The kernel is numerically identical to its 1985 self, which is exactly the problem. New tomography datasets, including those from dense broadband arrays, imply upper-mantle velocities that differ from the old reference by roughly two to five percent in some depth ranges. That is small enough to hide inside noise for many applications and large enough to matter for inversions. The mismatch surfaced only when a team ran a manual audit, comparing the kernel's predictions against a current model.

The Solver Outlives Its Physics

Legacy Fortran solvers are unusually durable. A codebase compiled in 1985 still runs on a laptop today, which is a remarkable property and also a trap. The unit tests that ship with it were written to confirm the code matched the physics of its era. They do that faithfully. What they no longer do is confirm the code matches physics as currently measured.

The gap is an epistemic one. A passing test suite tells you the numerical output has not changed since the baseline was recorded. It says nothing about whether the baseline was ever right, or whether the world has moved on. Reproducibility in the bitwise sense and physical validity are different claims, and only one of them is usually tested.

This matters because computational results feed into published findings. When a solver drifts, the drift propagates into papers, and the papers cite the solver as validated. The validation, on inspection, often reduces to a test file that has not been touched since the Reagan administration.

Unit Tests Codify Stale Assumptions

A regression test works by freezing behavior. You run the code, record the output, and assert that future runs match. That is the right design for catching accidental edits. It is the wrong design for catching deliberate physics updates, because any update that changes the output will fail the test, and the fastest way to make the suite green is to revert.

Test oracles carry the assumptions of their authors. A 1980s benchmark value embeds the constants, the Earth model, the boundary conditions, and the numerical scheme of that year. When the constants change, the oracle does not. The test still passes, because the code still produces the old number. The old number is now wrong.

Climate models illustrate the pattern. The Community Climate System Model version 3 (CCSM3), released in 2004, included a parameterization of cloud radiative properties tuned to satellite observations from the 1980s. That tuning was correct for its dataset. Successor datasets from the 2000s disagreed in some regimes. The regression suite, if it existed, would not flag the disagreement because it was never built to look outside the code.

Why Reproducibility Tools Miss the Gap

Containers and continuous integration preserve code, not physics. A Docker image pins library versions and compiler flags. It does not pin the scientific assumptions embedded in a hardcoded constant. Two researchers can run the same container and get bitwise identical output while both are wrong in the same way.

Version control tracks lines, not meanings. A commit that changes a constant from 6.371 to 6.378 megameters is a one-line diff. The commit message rarely explains which measurement motivated the change, or which measurements the new value supersedes. Six months later, the provenance is gone.

Peer review rarely inspects test oracles. Reviewers check the method and the results. They seldom open the test file, and when they do, they see passing tests and move on. Archiving guidelines, meanwhile, emphasize bitwise reproducibility, which is valuable but orthogonal to whether the physics still holds.

There is a reasonable defense of the current practice. Bitwise reproducibility is hard enough to achieve, and demanding physical revalidation of every archived code would make archiving impractical. The counterargument is that a reproducible wrong answer is not obviously better than an irreproducible one.

Signs Your Test Suite Has Become a Fossil

Several symptoms recur. Tests reference physical constants that no longer match the values in current use, or that were never given units. Expected outputs contain hardcoded parameters that appear nowhere in the documentation. Changing the input dataset does not fail any test, which means the tests never exercise the data path where the physics lives.

Documentation is another tell. If the cited literature predates 1990 and the field has published revised values since, the test suite is probably encoding the old consensus. That is not automatically wrong. Some old values remain the best available. The question is whether anyone checked.

A related pattern appears in long-running laboratory operations, where a per-diem rate or a colony protocol can outlive the findings it was designed to support, as described in a 1986 primate lab's per diem. The administrative shell persists after the science moves on.

None of this is a criticism of the original authors. They wrote tests appropriate to their time. The failure is institutional: nobody was assigned to revisit the oracles.

Auditing Test Oracles

Audit test oracles against current physics, not just against prior output. Pick the ten tests most central to your solver's claims and ask what physical assumption each one encodes. If you cannot answer, that test is a candidate for rewriting or retirement. For example, in the ray-tracing kernel, the oracle might assume a constant velocity gradient that no longer matches tomographic inversions; rewriting it to accept a velocity model as input would make the assumption explicit and testable.

Add integration tests that consume modern datasets, and separate numerical regression from physical validity in your test naming and reporting. A unit test that checks a function against a 1985 benchmark is useful for regression. A test that checks the full pipeline against a recent observational product catches drift. The two serve different purposes and both belong in the suite. A test named test_travel_time_matches_reference should say which reference, and when it was current. Make the distinction visible in CI output so a green build is not mistaken for a validated model.

Document assumption provenance in code comments. When you change a constant, record the measurement, the paper, and the date. A future reader should be able to reconstruct why the value is what it is without archaeology. Re-run benchmarks with updated constants on a schedule—annually is a reasonable default for slow-moving fields. The point is not to chase every revision but to notice when the gap between your code and the literature has grown large enough to matter.

There is disagreement about how much of this belongs in the code versus the paper. Some argue the code should be physics-agnostic and the paper should carry the assumptions. Others argue the assumptions must live where the tests can see them. Both positions have merit, and the right answer probably depends on how long the code is expected to survive. Most code survives longer than anyone plans.