Metamorphic testing: how to check code when you can't predict the output

Most testing advice assumes you already know the right answer. You call the function, you assert it equals the value you wrote down, green.

That assumption breaks the moment the output is hard to predict: a ranking function, a route planner, a physics simulation, a machine-learning classifier, an LLM. You can't write assert answer == 42 when you don't know 42, and eyeballing a handful of outputs doesn't scale. This is the test oracle problem, and it's where most "we can't really test this" conversations quietly end.

Metamorphic testing is a way through it. The idea is small: stop asserting the output of one input, and start asserting a relationship between the outputs of related inputs. You don't need to know the right answer. You only need to know how the answer should change when you change the input in a known way.

A concrete example

Say you have a distance(a, b) that calls a maps API for driving distance. You can't hardcode the kilometres between two addresses; they change, and you don't know them up front. But two things must hold regardless of the actual number:

• Symmetry (roughly): distance(a, b) and distance(b, a) should be within a small tolerance.
• Waypoint monotonicity: going a -> c -> b can never be shorter than a -> b.

Neither assertion needs the true distance. Each is a metamorphic relation: a rule linking the output of a transformed input back to the original.

from hypothesis import given, strategies as st

places = st.sampled_from(KNOWN_PLACES)

@given(a=places, b=places, c=places)
def test_waypoint_is_never_shorter(a, b, c):
    direct = distance(a, b)
    detour = distance(a, c) + distance(c, b)
    assert detour >= direct - TOLERANCE

If someone swaps a + for a - in the accumulator, or truncates a leg, this fails, and you never wrote down a single expected distance.

Relations worth stealing

The real skill is choosing relations. A few that generalise:

• Numeric / scientific: sin(x) == sin(pi - x); doubling every input to a linear model doubles the output; sorting a permutation of a list gives the same multiset.
• Search / ranking: adding a more specific term should not grow the result set; re-running the same query returns the same top-k; permuting equally-relevant terms shouldn't reorder results.
• ML classifiers: appending a neutral sentence to a review shouldn't flip its sentiment; replacing a word with a synonym shouldn't change the class. These are robustness relations, and they catch brittle models that example-based tests sail past.
• LLM / RAG: paraphrasing the question shouldn't change a factual answer; shuffling the retrieved chunks shouldn't change a grounded answer; asking in another language should preserve the facts.

That last group is why I keep coming back to this. When the system under test is itself probabilistic, you often can't assert the output at all. But you can assert that two equivalent inputs produce equivalent outputs, and then measure how often that breaks.

The bug example-based tests miss

Example-based tests check the points you chose. They are strongest exactly where you already suspected a problem, and blind everywhere else. Metamorphic relations check the shape of behaviour across inputs you never enumerated.

A pattern I have hit in the wild: a recommendation scorer that looked correct on every hand-written case, but violated "adding an item the user already owns shouldn't raise its score." No single example caught it, because nobody thought to write that example. A random input pair plus the relation caught it on the first run.

Running it

You don't need a framework. A metamorphic test is an ordinary test that (1) builds an input, (2) transforms it by a known rule, (3) runs both, (4) asserts the relation. Property-based tools like Hypothesis (Python) or fast-check (JS) pair well, because they generate the input pairs for you and shrink a failure down to a minimal reproducing case.

In CI, treat a relation violation like any other failure. When the system is probabilistic, track a violation rate against a threshold instead of a hard pass/fail, the same way you would budget flakiness.

Honest limits

Metamorphic testing finds violations of relations, not correctness. A system can satisfy every relation you wrote and still be wrong in a way you never encoded. Trivial relations like f(x) == f(x) give false comfort. And a failing relation tells you that something changed which shouldn't have, not which side is wrong, so you still have to debug.

But for anything where you cannot write down the answer, it turns "untestable" into "testable," and it does so without a human labelling thousands of outputs. That is a good trade.

If you build or test systems without a reliable oracle, which relations do you lean on? I'm collecting the ones that generalise across domains.