Evals are the new PRD
When a feature's output is non-deterministic, your spec can't be a list of screens. It has to be a definition of 'good' the whole team agrees on — which is what an eval really is.
For a deterministic feature, a PRD describes behavior: click this, see that. For an AI feature, behavior isn't fixed — the same input can produce different outputs, and "correct" lives on a spectrum. So what does a spec even mean?
It means the eval.
The spec is the definition of good
An eval set is a collection of inputs, expected qualities, and a way to score how close the output gets. Writing one forces the exact conversations a PRD used to: What counts as a good answer here? What's a tolerable failure versus a trust-ending one? Who decides?
If your team can't write the eval, you don't actually agree on what you're building. You just agree on the demo.
Acceptance criteria for the non-deterministic
The shift is from "does it do X" to "how often, and how badly does it fail when it fails." That reframes product decisions:
- A feature that's right 95% of the time but catastrophically wrong 5% may be worse than one that's right 85% and gracefully uncertain the rest.
- The cost of each failure mode belongs in the spec, not the postmortem.
Why PMs should own this
Evals are where product judgment meets the model. Engineers can build the harness; only the product owner can say what "good" means for this user, this decision, this risk. Hand that off and you've outsourced the most important product decision you have.
Write the eval first. It's the realest PRD you'll ever ship.