How to retire a rule everybody believes

Every team I have worked with runs on a handful of rules that nobody can source. Close it inside this many days. Never discount past this point. If they have not done that by week three, walk away. The rules are usually not stupid. Someone noticed a real pattern once, said it in a meeting, and it hardened.

The trouble is that a rule like this quietly reorganizes everything around it. People start managing to the rule instead of to the outcome. Process gets built on it. And because it feels like experience rather than a hypothesis, nobody thinks to check.

We had one of these. The belief was that a proof of concept needed to close inside a forty eight day window, and that a POC running longer than that was effectively lost. A lot of behaviour followed from it, including pressure to shut POCs down at the deadline.

I did not think the rule was obviously wrong. I thought it was untested, which is a different thing.

Why you cannot just compare the two groups

The naive version of this analysis takes about ten minutes. Split deals into those that closed inside the window and those that did not, compare win rates, and you will almost certainly find that the fast ones won more often. Which proves nothing.

The deals that move fast are different deals. They tend to have a clearer problem, a sponsor who can sign, fewer stakeholders, less custom work. Those same properties make them more likely to be won regardless of speed. So the fast group looks better because of what it was, not because of how long it took. Confuse the two and you will spend the next year pushing teams to hurry deals that were never going to behave.

This is the ordinary trap in almost every operational question I get asked. The groups being compared were never comparable to begin with.

What I did instead

I worked across 396 deals and used three approaches that attack the problem from different angles.

Propensity score matching, which pairs each deal that had the treatment with a similar deal that did not, matched on the things that predict getting the treatment in the first place. Inverse probability weighting, which keeps the whole sample but reweights it so the two groups look alike on those same characteristics. And straightforward regression adjustment, controlling for the confounders directly.

Three methods because no single one is trustworthy on its own. Each makes different assumptions and fails in a different way. If all three point the same direction and roughly the same size, you have something. If they disagree, you have learned that your answer depends on your assumptions, which is itself worth knowing before anyone acts on it.

What came back

Two findings, and the second one is the point.

Deals where the proof of concept actually succeeded showed around a 58 point higher win rate, with a 95% confidence interval of plus 51 to plus 65. A large effect, consistent across all three methods, and not really a surprise to anyone.

The forty eight day window showed no significant effect at all. Once you accounted for what kind of deal it was, how long the POC took did not predict whether the deal was won.

So the rule the organization had built process around was measuring the wrong thing. Length was a visible proxy for something invisible. What mattered was whether the POC succeeded, and success had very little to do with how fast it ran.

The conversation is the output

I want to be careful here, because it would be easy to tell this as a story about analysis winning an argument. It was not.

A confidence interval does not change anyone's mind on its own. What changed the conversation was that the finding gave people something better to do. If I had only shown that the forty eight day rule did not work, the honest response would have been: fine, so what do we manage to now. Nothing. Taking away a rule and offering nothing in its place is not an improvement, and teams are right to resist it.

Because the same analysis showed that POC success was where the effect lived, the recommendation was not stop doing something. It was move your attention here instead. That is a conversation people can act on.

I also tried hard not to overclaim. This was quasi experimental, not a randomized test. It rests on the assumption that we measured the things that drive both POC outcome and deal outcome, and we almost certainly missed some. I said that plainly, including the part about what would have made it stronger. In my experience stating the limits buys more credibility than hiding them, because the first smart person in the room is going to find them anyway, and it is much better if you found them first.

If you want to try this

Pick the rule your team repeats most often and has never tested. There is always one. Write down what it predicts, then ask the boring question: are the two groups I am about to compare actually comparable, and if not, what makes them different.

Most of the work is in that question. The methods are learnable in a weekend. Knowing what you are controlling for, and being honest about what you could not control for, is the part that takes judgment.

And before you start, work out what you will offer if the rule turns out to be empty. If the answer is nothing, the analysis will be correct and will change absolutely nothing, which is the most common way good work gets wasted.