Scaling insights without scaling analysts

Here is a pattern I have seen more than once. A company grows. More teams, more managers, more regions. Each of them wants to know how they are doing. So they ask the analytics team. The analytics team is good, so the answers are good, so more people ask. Within a year the team is spending most of its week answering variations of the same question for different audiences, and nobody has time left to look at anything new.

The usual diagnosis is headcount. We need more analysts. Sometimes that is true. More often the queue is not a capacity problem, it is a design problem, and hiring into it just makes a bigger queue.

The question behind the question

When you actually read a month of incoming requests, most of them are not research. They are lookups. Where does my team stand. Are we ahead or behind. Which accounts should I be worried about. Those are not questions that need an analyst's judgment. They need a reliable, trusted place to look.

The requests that genuinely need an analyst are the other kind. Why did this change. Is this effect real or noise. What happens if we stop doing the thing we have always done. Those are worth a person's full attention, and they are exactly the ones that get starved when the queue is full of lookups.

So the job is not to answer faster. It is to separate the two, and get the first kind out of the queue entirely.

What I actually did

At BrowserStack I built a self-serve layer for this. Not a dashboard, which we already had plenty of. A layer that let a manager answer their own standing questions without filing a request.

A few things mattered more than I expected.

Start from the manager's week, not the data model. I did not begin with what was available in the warehouse. I began with what a pod manager has to decide on a Monday morning. That turned into a set of coverage measures, fifteen of them, that mapped to how the teams were actually run. Fifteen was a deliberate number. Enough to cover the real questions, few enough that people could hold the whole set in their head.

Make the time frame match the planning cycle. The pacing ran on a thirteen week view because that is the window the teams plan in. A monthly view would have been easier to build and would have been quietly useless, because nobody managed to a month.

Build for the pods, not the org chart. Twenty four manager pods across twenty two teams, each with slightly different motions. The temptation is to build one view and tell everyone it applies to them. What worked better was a common spine of definitions with enough slack that each pod could see itself in it.

The result, after it settled, was around thirty five people using it every week and roughly eighteen analyst hours a week that came back to the team. Those eighteen hours are the part I care about. They did not disappear into nothing. They went into the second kind of question.

Adoption is the deliverable

This is the part I underrated at first. I thought the deliverable was the system. It was not. The deliverable was people using the system instead of asking me.

A self-serve layer that nobody trusts is worse than the request queue, because now you have both. People still ask, and you also have a thing to maintain. The failure mode is not technical. It is that someone looks at the number, it does not match what they had in their head, they cannot work out why, and they quietly go back to emailing an analyst.

So the unglamorous work turned out to be the real work. Making every number traceable to how it was defined. Making the definition visible next to the number rather than buried in a doc. Sitting with the managers whose numbers looked wrong and finding out whether the system was wrong or their mental model was. Sometimes it was the system. Often it was a definition nobody had ever written down.

What I would tell someone starting this

Do not begin with tooling. The tooling question is the least interesting one and it will absorb all your attention if you let it.

Begin by reading your own inbox. Sort a month of requests into lookups and real questions. Count them. That number is your business case, and it is usually more persuasive than anything you could argue in the abstract.

Then pick the smallest set of measures that covers most of the lookups, agree what they mean with the people who will be judged by them, and put them somewhere those people already go. Resist adding more. Every measure you add is one more thing that can be wrong, one more definition to defend, and one more reason for someone to stop trusting the whole thing.

And measure yourself on the queue, not on the build. If the requests have not dropped, it has not worked yet, however good the system looks.