Learning, not failure

Context

PMs are writing down success measures for every feature they build, but they aren't attending to the results - instead, they're moving on the next thing.

Discussion

My old boss Zach Nies teaches startups and growth companies about how to use the scientific method. In his courses he often shared the first minute of this lecture from the late physicist Richard Feynman:

Zach always reminded us that learning happens when we compare our expectations to what actually happens. He also liked to share a Daniel Kahneman quote:

"Hindsight bias makes surprises vanish."

That is, if you don't write down your hypothesis before you take your measurement, you will just justify whatever happens as obvious and miss the chance to learn something.

So, how to help our PMs to learn from their experiments?

Play

There are 4 ways I teach PMs to learn and adapt:

• Use a review card in your feature/Epic/MBI template
• Set aside 30 mins for a 1:1 conversation with a single PM about their results
• Create a regular 'learning call' with 5-6 PMs where they share what they've learned with each other
• Use a weekly standup around a Level 2 (feature) Kanban board to share news about results

Review Card

Add a simple section to the bottom of your MBI template and ask the PMs to fill it out before you meet with them. Mine is modeled after an old PDF from Strategyzer. Here's a completed example:

Coaching 1:1 after feature release

PMs will almost never set the right success criteria the first time they try. You need to have a conversation with them after the feature is launched to guide them through the learning process. Book 30 minutes with each PM (at least once per PM) to guide them through what it's like to evaluating their own results.

In that conversation, you ask,

• How did we do? Did we meet the criteria?
• Do you care whether we met them? Or did we realize a different goal is more important?
  • Sometimes the PM is happy to claim credit for success, but when you ask them if they care, they will admit that the result isn't actually the one that matters.
• What are you still hoping to learn?
  • How can we learn it sooner?
• What instrumentation (tracking) did we forget to add?
  • Can we still add it? Is it worth the effort?
  • Often they will neglect a 1-hour engineering task because they are so excited to move to the next thing.
• If we didn't meet our criteria, is it worth it to wait and watch?
  • Or have we reached the end of our learning?
  • Often they will waste a bunch of time and money studying something that they already know won't have any business impact, but a people-pleasing PM might need your permission to let it go.
• If you were back at the start, what success criteria do you wish you had set?
• What changes will you make for other success criteria for other features/projects?

Regular Learning Call

Every 2 weeks, gather all of your PMs for 45 minutes to discuss what they have learned from recently shipped features. Do it round robin style, asking each PM to talk about their learnings. They'll ask questions of each other, and over time challenge each other on better success criteria. Not every PM will have something to share every time, but someone will, and this is a great discussion to have. I do not recommend including stakeholders (because this is a learning opportunity - keep safety up by keeping it inside the team) but your CPO or CTO might love joining (depending on their personality).

Weekly Standup

If your tech department is up to about 50 people (product, engineering and design) you can have a weekly 10 minute standup around your Level 2 (feature) Kanban board to share progress. Typically this would be with 10-20 people, including PMs, engineering leaders, and key designers. This can also be a great time for PM to share news about results. You'll need a 'measuring' or 'learning' column after release before moving features to 'done'.

Cautions and Caveats

This is a low-risk play, but watch safety closely. If you are discussing results with too many senior people watching, PMs will not be able to talk honestly about their learnings. It's fine to discuss results with executives, but you might need a private discussion first to help the PMs get more comfortable sharing what they have learned - especially if they feel it is bad news.

It is much easier to share 'bad news' with grouchy execs if you follow the above template. If your PM just says 'We didn't meet the target', they're creating an opening to be yelled at. Instead, if they say 'Here's what we observed, here's what we've learned, and here's what we're already doing about it' they are less likely to unnecessarily activate the exec's lizard brain.