Why Judging Product Managers Is Hard
It’s easy to believe that product managers with long tenures at established software companies like Meta, Google, Netflix, and Amazon are awesome at what they do. We might also assume the same for product managers from well-funded late-stage venture-backed startups like Stripe, Airbnb, Coinbase, SpaceX, or JUUL. The trouble is, working in successful places does not necessarily mean the product people are the reason for the success.
I personally found this realization hard to swallow — after all, I learned how to be a product leader at Palantir, a company that raised billions in venture funding. When I look for role models, I try to find people with repeated success stories, ideally in completely different contexts. The reason is simple — we (and others evaluating us) overestimate our agency in success when we are part of a larger narrative of success. If we are part of a success story, we assume that ALL of our decisions in that context were great, and we tend to assume that we were a key reason for the success. Conversely, if we’re part of a failure, we tend to assume that nothing we did was right. It’s not impossible that these extremes are true, but it’s not likely even for the best of the best or the worst of the worst.
Let’s conceive of a tangible example with fictitious numbers: suppose a product manager needs to do 1,000 things per year and at least 700 of those must be great for success.
The first issue is that we rarely know in the moment which were the 700 good and the 300 bad. More importantly, if 700 of 1,000 efforts need to be great for success in a vacuum, it may be the case that 400 strong ones out of 1,000 still produces success when the person is inside of an already successful environment. That individual is buoyed by so many factors they don’t impact directly that protect their record of achievement even when they are otherwise failing. For example, great engineers, great sales people, strong product-market-fit, and macro geopolitical pressures can all make a 400 out of 1,000 year look and feel like a perfectly executed set of wins.
The converse is also true — the product managers who face headwinds in each of these categories may make 900 out of 1,000 great decisions and the company may still fail for all the same reasons, and that product manager (or the next founder to hire them) may incorrectly conclude that they are less amazing than the others. This phenomenon can make great product managers exhibit imposter syndrome if they aren’t associated with publicly recognized success stories, and it can also create some seriously inflated egos among the ones who are.
Of course, the best of the best are like Michael Jordan and the 1984 Chicago Bulls; they make the team/product great by producing 1,500 units of great work and this counteracts the effects of the tough environment.
Let’s dive into a real story of this in action. In the early days at Palantir, we on the Forward Deployed Engineering team (as distinct from the Product team) spent a lot of time in the field building extensions of our product to cover the last mile of customer analytic needs. Our success rate on these custom apps was phenomenal. Now let’s fast forward a few years to a time at Palantir when I led a product effort to provide something other than a data analytics extension to a new customer. Even with the same people, the same culture, and the same tools, we did not immediately win; in fact, our first launch was mediocre. I wrestled with this a lot and realized something crucial and humbling: if the product we were building was already based on having that customer’s data integrated for the first time from many silos, the chance of success was always going to be very high relative to true 0 to 1 innovation. We were innovating inside a small envelope of existing product-market-fit on problems that were already understood. On the new initiative, we didn’t have that success to build on, and we had to rapidly raise our product skills to move the needle. The moral of the story is that without the second experience, we might have assumed that our product skills were undefeated merely because we did things that were successful without taking the time to dissect how much agency we contributed. Of course, it’s still valuable to build the best thing on top of product-market-fit, that’s an in-demand and crucial product skill, but it’s far from the whole product enchilada.
Let’s look at Meta now. From a product perspective, Meta’s product Facebook is quantitatively the most successful consumer product on planet Earth. Its yearly growth is higher than almost any other product’s total adoption, and it has translated billions of users into $100s of billions in revenue. For the sake of this example, we won’t address whether it’s a force for good or bad in the world, just the objective, numerical success of the thing from a product perspective from 2004 through 2021.
With this context, suppose you’re a product manager on Facebook Marketplace in 2016. It doesn’t matter externally what you did or did not accomplish because Facebook grew by a half billion users that year, so you are a product deity in the eyes of everyone who meets you. But are you? First, how many of those half billion users joined the platform BECAUSE of what you did with the Facebook Marketplace? “But 100M people used the marketplace that year [guessing, I don’t know the actual stat] so I’m better than a product manager at a seed-stage venture that has less than 5,000 users for sure,” you tell yourself. Right, but you got 100M users because 1 in 20 people who already use and love Facebook saw a new icon on the navigation bar and they clicked it. You are not a genius (necessarily). You are not “better” than the other guy. Nonetheless, the public signaling of monumental company success or huge capital raises makes you feel like you are, and it makes the rest of the world implicitly trust that you are qualified without further scrutiny.
I’d like to extend the Facebook growth example one layer deeper. Now let’s suppose you worked as the product manager for Facebook birthday reminders in 2011. It doesn’t cost users anything to use this feature or give it a test drive, and on day one, you have the users’ entire lists of friends’ birthdays. Let’s say you win 10M users in the first month with this feature. Are you a product savant as a result? Are you better than the product person who won 1,000 paying users for a new product that’s never existed before? Your users did not have to sign up for your service (they were already Facebook users) and they didn’t have to trade money for your product, so what have you proved about your skills? Facebook, as an example, is all about engagement (more time in the app so people see more ads so Meta makes more revenue), not conversions (a new person deciding to buy your product because it solved something crucial), so the entirety of the experience there for the majority of their product managers lives in this tiny slice of what product work is all about. If you need to drive engagement with a product that has already achieved strong product-market-fit with a large user-base then a Facebook product alumnus could be the perfect fit.
Conversely, to make the point, you could have fully researched and built a product no one ever saw before and sold the first 1,000 units through hand-to-hand combat with users, made people’s lives better, and then you could still flunk as a Facebook product manager because you’ve never done heavy data analytics on billions of users before. I’m not for a second saying that all product people from reputable places are bad and all scrappy ones are good, nor the reverse; I’m just asking everyone to slow down and dig deeper because a product person’s true experiences and accomplishments are deeply colored by where they did the work and the phase of the company they did it at.
To be clear, self reflective product people generally know when they are out of their element, so it’s more about avoiding the people who don’t know the difference and helping Founders who may be choosing or evaluating product people for the first time to understand who they are getting and the relative strengths they will have. It’s not easy to parse the difference, but it’s not impossible either. I recommend trying any of the following:
- Drill into the outcomes the specific product produced under the product manager’s tenure. Attempt to decompose and remove the macro factors. Were they riding a wave or did they create a wave?
- Ask them to compare their outcomes to those of their peers. While winning 10M users for a new Facebook product enhancement may be hard to judge, if everyone else earned 100M on their launches, we can better judge where this product manager stands.
- Deeply understand the stage and phase of the product work the product manager has executed and compare that to your challenge.
- Try to parse how much they learned from the customer and market during their work and how those learnings impacted success and actual changes to the product.
- Get the product manager to tell stories of 0 to 1, not just 1 to n. Press them to justify why they believe they were starting from 0 and not 1.
- Find out how many places and environments the product manager has operated in as this will help to elucidate how likely they are to understand the differences across contexts.
At the end of the day, very few product managers are great at every style or every skill at every stage. With this checklist, however, you can proactively look for the ones who know how to crush the phase you’re at. At a minimum you can weed out the ones who aren’t self reflective enough to know they likely weren’t always the reason for the success or failure by themselves.
Hi! If you enjoyed these insights, give us a follow, and if you are interested in tailored support for your venture, please visit our website at First Principles, where we focus on product to help the world’s most ambitious Founders make a difference.
