This is a guide to evaluating product/feature ideas, meant for product managers and product owners who have taken a start in this career, and are looking to up their product management game, as well as build their leadership skills. This is not a guide on how to conduct user interviews or A/B testing, and assumes that you know that already.

What is the core job of a product manager (PM)?

The number one job of a PM is to make sure that the team is working on the next most valuable thing. Knowing what is the most valuable thing to work on, is not an intuition-based determination. This is where a distinction between ‘problems’ and ‘solutions’ is useful. Knowing what are the most important problems to solve, allows you to explore solutions to those problems.

Ideally, we would move from a problem definition to exploring solutions, but that’s not a luxury most PMs have. In reality, it is likely that you are getting bombarded with many solution ideas but without an implicit or explicit understanding of the problem being solved. If you give in to this bombardment, your team may be spending time on sub-optimal or even detrimental activities that serve no good for the customers and the company. So, to execute well on your core job, you need to be able to evaluate product and feature ideas effectively.

Ideas come from various sources. Although Marty Cagan seems to suggest that the sales department may not be the best source of ideas, that mindset can create unnecessary rifts between product and sales. Donald Asher (in a totally unrelated book called “Cracking The Hidden Job Market”) says, “The value of information is independent of the source of information”. I would encourage you to not pre-maturely shut out ideas based on where they are coming from, but assess them as you would assess your own.

You also need to be able to move the conversation back from solutions to problem definition when it’s not well defined. I have been in numerous conversations where the question “what is the problem we’re trying to solve?” invites blank stares — either because no one has really thought about it, or the answer is in someone’s head but not shared across the team. This article will help you tactfully move the discussion from solution back to problem definition. However, sometimes it’s a company culture that is preventing product-led thinking. If you find yourself repeatedly struggling despite your most tactful attempts at steering the conversation in the right direction, you might have to consider whether this is the right place for you.

The Risk Exploration Framework

The following matrix sets a framework with some questions that will help you attack product/feature ideas in a systematic way. This matrix combines some elements of the Socratic method of questioning and Marty Cagan’s four types of product discovery risks, along with my own experience. If you haven’t yet read ‘Inspired’ by Marty Cagan, I suggest you do so, to fully appreciate this discussion. But as a 30 second refresher, the four types of risks include Value risk (whether customers will buy it or users will choose to use it), Usability risk (whether users can figure out how to use it), Feasibility risk (whether engineers can build what we need with the time, skills and technology we have) and Business viability risk (whether this solution also works for the various aspects of our business).

For the sake of understanding, this framework is presented as a somewhat linear, but any PM with real life experience knows that you need to be flexible and jump back and forth according to the situation. A quick 2-minute review of this framework can help you get ready for a discussion with other team members about a new product/feature idea or solution.

1. Clarifying the idea and its origin

• What exactly is the idea? Vaguely described ideas can mean different things in people’s minds. It’s amazing how far some ideas can go through development until someone realizes that people have been talking about different things. Pushing for specific descriptions, specific examples, or user flows can be extremely helpful. It can be disarming as well for someone who is coming in with a pre-conceived but not well-thought-out solution. “We need a chatbot” is not good enough. What does the chatbot do for the user? Where, when and how is the user using it?
• What is the context and origin of this idea? How did this come about? What business context or problem triggered this? This is hinting at the problem definition. If the answer to this question is blank, you’re probably looking at some fancy idea you or someone had. But more often, this question will trigger interesting debate about the actual problem (and whether it really is a problem worth solving).
• Why is this a good idea in terms of value, usability, feasibility, and viability? This question is attempting to disclose the assumed ‘business case’ or ‘benefit statement’ of this idea. In particular, around usability, it’s helpful to ask: Is this a good idea in the context of the user’s environment at workflows?

2. Exposing and challenging assumptions

I’ve found this to be the easiest point of intervention when some stakeholders are pushing on solutioning pre-maturely. “Noting down assumptions” is usually acceptable even when you are dealing with stakeholders who come from a non-agile background.

• What must be true for this idea to provide the value we think it will? What if that’s not true?
• What conditions must be true such that this product/feature is usable by the intended users? Why do we think those conditions are true? What if that’s not the case?
• How do we know this idea is technically feasible? What if it turned out to be more complex and difficult to implement? Have we talked to engineering? There are some situations where engineering input is less needed e.g. when the idea turns out to be more focused on people and processes rather than building software. Nevertheless, it’s still best to have engineering by your side to confirm that.
• How do we know this idea is marketable and sellable? Do we know the legal, compliance and security concerns? What if I thought the opposite? Depending on the idea in question, it may be worth pulling in the relevant stakeholders very early on e.g. if your solution involves tracking users’ location by GPS, it’s best to clear security, legal and compliance concerns before you invest any more energy into it.

3. Looking for evidence and alternative evidence

Humans naturally suffer from confirmation bias. We believe what we want to believe and then we focus on facts, opinions and evidence that support our beliefs. Also, once someone has spent considerable effort thinking about and defining a solution, they are likely to get emotionally attached to it. For these reasons, it is important to candidly ask these two fundamental questions:

• What data points do we have that support this idea?
• What data points do we have that refute this as a good idea?

It’s easy to do a poor job of answering the second question if some people are already strongly convinced. That’s why it is good to have some dissenters in the team. The bigger the decision, the more important this is. In the book “Decisive”, Dan and Chip Heath analyze Quaker company’s decision about Snapple acquisition where there were no dissenters and no alternatives considered — the deal resulted in loss of $1.5 billion in three years.

Often these questions will also reveal what evidence you don’t have:

• What data do we not have that would be useful to evaluate this idea?

It’s good to note these information gaps for later exploration, unless something can be answered with a quick phone call or internet search. The riskier the assumption that is bridging an information gap, the more important it is to address that gap through discovery process.

One more thing: In fast paced tech companies, institutional memory of product changes may be lacking. But if at all possible, it is very much worth your time to find out how a previous similar product change paned out, and why.

4. Unraveling consequences and implications

This exercise will reveal many different assumptions and risks. Not all of them carry the same weight, and it is not possible to address every little thing. Thinking about consequences, second order effects, and alignment with corporate strategy can help decide which risks are worth addressing:

• Which assumptions/risks are the most consequential?

To answer the above, it is helpful to think across the four types of risks.

• What if this idea doesn’t end up providing the expected value? Usually this is the most important type of risk to consider. What are the consequences for the team and the business? How much time and resources might be lost? Would it adversely impact brand value? Also think about your leadership development — is this a high stakes, high visibility project failing which will damage your reputation?
• What if product usability turns out to be poor? Often the consequences are user frustration, low engagement and eventually impact on revenue. Fortunately, these are often more reversible and easier/cheaper to fix.
• What if the technical effort turns out to be much higher than anticipated? Maybe a hacky solution works in the short term but at the expense of creating long term technical debt. Patchwork in software encourages more patchwork until the system becomes so bloated that it requires a large multi-year effort to revamp everything. Yet teams might still be ok with taking on technical debt for a feature that is expected to have a short life.
• What if the product/feature creates consequences not well aligned with marketing, sales and customer success strategy of the company? Such misalignment can derail otherwise successful products, and create internal friction between teams that is difficult to remedy. Depending on the feature/product and industry, legal and compliance consequences may be too significant to ignore. For example, a medical software hosting patient data cannot ignore privacy and security concerns.

5. Experimentation and learning

There are two questions here:

• How can this be tested cheaply and quickly?
• Which tests will lead to most important learning and de-risking?

Some teams do “pilot” testing of an idea but in their minds are already convinced that it’s a great idea. The pilot is designed as if it’s a soft launch rather than an experiment. This leads to expensive work upfront that may require revision or become throwaway. For a PM it can be challenging to defend ‘wasting’ time on testing when the team already ‘knows’ what to build. This is where the previous steps of exposing assumptions collaboratively should help build some healthy self-doubt among the stakeholders.

• How can we test this idea for value? We need to find easy ways to do some fundamental tests: Do (potential) customers even have the problem we’re trying to solve? Would they care to buy such a product? There are many qualitative and quantitative techniques that can be used, that are outside the scope of this article. However, testing is not enough; you need to learn from testing, and that is possible only if you are looking for a signal. Depending on the fidelity of the idea and experiment, the signal you’re looking for could be just a qualitative one e.g. do people with acute disease like to record their symptoms?, or it could be a more sophisticated test of statistical significance e.g. A/B testing some versions of a webpage.
• How can we test this idea for usability? Can you throw together some quick prototype that helps you test the user flows? Doing lo-fi prototypes is especially important when playing with new ideas and user flows (as opposed to fine-tuning some aesthetics on an already live product, where you really need the hi-fi design for testing).
• How can we test this idea for technical feasibility? This isn’t a binary question — it’s usually possible to build it, unless engineering is being asked to build a time travel machine. The real question is: how difficult or easy is it to build for the engineering team? and how well does it fit (or not) with the software architecture in place? Testing for feasibility can range from just a conversation with experienced engineers to a spike ticket that involves some exploration / investigation to building an actual proof-of-concept that can demonstrate the core functionality needed. As a PM, you have to consider the increasing amount of investment as you move to the right. Your requests should match the stage at which the idea is. This could also be a build vs buy question. It may be hard to build, but easy to buy. However, 3rd party services usually involves integration with existing systems, and that in itself can be expensive and should be considered as part of ‘technical feasibility’ when going for a buy instead of build. A written description of what you want, a user flow diagram, a quick mockup or other such artifacts can help engineering team do a better job of estimating technical feasibility.
• How can we check off business viability concerns? It depends on the exact risks in consideration. The most common questions are around marketing, sales, customer success and finance. Sales team may be more interested in delivery timelines, but this is not the time for committing to deadlines before de-risking ideas as much as possible. Customer success may tell you that the idea is great but will come with a significant support cost. Is that acceptable? Legal, information security, and other compliance concerns also need to be reviewed. There might be hard boundaries that cannot be crossed, or it may be a question of how much risk a company is willing to take.

6. Opportunity cost and tradeoffs

It is also possible that after this exercise, some ideas will clearly be non-starters or just not worth the benefit/risk. And that’s a good outcome! (and hopefully other stakeholders you’ve involved in the process are also aligned). Now you won’t be spending more effort on something of low value. More often though, at this point you’ll be thinking about put this idea/feature on the roadmap (as a discovery item). Resources are often limited and thus you will have two broad questions to answer:

• What are we willing to give up in our existing roadmap, in order to take on this new project?
• Is there anything else even higher value than this, that we should/could be taking on instead?

These questions can be complex for a number of reasons, which warrant a separate article in itself. But in short, you have to consider short vs long term value, complexity and effort, alignment with OKRs (if you have those) and strategy and more. The most important question being:

• How well does this idea align with our product vision and strategy?