Are We Asking the Right Questions? The Case Study Interviews for Product Managers.

listen to the Podcast by NotebookLM

"How would you design an elevator for a 50-story building"

This was the case study question posed to my mentee during her interview for an e-commerce product manager role. Like many companies, the interviewer was trying to evaluate her product craft and ability to think on her feet. But was this actually measuring what mattered for the job?

In my decade(s) of product management experience, I've seen case study interviews used as both powerful evaluation tools and counterproductive time-wasters. The difference often comes down to one factor: relevance.

I discussed my opinion on product craft in an earlier post, so I won't delve into that topic here. Instead, I'll focus on the merits I find in case study interviews. 

The Merits of Case Study Interviews

A quick search on Perplexity for types of case study interviews for product managers yields the following categories:

• Product Design
• Product Improvement
• Product Strategy
• Metrics and Data Analysis

"These types of interviews have merit and can provide value. By examining relevant scenarios, they can yield valuable insights when used appropriately. But what happens when these case studies miss the mark?

Irrelevant Case Studies

In practice, I often find that case studies are used in irrelevant contexts, merely to gauge a candidate's product sense or craft. This approach can be counterproductive and may not accurately assess a candidate's abilities. To illustrate this point, let's look at some examples from my mentees and colleagues:

• Elevator Design for an E-commerce Role: A candidate interviewing for an e-commerce company is asked, "How would you design an elevator for a 50-story building?"

• VR Launch Strategy for a Furniture Retailer Position: Another candidate, interviewing with an online furniture and home products retailer, is given a couple of days to develop a one-hour presentation. The task is to demonstrate their "product craft" by recommending how Epic should launch VR (virtual reality) support in Fortnite for a truly immersive experience.

• Pizza Parlor Website for an Online Grocery Role: An online grocery company asks candidates to design a direct-to-consumer website for a new pizza parlor as their "think-on-your-feet" PM case study.

These scenarios highlight a common issue: relevance.

The Importance of Relevance

In each of these cases, the case study is disconnected from the company's core business and does not reflect the challenges the candidate would face in the role. This approach can lead to inaccurate assessments and may disadvantage candidates who are unfamiliar with the unrelated domain. Conversely, the candidate may come out of the interview process with the wrong assessment of what the role will entail. 

In the pizza parlor website example, it was standardized across all PM interviews in my colleague's company. She encountered an issue because she owned backend services and led a team of technical PMs. At their six-month checkpoint, two new PMs she hired expressed disappointment because they expected to be working on customer-facing features. Now, don't ask me how launching VR support in Fortnite will make me a better PM to sell furniture online.

To truly evaluate a candidate's abilities, case studies should be relevant to the company's industry and the role's responsibilities. This not only allows candidates to demonstrate their skills in a meaningful context but also provides the interviewer with insights into how the candidate would perform in real-world scenarios pertinent to the position.

The Challenge of Relevant Case Studies

Having been on the hiring side, I realize this is a fine line to tread. If we ask case study questions that are too close to the business at hand, I can foresee the headlines: "Company X is asking candidates to solve their problems disguised as case study questions... candidates should be paid for their answers..."

But I believe candidates who apply to a role because their experience or objectives align with the company's products and mission may appreciate this approach more. It demonstrates that the company values relevant expertise and provides candidates with an opportunity to showcase their skills in a context that matters. However, despite understanding the potential benefits, I don't like case study interviews.

My personal preference

I don't favor case study interviews. Even when they are related to the role, as an interviewer, I'll be making up data and scenarios. These would be based on my own experience, but given the same facts, scenarios, or data, different people will have different reactions and will tackle them differently. I have my own biases. Should I continue to nudge the candidate in the direction I think they should be going to solve my fictional case study? Is my solution or my preference the right approach?

To be transparent—and perhaps this is my bias—as a candidate, I had a low success rate with case study questions. I struggle with them. It is just not the way I operate as a product manager. I need to digest all available information, immerse myself in the space I am operating in, and conduct research. A picture, an opinion, ideas, and plans start to emerge in my head.

As an interviewer, I prefer to use behavioral-based interview questions—the classic 'tell me about a time...' prompts. This preference stems from my belief that behavioral questions elicit real-life examples from candidates in their domain and describe how they deliberately delivered results, solved customer problems, identified KPIs, and so on. I can inspect their reasoning, the approaches they considered, the specific role they played, and, in retrospect, what they would have done differently.

As a hiring manager, I am interested in understanding the candidate's thought process in solving real problems, not how well they apply a product sense framework. I want to see how they use what they know about their problem space, their product, and their customers to deliver an optimal solution—not how well they think on their feet in a hypothetical scenario. This approach aligns with a more authentic assessment of their capabilities.

Conclusion

Case study interviews can be a valuable tool in assessing a candidate's abilities, but only when they are relevant to the role and the company's industry. Irrelevant case studies not only fail to provide meaningful insights but can also mislead both the interviewer and the candidate about their suitability for the position. Behavioral-based interview questions, grounded in real experiences, offer a more accurate gauge of a candidate's capabilities and fit within the team. As hiring managers, we should strive to create an interview process that genuinely reflects the challenges and opportunities of the role, enabling candidates to showcase their true potential.