top of page
Search
  • Writer's pictureProductance

The Most Common User Testing Mistake That PMs Make

For some of you fellow PMs out there, this first part might seem obvious. Your first thought will be, "Well duh! Tell me something I don't know already." To you I say, while this might seem too self-evident, I can tell from first-hand experience that many product managers and product owners will need to hear that.




1. User Testing is Invaluable:


This is the obvious part. Your MVP is a test, so make sure you test it. It's been my experience that some product teams skip the user testing step of the Lean Product Process. I believe there are three common reasons behind this absurd decision: (1) saving money, (2) worried about potential findings, and (3) "it's just a new feature" or "a product update." To that I say, (1) you'll lose more money by launching an untested MVP/product update without testing the many hypotheses and assumptions you've made, (2) you should be more worried about the damage to your product's reputation if you launch an untested product, and (3) you should be even more worried about those damages and the effects they could have on your business (lower retention rate, increased number of complaints...). I have seen the scenario (3) play out with at least two different products I've worked on.


User feedback is invaluable because it identifies flaws and pain-points that only a customer does. Product teams are often too familiar with the product or MVP to perceive these areas of growth. With a completely new product, PMs develop product blindness: Dan Olson defines these as blind spots for the issues that a new user will readily encounter within minutes of using the product. With a product update like in scenario (3), PMs can also develop some overconfidence bias. By testing your product, a new user might point out a weakness that existing users have become blind towards as well, while a current user might go as far as completely invalidating your hypothesis that the new update or feature is valuable to begin with. So, make sure you to perform your invaluable user tests. It's not easy to design effective user tests, but I will be sharing some tips on how to best run and design these tests successfully.


2. The Most Common User Testing Mistake that PMs Make:


When running your invaluable user tests, it is of the utmost importance that you design and conduct these tests in such a manner that allows you to differentiate between feedback on usability and feedback on product-market fit. The first kind of feedback tells you the extent to which users understand and find your product easy to use, whereas the latter speaks to the extent to which users find your product valuable to them.


If the MVP that's being tested provides a poor UX, that will most likely translate to negative usability feedback which in turn often prevents users from seeing and commenting on the value that your product (aims to) deliver. Another factor that could potentially prevent your users from providing feedback on the product-market fitness of your MVP is poor messaging (especially on a landing page; if you have one). Furthermore, even with a satisfactory design and UX, and targeted and effective messaging, product-market fit feedback is not easily provided unless it is deliberately sought after. People, especially strangers, are usually nice and won't tell you they don't find any value to your product. PMs on the other hand can sometimes mistakenly infer that a smooth UX (lack of usability issues) means achieving product-market fit.


Therefore, when running these tests, one should deliberately seek product-market fit feedback. My suggestion is to design user tests that deliberately differentiate usability and product-market fitness. For instance, prepare test scripts with open-ended questions and categorize them into usability and product-market fit. To the adrenaline junkies out there, you can further breakdown the second category of questions into: underserved user needs (needs being served by the product), value proposition (messaging), feature set (value breakdown by feature).

10 views0 comments
Post: Blog2 Post
bottom of page