Visit the ARCHIVE

Everyone knows that Steve Jobs “didn’t believe in market research.” At the same time, we also know that an Apple product goes through countless iterations before and even after launch. What’s the difference between testing and continuous prototyping? On the one hand: Nothing. Prototyping is a form of testing. And on the other hand? Well, everything.

For perspective, let’s turn on the TV. Here’s what Gavin Polone, a producer of such hits as the Gilmore Girls and Curb Your Enthusiasm, has said:

“I have found some value in testing. It can show what parts of a story are confusing; it can give you an idea of where things slow down too much and become boring; and, most usefully, it can tell you if a joke is funny or not. Either the viewers laugh or they don’t. But the post-screening focus-group conversations are pretty worthless. People feel like they have to talk, so they come up with criticisms when they might otherwise have none, and there are always those who have some agenda they like to forward or are just showing off now that they’ve been given an audience; one person among twelve can easily move a discussion to distraction…”

In spite of the fact that it sounds like he got stuck with a lousy moderator, I think Polone hits the nail on the head when he says what such research is good for. Do the jokes work? Is the story confusing? Does this thing work the way we intended it to? In other words, he finds value in prototyping, but less value in an old-school, thumbs-up/thumbs-down conception of “testing.”

Generally speaking, there are three key differences between traditional testing and prototyping.

  1. Traditional testing means exposing the exact same stimuli to all respondents – in fact, if you don’t, it’s considered a faulty test. Prototyping means actively evolving stimuli over time. If you don’t iterate, you’re not prototyping. The result is that, unlike with traditional testing, you come out of prototyping with something measurably better than what you brought in.
  2. Traditional testing is more concerned with the whole. Prototyping is more concerned with the details. Behavioral research shows again and again that new things are difficult for people to judge, because they are unfamiliar. This is one reason why using testing to predict the real-world performance of a new product or service can be so unreliable. Prototyping sidesteps this by focusing on the functional details that serve to make even strange new ideas understandable and approachable.
  3. Testing attempts to pick a winner between competing options. Prototyping is about approaching the best possible state.  Testing is often structured as a horse race. There is only one winner. Prototyping offers a much more nuanced view. Indeed, bringing multiple options to the table can be highly valuable when prototyping – you can see what works well in one, and apply it to another.

I’m a big, big fan of prototyping nearly everything. With clients, I routinely go through dozens of drafts of a value proposition until everyone agrees it’s just right. As both a marketer and a writer, I like to think I have a good ear for how language works, but I can’t know for sure until I see how people respond. For instance, I once worked on a positioning statement for a company that outsourced back-office services for small businesses. In communicating the company’s offering to customers, we used the phrase “So you can relax” – the basic idea being that the company’s services relieved business owners from the needless stress of managing some things themselves. Sharing our prototype with target customers, the reaction was swift and brutal: small business owners were deeply offended by the suggestion that they weren’t relaxed to begin with.

The overall proposition turned out to be strong; the way we were expressing it turned out to be wrong. We would never have known without prototyping.


The post includes an excerpt from the article “Everyone Should Vote on Pilots (But Not in Focus Groups)” by Gavin Polone, New York Magazine (May 21, 2012)