26 Apr 2010
The Wall Street Journal’s Venture Capital Dispatch reports on how Aardvark, the social question asking and answering service recently acquired by Google, used a Wizard of Oz prototype to learn about how their service concept would work without building all the tech before knowing if it was any good.
Aardvark employees would get the questions from beta test users and route them to users who were online and would have the answer to the question. This was done to test out the concept before the company spent the time and money to build it, said Damon Horowitz, co-founder of Aardvark, who spoke at Startup Lessons Learned, a conference in San Francisco on Friday.
“If people like this in super crappy form, then this is worth building, because they’ll like it even more,” Horowitz said of their initial idea.
At the same time it was testing a “fake” product powered by humans, the company started building the automated product to replace humans. While it used humans “behind the curtain,” it gained the benefit of learning from all the questions, including how to route the questions and the entire process with users.
This is a really good idea, as I’ve argued before on this blog and in a chapter for developers of mobile health interventions. What better way to (a) learn about how people will use and experience your service and (b) get training data for your machine learning system than to have humans-in-the-loop run the service?
My friend Chris Streeter wondered whether this was all done by Aardvark employees or whether workers on Amazon Mechanical Turk may have also been involved, especially in identifying the expertise of the early users of the service so that the employees could route the questions to the right place. I think this highlights how different parts of a service can draw on human and non-human intelligence in a variety of ways — via a micro-labor market, using skilled employees who will gain hands-on experience with customers, etc.
I also wonder what UIs the humans-in-the-loop used to accomplish this. It’d be great to get a peak. I’d expect that these were certainly rough around the edges, as was the Aardvark customer-facing UI.
Aardvark does a good job of being a quite sociable agent (e.g., when using it via instant messaging) that also gets out of the way of the human–human interaction between question askers and answers. I wonder how the language used by humans to coordinate and hand-off questions may have played into creating a positive para-social interaction with vark.
Posted by Chris Streeter on April 26th, 2010 at 9:31 pm
Yeah, thats a good point. If you’re using turkers to do the filing, then you don’t see yourself how people are using the service. Though, if your traffic is big enough, then you have turkers do a majority of the filing, and then your own employees doing some subset to learn about how the service is used. Sort of how Facebook started out, by seeing how users were using the site, and then responding to them: http://techcrunch.com/2010/04/21/facebook-vp-chris-cox-on-the-companys-past-future-and-recipe-for-success/
Posted by Joan Griffin on April 27th, 2010 at 7:17 pm
Interesting… using humans… pretending to be computers… to see how people will like a computerized product/service that imitates humans answering one another’s questions… human-human interface imitating human-human interface while pretending to be computers interfacing with humans…
Sorta like using human tellers hidden behind fake ATM machines taking your card and slipping money out to you… or human telephone operators faking computer generated voices that say “press 1 for this and 2 for that”… or a human passenger in the car with you, holding a map and pretending to be a GPS unit mounted on the dash…
Posted by Dean Eckles on April 27th, 2010 at 9:13 pm
I like the examples!
As for imitating humans answering one another’s questions, the questions actually get answered by humans in all versions, but the computer agent routes the questions and serves as an intermediary between askers and answerers.
Posted by 8 Lessons Learned from Zynga about Virality | Amit Klein on June 17th, 2010 at 2:58 am
[…] The guys at Aardvark (another product I love) call this Wizard of Oz testing. […]
Posted by Fake it till you make it - Startup Giraffe on March 8th, 2012 at 10:41 am
[…] (source: Dean Eckles) […]