Dean Eckles on people, technology & inference

Aardvark’s use of Wizard of Oz prototyping to design their social interfaces

The Wall Street Journal’s Venture Capital Dispatch reports on how Aardvark, the social question asking and answering service recently acquired by Google, used a Wizard of Oz prototype to learn about how their service concept would work without building all the tech before knowing if it was any good.

Aardvark employees would get the questions from beta test users and route them to users who were online and would have the answer to the question. This was done to test out the concept before the company spent the time and money to build it, said Damon Horowitz, co-founder of Aardvark, who spoke at Startup Lessons Learned, a conference in San Francisco on Friday.

“If people like this in super crappy form, then this is worth building, because they’ll like it even more,” Horowitz said of their initial idea.

At the same time it was testing a “fake” product powered by humans, the company started building the automated product to replace humans. While it used humans “behind the curtain,” it gained the benefit of learning from all the questions, including how to route the questions and the entire process with users.

This is a really good idea, as I’ve argued before on this blog and in a chapter for developers of mobile health interventions. What better way to (a) learn about how people will use and experience your service and (b) get training data for your machine learning system than to have humans-in-the-loop run the service?

My friend Chris Streeter wondered whether this was all done by Aardvark employees or whether workers on Amazon Mechanical Turk may have also been involved, especially in identifying the expertise of the early users of the service so that the employees could route the questions to the right place. I think this highlights how different parts of a service can draw on human and non-human intelligence in a variety of ways — via a micro-labor market, using skilled employees who will gain hands-on experience with customers, etc.

I also wonder what UIs the humans-in-the-loop used to accomplish this. It’d be great to get a peak. I’d expect that these were certainly rough around the edges, as was the Aardvark customer-facing UI.

Aardvark does a good job of being a quite sociable agent (e.g., when using it via instant messaging) that also gets out of the way of the human–human interaction between question askers and answers. I wonder how the language used by humans to coordinate and hand-off questions may have played into creating a positive para-social interaction with vark.

5 thoughts on “Aardvark’s use of Wizard of Oz prototyping to design their social interfaces

  1. Yeah, thats a good point. If you’re using turkers to do the filing, then you don’t see yourself how people are using the service. Though, if your traffic is big enough, then you have turkers do a majority of the filing, and then your own employees doing some subset to learn about how the service is used. Sort of how Facebook started out, by seeing how users were using the site, and then responding to them:

  2. Interesting… using humans… pretending to be computers… to see how people will like a computerized product/service that imitates humans answering one another’s questions… human-human interface imitating human-human interface while pretending to be computers interfacing with humans…

    Sorta like using human tellers hidden behind fake ATM machines taking your card and slipping money out to you… or human telephone operators faking computer generated voices that say “press 1 for this and 2 for that”… or a human passenger in the car with you, holding a map and pretending to be a GPS unit mounted on the dash…

  3. I like the examples!

    As for imitating humans answering one another’s questions, the questions actually get answered by humans in all versions, but the computer agent routes the questions and serves as an intermediary between askers and answerers.

Leave a Reply

Your email address will not be published. Required fields are marked *

Scroll to top