Aardvark’s use of Wizard of Oz prototyping to design their social interfaces
The Wall Street Journal’s Venture Capital Dispatch reports on how Aardvark, the social question asking and answering service recently acquired by Google, used a Wizard of Oz prototype to learn about how their service concept would work without building all the tech before knowing if it was any good.
Aardvark employees would get the questions from beta test users and route them to users who were online and would have the answer to the question. This was done to test out the concept before the company spent the time and money to build it, said Damon Horowitz, co-founder of Aardvark, who spoke at Startup Lessons Learned, a conference in San Francisco on Friday.
“If people like this in super crappy form, then this is worth building, because they’ll like it even more,” Horowitz said of their initial idea.
At the same time it was testing a “fake” product powered by humans, the company started building the automated product to replace humans. While it used humans “behind the curtain,” it gained the benefit of learning from all the questions, including how to route the questions and the entire process with users.
This is a really good idea, as I’ve argued before on this blog and in a chapter for developers of mobile health interventions. What better way to (a) learn about how people will use and experience your service and (b) get training data for your machine learning system than to have humans-in-the-loop run the service?
My friend Chris Streeter wondered whether this was all done by Aardvark employees or whether workers on Amazon Mechanical Turk may have also been involved, especially in identifying the expertise of the early users of the service so that the employees could route the questions to the right place. I think this highlights how different parts of a service can draw on human and non-human intelligence in a variety of ways — via a micro-labor market, using skilled employees who will gain hands-on experience with customers, etc.
I also wonder what UIs the humans-in-the-loop used to accomplish this. It’d be great to get a peak. I’d expect that these were certainly rough around the edges, as was the Aardvark customer-facing UI.
Aardvark does a good job of being a quite sociable agent (e.g., when using it via instant messaging) that also gets out of the way of the human–human interaction between question askers and answers. I wonder how the language used by humans to coordinate and hand-off questions may have played into creating a positive para-social interaction with vark.
Source orientation and persuasion in multi-device and multi-context interactions
At the Social Media Workshop, Katarina Segerståhl presented her on-going work on what she has termed extended information services or distributed user experiences — human-computer interactions that span multiple and heterogeneous devices (Segerståhl & Oinas-Kukkonen 2007). As a central example, she studies a persuasive technology service for planning, logging, reviewing, and motivating exercise: these parts of the experience are distributed across the user’s PC, mobile phone, and heart rate monitor.
In one interesting observation, Segerståhl notes that the specific user interfaces on one device can be helpful mental images even when a different device is in use: participants reported picturing their workout plan as it appeared on their laptop and using it to guide their actions during their workout, during which the obvious, physically present interface with the service was the heart rate monitor, not the earlier planning visualization. Her second focus is how to make these user experiences coherent, with clear practical applications in usability and user experience design (e.g., how can designers make the interfaces both appropriately consistent and differentiated?).
In this post, I want to connect this very interesting and relevant work with some other research at the historical and theoretical center of persuasive technology: source orientation in human-computer interaction. First, I’ll relate source orientation to the history and intellectual context of persuasive technology. Then I’ll consider how multi-device and multi-context interactions complicate source orientation.
Source orientation, social responses, and persuasive technology
As an incoming Ph.D. student at Stanford University, B.J. Fogg already had the goal of improving generalizable knowledge about how interactive technologies can change attitudes and behaviors by design. His previous graduate studies in rhetoric and literary criticism had given him understanding of one family of academic approaches to persuasion. And in running a newspaper and consulting on many document design (Schriver 1997) projects, the challenges and opportunities of designing for persuasion were to him clearly both practical and intellectually exciting.
The ongoing research of Profs. Clifford Nass and Byron Reeves attracted Fogg to Stanford to investigate just this. Nass and Reeves were studying people’s mindless social responses to information and communication technologies. Cliff Nass’s research program — called Computers as (or are) Social Actors (CASA) — was obviously relevant: if people treat computers socially, this “opens the door for computers to apply […] social influence” to change attitudes and behaviors (Fogg 2002, p. 90). While clearly working within this program, Fogg focused on showing behavioral evidence of these responses (e.g., Fogg & Nass 1997): both because of the reliability of these measures and the standing of behavior change as a goal of practitioners.
Source orientation is central to the CASA research program — and the larger program Nass shared with Reeves. Underlying people’s mindless social responses to communication technologies is the fact that they often orient towards a proximal source rather than a distal one — even when under reflective consideration this does not make sense: people treat the box in front of them (a computer) as the source of information, rather than a (spatially and temporally) distant programmer or content creator. That is, their source orientation may not match the most relevant common cause of the the information. This means that features of the proximal source unduly influence e.g. the credibility of information presented or the effectiveness of attempts at behavior change.
For example, people will reciprocate with a particular computer if it is helpful, but not the same model running the same program right next to it (Fogg & Nass 1997, Moon 2000). Rather than orienting to the more distal program (or programmer), they orient to the box.1
Multiple devices, Internet services, and unstable context
These source orientation effects have been repeatedly demonstrated by controlled laboratory experiments (for reviews, see Nass & Moon 2000, Sundar & Nass 2000), but this research has largely focused on interactions that do not involve multiple devices, Internet services, or use in changing contexts. How is source orientation different in human-computer interactions that have these features?
This question is of increasing practical importance because these interactions now make up a large part of our interactions with computers. If we want to describe, predict, and design for how people use computers everyday — checking their Facebook feed on their laptop and mobile phone, installing Google Desktop Search and dialing into Google 411, or taking photos with their Nokia phone and uploading them to Nokia’s Ovi Share — then we should test, extend, and/or modify our understanding of source orientation. So this topic matters for major corporations and their closely guarded brands.
So why should we expect that multiple devices, Internet services, and changing contexts of use will matter so much for source orientation? After having explained the theory and evidence above, this may already be somewhat clear, so I offer some suggestive questions.
- If much of the experience (e.g. brand, visual style, on-screen agent) is consistent across these changes, how much will the effects of characteristics of the proximal source — the devices and contexts — be reduced?
- What happens when the proximal device could be mindfully treated as a source (e.g., it makes its own contribution to the interaction), but so does a distance source (e.g., a server)? This could be especially interesting with different branding combination between the two (e.g., the device and service are both from Apple, or the device is from HTC and service is from Google).
- What if the visual style or manifestation of the distal source varies substantially with the device used, perhaps taking on a style consistent with the device? This can already happen with SMS-based services, mobile Java applications, and voice agents that help you access distant media and services.
References
- This actually is subject to a good deal of cross-cultural variation. Similar experiments with Japanese — rather than American — participants show reciprocity to groups of computers, rather than just individuals (Katagiri et al.) [↩]