Ready-to-hand

Dean Eckles on people, technology & inference

HCI

Using a Wizard of Oz technique in mobile service design: probing with realistic motivations

As I’ve blogged before, I spoke at the Texting 4 Health conference on the topic of research methods for mobile messaging. One method I covered was an interesting use of Wizard of Oz techniques for designing mobile services. I’ve since started getting some of this material in writing for the Texting 4 Health book. Here is a taste of that material, minus the health-specific focus and examples.
——-
Just like the famous Wizard of Oz, one can simulate something impressive with a just a humble person behind the curtain — and use this simulation to inform design decisions. When using a Wizard of Oz technique to study a prototype, a human “wizard” carries out functions that, in a deployed application or service, would be handled by a computer. This can allow evaluating a design without fully building what can be expensive back-end parts of the system (Kelley 1984). The technique is often used in recognition-based interfaces, but it also has traditional applications to identifying usability problems and carrying out experiments in which the interaction is systematically manipulated.

Wizard of Oz techniques are well suited to prototyping mobile services, especially those using mobile messaging (SMS, MMS, voice messaging). When participants send a request, a wizard reads or listens to it and chooses the appropriate response, or just creates it on-the-fly. Since all user actions in mobile messaging are discrete messages and (depending on the application) the user can often tolerate a short delay, a few part-time wizards, such as you and a colleague, can manage a short field trial. As you’ll see, this can be used for purposes beyond many traditional uses of a Wizard of Oz.

Probing photo consumption needs with realistic motivations
One use for this technique in designing a mobile messaging service is a bit like a diary study. In designing an online and mobile photography service, we wanted to better understand what photos people wanted to view and what prompted these desires.1 Instead of just making diary entries, participants actually made voice requests to the system for photos – and received a mobile message with photos fitting the request in return. We didn’t need to first build a robust system that could do this; a few of us served as wizards, listening to the request, doing a couple manual searches, and choosing which photos to return on demand. Though this can be done with a normal voice call, we used a mobile client application that also recorded contextual information not available via a normal voice call (e.g. location), so that participants could make context-aware requests as they saw fit (e.g. “I want too see photos of this park”)

In this case, we didn’t plan to (specifically) create a voice-based photo search system; instead, like a diary study, this technique served as a probe to understand what we should build. As a probe it provided realistic motivations for submitting requests, as the request would actually be fulfilled. This design research, in additional to other interviews and a usability study, informed our creation of Zurfer, a mobile application that supports exploring and conversing around personalized, location-aware channels of photos.
It is great if the Wizard of Oz prototype is quite similar to what you later build, but it can yield valuable insights even if not. Sometimes it is precisely these insights that can lead you to substantially change your design.

This study design can apply in designing many mobile services. As in our photos study, participants can be interviewed about the trigger for the requests (why did they want that media or information) and how satisfied they were with the (human-created) responses.2

Kelley, J.F. (1984). An iterative design methodology for user-friendly natural language office information applications. In ACM Trans. Inf. Syst., vol. 2, pp. 26-41.

  1. This study was designed and executed at Yahoo! Research Berkeley by Shane Ahern, Nathan Good, Simon King, Mor Naaman, Rahul Nair, and myself. []
  2. Participants were informed that their requests would be seen by our research staff. Anonymization and strict limits of who the wizards are is necessary to protect participants’ privacy. Even if participants are not informed that a wizard is creating the responses until they are debriefed after the experiment, participants can nonetheless be notified that their responses are being reviewed by the research team. []

Riskful decisions and riskful thinking: Donald Davidson and Cliff Nass

Two personal-professional narratives that I’ve been somewhat familiar with for a while have recently highlighted for me the significance of riskful decisions and thinking in academia. I think the stories are interesting on their own, but they also emphasize some questions and concerns for the functioning of scholarly inquiry.

The first is about the American philosopher Donald Davidson, whose work has long been of great interest to me (and was the topic of my undergraduate Honors thesis). The second is about Cliff Nass (Clifford Nass), Professor of Communication at Stanford, an advisor and collaborator. The major published source I draw on for each of these narratives is an interview: for Davidson’s story, it is an interview by Ernest Lepore (2004), a critic and expositor of Davidson’s philosophy; for Cliff Nass, it is an interview by Tamara Adlin (2007). After sharing these stories, I’ll discuss some similarities and briefly discuss risk-taking in decisions and thinking.

Donald Davidson is considered one of the most important and influential philosophers of the past 60 years, and he is my personal favorite. Davidson is often described as a highly systematic philosopher — uncharacteristically so for 20th century philosophy, in that his contributions to several areas of philosophy (philosophy of language, mind, and action, semantics, and epistemology) are deeply connected in their method and the proposed theories. He is the paradigmatic programmatic philosopher of the 20th century.

Despite this, Davidson’s philosophical program did not emerge until relatively late in his career. The same is true of his publications in general. Only after accepting a tenure track position at Stanford in 1951 (which was then still up-and-coming, though quickly, in philosophy) did he begin to publish (nothing was even in the “pipeline” previous to this). This began under the wing of the younger Patrick Suppes, with whom Davidson co-authored a book (1957) on decision theory. His first philosophical article appears in 1963 (which he authored alone only through an unexpected death). As Davidson puts it in an interview with Ernest Lepore, “I was very inhibited so far as publication was concerned” and was worried “that the minute I actually published something, everyone was going to jump on me” (Davidson 2004).

Then Davidson published “Actions, Reasons and Causes” (1963), twelve years after joining the Stanford faculty. It argues against the late-Wittgensteinian dogma that reasons are not also causes. It is only with this paper that there was a publication by Davidson that drew significant attention from the community (beginning with a presentation of the paper at a meeting of the American Philosophical Association). This paper has been hugely influential and alone identified Davidson as an important thinker in the field, though he was surprised the reception was not as overwhelming as he had thought: “I didn’t realize that if you publish, as far as I can tell, no one was going to pay any attention.” Many responses, both positive and critical, did eventually come, and Davidson went on to publish many highly influential papers, reaching the height of his immense scholarly influence in the 1970s and 1980s.

Clifford Nass is widely known researcher in the psychology of human-computer interaction (HCI). With Byron Reeves, he wrote The Media Equation (1996), which presents research carried out at Stanford University on how people respond in mediated interactions (e.g. with computers and televisions) by overextending social rules normally applied to other people. This hints at the (here simplified) straight, bold line of Nass’s research program: take a finding from social psychology, replace the second human with a computer, see if you get the same results. This exact strategy has been modified and expanded from, but the general consistency of Nass’s program over many years is striking for HCI: unlike in psychology, for example, in HCI there are many investigators seeking low-hanging fruit and quickly moving on to new projects.

Nass likes to refer to his “accidental PhD”, as he hadn’t intended to get a PhD in sociology. After working for a year at Intel, he was planning to matriculate in a electrical engineering PhD program, but an unexpected death postponed that. “[J]ust to bide my time and to have some flexibility, I ended up doing a sociology degree,” says Nass. He did his dissertation on the role of pre-processing jobs in labor, taking an approach that was radical in its elimination of a role for people and that connected with contemporary research by social science outsiders doing “sociocybernetics”. With such a dissertation topic (and the dissertation itself unfinished), finding a job did not seem easy at the outset: “It’s a nutty topic. I was going to be in trouble getting jobs. I had published stuff and was doing work and all that, but my dissertation was so weird” (Adlin 2007).

There was, however, a bit of luck, well taken advantage of by Nass: the Stanford Communication Department was under construction and looking to hire some folks doing weird work. So when Nass interviewed, impressing both them and the Sociology Department, he got the job, despite knowing nothing about Communication as a discipline and having been to no conferences in the field. After beginning at Stanford, Nass was seeking a research program, as clearly there was something wrong, at least when it came to getting it accepted for academic publication, with his previous work: “I was having a terrible time getting my work accepted. In fact, to this day I’ve still never published anything off my dissertation, 20-odd years later. Because again, no field could figure out who owned the material. I got reviews like, ‘This work is offensive.‘”

But Nass couldn’t settle on any normal research program. He wanted to examine how people might treat computers socially. Getting funding for this work wouldn’t have been easy, but he got a grant that the grant administrator described as the 1 of 35 given that they chose to give to the “weirdest project that was proposed”. It wasn’t all easy from there, of course. For example, it took some time to design and carry out successful experiments in this program — and even longer to get the results published. But this risk-taking in distributing this grant helped enable the work to continue.

Cliff Nass is very clear about the role riskful decisions, in admissions, hiring, and funding, played in his success:

I was very lucky. I fear that those times are gone. I really do fear to a tremendous degree that the risk-taking these people were willing to do for me, to give me an opportunity, are gone. I try to remember that. […]

I benefited from the willingness of people to say, “We’re just going to roll the dice here.”

Of course, it isn’t just Cliff who got lucky; in a big sense we all did. His work has been an important influence in HCI and has contributed to our stores of both generalizable knowledge and new lenses for approaching how we get on in the world.

What does it mean for academic research, and science generally, if this choice and ability to take these risks evaporates? There is incredible competition for academic positions now, more so in some fields than others. And the best tool in getting a job is a whole list of publications accepted in important, mainstream journals in the field. There is a lot written about the competition for academic jobs and criteria for wading through applicants to sometimes a safe option. There are case studies of families of disciplines; for example, a study of the biosciences argues that market forces are failing to create sufficient job prospects for young investigators (Freeman et al. 2001).

I won’t review them all here. Instead I suggest an article for general readers from The New York Times about state and regional colleges’ use of non-tenure track positions, which has an impact of the institutions’ bottom line and flexibility (Finder 2007). This is part of a wider trend in how tenure is used that also impacts the academic freedom and resources that scholars have to pursue new research (Richardson 1999).

Enabling riskful thinking
Hans Ulrich Gumbrecht argues that “riskful thinking” is central to the value of the humanities and arts in academia. He defines riskful thinking as investigation that can’t be expected to produce results interpretable as easy answers, but that instead is likely to produce or highlight complex and confusing phenomena and problems. But I think that this is more broadly true. Riskful thinking is critical to interdisciplinary and pre-paradigmatic sciences, or disciplines long doing normal science but in need of a shake-up. These are situations where compelling phenomena can become paradigmatic cases for study and powerful vocabularies can allow formulating new problems and theories.

What threatens riskful thinking, and how can we enable it? What is so great about riskful thinking anyway, and what makes some riskful thinking so successful, while much of it is likely to fail? At Nokia Research Center in Palo Alto, our lab head John Shen champions the importance of risk taking in industry research, but also argues that risk-taking is often misunderstood and that it is only some kinds of risk-taking that are most important to cultivate in industry research.

Finally, a list of Davidson–Nass similarities, just for fun:

  • Both were hired to tenure track positions at Stanford, where they first did and published highly influential work
  • Both are easily and widely seen as highly programmatic, having defined a clear research program challenging to currently popular approaches and beliefs in their fields
  • Both had great difficulty finding early, publishable success with their research programs, even after ceasing their early work (Davidson: Plato, empirical decision theory; Nass: information processing models of the labor force)
  • Both had other draws and distractions (Davidson: business school, teaching plane identification in WWII; Nass: being a professional magician, working at Intel)
  • Both produced dissertations viewed by others in the discipline as odd (Davison: Quine “was a little mystified by my writing on this. He never talked to me about it.”; Nass: “my PhD thesis was so bizarre”)

References

Adlin, T. (2007). An interview with Cliff Nass. UX Pioneers. http://www.adlininc.com/uxpioneers/new_pioneers/interview_cliff_nass.html
Davidson, D. (1963). Actions, Reasons, and Causes. Journal of Philosophy, 60(23), 685-700.
Davidson, D., & Suppes, P. (1957). Decision Making: An Experimental Approach. Stanford University Press.

Finder, A. (2007, November 20). Decline of the Tenure Track Raises Concerns. The New York Times.

Freeman, R., Weinstein, E., Marincola, E., Rosenbaum, J., & Solomon, F. (2001). Careers: Competition and Careers in Biosciences. Science, 294(5550), 2293-2294.

Lepore, E. (2004). Interview with Donald Davidson. In Problems of Rationality, Oxford University Press, 2004, pp. 231-266.

Nass, C., Steuer, J., & Tauber, E. R. (1994). Computers are social actors. In Proc. of CHI 1994. ACM Press.

Reeves, B., & Nass, C. (1996). The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press.

Richardson, J. T. (1999). Tenure in the New Millenium. National Forum, 79(1), 19-23.
Sanford, J. (2000, November 17). ‘Elementary pleasures’ and ‘riskful thinking’ matter to Gumbrecht. Stanford Report.

Laptop + shopping cart = mobile

This guy can roll up with laptop and webcam to record robots (photo CC from violetblue):

But in Silicon Valley, combining laptops and shopping carts is just a way to get chores done. When at Whole Foods in Los Altos, I saw a man pushing a shopping cart with a laptop in the part where you can sit your toddler. I suppose he was reading a recipe or something. (I, and I’m sure other Valley folks, do that on a phone.)

A bit odd, but then again, I used to be (I’ve fallen off a bit) judicious about capturing the contents of my shopping cart with ZoneTag.

Texting 4 Health conference in review

As I blogged already, I attended and spoke at the first Texting 4 Health conference at Stanford University last week. You can see my presentation slides at SlideShare here, and the program, with links to the slides for most speakers is here.

The conference was very interesting, and there was quite the mix of participants — both speakers and others. There were medical school faculty, business people, people from NGOs and foundations, technologists, representatives of government agencies and centers, futurists, and social scientists. Everyone had something to learn — I know I did. This also made it somewhat difficult as a speaker because it is hard to know how best to reach, inform, and hold the interest of such a diverse audience: what is common ground with some is entirely new territory with others.

I think my favorite session was “Changing Health Behavior via SMS”. The methods used by the panelists to evaluate their interventions were both interesting to reflect on and good tools for persuading me of the importance and effectiveness of their work. One of my reflections was about what factors to vary in doing experiments on health interventions: there is (reasonable) focus on having a no-SMS control condition, and there are very few studies with manipulations of dimensions more fine-grained. Of course, the field is young and I understand how important true controls are in medical domains, but I think that real progress in understanding mobile messaging and designing effective interventions will require looking at more subtle and theoretically valuable manipulations.

You can see other posts about the conference here and here. And the conference Web site is also starting a blog to watch in the future.

Texting 4 Health

On February 29th I’m speaking at Texting 4 Health, a conference at Stanford University about using mobile messaging for health interventions and research. I’ll be talking about mobile messaging research methods I’ve used to study mobile persuasive technology. Like Mobile Persuasion 2007, it will feature a fast-paced, single-track program with time to meet and talk with participants from health, technology, policy, and research communities.

Scroll to top