Marginal evidence for psychological processes
Some comments on problems with investigating psychological processes using estimates of average (i.e. marginal) effects. Hence the play on words in the title.
Social psychology makes a lot of being theoretical. This generally means not just demonstrating an effect, but providing evidence about the psychological processes that produce it. Psychological processes are, it is agreed, intra-individual processes. To tell a story about a psychological process is to posit something going on “inside” people. It is quite reasonable that this is how social psychology should work — and it makes it consistent with much of cognitive psychology as well.
But the evidence that social psychology uses to support these theories about these intra-individual processes is largely evidence about effects of experimental conditions (or, worse, non-manipulated measures) averaged across many participants. That is, it is using estimates of marginal effects as evidence of conditional effects. This is intuitively problematic. Now, there is no problem when using experiments to study effects and processes that are homogenous in the population. But, of course, they aren’t: heterogeneity abounds. There is variation in how factors affect different people. This is why the causal inference literature has emphasized the differences among the average treatment effect, (average) treatment effect on the treated, local average treatment effect, etc.
Not only is this disconnect between marginal evidence and conditional theory trouble in the abstract, we know it has already produced many problems in the social psychology literature.1 Baron and Kenny (1986) is the most cited paper published in the Journal of Personality and Social Psychology, the leading journal in the field. It paints an rosy picture of what it is like to investigate psychological processes. The methods of analysis it proposes for investigating processes are almost ubiquitous in social psych.2 The trouble is that this approach is severely biased in the face of heterogeneity in the processes under study. This is usually described as problem of correlated error terms, omitted-variables bias, or adjusting for post-treatment variables. This is all true. But, in the most common uses, it is perhaps more natural to think of it as a problem of mixing up marginal (i.e. average) and conditional effects.3
What’s the solution? First, it is worth saying that average effects are worth investigating! Especially if you are evaluating a intervention or drug that might really be used — or if you are working at another level of analysis than psychology. But if psychological processes are your thing, you must do better.
Social psychologists sometimes do condition on individual characteristics, but often this is a measure of a single trait (e.g., need for cognition) that cannot plausibly exhaust all (or even much) of the heterogeneity in the effects under study. Without much larger studies, they cannot condition on more characteristics because of estimation problems (too many parameters for their N). So there is bound to be substantial heterogeneity.
Beyond this, I think social psychology could benefit from a lot more within-subjects experiments. Modern statistical computing (e.g., tools for fitting mixed-effects or multilevel models) makes it possible — even easy — to use such data to estimate effects of the manipulated factors for each participant. If they want to make credible claims about processes, then within-subjects designs — likely with many measurements of each person — are a good direction to more thoroughly explore.
- The situation is bad enough that I (and some colleagues) certainly don’t even take many results in social psych as more than providing a possibly interesting vocabulary. [↩]
- Luckily, my sense is that they are waning a bit, partially because of illustrations of the method’s bias. [↩]
- To translate to the terms used before, note that we want to condition on unobserved (latent) heterogeneity. If one doesn’t, then there is omitted variable bias. This can be done with models designed for this purpose, such as random effects models. [↩]
Academia vs. industry: Harvard CS vs. Google edition
Matt Welsh, a professor in the Harvard CS department, has decided to leave Harvard to continue his post-tenure leave working at Google. Welsh is obviously leaving a sweet job. In fact, it was not long ago that he was writing about how difficult it is to get tenure at Harvard.
So why is he leaving? Well, CS folks doing research in large distributed systems are in a tricky place, since the really big systems are all in industry. And instead of legions of experienced engineers to help build and study these systems, they have a bunch of lazy grad students! One might think, then, that this kind of (tenured) professor to industry move is limited to people creating and studying large deployments of computer systems.
There is a broader pull, I think. For researchers studying many central topics in the social sciences (e.g., social influence), there is a big draw to industry, since it is corporations that are collecting broad and deep data sets describing human behavior. To some extent, this is also a case of industry being appealing for people studying deployment of large deployments of computer systems — but it applies even to those who don’t care much about the “computer” part. In further parallels to the case with CS systems researchers, in industry they have talented database and machine learning experts ready to help, rather than social science grad students who are (like the faculty) too often afraid of math.
Economic imperialism and causal inference
And I, for one, welcome our new economist overlords…
Readers not in academic social science may take the title of this post as indicating I’m writing about the use of economic might to imperialist ends.1 Rather, economic imperialism is a practice of economists (and acolytes) in which they invade research territories that traditionally “belong” to other social scientific disciplines.2 See this comic for one way you can react to this.3
Economists bring their theoretical, statistical, and research-funding resources to bear on problems that might not be considered economics. For example, freakonomists like Levitt study sumo wrestlers and the effects of the legalization of abortion on crime. But, hey, if the Commerce Clause means that Congress can legislate everything, then, for the same reasons, economists can — no, must — study everything.
I am not an economist by training, but I have recently had reason to read quite a bit in econometrics. Overall, I’m impressed.4 Economists have recently taken causal inference — learning about cause and effect relationships, often from observational data — quite seriously. In the eyes of some, this has precipitated a “credibility revolution” in economics. Certainly, papers in economics and (especially) econometrics journals consider threats to the validity of causal inference at length.
On the other hand, causal inference in the rest of the social sciences is simultaneously over-inhibited and under-inhibited. As Judea Pearl observes in his book Causality, lack of clarity about statistical models (that social scientists often don’t understand) and causality has induced confusion about distinctions between statistical and causal issues (i.e., between estimation methods and identification).5
So, on the one had, many psychologists stick to experiments. Randomized experiments are, generally, the gold standard for investigating cause–effect relationships, so this can and often does go well. However, social psychologists have recently been obsessed with using “mediation analysis” to investigate the mechanisms by which causes they can manipulate produce effects of interest. Investigators often manipulate some factors experimentally and then measure one or more variables they believe fully or partially mediate the effect of those factors on their outcome. Then, under the standard Baron & Kenny approach, psychologists fit a few regression models, including regressing the outcome on both the experimentally manipulated variables and the simply measured (mediating) variables. The assumptions required for this analysis to identify any effects of interest are rarely satisfied (e.g., effects on individuals are homogenous).6 So psychologists are often over-inhibited (experiments only please!) and under-inhibited (mediation analysis).
Likewise, in more observational studies (in psychology, sociology, education, etc.), investigators are sometimes wary of making explicit causal claims. So instead of carefully stating the causal assumptions that would justify different causal conclusions, readers are left with phrases like “suggests” and “is consistent with” followed by causal claims. Authors then recommend that further research be conducted to better support these causal conclusions. With these kinds of recommendations awaiting, no wonder that economists find the territory ready for taking: they can just show up with econometrics tools and get to work on hard-won questions that “rightly belong to others”.
- Well, if economists have better funding sources, this might apply in some sense. [↩]
- For arguments in favor of economic imperialism, see Lazear, E.P. (1999). Economic imperialism. NBER Working Paper No. 7300. [↩]
- Or see this comic for imperialism by physicists. [↩]
- At least by the contemporary literature on what I’ve been reading on — IVs, encouragement designs, endogenous interactions, matching estimators. But it is true that in some of these areas econometrics has been able to fruitfully borrow from work on potential outcomes in statistics and epidemiology. [↩]
- Econometricians have made similar observations. [↩]
- For a bit on this topic, see the discussion and links to papers here. [↩]
Keyword searching papers citing a highly-cited paper with Google Scholar
[Update: Google Scholar now directly supports this feature, check the box right below the search box after clicking “Cited by…”.]
In finding relevant research, once one has found something interesting, it can be really useful to do “reverse citation” searches.
Google Scholar is often my first stop when finding research literature (and for general search), and it has this feature — just click “Cited by 394”. But it is not very useful when your starting point is highly cited. What I often want to do is to do a keyword search of the papers that cite my highly-cited starting point.
While there is no GUI for this search within these results in Google Scholar, you can actually do it by hacking the URL. Just add the keyword query to the URL.
This is the URL one gets for all resources Google has as citing Allport’s “Attitudes” (1935):
http://scholar.google.com/scholar?cites=9150707851480450787&hl=en
And this URL searches within those for “indispensable concept”:
http://scholar.google.com/scholar?hl=en&cites=9150707851480450787&q=indispensable+concept
In this particular case, this gives us many examples of authors citing Allport’s comment that the attitude is the most distinctive and indispensable concept in social psychology. This example highlights that this can even just help get more useful “snippets” in the search results, even if it doesn’t narrow down the results much.
I find this useful in many cases. Maybe you will also.
Naming this blog “ready-to-hand”: Heidegger, Husserl, folk psychology, and HCI
The name of this blog, Ready-to-hand, is a translation of Heidegger’s term zuhanden, though interpreting Heidegger’s philosophy is not specifically a major interest of mine nor a focus here. Much has been made of the significance of phenomenology, most often Heidegger, for human-computer interaction (HCI) and interaction design (e.g., Winograd & Flores 1985, Dourish 2001). And I am generally pretty sympathetic to phenomenology as one inspiration for HCI research. I want to just note a bit about the term zuhanden and my choice of it in a larger context — of phenomenology, HCI, and a current research interest of mine: cues for assuming the intentional stance toward systems (more on this below).
The Lifeworld and ready-to-hand
Heidegger was a student of Edmund Husserl, and Heidegger’s Being and Time was to be dedicated to Husserl.1 There is really no question of the huge influence of Husserl on Heidegger.
My major introduction to both Husserl and Heidegger was from Prof. Dagfinn Føllesdal. Føllesdal (1979) details the relationship between their philosophies. He argues for the value of seeing much of Heidegger’s philosophy “as a translation of Husserl’s”:
The key to this puzzle, and also, I think, the key to understanding what goes on in Heidegger’s philosophy, is that Heidegger’s philosophy is basically isomorphic to that of Husserl. Where Husserl speaks of the ego, Heidegger speaks of Dasein, where Husserl speaks of the noema, Heidegger speaks of the structure of Dasein’s Being-in-the-world and so on. Husserl also observed this. Several places in his copy of Being and Time Husserl wrote in the margin that Heidegger was just translating Husserl’s phenomenology into another terminology. Thus, for example, on page 13 Husserl wrote: “Heidegger transposes or transforms the constitutive phenomenological clarification of all realms of entities and universals, the total region World into the anthropological. The problematic is translation, to the ego corresponds Dasein etc. Thereby everything becomes deep-soundingly unclear, and philosophically it loses its value.” Similarly, on page 62, Husserl remarks: “What is said here is my own theory, but without a deeper justification.” (p. 369, my emphasis)
Heidegger and his terms have certainly been more popular and in wider use since then.
Føllesdal also highlights where the two philosophers diverge.2 In particular, Heidegger gives a central role to the role of the body and action in constituting the world. While in his publications Husserl stuck to a focus on how perception constitutes the Lifeworld, Heidegger uses many examples from action.3 Our action in the world, including our skillfulness in action constitutes those objects we interact with for us.
Heidegger contrasts two modes of being (in addition to our own mode — being-in-the-world): present-at-hand and ready-to-hand (or alternatively, the occurant and the available (Dreyfus 1990)). The former is the mode of being consideration of an object as a physical thing present to us — or occurant, and Heidegger argues it constitutes the narrow focus of previous philosophical explorations of being. The latter is the stuff of every skilled action — available for action: the object becomes equipment, which can often be transparent in action, such that it becomes an extension of our body.
J.J. Gibson expresses this view in his proposal of an ecological psychology (in which perception and action are closely linked):
When in use, a tool is a sort of extension of the hand, almost an attachment to it or a part of the user’s own body, and thus is no longer a part of the environment of the user. […] This capacity to attach something to the body suggests that the boundary between the animal and the environment is not fixed at the surface of the skin but can shift. More generally it suggests that the absolute duality of “objective” and “subjective” is false. When we consider the affordances of things, we escape this philosophical dichotomy. (1979, p. 41)
While there may be troubles ahead for this view, I think the passage captures well something we all can understand: when we use scissors, we feel the paper cutting; and when a blind person uses a cane to feel in front of them, they can directly perceive the layout of the surface in front of them.
Transparency, abstraction, opacity, intentionality
Research and design in HCI has sought at times to achieve this transparency, sometimes by drawing on our rich knowledge of and skill with the ordinary physical and social world. Metaphor in HCI (e.g., the desktop metaphor) can be seen as one widespread attempt at this (cf. Blackwell 2006). This kind of transparency does not throw abstraction out of the picture. Rather the two go hand-in-hand: the specific physical properties of the present-at-hand are abstracted away, with quickly perceived affordances for action in their place.
But other kinds of abstraction are in play in HCI as well. Interactive technologies can function as social actors and agents– with particular cues eliciting social responses that are normally applied to other people (Nass and Moon 2000, Fogg 2002). One kind of social response, not yet as widely considered in the HCI literature, is assuming the intentional stance — explanation in terms of beliefs, desires, hopes, fears, etc. — towards the system. This is a powerful, flexible, and easy predictive and explanatory strategy often also called folk psychology (Dennett 1987), which may be a tacit theory or a means of simulating other minds. We can explain other people based on what they believe and desire.
But we can also do the same for other things. To use one of Dennett’s classic examples, we can do the same for a thermostat: why did it turn the heat on? It wanted to keep the house at some level of warmth, it believed that it was becoming colder than desired, and it believed that it could make it warmer by turning on the heat. While in the case of the thermostat, this strategy doesn’t hide much complexity (we could explain it with other strategies without much trouble), it can be hugely useful when the system in question is complex or otherwise opaque to other kinds of description (e.g., it is a black box).
We might think then that perceived complexity and opacity should both be cues for adopting the intentional stance. But if the previous research on social responses to computers (not to mention the broader literature on heuristics and mindlessness) has taught us anything, it is that made objects such as computers can evoke unexpected responses through other simplier cues. Some big remaining questions that I hope to take up in future posts and research:
- What are these cues, both features of the system and situational factors?
- How can designers influence people to interpret and explain systems using folk psychology?
- What are the advantages and disadvantages of evoking the intentional stance in users?
- How should we measure the use of the intentional stance?
- How is assuming the intentional stance towards a thing different (or the same) as it having being-in-the-world as its mode of being?
References
- But Husserl was Jewish, and Heidegger was himself a member of the Nazi party, so this did not happen in the first printing. [↩]
- Dreyfus (1990) is an alternative view that takes the divergence as quite radical; he sees Føllesdal as hugely underestimating the originality of Heidegger’s thought. Instead Dreyfus characterizes Husserl as formulating so clearly the Cartesian worldview that Heidegger recognized its failings and was thus able to radically and successfully critique it. [↩]
- It is worth noting that Husserl actually wrote about this as well, but in manuscripts, which Heidegger read years before writing Being and Time. [↩]