The challenge of representative design in psychology and economics

Abstract

The demands of representative design, as formulated by Egon Brunswik (1956), set a high methodological standard. Both experimental participants and the situations with which they are faced should be representative of the populations to which researchers claim to generalize results. Failure to observe the latter has led to notable experimental failures in psychology from which economics could learn. It also raises questions about the meaning of testing economic theories in “abstract” environments. Logically, abstract tests can only be generalized to “abstract realities” and these may or may not have anything to do with the “empirical realities” experienced by economic actors.

Economists are adept at handling the requirements of classic, factorial experimental designs. However, a major problem with factorial experiments is that the basic logic (i.e., the orthogonal variation of variables) precludes generalizing results outside the laboratory (Brunswik, 1956). The rationale does seem impeccable. By varying one variable at a time and holding all others constant, one can isolate the effects. However, outside the laboratory all other variables are not constant and variables are not orthogonal. Thus, estimates of the sizes of effects (based on the experiment) are subject to other forces. You can design factorial worlds within an experiment but this may not have much to do with what happens outside it.

On the positive side, it should be noted that economists have looked at whether results that looked surprising within experimental laboratories (from the viewpoint of economic theory) can also be observed in more realistic environments, i.e., attempts have been made to demonstrate external validity. As a case in point, consider judgments of willingness-to-pay and willingness-to-accept on issues as trivial as small gambles to those as consequential as compensation awards in civil trials. Whereas the response mechanisms can be justified by economic theory, people are not machines that necessarily produce appropriate responses. Indeed, most people have limited experience with such mechanisms and responses are often sensitive to normatively irrelevant considerations whether in or outside the experimental laboratory. For good examples, see Sunstein, Hastie, Payne, Schkade and Viscusi (2002). In addition, consider evidence that supports prospect theory “in the wild” (Camerer, 2000) or empirical studies in behavioral finance (Barberis & Thaler, 2003).

Download full text article here