PhD Project

16 september 2005

PHD PROJECT

Tentative title of the PhD thesis and contents

Background

Survey research is by far the most commonly used data collection method for the social sciences. They have revolutionized social science since their introduction in the 1940s, and also changed democratic societies. For starters surveys provided the gold standard for measuring citizen opinions that are at the heart of democratic deliberation. And surveys also provided a powerful technique for ensuring the openness and transparency of the democratic process through studies of democratic institutions. No other social science method has proven so valuable. Surveys provide some of the best tests of different kinds of theories, ranging from rational choice decision making to theories about representation, tolerance, political protest, elections, agenda setting, and the impacts of political campaigns. The conceptual richness of survey work rests upon over 40 years of thinking about how to measure democratic values, tolerance, participation, attitude constraint, party identification, ideology, and many other concepts. Surveys have become even stronger methods for confirming theories with the development of sophisticated quasi-experimental designs, the incorporation of experiments within surveys, and the addition of contextual data to survey information. Finally, results from surveys are at the core of the journalistic understanding of politics and survey results have informed constitution writing and the writing of history.

Despite the importance of survey research as a major tool for opinion research, there are also strong doubts about the usefulness of results from survey research. It is a well-known problem that responses to survey questions, especially questions about political issues, can show a great deal of response instability across waves of panel data. Social scientists have been aware of this since Converse's paper (1964) on mass belief systems. This essay ‘The Nature of Belief Systems in Mass Publics’ is the starting point for my study into the survey response process. In that essay Philip Converse raised serious doubts about the capabilities of the mass public as democratic citizens. He based this idea on survey data, in which he found that across time responses to the same (political) attitude questions were very unstable and within an interview respondents showed little constraint with respect to (logically) related attitudes. This debate has spread across several scientific disciplines and for me the most interesting part of the debate is about measurement error as an explanation for the instability and low constraint. In the measurement error approach it is assumed that it are not the citizens who are unsophisticated and not equipped to participate in the democratic debate, but it are the survey questions that are unreliable instruments to measure opinions and attitudes. Hence respondents cannot provide – due to the vague and unreliable questions - the same response (within a limited time-span) to the same question, although they have the same attitude/opinion. So first we observe that survey responses are very unstable, but after correction for measurement error the attitudes are stable. This approach was questioned because the amount of estimated measurement error for a single question was in general very large, sometimes up to 80%. So according to this approach most of what we measure is error. This raises the question whether we should still spend money on survey research for the measurement of attitudes and opinions. A different approach entered the field that had its origins in cognitive psychology. In that approach measurement error is neglected and the models that explain question-answering are very sophisticated but also complicated. Furthermore, instead of focusing on the population (public opinion) these models focused on individuals. In these approaches it is assumed that individuals hold all kinds of considerations about the topic in the survey question, and some of them are triggered and some of them are not. The ones that are activated form the response, but there are many variations on this theme. A debate that partly stems from decision-making theory was also introduced in the survey field, and this debate was on rationality and emotion as explanations for survey responses.

It should be obvious that describing the process of answering survey questions is not straightforward and does – unfortunately – not restrict itself a one scientific field. There is no room here to mention all the debates that are going on in the different fields that are linked to survey question answering, but the impression given above summarizes the main issues in the debate that are of interest to me and my research.

The VAS model

Given the different and sometimes contradictory theories regarding the issue of response instability, it would be attractive if a theory could be developed for the response process in survey research on the basis of the more recent ideas from the cognitive sciences with respect to the working of the memory. It would also be attractive if a research design and model could be developed that would enable us to estimate some of the basic characteristics governing the development of opinions and the response process. With that design we should be able to differentiate between unique and stable components of an opinion. And, we should be able to differentiate between the opinion at a point in time, a model stable opinion, an attitude and ideology and the measurement error component in the response.

The starting point for such a theory is that words in a survey question are represented in the brain by a network of nodes. These nodes contain cognition and affect and they can activate other nodes that are connected to it once a certain level of activation is reached. The evaluation of the total of nodes that are still activated after the survey question is understood by respondent is what we refer to as the opinion at a specific point in time. The time-restriction is necessary because it is very well possible that the context of the survey question changes with time, and hence also the nodes that are activated might change. But this opinion is still not the same as the response on paper. A choice has to be made between the response options. This process requires the matching of the opinion with one of the response options. It is assumed that measurement error occurs here due to the fact that it is easier to match an opinion on one response scale rather than on another.

Based upon this theory a model is build that takes into account the major forces governing the response process. With this model it becomes possible to estimate the stability of public opinion and the size of the effect of measurement error on the response. The latter being an indication of the quality of the survey question. In addition the model allows for the estimation of the crystallization of the opinion/attitude. The crystallization is an indication of the impact of contextual time-restricted influences on the opinion. The model is presented below as a set of three equations.

        S_t    =  S_t-1   + N_t

        O_t   =  S_t      + U_t

        R_tm  = O_t   + e_tm     

In the first equation of the variable S_t, this variable represents evaluations of information that is stable for each individual, this variable will be referred to as the stable opinion. This variable is the addition of the stable opinion at a previous point in time and the variable N_t. The latter variable represents evaluations of information that for each individual is new at that point in time and that will remain at least until the next point in time. This variable will be referred to as the new stable opinion. Furthermore, the variable O_t is introduced that represents the evaluation of information that for each individual is both stable (and new) and unique. This variable will simply be referred to as the opinion. The variable U_t represents the evaluation of information that for each individual is unique at that time, and will be referred to as the unique opinion. And in the last equation of the model the response variable R_tm is found, which represents the opinion expressed on a specific response scale m. It can be derived that the opinion is obtained by subtracting the measurement (e_tm) error from the response.

In order to make the link between the model presented here and quality criteria often used in research (validity and reliability) the variables in this model are standardized. To standardize the variables they first have to be expressed as deviation scores, so that their means will be zero. These variables are denoted with a ‘d’ in the superscript. Then all the variables are standardized - except for N_t, U_t, and e_tm. This leads to the set of equations presented below by dividing them by their standard deviation. In doing so, the effect parameters that were equal to 1 in model 1 become equal to the ratio of the standard deviation of the causal variable divided by the standard deviation of the effect variable. This result of standardization is presented in model 2.

        S^*_t   = s_{t,t-1  *}  S^*_t-1      +    N^’_t

        O^*_t  = c_{t     *}   S^*_t      +    U^’_t

        R^*_tm = q_{tm   *}   O^*_t     +    e^’_tm

Where:

S^*_t    = S^d_t / σ_St                   s_t,t-1  = σ_St-1 / σ_St                   N^’_t   = [1 / σ_St] _* N_t

O^*_t   = O^d_t / σ_Ot                c_t       = σ_St / σ_Ot                     U^’_t   = [1 / σ_Ot] _* U_t

R^*_tm  = R^d_tm / σ_Rt               q_tm    = σ_Ot / σ_Rt                     e^’_tm  = [1 / σ_Rtm] _* e_tm

S^*_t-1  = S^d_t-1 / σ_St-1

The coefficients in the model presented above are linked to commonly used concepts in survey research:

• The coefficient q_tm is the quality coefficient of measurement method m at time t. The square of this coefficient is the strength of the relationship between the opinion and the response, and q²_tm we call the quality of the measurement instrument. By definition, the quality is the complement of measurement error variance. This follows from the formula: q²_tm = 1-var(e_tm).

• The coefficient c_t is called the crystallization coefficient of the stable opinion at time t. The square of this coefficient is the strength of the relationship between the stable opinion and the opinion at time t, and c²_t we call the crystallization of the opinion. By definition, the crystallization is the complement of the variance of the unique opinion. This follows from the formula: c²_t = 1-var(U^’_t).

• The coefficient s_t,t-1 is the stability coefficient of the stable opinion. The square of this coefficient is the strength of the relationship between the stable opinion at time t-1 and t, and s²_t,t-1 we generally call the stability of the stable opinion. By definition, the stability is the complement of the variance of the new stable opinion. The variance of N indicates the amount of change in the stable part. This follows from the formula: s²_t,t-1 = 1-var(N^’_t).

A design that allows the estimation of the parameters contains six observations using the same question, in a three-wave panel. In each wave this question should be repeated¹. Data from that design could be analyzed with the path model presented in figure 1. Feldman suggested a similar model in 1989; however that model was not identified because only 1 indicator was specified for each opinion.

Figure 1: The VAS model.

The above model is called VAS model, to refer to its inventors, namely Van der Veld and Saris. An interpretation of the VAS model will be given starting at the bottom. Respondents are asked to provide two responses in each wave (interview) on the same question, but not necessarily the same survey item. So the measurement methods may differ within and across the waves. The responses to a specific survey item at time t observed with measurement method m are denoted by R^*_tm. The responses contain measurement errors (e^’_tm). If the responses are corrected for measurement errors, one obtains the opinion (O^*_t) at time t. The opinion at time t is the (weighted, due to the coefficients) addition of the unique opinion (U^’_t) and the stable opinion (S^*_t). The stable opinion in turn is the (weighted, due to the coefficients) addition of the stable opinion S^*_t-1 at time t-1 and the new stable opinion (N^’_t).

¹ A design with repeated observations within the wave can lead to a practical problem concerning the independence of the repeated observations. That is when memory effects play a role in the second observation, then the observations are said to be dependent. To prevent these memory effects one should make the time between the repeated observations as long as possible so that the previous response cannot be remembered. However, the time between the observations cannot be too long, because then the opinion (O^*_t) may change. Meurs van and Saris (1995) have shown that memory effects are virtually gone after 20 minutes if similar questions have been asked in between the two observations and the respondents have no extreme opinions. In this study the first measure is always observed at the very beginning of the interview and the second observation at the end. The average length of the questionnaires that are used in this study was approximately 50 minutes. It is therefore fair to assume that problems due to memory effects will not occur.