SURVEY QUALITY
Developing surveys is sometimes referred to as an art. The reason for this is that it seems that we lack the right information and instruments to asses the quality of our surveys. And this lack of information leads to a practice where many researchers (in business but also in science) still tend to come up with a set of nice questions they just happen to like and that ought measure the variables they want to measure. This generalization is likely to insult those who take survey development very seriously, and who don’t just treat survey design as an art. But unfortunately many researchers still do. This despite the fact that methodological research has revealed a lot about the effects of question wording. It is well known that small changes in question wording can lead to remarkable differences in response distributions. And that specific formulations can even affect the relations between different variables.
All the methodological studies did not seriously change the way we develop survey questions. The exception might be that nowadays there are so-called cognitive labs were survey questions are pre-tested on their linguistic and cognitive difficulty. Although such pre-tests can be very useful, they only provide information about what goes wrong, not necessarily why it goes wrong and how it can be fixed. And that is of course what we would like to know, and probably do know given the amount of methodological experiments that have been done in the past 50 years. Taken all together it is not surprising that the results of the methodological studies are for the largest part ignored by those who develop surveys, because the matter is extremely complex. Designing surveys requires many choices to be made with respect to question order, question wording, response scale type, length of the question, open or closed ended question, etc., etc. Each of these choices have their own effect on the quality of a survey question and it is impossible to know all these effects, let alone to take all those effects into account when developing surveys. But if we do ignore the information and the effects of our decisions on the quality of survey questions then survey development will remain an art. To make survey design a more scientific activity requires a systematic enterprise to develop an inventory with information about the effects of question characteristics on the quality of survey questions.
This inventory already exists and has been build with:
-
A large set of multitrait multimethod experiments that enables the estimation of the quality (reliability and validity) of questions.
-
A system developed for coding characteristics of survey questions;
-
A meta-analysis that links the question characteristics and the estimated quality of the survey questions.
This inventory is used in a computer program (SQP) that is able to read survey questions and that can also decompose survey questions into their characteristics. Because the effects of the question characteristics on the quality are stored in the inventory, it is possible to predict the quality of the survey questions. And these quality predictions can warn researchers of low quality survey items before their data is collected and provide suggestions for improving the quality of survey questions. In this way we hope that survey development will become more a scientific activity and not merely an art.
The following will be added later to this page:
Overview of all multitrait multimethod experiments that have been done so far
The automatic and semi-automatic SQP program.
The inventory of quality estimates.
Paper on how to use the inventory in your studies.