It is well established that the wording of the questions, the order in which they are asked and the number and form of alternative answers offered can influence results of polls. For instance, the public is more likely to indicate support for a person who is described by the operator as one of the "leading candidates". This support itself overrides subtle bias for one candidate, as does lumping some candidates in an "other" category or vice versa. Thus comparisons between polls often boil down to the wording of the question. On some issues, question wording can result in quite pronounced differences between surveys.
[20][21] This can also, however, be a result of legitimately conflicted feelings or evolving attitudes, rather than a poorly constructed survey.
[22]
A common technique to control for this bias is to rotate the order in which questions are asked. Many pollsters also split-sample. This involves having two different versions of a question, with each version presented to half the respondents.
The most effective controls, used by
attitude researchers, are:
- asking enough questions to allow all aspects of an issue to be covered and to control effects due to the form of the question (such as positive or negative wording), the adequacy of the number being established quantitatively with psychometric measures such as reliability coefficients, and
- analyzing the results with psychometric techniques which synthesize the answers into a few reliable scores and detect ineffective questions.
These controls are not widely used in the polling industry.[
why?]. However, as it is important that questions to test the product have a high quality, survey methodologists work on methods to test them. Empirical tests provide insight into the quality of the questionnaire, some may be more complex than others. For instance, testing a questionnaire can be done by:
- conducting cognitive interviewing. By asking a sample of potential-respondents about their interpretation of the questions and use of the questionnaire, a researcher can
- carrying out a small pretest of the questionnaire, using a small subset of target respondents. Results can inform a researcher of errors such as missing questions, or logical and procedural errors.
- estimating the measurement quality of the questions. This can be done for instance using test-retest,[23] quasi-simplex,[24] or mutlitrait-multimethod models.[25]
- predicting the measurement quality of the question. This can be done using the software Survey Quality Predictor (SQP).[26]