Exploring the effect of a middle response category on response style in attitude measurement

Perspectives on the concept of response style differ and range from minimizing-or even neglecting-the problem to highlighting it. Among the latter are those who argue that response styles are meaningful personality constructs, and as such are potentially measurable (e.g. Couch and Keniston 1960; Jackson and Messick 1965). From this perspective it is not justified to refer to the phenomenon as a response ‘style’ but as a response ‘trai’ that is not intended to be measured by items of a particular measurement instrument. The second perspective that recognizes the concept of response style only thinks of it as a statistical nuisance that should be controlled for in empirical models. As Mellenbergh (2001) has pointed out response bias arises in a situation in which a person responds to item properties other than the location of that item in the construct to be measured. In this second perspective response style is an artefact in measurement that depends on characteristics of the instrument that is used. Finally, some researchers question the meaningfulness of the concept of response bias altogether. Nunnally (1978) for instance, argues that there is little evidence that agreement tendency is an important measure of personality. He also claims that agreement bias is not an important source of systematic invalidity in measuring constructs. Rorer also questions the relevance of the concept by referring to ‘the great response-style myth’ (1965). Without going into detail, , Rorer’s arguments also relate to the lack of consistent empirical evidence, a finding that is recently confirmed by Ferrando et al. (2004) when they researched convergent validity of acquiescence in balanced and separate acquiescence scales. Nevertheless, in our opinion one should be prudent to generalize the findings from these studies. These findings do not suggest that one may feel confident in ignoring possible bias due to response style in survey research. After all, regardless whether one ‘beliefs’ that response style is a personality construct, a statistical nuisance, or even a non-significant factor, it is important to rule out that response style biases the measurement of constructs or implies misspecification of the relationship of content traits with covariates. Already in 1981, in their standard work on question form, wording and context in attitude surveys, Schuman and Presser warned that even the ‘dismissal of the importance of acquiescence in psychological investigations is not incompatible with the assumption of survey researchers that acquiescence is quite important in survey data’ (pp. 205–206). They refer to survey-based interpretations of acquiescence that hypothesize an inverse association with education. Furthermore, response styles may be greatest when vague, ambiguous or difficult to answer items are involved. From political science research, for instance, we have learned that the political value structure of respondents involved in politics can differ from general public (e.g. Inglehart 1990, chapter 9). Hence, it remains important to account for possible response bias in measuring constructs and estimating the effect of covariates.

It does not come as a surprise, given the aforementioned controversies regarding response style, that there is no single accepted statistical procedure to take response bias into account in attitude survey (Cheung and Rensvold 2000). However, within the context of structural equation modelling an approach has been suggested that is promising (Billiet and McClendon 2000; Moors 2003). Promising in the sense that the approach does not explicitly ally with a particular perspective on response styles as discussed in the previous section. Billiet and McClendon (2000) define the basic premises of this approach in the context of acquiescence response style. Their key argument is that it is possible to model a response style factor by means of a structural equation modelling of two balanced sets of items measuring two different constructs. As such, they combine the two major procedures for assessing acquiescence (Ferrando et al. 2004), i.e. (a) the balanced scale procedure and (b) the separate acquiescence scale approach. A balanced scale includes items that are negatively as well as positively worded and that are supposed to measure the same latent construct. It is assumed that acquiescence to the items in the positive direction is cancelled out by acquiescence to the items in the negative direction. One major critique, however, is that it is fairly difficult to operationalize complete reversals of items and hence that positively and negatively worded items may measure different or only partly related constructs. In Billiet and McClendon’s procedure this critique is overcome by combining balanced items with the idea that items should be pooled from heterogeneous sets. Pooling items from heterogeneous sets allows operationalizing a separate acquiescence scale. Hence, Billiet and McClendon estimate a confirmatory three-factor model including two so-called ‘content’ factors and one so-called ‘style’ factor. To identify the latter style factor they impose equality restrictions on the factor weights of all items of the two sets of questions. Hence, regardless of the content of the items the same factor loading is imposed on all positive and negative issues of two sets of items. Obviously, this style factor highly correlates with an index that counts the number of ‘completely agree’ answers on all items of the two sets of questions. Moors (2003, 2004) adapted the same rational within the context of a LC confirmatory factor approach to model extreme response bias. This approach is also adopted in this research. Conceptually a latent-class factor approach for analysing the latent structure of categorical variables is similar to the confirmatory factor (Lisrel) approach for the analysis of continuous variables. Explaining the technical details of the method is beyond the purpose of this paper. For the reader who is interested we refer to he following references that provide amore in-depth reading on the subject: Heinen (1996), Vermunt (1997) and Magidson and Vermunt (2001). However, we need to point out the advantage of the LC-factor approach adopted in this paper compared to the more commonly known confirmatory factor approach with Lisrel. The major difference is that a LC-factor approach does not use a correlation or variance/covariance matrix as an input. Instead it analyses the underlying pattern in the cross-classification of the responses pertaining to the manifest variables of interest. For example, a LC-factor model including four items, each of which has five response categories, involves analysing a ‘5 × 5 × 5 × 5’ table. Analysing such a cross-classification is perhaps more ‘complex’ to understand, but at the same time more informative as is demonstrated in Table 1.

Table 1 represents adjusted residuals of the crosstabulation of two ethnocentric-worded questions that are used in this research (cfr. infra). As is clear from the table, a positive association between the two items is observed. Such a relationship can be represented by one ordinal (gamma) or one linear (Pearson) correlation. The information that is ignored with these two summary measures, however, is that the residuals do not continuously decrease the further one moves from the main diagonal. For instance, respondents who fully agree with the ‘cultural threat’ issue are less likely to ‘disagree’ than to ‘completely disagree’ with the issue of ‘job threat’. This pattern suggests that an extreme response style might have influenced the way respondents answered to the questions (Moors 2003). A LC-factor approach with nominal indicators uses the full information from multiple frequency tables. By consequence, an effect of a LC-factor on a response variable is represented by several coefficients, i.e. equal to the number of response categories. Hence, if 5-response categories are offered, five coefficients are necessary to estimate the effect of one LC-factor on one item. Such a model is less parsimonious, but-as will be demonstrated in this paper-is flexible in diagnosing response styles.

Research on number of response categories has primarily focused on the reliability of attitude scales. Most research on this topic can be found within the psychological and psychometric literature; in the context of survey research this topic is less covered (Alwin 1992). Information theorists would probably argue that this question is trivial since the more response categories that are presented the more bits of information are conveyed. In survey research, however, the key question is whether there are an optimal number of response categories. This question is raised, not only from a statistical point of view, but also from a cognitive point of view (Alwin 1992). After all, too many categories may go beyond a respondent’s ability to distinguish among categories. Hence, recent research has focused attention to detecting a sufficient number of response categories that optimizes reliability and at the same time does not cause unnecessary burden upon a respondent (Viswanathan et al. 2004). The ‘ideal’ number of response categories, however, has not (yet) been established. Presumably, this ‘ideal’ number may also depend on the nature of attitude questions that is asked. Nevertheless, Alwin (1992) has suggested that this number might range between 4 and 7 categories with little left to gain in increasing the number higher than 7.

The question of offering a middle ‘neutral’ response alternative-which is the central focus of our research-is related to the aforementioned issue regarding the number of response categories in attitude research. Previous research has demonstrated that when this middle option is offered, it is far more likely to be chosen. Furthermore, it is argued that people who select a middle response alternative do not necessarily answer the question in the same way as other respondents if forced to choose sides on the issue (Bishop 1987; Kalton et al. 1980). The discussion about offering a middle response category may also extend to the validity of a measurement model. Hurley (1998) has argued that a mild response style can be regarded as the counterpart of the extreme response style. A mild response style implies a tendency to use the middle categories, while avoiding the extremes of a scale. As Presser and Schuman (1980) argued, less intense respondents are more affected by the presence or absence of a middle response category than respondents that feel strongly about the attitude. To some extent the mild response style is the conceptual counterpart of extreme response behaviour. In this research, we experimented with 5- vs. 6-response categories. Given the previous arguments we expect that offering 6-response categories will more easily allow to observe a mild response style as the counterpart of an extreme response style.

A short questionnaire was designed and administered in the context of the CentER-panel web-survey that was established in 1991. The random sample consists of over 2,000 households in The Netherlands representative of the Dutch population. On a weekly basis panel members complete a questionnaire on the Internet from their home. Households without Internet access are supplied with a set-top box with which questionnaires can be completed using a television screen as a monitor. A benefit of the web-based design is that it is easy to implement a split-ballot design in which respondents are randomly assigned to answer one of two versions of the questionnaire. The same questionnaire was administered only changing the number of response categories, i.e. 5 vs. 6. Including our split-ballot experiment in a representative sample survey also increases the validity of our findings compared to other research that often uses homogeneous student populations to conduct this type of experimenting. In Table 2, we compare the two samples with regard to the main covariates that are used in this research.

From Table 2 we read that the two samples are fairly comparable. The age distribution, however, slightly differs especially at the oldest age category. Differences between the other age categories proved not to be significant.

The questionnaire included two sets of four questions that were intended to measure ‘familistic attitude’, i.e. attitudes towards family and children, and ‘ethnocentrism’, i.e. attitudes towards immigrants. Each set included two positively (+) and two negatively (−) worded items. An overview of these items is presented in Table 3.

The response scales that were presented to the respondents were fully labelled, distinguishing between ‘completely disagree’, ‘disagree’, ‘agree’ and ‘completely agree’ in both versions. A ‘neutral’ category was included in the 5-categories version; this was substituted by ‘rather disagree’ and ‘rather agree’ in the 6-categories version.

Latent Gold 4.0 was used to estimate a confirmatory LC-factor model with one \lsstyle\rs factor influencing all eight items, one \lscontent\rs factor influencing the responses on the four \lsfamilistic\rs attitudes, and a second \lscontent\rs factor that influences the responses to the four \lsethnocentric\rs attitudes. The two content factors are allowed to correlate. This type of analysis in which the manifest items are treated as nominal response variables, and the LC-factor as a discrete interval variable is referred to as a latent trait approach. For ease of expose, assume a model with two sets of items (A and B) and two LC-factors (X). Then the conditional response probabilities of this latent-class factor model can be written as:

The response probabilities of this model are restricted by means of logit models with linear terms:

Since a LC-factor approach assumes that the factors are discrete interval (or ordinal) variables, the two-variable terms (e.g. \( \beta _{A1}^1 \cdot \upsilon _{X1} \) .) are restricted by using fixed category scores for the different categories of the LC-factor. Equidistant scores υX range from 0 to 1, with the first category of a factor getting the score 0 and the last category the score 1. Hence, a LC-factor with, for instance, four categories gets the scores 0, 1/3, 2/3 and 1. As such the categories of the factors are ordered by the use of fixed equal-interval category scores. The \gb’s indicate the strength of the relationship between factors and response variables. Equation (2) identifies a confirmatory LC-factor model with factor X₁ influencing the response probabilities of items A₁ and A₂, and factor X₂ influencing items B₁ and B₂.

In preparation of this paper, we estimated several models that differed in the number of categories of the LC-factors. We tested models with two, three and four categories. Results of the two-category model differed slightly from the models in which LC-factors were identified with three and four categories. Results of the three and four categories models were highly similar. The analyses reported in this paper identified LC-factors with four ordered categories. The covariates that are listed in Table 2 are also included in the analysis. After all, we have argued that response styles may bias the measurement of constructs as well as the effect of covariates. In the next section, we first discuss the comparison of the measurement models of the two samples. Then we focus on covariate effects.

There are different possibilities to present the LC-factor results. One obvious choice would be to present the β’s as defined in Eq. 2. Since our analysis includes 5 or 6 coefficients for each item per factor, this would result in a huge table, which would be difficult to interpret unless one is familiar with the method. Furthermore, the β’s do not have an upper limit which makes them more difficult to interpret and compare. For these reasons we have opted for a graphical presentation of the probability means (Figs. 1, 2). A probability mean is the mean latent-class factor score for each response category and ranges from 0 to 1. The order of the response categories of the negatively worded item has been reversed in Figs. 1 and 2 to be consistent in content to the positively worded items. Hence, the order for negatively worded items ranges from ‘completely agree’ (c.a.) to ‘completely disagree’ (c.d.), whereas the positively worded items range from ‘completely disagree’ (c.d.) to ‘completely agree’ (c.a.). Figure 1 compares the results of the effect of the extreme ‘style’ LC-factor on the eight items. Figure 2 includes the comparison of the two ‘content’ LC-factors, i.e. ‘familistic values’ and ‘ethnocentrism’.

The main finding from Fig. 1 is that regardless of the number of response categories that is used, an extreme response style is clearly observed. The U-shape curve indicates that the mean probability is highest among the two extreme response categories ‘completely agree’ and ‘completely disagree’. The principal difference between the two question formats is how the intermediate response categories position. When 5-response categories are offered, there is little difference between the ‘neutral’ category on the one hand, and the ‘agree’ or ‘disagree’ categories on the other. There are three exceptions in which the mean probability for ‘disagree’ is (somewhat) higher, i.e. ‘working mother’, ‘job orientation’ and ‘trust’. It is difficult to pinpoint the rational of these exceptions. Nevertheless, what can be concluded from comparing the intermediate response categories in Fig. 1(a) is that a ‘modest’ response style is not unequivocally observed-assuming of course, that such a modest response style leads to a greater prevalence of a neutral response. In Fig. 1(b) in which the results of the 6-response categories sample are summarized, a more consistent picture emerges. First of all, it confirms the findings from Fig. 1(a) that an extreme response style is clearly observed. Second, one particular response category clearly positions between the two extreme response on the one hand, and the other response categories on the other hand. This intermediate response category is not related to a particular response number and label. Rather it corresponds with ‘disagree’ for the negatively worded questions and ‘agree’ for the positively worded questions. Hence, it is in consistency with the content of the question, i.e. reflecting higher agreement with ‘familistic values’ and ‘ethnocentrism’. Only as far as the ‘cultural threat’ issue is concerned, this is less strongly observed. There is also a slight tendency for the two middle response categories to have the lowest mean probability values. However, to conclude from this that a ‘modest’ response style is emerging is difficult. For this, we need a more clear-cut difference of the two middle response scales with both adjacent ‘agree’ and ‘disagree’ categories. In this study, this is only the case with one of these adjacent categories. However, this is not to say that a ‘modest’ response style is non-existing either. After all, it is possible that lower agreement with ‘familistic’ and ‘ethnocentric’ issues may reflect a ‘modest’ response style. Future exploration of the issue by increasing the number of response categories may provide some answer.

The two ‘content’ factors are reproduced by the second and third LC-factor of the model. The general finding is that moving from left to right in response categories presented in Fig. 2, the mean probability scores of the LC-factors also increase, which is in consistency with the content of each item. There are, however, some differences that need a closer look. As far as the ‘familistic values’ LC-factor is concerned, there is a clear difference between the 5- and 6-response categories models. The difference is that in the 6-response categories model the mean probability score of the ‘job satisfaction’ issue first increases as the level of disagreement increases, but this is reversed at the end of the scale. The mean probability score of the ‘completely disagree’ category is lower than the score of the ‘disagree’ category. The same pattern is observed for the first issue ‘working mother’, but in this case the differences are very small. A likely explanation of this finding could be that the set of ‘familistic issues’ is less homogeneous than the set ‘ethnocentric’ issues. Taken at face value the ‘working mother’ and ‘pre-school child’ issues refer to the impact of a working mother on children; the other two issues refer to the conflicting roles of family versus job for women. Furthermore, inspecting the pseudo-R ² values (Table 4) for each adds support to this interpretation. These pseudo-R ² values indicate the ‘explained variance’ of the items by the LC-Factors. The explained variance of the two items of the familistic values dimension, i.e. ‘family orientation’ and ‘job orientation’, is lower than the explained variance of the other items in the analyses. This was true for both versions of the questionnaire. The fact that the more detailed six categories scale revealed a non-ordinal relationship of ‘job orientation’ with the latent content factor of familistic values might merely point to this issue.

There is little difference in how the two versions of the ‘ethnocentric’ items relate to the corresponding LC-factor, i.e. probability means increase in consistency with the ordered response categories. An interesting observation is that in the model with 5 response categories there is a large gap between the first and second response category, after which differences between adjacent categories decrease. In the model with 6 response categories, the adjacent categories are more evenly spread.

From the previous section we have learned that the measurement models of the 5- and 6-response scales were similar and at the same time revealed particularities. The next step is to compare the two models as far as the effect of selected covariates are concerned. In Table 5, we present the effect of gender, age and education on the extreme response style factor (LC-factor 1), the familistic orientation (LC-factor 2) and ethnocentrism (LC-factor 3). Recall that the figures presented in Table 5 are the structural part of the LC structural equation model that was also used to identify the LC-factors. Drawing an analogy with Lisrel modelling, Hagenaars (1990, see also: Goodman 1974) has referred to this model as a ‘modified Lisrel approach’. The three covariates are treated as categorical. Deviation coding was used which means that the overall effect (beta) of each covariate is fixed to zero, which is the reference value. Associated standard errors are presented, as well as the p-value of the Wald-statistic which indicates an overall significance of a covariate.

A first general finding is that the extreme response LC-factor does not significantly relate to any of the three covariates, suggesting that this response style may reflect a personality characteristic which has little to do with social background. In both models, gender is related to the second LC-factor ‘familistic values’ withmen beingmore ‘conservative’ than women. The education variable was not significant in the 5-response categories analysis although the beta’s suggest that ‘familistic values’ decrease with educational level. In the model with 6-response categories, education was significant. The two categories of education that stand out are (a) the highest level of university education (β = −1.112) that is the least ‘conservative’ in ‘familistic values’ and (b) the lower category of pre-vocational education (β = +1.154) that is the most conservative. Age differences in ‘familistic values’ were not significant.

As far as the third ‘ethnocentrism’ LC-factor is concerned education was the single most important covariate in both samples. Lower levels of education showed the highest level of ethnocentrism, whereas the higher educated were the least ethnocentric. The 5-response categories model indicates that ethnocentrism decreases with age, except for the oldest category. Recall that this effect is controlled for educational differences. In the 6-response categories model the overall effect of age is not significant, although the youngest category has also a higher level of ethnocentrism similar to the 5-categories sample.