The relationship between parties and their supporters is central to democracy and ideological representation is among the most important of these linkages. We conduct an investigation of party-supporter congruence in Europe with emphasis on the measurement of ideology and focusing on the role of party system polarization, both as a direct factor in explaining congruence and in modifying the effects of voter sophistication. Understanding this relationship depends in part on how the ideology of parties and supporters is measured. We use Poole’s Blackbox scaling to derive a measure of latent ideology from voter and expert responses to issue scale questions and compare this to a measure based on left–right perceptions. We then examine how variation in the proximity between parties ideological positions and those of their supporters is affected by the polarization of the party system and how this relationship interacts with political sophistication. With the latent ideology measure, we find that polarization decreases party-supporter congruence but increases the effects of respondent education level on congruence. However, we do not find these relationships using the left–right perceptual measure. Our findings underscore important differences between perceptions of left–right labels and the ideological constraint underlying issue positions.