In examining the factors that contribute to electoral success in congressional elections, legislative scholars often consider the actions of elected representatives; however, other research suggests that one must consider what challengers are (or are not) doing as well. For instance, inexperience and poor funding can significantly inhibit challenger success. We expand this list of potential shortcomings by arguing that ideological congruence with a constituency may be another factor in explaining challenger defeat. Using ideology measures derived from campaign contributions, we find that unsuccessful challengers in the U.S. House are generally more extreme than those who win, but ideological extremity is not a disadvantage to those seeking to represent an extreme constituency. More importantly, our existing political institutions may actually serve to mitigate the already high levels of partisan polarization in Congress.