Women through the glass ceiling: gender asymmetries in Wikipedia

Assessing the notability of people is a difficult task. Fortunately, Wikipedia and search engines like Google allow us to gauge public interest in different people and from different locations over time. Such signals can be employed as proxies for the notability of people. These proxy measures are noisy and may also be biased, since they reflect the interests of Google users or Wikipedia editors, which in turn are influenced by many factors. Nevertheless, both signals that we explore let us compare the public interest in men and women. While our analysis allows us to quantify the existence of a glass-ceiling effect, it does not permit an assessment of its origin. It could be that Wikipedians unconsciously apply different thresholds for men and women or that Wikipedia only reflects the glass ceiling of our society and other media, which only document the life of women who have higher capacities and abilities than men which are covered.

Concretely, we use the following external and internal proxy measures:

Google search volume: The Google trend³ data gauge the interest of Google users between 2004 and 2015. Google trend data serve as an external proxy for the public interest toward a person, or information need about that person, and can be measured in different countries and at different points in time.

For a random sample of around 5,000 people born after 1900 and before 2000 we collected Google trend data using the full name of the person as input. Google trends shows how often search terms are entered in Google relative to the total search volume in a region or globally. Using full names as search terms will of course introduce noise since several people may share the same name. However, a similar level of noise can be expected for men and women.

We count the number of countries and the number of months between January 2004 and October 2015 (from a worldwide perspective) that reveal a relative search volume above a threshold chosen by Google. The Google threshold is relative to the total number of searches in the region and month under consideration. To explore whether the number of countries and number of months in which we observe search volume above the threshold is influenced by gender, we fit two negative binomial regression models that both use gender as the independent variable. We also used a linear regression model and obtained similar results, but a loss of power.

After the investigation of potential differences in entry barriers, let us focus on the lexical presentation of those who made it into Wikipedia. Language use is reportedly different when speaking about different genders [9]. For example, the Finkbeiner test [10] suggests that an article about a woman often emphasizes the fact that she is a woman, mentions her husband and his job, her children and childcare arrangements, how she nurtures her underlings, how she is taken aback by the competitiveness in her field, and how she is such a role model for other women. Historian Gillian Thomas investigated the role of women in Encyclopaedia Britannica, finding that as contributors, women were relegated to matters of ‘social and purely feminine affairs’ and as subjects, women were often little more than addenda to male biographies (e.g., Marie Curie as the wife of Pierre Curie) [11].

Beside topical bias, previous research also suggests that linguistic biases may manifest when people describe other people that are part of their in- or out-group [12]. Linguistic bias is a systematic asymmetry in language patterns as a function of the social group of the persons described, and is often subtle and therefore unnoticed. The Linguistic Intergroup Bias (LIB) theory [13] suggests that for members of our in-group, we tend to describe positive actions and attributes using more abstract language, and their undesirable behaviors and attributes more concretely. In other words, we generalize their success but not their failures. Note that verbs are usually used to make more concrete statements (e.g., ‘he failed in this play’), while adjectives are often used in abstract statement (e.g., ‘he is a bad actor’). Conversely, when an out-group individual does or is something desirable, we tend to describe them with more concrete language (we do not generalize their success), whereas their undesirable attributes are encoded more abstractly (we generalize them). Maass et al. point out that LIB may serve as a device that signals to others both our status with respect to an in- or out-group, as well as our expectations for their behavior and attributes [13]. Our expectations are of course not only determined by our group-membership but also by the society in which we live. For example, in some situations or domains not only men but also women may expect other women to be inferior to men.

While it is well known that topical and linguistic biases exist, it is unknown to what extent these biases manifest in Wikipedia. To investigate this question we compare the overview of biographies about men and women in the English Wikipedia. The overview (also known as lead section) is the first section of an article. According to Wikipedia, it ‘should stand on its own as a concise overview of the article’s topic. It should define the topic, establish context, explain why the topic is notable, and summarize the most important points.’⁴ We focus on the lead section for two reasons. On one hand, the first part of the article is potentially read by most people who look at the article. On the other hand, Wikipedia editors need to focus on what they consider most important about the person, and biases are likely to play a role in this selection process.

Topical bias: To unveil topical biases in Wikipedia content, we analyze the following three topics that could be over-represented in articles about women according to what is suggested by Thomas’s observations in Britannica and the Finkbeiner test:

To associate words with these topics (plus an unrelated category, other), we follow an open vocabulary approach [14]. Because we want to include concepts that may comprise more than one word, we consider n-grams with \(n \leq2\). We then analyze the association between the top 200 n-grams for each gender and the four topics (gender, relationship, family, or other). To rank the n-grams for men and women we use Pointwise Mutual Information [15]. PMI measures the relationship between the joint appearance of two outcomes (X and Y) and their independent appearances. It is defined as: where, in our case, X is a gender and Y is an n-gram. The value of \(P(X)\) can be estimated from the proportions of biographies about men and women, and the other probabilities can be estimated from n-gram frequencies. PMI is zero if X is independent of Y, it is greater than 0 if X is positively associated with Y, and it is smaller than 0 if X is negatively associated with Y. We exclude words that appear in biographies from one gender only, because such words have undefined PMI for the other gender, and thus the comparison is not meaningful. We are interested in words/n-grams that may appear in any gender, and which presumably could be independent of gender. Finally, we compare the proportion of topics that are present in the top 200 n-grams that we associated with men and women using chi-square tests. In the absence of topical asymmetries, one would expect to observe only minor differences in the proportions of topics for men and women.

Linguistic bias: To measure linguistic bias, we use a lexicon-based approach and syntactic annotations to detect abstract and subjective language as proposed by Otterbacher [12]. The level of abstraction of language can be detected through the syntactic class of terms, where adjectives are the most abstract class, as for example comparing ‘is violent’ with ‘hurt the victims’ [16].

To test for the existence of linguistic biases in Wikipedia, we quantify the tendency of expressing positive and negative aspects of biographies with adjectives, as a measure of the degree of abstraction of positive and negative content. We quantify the tendency to use abstract language in each class as the ratio of adjectives among positive and negative words. To do so, we detect positive and negative terms taken from the Subjectivity Lexicon [17]. For each term that in the lexicon, we check if it is an adjective or not based on part-of-speech tags [18].

After processing the text, we count for each biography the numbers of positive \(W_{+}\) and negative \(W_{-}\) words, and from those the numbers of positive adjectives \(A_{+}\) and negative adjectives \(A_{-}\). We combine these counts into ratios of abstract positivity and negativity computed as \(r_{+}=A_{+}/W_{+}\) and \(r_{-}=A_{-}/W_{-}\). This way, we quantify the tendency to generalize positive and negative aspects of the biographies, with the purpose of testing if this generalization depends on the gender of the person being described.

The presence of gender stereotypes and sexism and the Linguistic Intergroup Bias (LIB) theory suggest that abstract terms would be more likely to be used to describe positive aspects in the biographies of men than in biographies of women. Similarly, abstract language would be more likely to describe negative aspects in the biographies of women in comparison to biographies of men. We test this hypothesis first through a chi-square test on the aggregated ratios of adjectives over positive and negative words in all biographies of each gender. To test if the bias appears at the individual level, we then focus on biographies with at least 250 words and one evaluative term, testing if the measured \(r_{+}\) and \(r_{-}\) depends on gender while controlling for professions and the century in which a person was born.

Structural properties impact how visible and reachable articles about notable men and women are, since users and algorithms rely on this information when navigating Wikipedia or when assessing the relevance of content within a certain context. For instance, search result rankings are often informed by centrality measures such as PageRank. Furthermore, search results show meta-data when the query is related to notable personalities (using, e.g., the Google Knowledge Graph [19]). These examples show that gender inequalities that manifest in the structure of Wikipedia may have important implications since they impact the information consumption process.

Hyperlink network: We build a network of biographies using the hyperlink structure among Wikipedia articles about people in the English language edition. Concretely, we use the structured links between the canonical URLs of articles provided by DBpedia, where redirects are resolved. On this network we perform two different analyses: first, we explore to what extent the connectivity between people is influenced by gender, and second, we investigate the relation between the centrality of people and their gender. To this end, we compute the PageRank of articles about people. PageRank is a widely used measure of network centrality [20, 21]. To explore potential asymmetries in network centrality, we sort the list of biographies according to their PageRank values in descending order. We estimate the fraction of biographies that are about women at different ranks k. In the absence of any kinds of inequality, whether endogenous or exogenous to Wikipedia, one would expect the fraction of women to be around the overall proportion of women biographies, irrespective of k.

To discern whether the observed asymmetries with respect to gender go beyond what we would expect to observe by chance, we compare our empirical results with those obtained from baseline graphs that are constructed as follows:

We measure the ratio between men and women as a function of the number of language editions in which they are depicted. If the Wikipedia entry indeed functions as a glass ceiling, we expect to see a larger gender gap for ‘local heroes’ than for ‘superstars,’ because fewer female local heroes would be able to overcome the glass ceiling. The exclusion of less notable women would also imply that, on average, women in Wikipedia should be more notable than their male counterparts. On the contrary, the inclusion of less notable men would decrease the average notability of men in Wikipedia.

Figure 1 shows that since 1900, the gap between men and women is indeed larger for people with low or medium level of global notability than for the ‘global superstars,’ compared to a baseline. If we focus on strictly local heroes (people who only appear in one language edition), the men to women ratio is larger than expected by chance. In the population of people born since 1900, men are 5.62 times more likely than women to be included in Wikipedia if they are only included in one language edition. By random chance (estimated by reshuffling the gender) we would expect a ratio of 4.94 for those people. This means that the population is 15.1% women versus the expected 16.8%, i.e., women are around 10% less likely to be included than we would expect by chance. This difference is important because almost half of our population (45% of men and 40% of women) belongs to the group of strictly local heroes. For global superstars, the gap tends to be smaller than expected.

We also find a higher than expected gap for strictly local heroes born before 1900. The men/women ratio is 13.28 versus an 11.73 baseline. In this case women are about 11% less likely to be included as local heroes than we would expect by chance. Again, a large portion of our population belongs to this group (44% of men and 39% of women). The main difference between the two populations in Figure 1 is that the gender gap for people born before 1900 does not decrease systematically with increasing notability.

A possible explanation for the high men-to-women ratio for local heroes is that the entry barrier into Wikipedia is higher for women than for men. Note that people can also create articles about themselves in Wikipedia; men are on average more self-absorbed than women [24], and thus may be more likely to create articles about themselves. Another possible explanation is that more information may be available online about less notable men than about less notable women. Since Wikipedia editors rely on secondary information sources, their decisions also reflect the biases that exist in other media.

To further quantify the glass-ceiling effect while controlling for other factors that may potentially explain our results (e.g., profession and age), we use a negative binomial regression model and explore the effect of gender on the number of language editions including a person. We performed three different regressions: one for people born before 1900 (\(N_{b}\)), one for people born since 1900 (\(N_{a}\)), and one for the entire dataset (N). The coefficients that are reported in Table 2 can be interpreted as follows: if all other factors in the corresponding model were held constant, an increase of one unit in the factor (e.g., from male to female, from Person to Scientist, etc.) would increase the logarithm of the number of editions by the fitted coefficient β. The Incidence Rate Ratio (IRR) of each factor is obtained by exponentiating its coefficient.

The regression from the full dataset (last column in Table 2) reveals that being female makes a biography increase its edition count by an IRR of 1.13, all other parameters equal. This effect is significant (\(p < 0.001\)), indicating that women in Wikipedia are 13% more notable than their male counterparts. If we only look at people born since 1900, we see that women are 12% more notable than men, while limiting our dataset to people born before 1900 indicates that women are 4% less notable than men. For people in Wikipedia born before 1900, being a female decreases the chances of notability, as one would predict based on the historical exclusion of women [25]. Conversely, for people in Wikipedia born since 1900, being female increases the chances of notability. Due to the noted relation between being historic and global notability (see Figure 2), we cannot claim a glass-ceiling effect for inclusion in Wikipedia of women born prior to 1900.

We also observe interesting differences for professions. For example, being a philosopher has the strongest positive effect on being of global importance (\(\mathrm{IRR} = 4.8\), \(p < 0.001\)), while being a journalist has the strongest negative effect on global importance (\(\mathrm{IRR} = 0.37\), \(p < 0.001\)). This indicates that people with certain professions are more likely to be recognized globally if they contributed something, while others are more likely to be recognized locally. While we do observe interesting differences among professions, further analysis is necessary to investigate whether professional differences in notability are confounded by the average birth decade. For instance, a quarter of the top 100 historical figures are philosophers [26], while journalists are more likely to have become famous in recent years.

The model further indicates that the decade when a person was born is negatively associated with notability (\(\mathrm{IRR} = 0.99\), \(p < 0.001\)); the more historic a person is, the more notable they are from a global perspective. This is expected: people from older centuries appear on Wikipedia because their ideas and actions have transcended time (through secondary sources). Conversely, people of recent fame can be notable in terms of availability of secondary sources, but not necessarily because their ideas will remain valuable in time. Interestingly, we find that the birth decade factor has a different effect when we look at people pre-1900 and post-1900. For people born before 1900, as with the global dataset, being historic is associated with notability (\(\mathrm{IRR}_{b} = 0.98\), \(p < 0.001\)). When we consider people born since 1900 we find that Wikipedia developed a ‘recency bias’; people in this group are slightly more notable if they were born more recently (\(\mathrm{IRR} = 1.01\), \(p = 0.008\)). A possible explanation is that younger people may benefit from the greater availability of digital information about them or generated by them, making them more likely to be recognized by Wikipedia editors.

Let us next compare the external notability proxy (based on geographic and temporal search interest) of a random sample of men and women in Wikipedia born since 1900. Table 3 shows that women in Wikipedia are slightly more of interest to the world according to Google’s relative search volume statistics. Both coefficients are significantly positive: on average, women are of interest in more regions (\(\mathrm{IRR} = 1.555\)) and during more months (\(\mathrm{IRR} = 1.322\)). The mean number of regions with search volume above the Google threshold is 2.10 for women, 1.56 for men; the median is zero for both. The mean number of months during which we observe a global search volume above the Google threshold is 34 for women, 30 for men. The median number of months is one for women and zero for men.

While our results suggest that the gender of a person that made it into Wikipedia is significantly related to the number of regions and months in which this person is of interest, we cannot exclude other confounders. For example, women included in Wikipedia tend to be born in recent years (see Figure 3) and people born in recent years may have received more attention on Google between 2004 and 2015. Controlling for year of birth and profession was not possible due to the technical challenges of collecting large amounts of Google trend data. Focusing on sub-samples of people who are born in the same year and share the same profession may allow to address these confounding factors future research.

Following the notability analysis, we must consider time as a confounding factor. We therefore consider two groups of biographies: those with birth date prior to 1900, and those with birth date from 1900 onwards. We estimated the PMI of each word and bi-gram in our vocabulary for each gender. Since the PMI give more weight to words with very small frequencies, we considered only n-grams that appear in at least 1% of men’s or women’s biography overviews. Our findings for each dataset are summarized as follows:

Figure 4 shows the n-grams that are strongly associated with each gender. The bi-grams that are strongly associated with women born before 1900 relate frequently to categories such as gender, family, and relationships. Words associated with men mainly relate to other categories, such as politics and sports. Table 4 shows the proportion of the top 200 n-grams that fall into each category, for both genders in both periods. The categories gender, relationship, and family are more prominent for women than men. However, the distributions of those categories are different in the two periods under consideration. The distribution is significantly different across genders only pre-1900, according to a chi-square test (\(\chi^{2} = 14.33\), \(p < 0.01\)). In prior work we have shown that the differences are significant if time is not considered [5] and that similar results hold for five other language editions [4].

Table 5 shows the ratios of abstract terms among positive and negative terms when aggregating all the text in the summaries of the biographies of men and women separately. One-tailed chi-square tests suggest that linguistic biases appear along the predicted directions: more abstract terms are used for positive aspects of men’s biographies and for negative aspects of women’s biographies. Effect sizes, measured by Cohen’s w, are very small, in line with the typically small effects in other studies in psycholinguistics. When measuring relative changes, we find that adjectives are almost 9% more likely to be used to describe positive aspects of men’s biographies, while 1.62% more likely to describe negative aspects in women’s biographies.

We apply linear regression in two models, one with \(r_{+}\) as dependent variable and another one with \(r_{-}\), expressed as a linear combination of gender, class, and century of birth. We focus on all biographies with valid birth dates and at least 250 words in their summary. Our results indicate that women’s biographies tend to have fewer abstract terms for positive aspects and more abstract terms for negative aspects, as predicted by the LIB (see Table 6). This effect is robust to the inclusion of control variables like profession and century of birth. We repeated the analysis using a logit transformation of \(r_{-}\) and \(r_{+}\), as well as with beta regression, finding the same results.

In total, the DBpedia dataset contains 340 attributes extracted from infobox templates. Of those attributes, 33 display statistically significant differences. Only 14 of them are present in at least 1% of the male or female biographies. These attributes are shown in Table 7. As in previous sections, we have estimated the significance of their differences for people born before and since 1900. An analysis of the entire dataset without considering time is presented in our previous work [5].

Due to the number of available attributes, the portion of biographies that contains each of them is small. Thus, instead of considering p-value correction, we discuss the statistically significant gender differences manifested in the meta-data to qualitatively assess whether they have significance in our context:

All differences found have large effect sizes (Cohen’s \(w > 0.5\)).

We constructed the empirical network from the inter-article links among 893,380 biographical articles in the English Wikipedia. After removing 192,674 singleton nodes (of which 15.3% were female), the resulting graph had n= 700,706 nodes (of which 15.6% were female) and 4,153,978 edges. All baseline graphs have the same number of nodes n and approximately the same mean degree \(k \approx4\) as the empirical network. The small world baseline has a parameter \(\beta= 0.34\) representing the probability of rewiring each edge. Its value was set using the Brent root finding method in such a way as to recover the clustering coefficient of the original network.

Figure 5 shows the top 30 men and women according to their PageRank. The top-ranked women are slightly less central than men, and the centrality of women decreases faster than that of men with decreasing rank. The top-ranked biographies are similar to those found in previous work [26, 27].

In addition to the full hyperlink network, we created two sub-networks: one only contains people born before 1900 and the other only contains people born since 1900. For each empirical network, we created several null models and compared the proportion of links within and across genders using a chi-square test. Table 8 indicates that in both empirically observed Wikipedia graphs, women biographies have more links to other women articles than one would expect by chance. A possible explanation for this asymmetry stems from the reported interests of female editors, who frequently edit biographies about women in Wikipedia [28].

The effect of structural differences on visibility can be analyzed in terms of how many women are ranked among the top biographies by centrality scores. Figure 6 displays the fraction of women in subsets of top-ranked biographies. For people born before 1900, the fraction of women in the top k biographies is below the expected ratio of 7.8% up to \(k \approx 10^{3}\), and above when lower-ranked biographies are considered. For people born since 1900, the fraction or women is below the expected ratio of 16.8% for the entire range of k. This indicates that the empirically observed structure of the Wikipedia hyperlink network puts women at a disadvantage when it comes to ranking algorithms, especially for women born since 1900. For people born before 1900, as k increases, the relative fractions of women among the top k biographies in the baseline networks converge to the expected ratios faster than in the empirical networks. This implies an asymmetry that cannot simply be explained by heterogeneities in the structure of the networks, since our baseline graphs preserve several characteristics of the empirical network, including the broad distribution of node degrees. Therefore one must conclude that there exists a bias in the generation of links by Wikipedia editors, favoring articles about men.