Diversity of editors and teams versus quality of cooperative work: experiments on wikipedia

Team diversity is one of the fundamental issues in social and organisational studies that has been broadly researched on (e.g. Parnas 1972; Sanchez and Mahoney 1996; Langlois and Garzarelli 2008). It has been broadly theorised and tested on virtual communities. One of the most burning questions concerns team coherence vs efficiency. There are two competing theories describing the efficient team organisation: modularity and integrity (Parnas 1972; Sanchez and Mahoney 1996). The first was introduced by David Parnas who suggested that co-dependence between “components” or “modules” (in our context this concept corresponds to an article on Wikipedia) should be eliminated by limiting the communication induced by the modules (Parnas 1972). In this approach participation in a module does not require knowledge about the whole system or other modules, e.g., Wikipedia users can co-author articles about social science without knowing anything about life sciences or mathematics. It leads to higher specialization and less diversity in individual performance. A modular approach enables more flexibility and decentralized management (Sanchez and Mahoney 1996).

In the integral mode the team members have diverse knowledge and skills. We aim to study whether modular/specialized or integral collaboration pattern is more successful in creating high-quality Wikipedia articles.

The contributions of our work include:

This article is a substantial extension of a conference paper (Baraniak et al. 2016) where the parts of the two first of the above contributions were preliminarily presented.

Our experimental results seem to positively confirm hypothesis that diversity of single editors and teams is positively related to the quality of their work and that diversity is usually more important than some seemingly more obvious attributes such as size or productivity of the team.

The general comparison of quality of classic and open-collaboration encyclopediae, in particular Britannica vs Wikipedia is discussed in Giles (2005) when it is observed that the quality of Wikipedia (in terms of number of errors) is not much lower than that of Britannica, which is a bit surprising result.

The problem of how the number of editors and the coordination method of their work influences the article quality is studied in Kittur and Kraut (2008). Two coordination methods are considered: the explicit one and the implicit one. In the latter one, the work is planned and coordinated by explicit communication between all the editors while in the second one the most of the work is organised and done by a small subset of the editor team. The presented results demonstrate that adding more editors can improve the article quality only if the applied work coordination method was appropriate. In particular the results indicate that the implicit coordination helps more in larger teams.

The interplay between the phenomenons of social influence and social preference based on similarity between the editors in the context of open collaboration in Wikipedia is studied in Crandall et al. (2008). The results indicate that both phenomenons play an important role in explaining the open collaboration patterns.

In Wilkinson and Huberman (2007) it is reported that the high-quality articles are those that are intensively edited and have high number of editors as compared to other articles of similar age. Our work shows that diversity is not less important in this context.

The important role of a diversity was noticed early not only in complex systems but also in other fields like Operation Research or Information Retrieval (Goffman 1964). One of the earliest successful applications of diversity-aware approach was reported in Carbonell and Goldstein (1998) in the context of text summarisation. Recently, diversity-awareness has gained increasing interest also in other information-related areas where the actual information need of a user is unknown and/or the user query is ambiguous so that a controlled level of diversity introduced to the results increases their quality. Examples range from databases (Vee et al. 2008) to Web search (Agrawal et al. 2009) or to the quite novel problem of graphical entity summarisation in semantic knowledge graphs (Sydow et al. 2013). A recent work (Strzezek et al. 2015) demonstrates that a controlled level of population diversity increases the performance of genetic algorithm for some hard optimisation problems.

The concept of diversity has also attracted interest also in the domain of open collaboration research, e.g. in Aggarwal (2014). From the open collaboration point of view, diversity can be considered from many perspectives, for example as a team diversity vs homogeneity or a single editor’s versatility (called “integrity” in that work) vs specialisation (called “modularity” in that work).

The positive role of team diversity was studied in Chen et al. (2010), where productivity and diversity of teams can be defined in a different sense and it is suggested that other variables may have influence on the quality of article.

In our work we use different definitions of diversity and its measures, since we quantify it with the use of the concept of entropy and on standard deviation, as will be explained in Section 2. Most importantly, in contrast to our work, the mentioned work studies the influence of diversity on the amount of accomplished work and withdrawal behaviour rather than the work quality that is considered here.

In contrast to our work most of previous works focus on diversity of editor teams in terms of categories such as culture, ethnicity, age, etc. López and Butler (2013) studies how the content diversity influences on-line public spaces in the context of local communities. A recent example, with a special emphasis on ad-hoc “swift” teams where the members have very little previous interactions with each other is Aggarwal (2014). Vasilescu et al. (2015) studied gender diversity relationship with work outcome.

A recent article (Ren et al. 2015) studies how tenure diversity and interest variety affect group productivity and member withdrawal and how the two types of diversity evolve over time. The results of this work seem to indicate the importance of the interest and experience diversity in online collaboration but does not directly address the issue of how it impacts the quality of the resulting articles that is the topic of this article.

In this section we explain the model of interest diversity that we apply in our approach. We use Wikipedia terminology to illustrate the concepts, however, our model can be adapted to other, similar open-collaboration cooperative work environments.

Let X denote a group of Wikipedia editors. Editors participate in editing Wikipedia articles. Each article can be mapped to one or more categories from a pre-defined set of categories C = {c ₁,…,c _k } that represent topics.

Each editor x∈X in our model is characterised by his/her editing activity i.e., all editing actions done by x. We assume that the interests of an editor x can be represented by the amount of work that x committed to articles in particular categories.

Let t(x) denote the total amount of textual content (in bytes) that x contributed to all articles co-edited (up to the moment of doing the analysis) and let t _i (x) denote the total amount of textual content that editor x contributed to the articles belonging to a specific category c _i .¹

Now, lets introduce the following denotation: p _i (x) = t _i (x)/t(x) and interpret it as representing x’s interest in category c _i . Henceforth, we will use a shorter denotation p _i for p _i (x) whenever x is understood from the context.

In this paper we also use some measures of diversity based on standard deviation. Standard deviation of numerical attribute X taking n values: X ₁,…,X _n is defined as where \(\text {avg}(X)=\frac {1}{n}{\sum }_{i=1}^{n}X_{i}\) is an arithmetic mean of attribute X. Standard deviation sd(X) measures how much (on average) an attribute varies around its arithmetic mean. Thus it can be seen as a natural measure of variability or dispersion of a numerical attribute. In our experiments, we will use standard deviations of the number of editors’ contributions in bytes and standard deviations of period lengths between the first and the last contributions (tenure, that may represent the experience of an editor).

We prepared datasets and preprocessed them to compute several attributes for editors and teams of editors. We also utilised information available in Wikipedia to attach a quality label to each article that is treated as the decision attribute in our analyses.

We first run statistical tools to preliminarily statistically analyse the relationship between diversity and other attributes and quality.

Next, for a deeper analysis, we additionally applied some more sophisticated statistical machine learning tools such as logistic regression, decision trees, random forests. The methods are described in more detail in Section 4.

In such models it is possible to objectively measure in various ways how strongly any attribute is correlated with the decision attribute (quality in our case). In particular, in some experiments we split our preprocessed data into training and test sets, built prediction models based on them and used the models to predict work quality based on the studied attributes.

The higher performance of such prediction, the stronger statistical relationship between the attributes (including diversity) and the decision attribute (quality). In addition we applied some other statistical tools to objectively measure how strongly diversity (and other attributes) affects the quality of work.

The activity of editors and their teams on Wikipedia are recorded and stored in Wikipedia dumps that are publicly and easily available. Wikipedia shares the latest dumps under the following URL address: https://​dumps.​wikimedia.​org/​.

To run our experimental study, we prepared ourselves two separate datasets by processing dumps of the Polish and German Wikipedia from March and September of 2015, respectively. We will refer to these two datasets as wiki-pl and wiki-de, respectively.²

We run all the experiments presented in this article on two different language versions of Wikipedia for greater reliability of the results.

Since the results (presented later on in this article) on both datasets wiki-pl and wiki-de are generally compatible, we assume that the choice of these particular language versions of Wikipedia does not significantly affect our general findings presented in this article.

We collected data about editors of articles, articles and editions of articles made by authors.

The datasets used in our article are summarised in the Table 1.

In this article we use the information assigned explicitly to Wikipedia articles to determine their quality. More precisely, the quality of articles is modelled based on the information given by Wikipedia community members, who evaluate articles as good and/or featured based on the following criteria explicitly defined by the Wikipedia community itself: We utilise the above two kinds of labels given by the Wikipedia community regarding the articles as the basic means of determining their quality.

Based on this, in our considerations and experiments we distinguish five quality classes of editors denoted as N, G ∪ F, G ∩ F, G, F as presented in Table 2.

Our definition of versatility (topical diversity) of an editor presented in Section 2.1 assumes the existence of topical categories.

In our model we utilised 12 main content categories for Polish Wikipedia and 8 main content categories for German Wikipedia accessible from the front page. We preprocessed our datasets wiki-pl and wiki-de to identify main categories assigned to considered articles. Table 4 presents the main categories for both datasets (they differ for language versions of Wikipedia).

Wikipedia articles are usually not directly tagged with any of these high-level categories. Only the most specific categories are assigned to the articles by Wikipedia community. Those are subcategories of more general categories, creating a structure of a directed graph. The nodes in this graph are categories and there is a directed arc from one vertex to another in such a graph if and only if the corresponding category is a subcategory of another. Starting from any node in the graph representing a lowest-level category directly assigned to a particular article, we employed a standard BFS (breadth-first search Cormen et al. 2001) graph search algorithm to assign top-level categories to this article. More precisely, the article was assigned all top-level categories reachible from the lowest-level category of this article by the BFS algorithm.

If the article was mapped to more than one category, the contribution size was split equally among them. Articles that couldn’t be classified were excluded from the dataset, as well as users whose production consisted of such articles exclusively. Also, only editions of the pages in the primary namespace were taken into account (that is “proper” articles and not, for example, discussion pages), because only these pages are evaluated with regard to their quality.

One of the main objects of study in our work is an editor of a Wikipedia article, i.e. a person who contributed to the work on the article. Size of editors contribution to article was counted as a sum of his editions to this article over all revisions in dump. We considered edition as size of one change by editor in one article. Wikipedia does not provide exact size of an edit and contribution so we had to count it. Every revision in a dump has an information about current size of text in bytes. Size of one edition made by editor was counted as difference of size between the last and current revision. Contribution is the amount of bytes changed by one editor in article. Datasets components presented in Table 5 was used to compute data for this part of the experiments. Anyone who made any change to article was treated as an editor even if it was just minor change like adding a comma, because criteria of good and featured articles include style and well-written.

Additionally, we gathered data about the editors’ gender that is available in our datasets. Not all editors share this information on their Wikipedia profiles, but enough information was available to perform some basic analysis.

The sampling frame (observation interval) was restricted to contributors who made at least one edition during the Wikipedia project lifetime.

In Section 6 we will present a series of experiments that will concern whole teams of editors.

To perform team-oriented experiments, some further data processing was needed. We used wiki-pl and wiki-de datasets again. For each dataset, we precomputed three components of the data shown in Table 6, and integrated them into one dataset.

Logistic regression (Hosmer et al. 2013) belongs to the most popular and successful classification models. Let C be the value of a binary class variable and x ₁,…,x _p be a set of numerical attributes. In logistic regression it is assumed that the posterior probability (the conditional probability of the class given attributes) is of the form where β ₀,β ₁,…,β _p are parameters. The parameters are usually estimated from data using a maximum likelihood method. The significance of the attribute in the model can be assessed using the Wald statistic (often denoted as z statistic). The z statistic for the j-th attribute is defined as the standardized estimator of the coefficient corresponding to the j-th attribute. The statistic is used to test the hypothesis β _j =0. For a large sample size, z statistic follows standard Gaussian distribution and thus the p-value of the statistic can be calculated. The smaller the p-value the more significant is the variable. Tables 10, 11, 16 and 17 contain values of z-statistics and the corresponding p-values from logistic regression.

Decision tree (Breiman et al. 1984) is an example of a non-linear classification model. In each node of the tree the data is split into two subsets according to the outcome of the test. The splits are performed in order to decrease the homogenity of the class distribution. The most popular measures of the class homogenity are: entropy and Gini index. The paths from root to leaves represent classification rules. The final decision is made based on the majority class in the given leaf. The optimal size of the tree can be determined using e.g. cost-complexity criterion. Figures 3 and 4 show the trees built based on our data.

Random Forest (Liaw and Wiener 2002) consists of many single decision trees. Each of the tree is built based on boostrap sample (sample drawn with replacement from the original data). In addition, the Random Forest use a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features. Random Forests corrects for a single decision trees’ habit of overfitting to their training data. The final classification rule is based on the majority voting of the trees. Random Forest can be used to assess the importances of the attributes. The two basic measures are described in Section 6.6.

Initially, we make some basic analysis of versatility level across all quality classes that were defined in Section 3.3. The results in a form of box-plots are presented in Figs. 1 and 2, for wiki-pl and wiki-de datasets, respectively. The results are encouraging for further analyses, since one can see on these figures that higher quality editors (classes F, G, F ∩ G, F ∪ G) tend to be more versatile (i.e. of higher entropy of interests): in terms of median, lower and higher quartiles (horizontal bars of the boxes). The situation is very similar for both analysed language versions of data.

The precise values of median versatility across the quality classes are presented in Table 7, first column. These results preliminarily indicate that diversity of interest seems to be correlated with quality.

Indeed, our analysis confirmed that productivity is another editor’s attribute that seems to be related to work quality. In the second column of Table 7 one can see that median productivity even stronger discriminates the quality classes than versatility.

As we will see in next experiments this seemingly superiority is misleading, since versatility better explains quality than productivity when more sophisticated statistical tools are applied.

Nonetheless, we selected productivity as the main “competitor” for versatility in the next experiments.

Some works study how gender relates to work outcome (e.g. Vasileseu et al. 2015). We split the editor data by their gender (Table 8, columns 1 and 2) for those editors who declared it and did an exploratory analysis concerning whether the relationship between versatility and quality differs between the both genders.

Comparison of editor versatility across all quality classes for both genders is presented in Table 8, columns 3 and 4, and indicates that there is no observable relationship between the gender and versatility across all classes. Versatility of women and men is more or less similar for each quality group and both examined datasets. We excluded gender factor from further experiments in this article.

The remaining experiments aim at studying on how accurately it is possible to predict the quality group of the editor based on his/her versatility and productivity. Such approach of applying prediction models makes it possible to make a deeper analysis of the relationship between the examined attributes and quality. In general, higher prediction performance in such models may be interpreted as a statistical signal of dependence. In addition, various statistics concerning the prediction models such as: p-values, z-values, estimated coefficients, give more precise information about the relationship between the attributes that was not available in simple exploratory analysis.

For clarity, in this series of experiments, we set the prediction problem as the binary classification problem, where class variable C = 1 corresponds to G ∪F (“high quality”) editors, whereas class C = 0 corresponds to the remaining ones. The classes are heavily unbalanced (there are many fewer cases in class C = 1, see Tables 3 and 9), which makes the problem challenging from the statistical point of view.

We use two classification models, which are among the most popular ones in the machine learning community: logistic regression (Hosmer et al. 2013) and decision trees (Breiman et al. 1984). These two classifiers represent different groups of methods: the former one is an example of linear classifier, as the hyperplane separating the classes is a linear function of attributes. The latter model is a non-linear classifier. We use implementations available in the R (R Core Team 2013): the glm{stats} function for logistic regression and rpart{rpart} function for the tree (Therneau et al. 2015) (CART trees). Since the training data is unbalanced, we assign larger weights to articles from rare class when fitting a model.

To assess the predictive power of the considered methods, we randomly split our data into training (50 % observations) and testing (50 % observations) sets. The training data is used to build models (i.e. to fit logistic regression and build a decision tree), whereas testing data is used to check the prediction accuracy.

The basic statistics presented in Table 7 indicate that productivity may be a factor as important as diversity in the context of work quality. To further examine this effect, we decided to use logistic regression to check how versatility, productivity and interactions between these two factors influence the quality group of editors which describes quality of his work and articles he edits. Tables 10 and 11 show the statistics from the fitted models: estimated coefficients (1st column), their standard errors (2nd column), Wald statistics, also called as “z-value” (3rd column) and the corresponding p-values (4th column). Small p-values indicate that the considered variables are statistically significant (i.e. are not “noisy”). High z-values indicate stronger relationship. Finally, the value of estimated coefficients objectively indicate how much one attribute affects another in the model. The sign of a coefficient represents the fact whether the influence is positive or negative. Row, corresponding to the most significant variable (we exclude the intercept coefficient as it is irrelevant in such consideration), is printed in bold. It is demonstrated that for both datasets versatility is the most significant variable, representing the strongest relation and it positively affects the quality in the model.

In this section we remind some basic machine learning concepts that we use to further analyse our results in prediction experiments presented in the next Sections.

Let C _i be the value of the class variable for the i-th observation in the test data and \(\hat {C}_{i}\) be the predicted class for the i-th observation in the test data. Let’s define the following quantities:

The letters ‘T’,‘F’, ‘P’, ‘N’ denote “true”, “false”, “positive” and “negative”, respectively. So, for example TP (“true positive”) is the number of cases correctly (“truly”) assigned to class C = 1 (“positive”), etc. We use the following basic evaluation measures on the testing data:

Precision measures how many articles are correctly predicted as C = 1 among those predicted as C = 1. Recall indicates how many articles are correctly predicted as C = 1 among all articles with label C = 1. In addition we calculate the F-measure, which is a harmonic mean of Recall and Precision. The higher the measures, the better the performance of the considered model. For highly unbalanced classes (such as in our data) it is hard to build a model that optimises both Precision and Recall. Notice that for highly unbalanced classes the simple accuracy rate is misleading since it is easy to achieve its high value by always predicting the major class. Thus why F-measure as the aggregation of both is usually applied in such situations. The above indices are commonly used in machine learning and information retrieval.

In this section we present the performance measurements for editor quality prediction experiments (Table 12). Precision is higher for the prediction based on the logistic regression model (about 87−88 %) than for the tree model (74−75 %), whereas recall is higher for the tree model (26−29 % for the tree model and 17 % for the logistic regression model). The tree model outperforms the logistic regression model with respect to F-measure, for both datasets.

Importantly, the results for wiki-pl and wiki-de datasets are quite similar which again supports the evidence that the choice of particular language version of Wikipedia does not affect the analysis.

Figures 3 and 4 present classification trees obtained in these experiments for wiki-pl and wiki-de datasets, respectively. They support the earlier observation that versatility is positively related to the high quality work or featured articles (class C = 1) as well as productivity.

In short, the presented prediction performance can be viewed as quite high if one takes into account high disproportion between class cardinalities. This can be interpreted as a signal of positive dependence between versatility and quality.

To complete the prediction-based analyses we present additional information about the prediction models that we obtained in our experiments.

This information is in the graphical form of Lift curves and ROC curves of the prediction models (Brown and Davis 2006). A lift curve graphically shows the precision (on the y-axis) with respect to the percentage of articles highest rated by the given model (on the x-axis). The precision in lift curve is calculated for the rule which assigns class C = 1 for articles highest rated by the given model (i.e. those with highest posterior probabilities).

ROC curve is another classical visualization tool, which shows the True Positive Rate (Recall) with respect to the False Positive Rate defined as follows: (fraction of articles incorrectly predicted as C = 1 among those with label C = 0).

In general, the higher the area below the ROC curve, the better is the prediction model, with (ideal) maximum being 100 % of the area of the square.

Observe, that the baseline (random) assignment of articles to classes would result in a horizontal line on Lift chart and a diagonal line on ROC chart. The random assignment can be seen as a baseline (Figs. 5, 6, 7 and 8).

Figures 5 and 7 show Lift curves for the users of the wiki-pl and wiki-de dataset. Figures 6 and 8 show the corresponding ROC curves. To further explain the Lift curve graphs, observe that Fig. 5 indicates that when we assign class C = 1 to, for example, 10 % of observations highest rated by our models, we achieve precision about 40 %. Similarly, when we assign class C = 1 to say 40 % of observations that are highest rated by our models, we achieve precision about 15 %, etc.

Interestingly, both classification models give similar results. The results are promising as for both datasets we achieve the accuracy value which is definitely above the baseline (random assignment).

This Section summarizes the experiments concerning single editors. We used two statistical models to verify how much diversity (versatility) influences the quality group of Wikipedia editors. In addition, we also tested whether the productivity is correlated with the quality of an editors’ group. It turns out that in both models, versatility is an important feature. In particular, versatility is the most significant variable according to the logistic model and it is also useful in the decision tree model. Analysis of an output from the logistic model indicates that versatility is positively correlated with the quality. Moreover, we tested the prediction performance of these two models. The values of the applied evaluation measures (Precision, Recall, F-measure) are very promising. They are much larger than for the baseline (random assignment of articles to classes describing quality). This is also confirmed by ROC and Lift curves. Both statistical models give comparable results and the conclusions are similar for both languages. The most important remark is that there is a strong dependence of versatility and work quality for editors.

In this section we introduce and compute several attributes for editors and teams that will be used in our statistical analyses.

In total, in this section we consider 10 team attributes that will serve as explanatory variables in our models and analyses. The attributes are presented in Table 13. In this table some attributes are based on standard deviation, i.e. they may be also viewed as representing team diversity measures. We present in bold all diversity-related attributes in the Table. For general description of diversity measures that we use, we refer the reader to Sections 2.1 and 2.2.

The order of the coming sections and experiments concerning teams is generally analogous to the one concerning editors with some necessary adaptations.

At the beginning of experiments we did an exploratory analysis of the mentioned team attributes (Table 13) and their relationship with work quality and found some of them as promising for discriminating quality classes. The results are presented in Tables 14 and 15 (where “sd” stands for “standard deviation”). Results for wiki-pl and wiki-de datasets signal similar relationships. For each dataset, we print in bold the group of all attributes that observably discriminate the normal (N) quality class versus higher quality classes. Out of 10 attributes only 5 in wiki-pl and 7 in wiki-de belong to this group. Interestingly, versatility does not belong to these groups, but other diversity-related attributes are highly present. As was with the case of editors, more sophisticated analysis will later demonstrate that team versatility actually is among the factors that are positively related with work quality.

More precisely, Tables 14 and 15 demonstrate that versatility, mean total productivity, standard deviation total productivity, mean tenure in Wikipedia and standard deviation tenure in Wikipedia seem to be indifferent between four group qualities. Versatility is just slightly higher for better quality articles than for normal ones. Total productivity of editors in teams seems to have no significant relationship with quality of articles. The most considerable differences are observed for the following attributes: mean productivity in article, standard deviation productivity in article, team size, mean tenure in article and standard deviation in article. It seems that productivity and tenure and their diversity have stronger relationship with quality, when measured in article than in the whole Wikipedia. It doesn’t matter how much work was done by editors in all articles but only productivity in particular one article has relationship with it’s quality. These results might indicate that “new” and “old” editors in an article through exchanging their experience create articles of better quality.

In this Section we fit a logistic regression model to the data using all 10 attributes described in Table 13 and G ∪ F as the target attribute.

Tables 16 and 17 show coefficient estimates, standard errors, Wald statistics (z-values) and their corresponding p-values for models.

Observed p-values demonstrate that almost all variables are statistically significant (assuming significance level 0.05), except mean total productivity for wiki-pl dataset and standard deviation tenure in Wikipedia for wiki-de dataset.

We highlight some more interesting observations in the tables by using bold print. Interestingly, in both datasets versatility has the absolutely highest positive coefficient of influence on quality, however it is statistically less significant than most of the other attributes. On the other hand, out of the three statistically strongest attributes (highest z-values) the majority (two) represent diversity-related attributes (standard deviations of productivity in article and of tenure or total productivity, depending on the dataset). In both datasets the remaining statistically strong (high z-value) attribute is team size, that is intuitionally obvious (large team likely improves the article).

In this Section we present experiments with prediction models concerning teams.

We used the aggregated data from the previous Section, split into training (50 % observations) and testing (50 % observations) datasets, and built logistic regression and Random Forest models (Liaw and Wiener 2002). Similarly as in case of editors, a response variable can take two values C = 0, which represents normal quality articles or C = 1, for both “higher” quality (G ∪F). In other words we want to predict the probability of being a G ∪F article produced by a team over the normal quality article.

As in the case of experiments for editors, we would like to verify how accurately it is possible to predict the quality of the article based on the features describing teams. Because the number of features is larger than for experiments with editors, instead of a single classification tree, we applied the Random Forest model, whose performance is usually superior to decision tree, and used the implementation available in the R package (RandomForest).

Table 18 shows evaluation measures for teams. The results indicate much lower prediction performance of logistic regression compared to the similar experiments concerning single editors. In general, Random Forest performs much better than logistic regression, the precision measure of Random Forest for both datasets is quite high here. It is larger for the wiki-pl data set (66.91 %) than for the wiki-de data set (58.93 %). Also other measures differ between the datasets. Thus, this experiment shows a different outcome than the corresponding experiment concerning single editors.

To further examine the prediction models we computed the Lift and ROC curves for our team quality prediction models (Figs. 9, 10, 11 and 12).

Figures 9 and 11 show Lift curves for teams of wiki-pl and wiki-de dataset. Figures 10 and 12 show the corresponding ROC curves. The results are significantly above the baseline. Note that, Random Forest outperforms logistic regression for both datasets. The ROC curves indicate that the Random Forest model performs better here than in the case of the experiments with single editors.

To additionally verify our hypothesis for teams, we assess the relevance of variables by using some variable importance measures available in the Random Forest model (Breiman 2001). The first measure (Imp1) is based on prediction error. Namely, for each tree, the prediction error on the out-of-bag portion of the data (data not used to build the model) is computed. Then the same is done after permuting the values of the given attribute (this makes the attribute irrelevant). The difference between the two are then averaged over all trees, and normalized by the standard deviation of the differences. The second measure (Imp2) pertains to average decrease of node homogeneity. Algorithms for constructing decision trees usually work top-down, by choosing an attribute at each step that best splits the set of observations. The quality of the split is measured using the decrease of node homogeneity, .e.g the difference between the class homogeneity in parent node and the child nodes. The class homogeneity is measured using entropy or Gini impurity measure. Large decrease indicates that the attribute is relevant. The average decrease of node homogeneity is taken over all splitting nodes and over all trees used to construct an ensemble classifier. Generally, the higher the value of the importance measures the stronger relationship with the predicted attribute (article quality).

Tables 19 and 20 show the results for wiki-pl and wiki-de dataset. For each dataset and importance measure we print in bold the attribute with the highest importance value.

All of the “winning” attributes in this analysis represent diversity-related attributes. Interestingly, for both datasets the winners are the same: versatility for the Imp1 measure and “standard deviation of productivity in article” for the second importance measure.

For both datasets, the diversity-related attributes like versatility and standard deviations are among the most significant variables according to either of the importance measures (Imp1, Imp2).

This result is especially significant, since we consider 10 attributes including such seemingly “strong” ones as “size of team” or tenure of editors. Diversity-based attributes turn out to be superior to them in this experiment.

This Section summarizes the experiments concerning teams of editors. Here our aim was to verify how different properties of teams (see Table 13), including diversity measures, influence the quality of articles. In this case we use two statistical models: logistic regression and random forest (more sophisticated ensemble of decision trees, tailored to the situation of larger number of attributes). The evaluation measures (Precision, Recall, F-measure) are again very promising. They are much larger than for the baseline (random assignment of articles to classes describing quality). Random forest outperforms the logistic regression significantly (this is clearly seen on ROC curves). As the performance of random forest was superior, we also calculated attribute importance measures based on random forest to check which attributes are useful for prediction of quality. It turns out that versatility is the most significant attribute according to the first measure. The experiments clearly indicate that diversity-related attributes of teams are strongly connected with the quality of the articles.