Corruption risk in contracting markets: a network science perspective

Our analytic framework measures corruption risk at a transaction level using data from public procurement contracts. We collected data on all public procurement contracts published in Tenders Electronic Daily¹ (TED), the official journal of public procurement contracts of the European Union, from 2008 to 2016. Both calls for tenders and the announcement of their award are published in TED. EU law requires that any public procurement contract exceeding an estimated value of 5.2 million Euro for works and 135 thousand Euro for services and supplies must be published on TED. TED estimates that over 400 billion Euro of total contract value is published in its pages every year, accounting for nearly 3% of EU GDP. Though the high thresholds exclude a significant number of contracts from our data, using only data from TED maximizes the international comparability of our analyses.

Our final dataset consists of 4,098,771 contracts awarded in 2008–2016. We consider 26 member states of the EU, excluding Luxembourg because of the relatively small number of contracts awarded there and Croatia because it only joined the EU in 2013. We also exclude contracts awarded by EU institutions (the Commission, Parliament, Council) as we aim to compare countries. The data are available for download at https://​zenodo.​org/​record/​3537986#.​XcmMPkX0mgk.

We processed the dataset to deduplicate the identities of each contract’s issuing buyer (i.e., public institution such as a ministry or city hall) and supplying winner (i.e., a private firm). Given that our analytic approach requires an accurate map of the interactions between issuers and winners, we developed a pipeline to maximize the accuracy of our deduplication, following the approach of Christen [56]. For each country, we preprocessed the text data for each entity, used machine learning to select both optimal string similarity measures and blocking methods, and selected a clustering threshold maximizing accuracy on a manually labeled subsample using the Dedupe computer software [57]. To give an example, the number of unique suppliers of contracts in France decreased from 364,125 to 200,584. For a more detailed summary of the procedure, see [58].

Besides the issuing buyer and winning supplier of each contract, we were able to extract several additional pieces of information, including the year the contract was awarded, the Common Procurement Vocabulary (CPV) code—a EU-wide taxonomy of procurement contracting goods and services [59], and the number of bidders competing for the contract. We use the latter information to score each contract for corruption risk.

We quantify the corruption risk of procurement contracts using a binary indicator tracking whether the contract attracted only a single bid in its competition. The use of single bidding as a signal for corruption risk or “undetected fraud” has recently been suggested by the European Court of Auditors [60]. Nevertheless, it is certainly not the case that any single instance of single bidding can be used as evidence for corrupt behavior. For instance, it may be the case that the government is purchasing a niche good or service with few suppliers on the market or under emergency circumstances—certainly it is more likely for such contracts to be awarded to a single bidder.

Two aspects of our data mitigate these limitations to the validity of single bidding as a signal of corruption risk. The first is that the high threshold of contract values insures both visibility and interest from the private sector. Secondly, the null models we create to benchmark our measures consider the CPV code of the contract, distinguishing between contracts for different categories of goods and services, such as medicine and furniture. In the case when data on the number of bidders are missing from a contract, we impute the country-level average single-bidding rate. This imputation strategy likely leads to an underestimate of corruption risk.

We plot the single-bidding rates with of each country over the 2008–2016 period in Fig. 1. We see significant variation, with single-bidding rates below 10% in some countries and over 30% in others. How do these measures of corruption risk correlate with the survey-based perception indicators mentioned in the previous section? In Fig. 2, we correlate the average single-bidding rates from 2008 to 2016 with Transparency International’s Corruption Perception Index (TI CPI) [32], the World Bank’s Control of Corruption Index (WB CoC) [33], Varieties of Democracy’s Corruption Index (V-Dem Corruption Index) [35], and the Quality of Government (QoG) Institute’s European Quality of Governance Index (European Qual. Gov. Index) [36] each measured in 2013 (this was the most recent year for which all five indicators were available). In each case, we find a significant correlation between country-level single-bidding rates and worse corruption or quality of government outcomes (Pearson correlation between .66 and .72, Spearman correlation between .74 and .76 in magnitude, all correlations significant at \(p< 0.05\)). We also find a significant correlation ( around − 0.7) between single-bidding rates and the recently developed Index of Public Integrity [61], which quantifies the extent to which the institutional framework of a country fosters integrity and blocks corruption.

Our data offer several advantages over the survey-based indicators. Surveys encode subjective perceptions about corruption that may be biased. For instance, past research by Olken found that perceived corruption exceeds actual corruption when ethnic diversity is high [11]. This is a significant obstacle when we would like to compare the rate of corruption across countries. Moreover, the way such indicators are calculated is often modified from year to year, again making comparison difficult. From our perspective, however, the most significant advantage of the single-bidder measure of corruption risk is that it is observed at a granular level; surveys are expensive and, with few exceptions [36], are not carried at the sub-national, e.g., regional level. We exploit that our indicator is available at the transaction level in the following section by introducing our network science-based analytic framework.

A common task of network analysis is to highlight the most central and active nodes. Some empirical networks have a group of densely connected actors at their centers, sometimes called cores [68]. Besides highlighting important actors, methods to detect network cores can be used to compare the degree of centralization in a network. In this subsection, we adapt a method known as k-shell decomposition to weighted bipartite networks in order to rank issuers and winners by their centrality in the network. We say that the most central actors form the core of the market. We measure and compare the relative sizes of the cores of each procurement market, interpreting a larger core as a signal of procurement market centralization. We find a strong correlation between market centralization and corruption risk.

In generic unweighted graphs, the concept of a coreness is defined iteratively [69]. To start, all nodes of degree 1 are assigned a core number of 1 and then removed from the graph. In the next step, all remaining nodes with degree 1 in the trimmed graph are also assigned core number 1 and then removed. This process repeats until all nodes have at least degree 2. Nodes with degree 2 are assigned core number 2, and they are the first nodes to be removed in the next iteration of removals. This process yields a hierarchical decomposition of the nodes in the network by their core number. The k-core of the graph refers to the subgraph consisting of nodes with core number at least k [70].

Our networks have two features that distinguish them from the graphs considered in this example. The first is that the edges include weights, encoding the volume of contracts between the focal issuer and winner. The second aspect is that the different node sets have different degree distributions. We tweak the concept of the core number and k-core in order to apply it to our networks.

Our method modifies the iterative procedure described above to consider the weighted degrees, defined as the count of contracts they are involved in as either issuers or winners, respectively, of nodes. Otherwise the method is identical. As the edge weights in our networks are integers (describing contracting volume), no further modification is necessary. This is a simplified version of the weighted k-shell decomposition of Garas et al. [71].

This procedure assigns each node in our network a weighted core number, measuring its centrality in the network. As we would like to partition the network into a core and periphery, we need to determine a cutoff for a node to be considered a member of the core. This cutoff should be a function of the size of the network as it would not be appropriate to use the same cutoff for networks of vastly different sizes. We also address the concern that issuers and winners have different degree distributions by setting two different cutoffs for core membership: one for issuers and one for winners. In the end, we choose to consider issuers as core issuers if their weighted core number exceeds the average degree of issuers. As winners tend to have lower average contract win counts, we only consider those winning twice the average degree as core winners.

We visualize the Hungarian core in Fig. 4, marking edges with above-market average single-bidding rates in red. The visualization highlights the importance of considering the volume of contracting between issuers and winners in our measure. Issuers and winners can have only a single neighbor and still be considered for core membership because they may have a high volume of contracts with that neighbor.

We observe that the core has a consistent size over time and remains densely connected in all years. We confirm that in general the weighted core subgraph has a much higher density in a summary statistics table included in Appendix. In general, between 20 and 60% of all contracts awarded in a given country are between core issuers and winners.

We now pose two questions about the relationship between network centralization and corruption risk measured by single bidding. First: What is the relationship between the degree of centralization of a market and its corruption risk. In Fig. 5, we plot the relationship between centralization, quantified by the share of all contracts between core issuers and winners, against corruption, measured by the overall single-bidding rate and the survey-based EQI measure. We find a significant positive relationship between procurement market centralization and corruption. In the political economy literature, there is an ongoing debate about the relationship between the centralization of government power and corruption. Our finding supports the side of the debate claiming that centralization facilitates corruption [72].

We are also interested in the distributional nature of corruption risk. Does centralization induce corruption by fostering corruption among core issuers and winners? We probe this question by comparing the rate of single bidding in the core against a null model that randomizes the distribution of single bidding. Specifically, we preserve network structure and the label of nodes as core or periphery, but randomly shuffle the single-bidding labels. We restrict the randomization by permuting single-bidder labels only across contracts with the same two-digit CPV code. This takes into account market-specific effects. For each market, we divide the observed rate of single bidding in its core \(SB_{\mathrm{core}}\) by the average rate of single bidding in its core over 1000 CPV-preserving randomizations \(\mu (SB_{\mathrm{core}}^{rand})\). We plot the results averaged over all years for each country in Fig. 6.

Unlike in the previous figure which indicated a strong relationship between market centralization and corruption risk, no clear pattern emerges. In some countries, corruption risk in the form of single bidding is overrepresented in the core. In other countries, it is underrepresented. For example, Czech Republic and Hungary have similar overall corruption risk scores, but in the former single bidding is more common between core issuers and winners, while in the latter it is more common between actors on the periphery. In the second panel of Fig. 6, we show that the size of the core does not have any straightforward relationship with the overrepresentation of corruption risk in the core.

Another dimension of potential heterogeneity in the distribution of corruption risk is its clustering. Previous research has found that corruption is significantly correlated in the network, with high corruption risk edges typically bunched together [14]. To demonstrate this idea, we plot the Hungarian procurement market in 2014 in Fig. 7. Coloring edges with above-market average rates of single bidding, a clear pattern emerges: The top left cluster of nodes has significantly higher rates of single bidding than other parts of the network. We propose to quantify this tendency, as well as the overall tendency of a procurement market to exhibit topological clustering using community detection. We first group the edges of the network into communities and measure the quality of this partition. We then calculate the variation of single-bidding rates across communities as a measure.

Community detection is a distinguished area of research interest in network science [73]. There are many methods to partition the nodes of networks into communities. As our object of interest, corruption risk, is an attribute of network edges, we should rather partition the edges of the network into communities. There are several well-studied methods used to cluster the edges of networks into so-called link communities. We adopt one approach based on line graphs [74, 75]. The line graph of a network is the network obtained if the edges if the original graph are considered as nodes and then connected if they share a node in the original graph. We transform each of our networks into the corresponding line graph, obtaining a new network for each one in which the nodes correspond to the original network’s edges or contract relationships. We can then apply any standard clustering algorithm on the new network to obtain an assignment of each issuer–winner edge to a community. We apply the Louvain method, a computationally fast and accurate method commonly used to detect communities in networks [76].

There is significant topological clustering in all countries and years, with modularity ranging from .55 to .80. For reference, the 2014 Hungarian market plotted in Fig. 7 has a modularity of .71. Given the partition of the edges of each network into communities, we can now quantify the extent to which single bidding varies across the different communities of a network.

The partition of edges in a network, denoting contracting relationships between issuers and winners naturally gives us a partition of all contracts awarded in a given market. We then calculate the coefficient of variation of single bidding across the clusters, defined as the standard deviation of the single-bidding rates across the contract clusters over the average. As clusters can have significantly different numbers, we actually calculate the weighted standard deviation \(\sigma ^{W}_{SB}\) and mean \(\mu ^{W}_{SB}\) of single bidding across clusters, defined as follows:andwhere C denotes the set of contract clusters, c is a specific cluster, and \(sb_{c}\) is the rate of single bidding in the cluster c. The weighted coefficient of variation, which in our context we refer to as the clustering of single bidding, is simply the ratio \(\sigma ^{W}_{SB}/\mu ^{W}_{SB}\).

As with our measure of centralization, we compare our observed clustering of single-bidding measure against a suitable null model. We again opt to randomize the single-bidder label on contracts within the 2-digit CPV classes to create a realistic yet randomized distribution to compare against the empirical data. Thousand times we shuffle the single-bidder label on all contracts and recalculate the clustering of single bidding for these randomized markets. We divide the observed clustering of single bidding by the average of the 1000 randomized clustering of single-bidding scores. We plot the result by country, averaged over 2008–2016, in Fig. 8. In every case, single bidding is non-trivially clustered within communities in the network. The magnitude of observed clustering ranges from 2 to over 8 times higher than expected in the sector-preserving randomization. We interpret this as significant evidence that corruption is not randomly distributed in the public sector.

The extent to which corruption risk clusters in the market has important policy implications. The greater the clustering of single bidding in a market, the more likely it is that investigations of the network neighbors of known corrupt actors will be successful. For example, returning to our visualization of the Hungarian market in 2014 (see Fig. 7), we do not only suggest that corruption risk is significantly higher in the northwestern community, but also that investigators of corruption in Hungary should consider this pattern of clustering in the future to shape their strategy.

We now compare our two measures of the distribution of corruption risk in markets. For simplicity, we consider only those countries with above average single-bidding rates in our data. We plot the centralization of corruption risk against its tendency to cluster for these countries in Fig. 9. We observe that although corruption risk is high in all countries, corruption risk within each country is distributed differently. Countries in the bottom right corner of the plot, such as Portugal and Italy, have higher corruption risk in the core of their markets and a weaker tendency for risk to cluster. Countries in the top and center of the plot such as Poland and Latvia have a neutral distribution of risk across the core and periphery of their markets, but overall risk has a strong tendency to cluster. Finally, countries such as Hungary, Slovakia, and Estonia have higher corruption risk in their peripheries and a moderate tendency for risk to cluster.

This emphasis on the distribution of corruption risk presents a novel way to think about corruption in a comparative manner. Though it may not be accurate to say that each country with high corruption risk is corrupt in its own way, these deviations suggest that corruption may be organized in different ways, likely reflecting political and economic constraints. The differences in topological structure (i.e., the degree of centralization, measured by the share of contracts in the core of the market, and the clustering of edges into communities measured by modularity) of the markets also indicate that interesting structures emerge in the mesoscopic level of these data: between the micro-level of individual transactions and the global picture of the entire market.