Performance Evaluation of Public-Private Partnership Projects from the Perspective of Efficiency, Economic, Effectiveness, and Equity: A Study of Residential Renovation Projects in China

Abstract

The performance evaluation index system was established on the basis of 4E (Efficiency, Economic, Effectiveness, Equity) theory to improve performance evaluation of Public-Private Partnership Projects for Residential Renovation (4P2R). Moreover, the index weighting model was demonstrated based on Ordered Weighted Averaging (OWA). Finally, the cloud model was used for performance evaluation, through the cloud uncertainty reasoning to determine the quantitative performance evaluation index value, the positive reverse cloud to calculate the qualitative performance evaluation index value, and the normal cloud to measure the project performance. The proof-of-concept application shows that the performance evaluation index system and evaluation model can better identify the deficiencies in the project, and provide targeted improvement measures for similar projects in the future.

1. Introduction

Local governments in China have been establishing Public-Private Partnership (PPP) projects owing to the increasing popularity and applicability of PPP model since September 2014. As of February 2018, the National PPP Integrated Information Platform Project Library of China has included 7477 PPP projects that have ~$1.8 trillion investment in total. With the ever-increasing demand for renovating old residential areas, Beijing, Harbin, and other regions in China have applied the PPP model to broaden financing channels and ease financial pressure. The main purpose of 4P2R is to renovate residential areas with long construction times, backward facilities, and imperfect property management, and solve the problems of “dirty, chaotic, and bad” neighborhoods [1]. By doing so, 4P2R can improve supporting facilities, achieve full coverage of property management, optimize the living environment, and enhance the residents’ quality of life and well-being [2,3].

Social capital has a significant role in PPP for the renovation of old residential areas (4P2R) through broadening the project financing channels, solving the project funding gap, and resolving the local government financial risks [4]. However, 4P2R has encountered problems such as slow progress, low public coordination, difficulty in implementing preferential policies, and low social responsibility. In addition, the above problems may cause quality risk, cost risk, and schedule risk in PPP projects. These risks may lead to issues such as quality defects, project funds waste, and delays in construction schedules [5]. In this regard, how to boost the investment benefits of 4P2R has attracted the attention of the academic community and industry [6]. Moreover, an effective performance evaluation approach can identify the problems, propose corresponding improvement measures, and provide a framework for follow-up projects [7]. Therefore, improving the performance evaluation system should be addressed to realize the full potential of 4P2R.

To date, few studies have been conducted to evaluate the performance of 4P2R. Although the Ministry of Finance defined the key points of PPP project performance evaluation, they have not detailed a specific outline, making it applicable to different types of PPP projects. On the other hand, several studies have been devoted to the performance evaluation of PPP projects regarding the early stage [8,9,10], the construction phase [11,12], and the later stage [13,14,15] of the project to lay the foundations for the performance evaluation of 4P2R. Analytic Hierarchy Process (AHP) [16], FANP-TOPSIS [17], Electre Tri-nC [18], and Grey Relational Analysis (GRA) [19] methods have also been applied for evaluating PPP projects’ performance. However, most of these methods use the scores that are subjectively determined by experts, relying on experts’ personal experience to calculate the weights. Moreover, these methods do not take into account the uncertainty of performance evaluation indicator values that lead to a biased evaluation.

In this study, we aim to establish the performance evaluation index system to achieve sustainable development of PPP projects. We first established an evaluation index system based on 4E theory from the perspective of efficiency, economy, effectiveness, and equity. Then, the OWA operator was used to calculate index weights objectively. Finally, we built a novel performance evaluation method by using the cloud model to overcome the existing methods’ inadequacy to determine index values.

2. Literature Review

2.1. The Performance Evaluation of PPP Projects

Existing studies focus on the performance evaluation of PPP projects regarding their different stages, i.e., early stage, construction phase and later stage.

Regarding projects in their early stage, Ismail et al. analyzed the key factors of (Value for Money) VFM as project operation, social benefit, and technical and environmental criteria using the principal component analysis method [8]. Cheung et al. analyzed the VFM of PPP projects on the basis of technological innovation, product performance, risk sharing, open bidding, and product level [9]. To study the influencing factors of VFM in PPP projects, Ismail used questionnaires and ranked the influencing factors through SPSS software (SPSS (SPSS Inc., Chicago, IL, USA)) [10]. The results suggest that competitive bidding and technical innovation of social capital are the two most important factors affecting VFM.

During the construction phase, Gohary et al. emphasized that effective communication can enhance the satisfaction of stakeholders and improve the efficiency of construction and operation, and achieve the project goals [11]. Moreover, Sanghi et al. believe that the government’s management level is a critical factor that determines the performance of PPP projects [12]. Therefore, in the construction phase of PPP projects, increasing the level of government management and promoting effective communication can improve the performance of PPP projects.

In the later stage of the project, Shen et al. developed the Sustainable Performance-based Assessment Model (SPbEM) to assess the sustainability of PPP projects by analyzing the investment distribution between the government and social capital [13]. It has been pointed out that each PPP project should look for key performance indicators in the contract and measure the life-cycle performance of the PPP project [14]. Others conducted a thorough review on the PPP-related normative literature, highlighting key success factors, the role of the public sector, selection of franchise rights, management of risks, issues relating to project cost and scheduling, and financing. Then, they proposed a dynamic conceptual scheme to assess the life cycle of PPP projects [15].

Existing studies suggest that the criteria for the performance evaluation studies concern various grounds that are by no means complete including economy, society, environment, efficiency, equity, effectiveness, and sustainability; they need to be considered in evaluating the performance of 4P2R.

2.2. 4E Theory

The 4E theory has been developed from 3E theory and the 3E system. The 3E theory that was proposed by Checkland [20] includes three aspects: economy, efficiency, and effectiveness. The economy measures its output; efficiency reflects the utilization of resources by the system; and effectiveness reflects the role of the lower-level system to the superior system in terms of output. The 3E theory is the proposed method of cognition and it can solve complex problems based on the soft system methodology from the systems perspective [21]. This method is used to analyze and understand the complex environment and problems faced by the system; consequently, logical and practical solutions can be developed [22]. At the same time, it can solve complex situations where traditional evaluation systems fail to be applied [23].

3E system is a project performance audit system based on 3Es—efficiency, economy, and effectiveness—as defined by the International Organization of Supreme Audit Institutions (INTOSAI) [24]. Efficiency measures the use of project resources and cost efficiencies, such as the project’s actual duration and qualified quality [25]. Economy focuses on the economics of the use of project resources and the economics of management, such as the use of project funds and the changes in the cost of project operations [26]. Effectiveness describes the completion of the expected goals from the perspective of project output and measures the relationship between actual benefits and expected benefits, such as product quality and efficiency, and increased employment [27].

With the development of performance evaluation theory, many scholars argued that the 3E evaluation system neglect fairness and democracy [28]. They built a 4E system to evaluate project performance by incorporating the fourth E: equity [29]. Equity can reflect the impact of a project on participants and society, including economic, social, and cultural fairness [30]. On the other hand, 4P2R makes use of a substantial investment, an extended period of construction and operation, and a large number of stakeholders, and it is easy to be affected by the internal and external environment. Therefore, we can comprehensively measure the performance level of 4P2R through the 4E theory.

3. Performance Evaluation Index System from the Perspective of 4E

An index system can be particularly important for evaluating the performance of 4P2R. It is vital to use indicators to reflect the intrinsic characteristics of 4P2R. However, due to the short implementation time of PPP projects, China has not yet established a unified index system for the measurement of 4P2R’s performance.

In the index system for evaluating the 4P2R’s performance in its construction stage, the principles of systematic, scientific, comparable as well as quantifiable, should be followed. We should also consider factors such as the construction purpose and characteristics of 4P2R, the diversity and particularity of stakeholders, and the degree of difficulty in obtaining relevant data. This study follows the research route of “Literature Survey → Field Investigation → Expert Inquiry → Indicators Selection” to determine the index system. First, we determined the objectives and indicators of performance evaluation through literature survey and field investigation. Then, each expert in the expert group independently evaluated the indicator through a questionnaire. To ensure the rationality and reliability of the indicator selection, we used a face-to-face interview approach to communicate with each expert after we collected data to confirm the evaluation results. They can provide professional advice for determining the indicator system by their rich experiences in the field of PPP projects. The expert group with a total of 14 members consisted of four scholars who also have many years of consulting experiences in PPP projects, five government staff and five project managers, all of whom have at least five years of comprehensive experience in PPP projects. Finally, we built an index system using four indicators that include efficiency indicators, equity indicators, economic indicators, and effectiveness index to evaluate the 4P2R performance (

Table 1. Index system for evaluating 4P2R performance.

First-Level Indicators	Two-Level Indicators
Efficiency indicators (F1)	Qualified rate of project quality (V11)
	The level of project safety management (V12)
	Project schedule payment rate (V13)
	Progress completion rate (V14)
Equity indicators (F2)	The rationality of risk sharing (V21)
	The rationality of concession period (V22)
	The rationality of charge mechanism (V23)
	The satisfaction level of public (V24)
	The satisfaction level of government (V25)
	The satisfaction level of social capital (V26)
Economic indicators (F3)	Tax contribution rate (V31)
	Asset-liability ratio (V32)
	Investment profit rate (V33)
	Employment contribution rate (V34)
	Operating cost change rate (V35)
	Actual investment change rate (V36)
	Degree of regional development (V37)
Effectiveness index (F4)	Housing repair	Energy-saving transformation (V41)
		Internal repair (V42)
		External repair (V43)
		Waterproofing and maintenance of roofs (V44)
	Living environment	Community greening (V45)
		Illegal buildings (V46)
		Sanitation facilities (V47)
	Infrastructure	The maintenance and renovation of water, electricity, heating, and gas (V48)
		The renovation of parking lots (V49)
		The reform of fire protection facilities (V410)
		The upgrading of residential roads (V411)
		The reconstruction of rain and sewage discharge facilities (V412)
	Operating maintenance	The maintenance of public facilities and equipment (V413)
		The maintenance of roads and ancillary facilities (V414)
		The quality of greening management service (V415)
		The quality of environmental health services (V416)
		The quality of security services (V417)

).

The 4P2R performance evaluation indicators are divided into four categories: efficiency, equity, economy, and effectiveness. By fully characterizing the intrinsic characteristics of 4P2R, the performance level of 4P2R can be measured comprehensively:

(1)

Efficiency indicators (F1): Efficiency indicators are mainly applied for reflecting the 4P2R performance in the transformation process regarding quality, schedule, cost, and safety management. In general, the objectives of project management include quality, progress, cost, and safety. This article selects indicators to measure the completion efficiency of the four major goals, such as the qualified rate of project quality (V11), the level of project safety management (V12), project schedule payment rate (V13), and progress completion rate (V14). The indicator (V11) is used to measure the qualified rate of 4P2R. The indicator (V12) is used to assess the safety management of 4P2R during the construction phase. The indicator (V13) is used to evaluate the timely payment of the 4P2R in the construction stage. The indicator (V14) is used to measure the completion degree of the project schedule.

Among them, qualified rate of project quality = Qualified number of single projects/Total number of single projects; project schedule payment rate = total progress paid/total amount of project investment completed; progress completion rate = actual construction time/the entire project schedule.

(2)

Equity indicators (F2): From the perspective of stakeholders, this type of indicators mainly analyzes project risk sharing, concession period, charge mechanism, as well as the satisfaction of stakeholders from the project. This paper selects six indicators: the rationality of risk sharing, concession period, and charge mechanism, in addition to the satisfaction of the public, government, and social capital. ① The rationality of risk sharing (V21) measures the rationality of risk sharing between the government and social capital in 4P2R; ② The rationality of concession period (V22) measures the rationality of the 4P2R concession period; ③ The rationality of charge mechanism (V23) measures the rationality of charging mechanism during the concession period after completing the renovation of the old residential areas; ④ Public, government and social capital satisfaction are used to assess the satisfaction of relevant stakeholders from 4P2R. Among them, the satisfaction level of the public (V24) and government (V25) focus on measuring the overall satisfaction of government departments and community owners on the effect of old residential areas renovation. Social capital satisfaction (V26) focuses on measuring the implementation of preferential government policies, and the cooperation of public with the project.

(3)

Economic indicators (F3): Economic indicators mainly reflect the cost and profit status of 4P2R and its contribution to the regional economic development. In this work, we chose seven indicators such as tax contribution rate, asset–liability ratio, investment profit rate, employment contribution rate, operating cost change rate, actual investment change rate, and degree of regional development. ① Tax contribution rate (V31) is used to measure the contribution of 4P2R to the tax revenue in the region; ② Asset–liability ratio (V32) is used to measure the debt level of 4P2R; ③ Investment profit rate (V33) is used to measure the profitability of 4P2R; ④ Employment contribution rate (V34) mainly measures the contribution rate of 4P2R to the public employment; ⑤ Operating cost change rate (V35) is mainly used to measure the cost change of 4P2R during the operation period, and operating cost change rate = (actual annual operating cost—planned annual operating cost)/planned annual operating cost; ⑥ Actual investment change rate (V36) is used to measure the deviation between the completed settlement price and the planned investment amount; ⑦ Degree of regional development (V37) is used to measure the degree of regional development contribution of 4P2R.

Among them, tax contribution rate = project tax amount/total regional tax revenue; asset–liability ratio = total project liabilities/total project assets; investment profit rate = the annual net income of the project/the project’s total investment; employment contribution rate = the number of employees increased by the project/the number of employees in the region where the project is located; actual investment change rate = (completed settlement price—planned investment)/planned investment.

(4)

Effectiveness index (F4): F4 indicators focus on assessing the renovation and remediation of 4P2R. During index selection, the research team conducts on-site investigations, consults literature, as well as academics, government staff, and project management personnel in related fields. The renovation and rectification of old buildings can be summarized as housing repair, living environment, infrastructure and operating maintenance. ① To achieve the integrity of the house and the improvement of its functionality, the main outline of house repair is to demolish, renovate and maintain the house. We based our evaluation on the following indicators: energy-saving transformation (V41), internal repair (V42), external repair (V43), and waterproofing and maintenance of roofs (V44); ② The main purpose of environmental renovation is to provide a livable environment for the old community residents. The main content is to clean up illegal buildings and structures, upgrade the district greening and improve sanitation facilities. Community greening (V45), illegal buildings (V46) and sanitation facilities (V47) have been included in this category; ③ Infrastructure is an engineering facility that provides services for residents. The transformation of infrastructure in old communities involves water, electricity, heating, gas, roads, parking lots and firefighting facilities. In this paper, six indexes were selected: the maintenance and renovation of water, electricity, heating and gas (V48), the renovation of parking lot (V49), the reform of fire protection facilities (V410), the upgrading of residential roads (V411) and the reconstruction of rain and sewage discharge facilities (V412); ④ The operation and maintenance management is critical for ensuring the normal operations of old communities, involving the maintenance of public facilities and equipment, roads and ancillary facilities, environmental health, greening, and security. Here, we focused on public facilities and equipment (V413), the maintenance of roads and ancillary facilities (V414), the quality of greening management service (V415), the quality of environmental health services (V416), and the quality of security services (V417). The indicators and their function are shown in

Table 2. The indicators and their function.

Indicator	Function
V41	To measure the energy-saving transformation of 4P2R
V42	To assess the internal repair of 4P2R
V43	To evaluate the external repair of 4P2R
V44	To measure the repair of roofing and waterproof
V45	To evaluate the community greening of 4P2R
V46	To measure the dismantling of illegal buildings in 4P2R
V47	To assess the arrangement of sanitation facilities
V48	To assess the maintenance and renovation of water, electricity, heating and gas
V49	To evaluate the renovation of parking lot
V410	To measure the reform of fire protection facilities
V411	To assess the upgrading of residential roads
V412	To evaluate the reconstruction of rain and sewage discharge facilities
V413	To assess the maintenance of public facilities and equipment
V414	To evaluate the maintenance of roads and ancillary facilities
V415	To measure the quality of greening management service
V416	To assess the quality of environmental health services
V417	To evaluate the quality of security services

.

4. Research Methods

This section explains the research methods used in the paper, including OWA operator and the cloud model. First, we analyzed the applicability of the methods. Subsequently, we evaluated the steps of OWA operator weighting method. Finally, we discussed the concept and application of the cloud model and three cloud generators.

4.1. Analysis of Research Methods’ Applicability

4.1.1. OWA Operator Weighting Method

The scientific rationale for index weight calculation is critical to ensure the accuracy of the evaluation results [31]. The weighting methods include objective weighting methods and subjective weighting methods. (1) Objective empowering methods, such as entropy weighting method [32] determine the index weight according to the degree of dispersion of the index. The degree of dispersion is proportional to the weight of the index. When the degree of dispersion is 0, the index does not work in the evaluation, and it is challenging to address the subjective preferences of the decision maker [33]; (2) Subjective weighting methods, such as Analytic Hierarchy Process (AHP) [34], could indicate decision makers’ subjective preferences. However, when the number of decision data made by decision makers is less, the maximum or minimum value has a greater negative impact on weighting and can reduce the objective determination of the weight value [35]. In other words, if the traditional weighting methods are used to assign weights to the indicators, deviations can easily affect the accuracy of the evaluation results. Therefore, numerous indicators should be taken into account to come up with an improved performance evaluation index system.

To solve the problems mentioned above, we used the OWA operator to weight the indicators. The calculation process of OWA operator weighting method is relatively simple and can better reflect the preference of decision makers [36]. The OWA operator weighting method has been applied successfully in the field of credit rating [37], strategic decision [38] and safety evaluation [39]. It is best suited for solving uncertain decision-making problems and can reduce the negative impact of the maximum and minimum values of decision data on the outcome [40].

4.1.2. Performance Evaluation Method Based on Cloud Model

Performance indicators are grouped into indicators that are qualitative and those that are quantitative. The values of qualitative indexes are determined by the expert group only once [41]. Affected by personal knowledge and experience, the scores given by experts have different degrees of randomness and ambiguity [42]. As a result, there are some deviations in the performance index values. These deviations can considerably affect the accuracy of performance evaluation results. The performance evaluation index system constructed in this paper includes many qualitative performance evaluation indicators. Therefore, scientific determination of the qualitative index values is key to ensuring the accuracy of project performance evaluation results.

We used the cloud model to measure index values and evaluate project performance. This model [43] converts between qualitative descriptions and quantitative ones; it has been applied to the fields of land evaluation [44], urban traffic flow forecasting [45], and smart distribution grid evaluation [46]. (1) To determine the value of qualitative indicators, we determined the scores of the performance indicators after repeated evaluations of the expert group through the combination the cloud model with the Delphi method. In the above process, we used the cloud diagram to visually control the convergence speed, quality, and direction of the expert scores, thereby ensuring the reliability of the scores; (2) In determining the value of qualitative indicators, we used the cloud uncertainty reasoning model to determine the value of quantitative indicators and maintain the consistency between the qualitative and quantitative indicators.

4.2. OWA Operator Weighting Method

Ordered Weighted Averaging (OWA) is an information aggregation method introduced by Yager [47]. It uses a particular method to reorder input elements based on attribute values and assigns different aggregation weights according to the location of data [48]. Many scholars have conducted in-depth research on the extended algorithm of OWA operator. The combination number OWA operator weighting model adopted in this paper was based on the classical OWA operator theory and used the combination number as the weighted vector to complete the data aggregation [49]. This method weights the decision data by the number of combinations and calculates each indicator’s weight. The specific steps are listed below:

(1)

First, we sorted the decision data $\left(e_1,e_2,\cdots ,e_n\right)$ of the indicators from the largest to the smallest, then numbered the data $\left(g_{0 },g_1,\cdots ,g_{n-1}\right)$ beginning from 0, and obtained the result $\left(g_{0 },g_1,\cdots ,g_{n-1}\right)$ that turned out to be $g_{0 }\ge g_1\ge \cdots \ge g_k\ge \cdots \ge g_{n-1}$.

(2)

The data $g_k$ was weighted by the number of combinations, and the equation ${\displaystyle \sum }_{\mathrm{i}=0}^{\mathrm{n}-1}{C}_{n-1}^i=2^{n-1}$ was obtained based on the number of combinations. The weight vector was determined by:

$${\beta }_{\tau +1}=\frac{C_{n-1}^{\tau }}{{{\displaystyle \sum }}_{\mathrm{i}=0}^{\mathrm{n}-1}{C}_{n-1}^i}=\frac{C_{n-1}^{\tau }}{2^{n-1}},k=0,1,2,\cdots ,n-1.$$

(1)

(3)

We weighted the decision data according to the weight vector ${\beta }_{\tau +1}$, and the absolute weights of the indicators were determined by:

$${\forall }_i={\displaystyle \sum }_{i=1}^n{\mathsf{\beta }}_{\mathsf{\tau }}{g}_{\tau }, \mathsf{\tau }ϵ\left[1,\mathrm{n}\right].$$

(2)

(4)

The relative weights of the calculated indicators were determined by:

$${\forall }_i^{*}=\frac{{\forall }_i}{{{\displaystyle \sum }}_{i=1}^m{\forall }_i}, i=1,2,\cdots , t.$$

(3)

Through the above steps, the weight vectors of the indicators of each level were obtained, thereby completing the weighting of the index system.

4.3. Cloud Model

Based on the theoretical knowledge of fuzzy mathematics and probability theory, Li et al. [50] proposed the cloud model. As a transformation model that is uncertain, this cloud model deals with both qualitative concepts and quantitative descriptions [51]. It can represent the process from qualitative concept to quantitative description and from quantitative description to qualitative conception [52]. The cloud model can prevent information loss due to the transformation processes and increase objectivity in comparison to other methods [53].

The cloud model uses three digital eigenvalues to represent language values: Expectation ($\mathrm{Ex}$) and Entropy ($\mathrm{En}$) as well as Hyper entropy ($\mathrm{He}$) [54]. Located in the center of gravity in the domain space, expectation ($\mathrm{Ex}$) is a sample value typically reflecting a qualitative concept. Entropy ($\mathrm{En}$) is used to describe the degree of uncertainty for a qualitative concept. This qualitative concept’s fuzziness and randomness are used to determine the Entropy (En). Hyper entropy ($\mathrm{He}$) is used to describe the degree of entropy’s uncertainty, which can be regarded as entropy’s entropy to reflect the thickness of clouds.

The cloud generation algorithm can be implemented in software and solidified into hardware. As a cloud generator, it comprises the positive cloud generator, reverse cloud generator, single condition, and single rule generator.

(1)

The positive cloud generator [55]

It is a process transferred from qualitative concept to quantitative representation. The model is used to generate cloud droplets from the cloud’s digital eigenvalues, as shown in Figure 1.

The algorithm for positive cloud generator is shown below:

Algorithm 1

Input: The positive cloud generator’s digital eigenvalues $\left(\mathrm{Ex},\mathrm{En},\mathrm{and}\mathrm{He}\right)$ are used as inputs. N is set as the number of cloud drops. Output: N and its corresponding membership degree. Step 1: The normal random number (${\mathrm{En}}^{*}$) is generated by taking $\mathrm{En}$ as expected value and $\mathrm{He}$ as mean square error, where ${\mathrm{En}}^{*}=N\left(\mathrm{En},\mathrm{He}\right)$; Step 2: The normal random number ($\mathrm{x}$) is produced by taking $\mathrm{Ex}$ as the expected value and ${\mathrm{En}}^{*}$ as mean square error, where $\mathrm{x}=N\left(\mathrm{Ex},{\mathrm{En}}^{*}\right)$; Step 3: Calculate $\mathrm{y}=\exp \left[-\frac{{\left(x_i-Ex\right)}^2}{2{\left({\mathrm{En}}^{*}\right)}^2}\right]$; Let (x,y) be a cloud drop, where x represents a specific quantitative value of the qualitative concept $\mathrm{C}$ and $\mathrm{y}$ represents the membership degree of the qualitative concept; Step 4: Repeat Step 1 to Step 3, the operation stops after generating the set of n cloud droplets.

(2)

The reverse cloud generator

As a transformation model, the reverse cloud generator implements quantitative values to qualitative concepts [56]. The precise data can be converted into qualitative concepts as digital eigenvalues $\left(\mathrm{Ex},\mathrm{En},\mathrm{He}\right)$, as shown in Figure 2.

The algorithm for reverse cloud generator is shown below:

Algorithm 2

Input: Enter n cloud drops, e.g., sample data $x_i\left(i=1,2,\cdots ,n\right)$ Output: The cloud model’s digital eigenvalues $\left(\mathrm{Ex},\mathrm{En},\mathrm{He}\right)$ Step 1: Take $\widehat{E}x=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{x}_i$ as the estimate of $\mathrm{Ex}$; Step 2: After removing the point of $\mathrm{y}$>0.999, the number of remaining cloud droplets is $\mathrm{m}$. Step 3: ${\mathrm{En}}^{*}$ is calculated by ${\mathrm{En}}^{*}=\frac{\left|x-\widehat{E}x\right|}{\sqrt{-2lny}}$; Step 4: Obtain $\widehat{E}n$ from $\widehat{E}n=\frac{1}{m}{\displaystyle \sum }_{i=1}^{m}E{n}_i^{*}$, where $\widehat{E}n$ is an estimate of $\mathrm{En}$; Step 5: Obtain $\widehat{H}e$ from $\widehat{H}e=\sqrt{\frac{1}{m}{\displaystyle \sum }_{i=1}^m{\left(E{n}_i^{*}-\widehat{E}n\right)}^2}$, where $\widehat{H}e$ is an estimate of $\mathrm{He}$;

(3)

The single condition and single rule generator

Based on the positive cloud generator, we constructed X-conditioned cloud generator ${\mathrm{CG}}_A$ and Y-conditioned cloud generator ${\mathrm{CG}}_B$, respectively, based on the single condition and single rule, and merged them to form a single condition with a single rule generator [57] in Figure 3.

The algorithm for the single condition and single rule generator is shown below:

Algorithm 3

Input: The quantitative data $x_A$, the digital eigenvalues $\left({\mathrm{Ex}}_A,{\mathrm{En}}_A,{\mathrm{He}}_A\right)$ of X-conditioned cloud generator ${\mathrm{CG}}_A$, the digital eigenvalues $\left({\mathrm{Ex}}_B,{\mathrm{En}}_B,{\mathrm{He}}_B\right)$ of Y-conditioned cloud generator ${\mathrm{CG}}_B$. Output: Under the condition that the degree of certainty is $\mathrm{u}$, the qualitative concept value of Y-conditioned cloud generator ${\mathrm{CG}}_B$ is $x_B$. Step 1: The normal random number $E{n}_A^{*}$ is generated by taking ${\mathrm{En}}_A$ as expected value and ${\mathrm{He}}_A$ as mean square error; Step 2: The degree of certainty is calculated by $\mathrm{u}=\exp \left[-\frac{{\left(x_A-E{x}_A\right)}^2}{2{\left({\mathrm{En}}_A^{*}\right)}^2}\right]$; Step 3: The normal random number ${\mathrm{En}}_B^{*}$ is generated by taking ${\mathrm{En}}_B$ as expected value and ${\mathrm{He}}_B$ as mean square error; Step 4: If the X-conditioned cloud generator ${\mathrm{CG}}_A$ is activated and then it rises in the direction of $x_A\le E{x}_A$, Y-conditioned cloud generator is activated, and the certainty value is $x_B=E{x}_B-E{n}_B^{*}\times \sqrt{-2lnu}$. Similarly, if the X-conditioned cloud generator ${\mathrm{CG}}_A$ is activated and then it drops in the direction of $x_A>E{x}_A$, Y-conditioned cloud generator is activated and the certainty value is $x_B=E{x}_B+E{n}_B^{*}\times \sqrt{-2lnu}$.

5. Performance Evaluation Based on OWA Operator and Cloud Model

5.1. Research Process

The research process on performance evaluation of 4P2R is shown in Figure 4. The detailed description of the research process is given from Section 5.2 to Section 5.5.

5.2. Determination of the Performance Evaluation Index System of 4P2R

We established the Performance Evaluation Index System including four first-level evaluation indicators and 34 second-level evaluation indicators. The indicator set $\mathrm{F}$ is a set of first-level performance evaluation indicators ${\mathrm{F}}_i$, where $\mathrm{F}=\left\{F_{1 },F_2,\cdots ,F_i\right\}$; the indicator set $V_i\left(i\in \left[1,t\right]\right)$ is a set of second-level performance evaluation indicators $V_{ij}\left(j\in \left[1,n_i\right]\right)$, where $V_i=\left(\begin{array}{ccc}{V}_{i1}& \begin{array}{cc}{V}_{i2}& \begin{array}{cc}{V}_{i3}& \cdots \end{array}\end{array}& V_{i{n}_i}\end{array}\right)$.

5.3. Using OWA Operator Weighting Method to Produce the Index Weights

The OWA operator weighting method was employed for determining the performance evaluation indicators weights. First, the OWA operator was used to obtain the weight value ${\forall }_i^{*}$ of the first-level indicator. Then, we calculated the weight values $Z_e$ of the secondary indicators where the second-level indicators belong. Finally, we used $\mathrm{W}={\forall }_i^{*}·Z_e$ to calculate the final weight of the secondary indicator.

5.4. Index Value Measurement Based on Cloud Uncertainty Reasoning and Positive Reverse Cloud

We used the cloud uncertainty reasoning to determine the quantitative performance evaluation index value, and the positive reverse cloud to calculate the qualitative performance evaluation index value.

5.4.1. Quantitative Index Measurement Based on Cloud Uncertainty Reasoning

The steps through which we established quantitative performance index value are outlined below:

(1)

Determining the comment set of the indicator and digital eigenvalues

Through cloud uncertainty reasoning model, quantitative index values were measured using the single condition and single rule generator. We used the comment $A_k$ to describe the quantitative indicators $V_{ij}$, in which the comment was $\mathrm{A}=\left(\begin{array}{ccc}{A}_1& \begin{array}{cc}{A}_2& \begin{array}{cc}{A}_3& \cdots \end{array}\end{array}& A_k\end{array}\right)$, and the digital eigenvalues of the comment $A_k$ was $AS{C}_{ij}^{A_k}\left({\mathrm{Ex}}_{ij}^{A_k},{\mathrm{En}}_{ij}^{A_k},{\mathrm{He}}_{ij}^{A_k}\right)$. If the quantitative index value was $x_{ij\left(s\right)}$, after calculation by the positive cloud generator, the degree of membership of the index value $x_{ij\left(s\right)}$ was 4, which corresponds to the comment $A_k$.

(2)

Determining the comment set of the score and digital eigenvalues

After calculating the degree of membership of index value $A_k$, we determined the comment set of score and the corresponding digital eigenvalues to determine the corresponding percentage value of the index value $A_k$. We used the comment $B_d$ to describe the percentage value of the index $V_{ij}$, in which the comment was $\mathrm{B}=\left(\begin{array}{ccc}{B}_1& \begin{array}{cc}{B}_2& \begin{array}{cc}{B}_3& \cdots \end{array}\end{array}& B_d\end{array}\right)$, and the digital eigenvalues of the comment $A_k$ was $S{C}^{B_d}\left(E{x}^{B_d},E{n}^{B_d},H{e}^{B_d}\right)$.

(3)

Measuring the value of the evaluation index

The cloud uncertainty reasoning was used to calculate the measurement value $r_{ij\left(s\right)}$ of the evaluation index. Among them, the form of uncertainty reasoning was; if $x_{ij\left(s\right)}\in A_k$, then $r_{ij\left(s\right)}\in B_d$, since there is a one-to-one relationship between $k$ and $d$.

5.4.2. Qualitative Index Measure Based on Positive Reverse Cloud

The steps by which we obtained qualitative performance index value are described below:

(1)

Qualitative numerical conversion to qualitative concept based on reverse cloud

First, we formed a group of experts; then, evaluation criteria were formulated, and a set of precise data was formed after the experts gave their scores; finally, we used the reverse cloud generator to transform the set of precise data into digital eigenvalues $\left(\mathrm{E}\widehat{x},\mathrm{E}\widehat{n},\mathrm{H}\widehat{e}\right)$ that can be used to represent qualitative concepts.

(2)

Transforming from qualitative concept to quantitative representation based on positive cloud

Based on the digital eigenvalues $\left(\mathrm{E}\widehat{x},\mathrm{E}\widehat{n},\mathrm{H}\widehat{e}\right)$, we ran the positive cloud generator to generate cloud droplets and formed cloud maps. Then, the mapping of qualitative concepts to quantitative representations was completed. In combination with the cloud map, we visually analyzed the gravity center of discourse domain, the range of the number domain, the condensation degree of cloud droplet, and the clouds’ thickness.

(3)

Determining the qualitative index value based on the Delphi method

To reasonably determine the qualitative index value, we used the cloud model to visually control the convergence speed, quality, and direction of the expert experience; and we used the Delphi method to collect feedback for the expert group on the score’s details. After repeated evaluations, it was unanimously approved by the expert group to obtain the Expectation $\left({\mathrm{Ex}}^{*}\right)$ and Entropy $\left({\mathrm{En}}^{*}\right)$ as well as Hyper entropy $\left({\mathrm{He}}^{*}\right)$ of the qualitative index value, and the Expectation $\left({\mathrm{Ex}}^{*}\right)$ was used as the quantitative value of the index.

5.5. Project Performance Evaluation Based on Normal Cloud Model

(1)

Building the domain of project performance indexes and the domain of comments

The index domain of project performance evaluation was described as $V_i=\left(\begin{array}{ccc}{V}_{i1}& \begin{array}{cc}{V}_{i2}& \begin{array}{cc}{V}_{i3}& \cdots \end{array}\end{array}& V_{i{n}_i}\end{array}\right)$, and the comment domain was $\mathrm{D}=\left(\begin{array}{ccc}{D}_1& \begin{array}{cc}{D}_2& \begin{array}{cc}{D}_3& \cdots \end{array}\end{array}& D_{\theta }\end{array}\right)$.

(2)

Determining standard normal cloud for project performance evaluation indicators

Set $x_{ij}^1$ and $x_{ij}^2$ respectively as the indicator’s upper and lower boundary values to the level of comment $V_{i{n}_i}$. Note the indicator is $V_{ij}$. Then, we measured the qualitative concept of the index $V_{ij}$ corresponding to the comment level $D_{\theta }$ of performance evaluation, which can be characterized by the normal cloud. Among them,

$$E{x}_{ij}=\left(x_{ij}^1+x_{ij}^2\right)/2.$$

(4)

Because the boundary value is a transition value from one level to another level, it is a fuzzy boundary and should belong to the corresponding two levels at the same time, that is, the degrees of membership of the two levels are equal. Thus, we can see:

$$\exp \left[-\frac{{\left(x_{ij}^1-x_{ij}^2\right)}^2}{8{\left(E{n}_{ij}\right)}^2}\right]\approx 0.5,$$

(5)

where

$$E{n}_{ij}=\frac{x_{ij}^1-x_{ij}^2}{2.355}.$$

(6)

$H{e}_{ij}$ is used to describe the entropy’s uncertainty, which is entropy’s entropy to reflect clouds’ thickness. In general, $H{e}_{ij}$ should be adjusted according to the uncertainty of the evaluation index. In general, $H{e}_{ij}$ should be adjusted according to the uncertainty of $E{n}_{ij}$, and its value has a certain relationship with $E{n}_{ij}$:$H{e}_{ij}=mE{n}_{ij}$. The range of $m$ is from 0.05 to 0.2 and needs to be adjusted according to the actual situation [58].

(3)

Performance evaluation of a single index

According to each index value of the item to be evaluated, a positive cloud generator was used to determine the membership matrix $\widehat{\partial }$ corresponding to each index for each level, e.g., $\widehat{\partial }={\left({\partial }_{ij}\right)}_{n\times m}$. For the improvement of the credibility of the assessment, the forward cloud generator should be run N times, and the mean value of the membership degree should be calculated as the evaluation result using ${\partial }_{ij}={\displaystyle \sum }_{k=1}^n{\partial }_{ij}^k/n$.

(4)

Comprehensive performance evaluation of the project

According to the membership matrix $\widehat{\partial }$ and index weights $W$ that have been obtained, we used Formula (7) to perform fuzzy transformation and obtained project performance:

$$\widehat{\phi }=W·\widehat{\partial }=\left({\phi }_1,{\phi }_2,\cdots ,{\phi }_{\theta }\right).$$

(7)

In Equation (7), ${\phi }_j\left(j=1,2,\cdots ,\theta \right)$ refers to the degree of membership of the j-th comment. Then, based on the principle of the membership’s maximum degree, ${\widehat{\phi }}_{\theta max}$ is the result of project performance evaluation.

6. Case Study

Zibo in Shandong, China aims to carry out a total of 241 residential renovation projects. The project involves a total of 174,000 households and a total area of 13.477 million square meters. The main renovations are housing repair, living environment, infrastructure, and operating maintenance. Among them, the ZD district involves 53 residential areas with a total investment of about ~$103 million. The PPP model was adopted to renovate the residential areas in two batches. The first batch of 28 projects was completed by the end of 2016, and the second batch of 25 project plans are due to be completed by the end of 2018. We studied the representative residential area E in the ZD district as the proof-of-concept demonstration of our performance evaluation index system and performance evaluation model.

6.1. Using OWA Operator Weighting Method to Determine the Index Weights

(1)

Determining the evaluation criteria

We set the importance of performance evaluation indicators to range from 0 to 10, the importance of evaluation levels as (low, low, general, high, and high), and the corresponding score range as ([0, 2], (2, 4], (4, 6], (6, 8], (8, 10]).

(2)

Scoring the importance of indicators

First, six experts were invited to form an expert group, including government department staff and university scholars. The expert group scored the first-level indicators according to the evaluation criteria. For the sake of simplicity, all scores were taken as an integer multiple of 0.5, e.g., 3.5, 5, and 7.5 (

Table 3. Score the importance of the first level index.

Indicator	Expert (1)	Expert (2)	Expert (3)	Expert (4)	Expert (5)	Expert (6)
F1	5.5	6	5	6	5	6.5
F2	8	7.5	8	8	7	7
F3	6	5	5.5	5	5	4.5
F4	8.5	9.5	9	9.5	9	9.5

).

(3)

Calculating the weight vector

First, we sorted the decision data of the indicators from the largest to the smallest and took indicator F1 as an example to obtain $g_k$ = (6.5, 6, 6, 5.5, 5, 5). Since the number of experts $n$ was 6, the weight vector was calculated according to Equation (1):

$$\mathsf{\beta }=\left(0.03125,0.15625,0.3125,0.3125,0.15625,0.03125\right).$$

(4)

Calculating the absolute weights and relative weights

We weighted the decision data according to the weight vector. The absolute weights of the indicators were as follows:

$${\forall }_1^{*}={\displaystyle \sum }_{i=1}^n{\beta }_j{g}_j=\left(0.03125,0.15625,0.3125,0.3125,0.15625,0.03125\right)·\left[\begin{array}{c}6.5\\ 6\\ \begin{array}{c}6\\ 5.5\\ \begin{array}{c}5\\ 5\end{array}\end{array}\end{array}\right]=5.359.$$

In the same way, we obtained: ${\forall }_2^{*}$ = 7.656, ${\forall }_3^{*}$=4.938, ${\forall }_4^{*}$ = 9.234.

According to Formula (3), we calculated the first-level index weight value: $\forall$ = (0.1971 0.2816 0.1816 0.3397).

The weighting method for second-level indicators was the same as described above, and the results were provided in

Table 4. The values and weights of second-level indicators.

Indicator	V11	V12	V13	V14	V21	V22
Indicator Value	85.98	88.03	67.85	74.34	78.5	69.35
Weight	0.0511	0.0490	0.0484	0.0486	0.0468	0.0495
Indicator	V23	V24	V25	V26	V31	V32
Indicator Value	72.57	76.35	82.56	72.99	62.05	72.4
Weight	0.0478	0.053	0.0417	0.0428	0.0162	0.0316
Indicator	V33	V34	V35	V36	V37	V41
Indicator Value	61.17	83.27	42.48	62.36	71.1	71.25
Weight	0.0173	0.0302	0.0265	0.0291	0.0307	0.0193
Indicator	V42	V43	V44	V45	V46	V47
Indicator Value	64.12	78.05	46.58	77.12	70.66	73.13
Weight	0.0182	0.0181	0.0197	0.0203	0.0191	0.0198
Indicator	V48	V49	V410	V411	V412	V413
Indicator Value	61.29	48.24	81.85	75.1	71.35	63.57
Weight	0.0204	0.0194	0.0201	0.0192	0.0211	0.0189
Indicator	V414	V415	V416	V417
Indicator Value	70.93	74.14	80.24	70.25
Weight	0.0235	0.0192	0.0190	0.0244

.

6.2. Project Performance Evaluation Based on Cloud Model

6.2.1. Quantitative Index Measure Based on Cloud Uncertainty Reasoning

(1)

Determining the comment set of the indicator and digital eigenvalues

We determined the comment set and the cloud model’s digital eigenvalues based on the project’s actual situation. Taking the indicator of “Qualified Rate of Project Quality (QRPQ)” as an example, the natural property value of this indicator for the old residential area E was determined to be 0.94. We used the comment to describe the percentage value of the index, where the comment was “Very High, High, Middle, Low, Very Low” and the digital eigenvalues of the comment was “(0.95, 0.0083, 0.001), (0.9, 0.0083, 0.001), (0.85, 0.0083, 0.001), (0.8, 0.0083, 0.001), (0.75, 0.0083, 0.001)”. The cloud chart is shown in Figure 5.

(2)

Determining the comment set of score and digital eigenvalues

For ensuring consistency between index level and score, the comment level of the quantitative index should have a positive correlation with the score level. That is, the review level increases with the score. We used the comment to describe the percentage value of the index, where the comment was “Very High, High, Center, Low, Very Low” and the digital eigenvalues of the comment was “(90, 3.33, 0.02), (70, 3.33, 0.02), (50, 3.33, 0.02), (30, 3.33,0.02), (10, 3.33, 0.02)”. The cloud uncertainty reasoning was used to calculate the measurement value of the evaluation index. As an example, we showed the second-level index QRPQ to establish the following inference rules:

• If QRPQ is “Very High”, the score is “Very High”;
• If QRPQ is “High”, the score is “High”;
• If QRPQ is “Middle”, the score is “Center”;
• If QRPQ is “Low”, the score is “Low”;
• If QRPQ is “Very Low”, the score is “Very Low.”

According to the relevant steps of the single condition single rule generator, the score of QRPQ is 85.98 through cloud uncertainty inference. Similarly, the quantitative scores of other quantitative indicators can be obtained, as shown in Table 4.

6.2.2. Qualitative Index Measure Based on Positive Reverse Cloud

(1)

Construction of the comment sets of qualitative indicators

In this paper, the expert evaluating method was used to determine the value of qualitative indicators. The comment of indicator was “Very High, High, Center, Low, Very Low” and the corresponding score range was “[0 20), [21 40), [41 60), [61–80), [81 100]”. A group of experts was formed to evaluate the qualitative indicators, including two government department staff, three university academics, and five experienced management personnel in construction, operation, and maintenance.

(2)

Determining the value of qualitative indicator value

Taking the indicator of “the level of project safety management” as an example, we calculated the reverse cloud generator and positive cloud generator, respectively, based on the scores of 10 experts. Through three rounds of expert scoring, three digital eigenvalues of the indicator (88.03, 1.57, 0.14) were obtained, and the expected value (88.03) was taken as the measured value of “the level of project safety management.”

The process was performed as follows: ① In the first round, due to the influence of personal experience and knowledge, experts had different views of “the level of project safety management” and needed to meet a second time. The specific performance of the first round was that the expert scores were relatively scattered, the digital eigenvalues of $\mathrm{En}$ and $\mathrm{He}$ were larger, and the cloud chart showed a fog shape, as shown in Figure 6a; ② After summarizing the scores, we gave feedback to the experts and invited the experts to conduct the second round of scoring. The digital eigenvalues showed that the $\mathrm{En}$ and $\mathrm{He}$ decreased, and the cloud began to be condensed from the fog to the standard normal cloud, as shown in Figure 6b; ③ After feedback was provided, the experts scored in the third round, and the entropy and Hyper entropy further decreased. Consequently, the cohesion of the cloud diagram was significantly enhanced. Figure 6c shows that the standard normal cloud representing the expert score was gradually formed.

6.3. Project Performance Evaluation Based on Normal Cloud

(1)

Building the comment domain and digital eigenvalues

The comment domain of its performance level was determined as “Very Low, Low, Middle, High, Very High”, while its standard normal cloud digital feature was “(10, 8.49, 2.01), (30, 8.49, 2.01), (50, 8.49, 2.01), (70, 8.49, 2.01), (90, 8.49, 2.01)”. The cloud chart is shown in Figure 7.

(2)

Performance evaluation of the project

According to each index value of the item to be evaluated, a positive cloud generator is used to determine the membership matrix corresponding to each index for each level. We repeatedly ran the forward cloud generator 2000 times to improve the credibility of the assessment. After calculation, first, we obtained the performance level of the first-level indicators of the project (Table 4). Then, we obtained the overall performance level of the project as “2.02 × 10⁵, 0.0139, 0.1185, 0.6734, 0.2630”, corresponding to the performance levels “Very Low, Low, Middle, High, Very High”, as shown in

Table 5. The results of project performance evaluation.

Indicator	Very Low	Low	Middle	High	Very High	Performance Level	Changing Trends
F	2.02 × 105	0.0139	0.1185	0.6734	0.2630	High	Very High
F1	2.07 × 1019	1.85 × 1011	6.43 × 106	0.0087	0.0457	Very High	High
F2	1.34 × 1012	1.37 × 106	0.0068	0.2176	0.0758	High	Very High
F3	1.76 × 105	0.0090	0.0436	0.1098	0.0286	High	Middle
F4	2.60 × 106	0.0049	0.0620	0.2426	0.0548	High	Middle

.

(3)

Analysis of the evaluation results

Table 5 shows that the project’s performance evaluation result is $\widehat{\phi }$ = 0.6734. The result shows that the overall performance level is “High” and the performance level of the project is on an upward trend, i.e., it tends to be “Very High.” ① The performance evaluation result of the first-level indicator F1 is ${\phi }_{F1}$ = 0.0457, and the performance level is “Very High”; ② The evaluation result of indicator F2 is ${\phi }_{F2}$ = 0.2176, and the performance rating is “High,” which shows an upward trend, i.e., it tends to be “Very High”; ③ The evaluation results of indicators F3 and F4 are, respectively, ${\phi }_{F3}$ = 0.1098 and ${\phi }_{F4}$ = 0.2426. Thus, the overall performance level is “High” and the performance level of the project is on a downward trend, i.e., it tends to be “Middle.”

According to the above evaluation, the project’s overall performance level is “High.” In the first-level indicators, the level of the efficiency index F1 is “Very High,” the level of performance of the “fairness” index F2, the “economic” index F3, and the “effectiveness” index F4 are all “High.” However, the level of performance of the “economic” index F3, and the “effectiveness” index F4 are on a downward trend, i.e., it tends to be “Middle.” After the analysis of the second-level indicators, its performance level “Operating cost change rate” in the “economic” index is “Low,” and the performance level of the second-level indicator “waterproofing and maintenance of roofs” and “the renovation of parking lots” are “Low.”

By identifying and analyzing the problems in the project, we propose the following improvement measures: follow-up projects should strengthen cost management in energy consumption, material procurement, and equipment maintenance, and control non-essential cost expenditures. In order to improve the roof waterproofing quality after the completion of the project, the follow-up projects should pay attention to the supervision of the construction process and material quality, and build a suitable quality supervision and management system. Relevant departments should rationally plan parking lots and strengthen parking management to avoid limited parking.

7. Conclusions

The performance evaluation of 4P2R can identify the shortcomings in the project, and provide means of analysis for the causes of the problems and related improvement measures. Our key conclusions can be stated as follows:

(1)

We built a performance evaluation index system for 4P2R based on the 4E (Efficiency, Economic, Effectiveness, Equity) theory.

(2)

The performance evaluation model was established based on OWA and the cloud models. We used OWA to analyze the evaluation index’s weight. To reasonably determine the index value, we used a positive reverse cloud to determine the qualitative index value and cloud uncertainty reasoning to obtain the quantitative index value. We also used the normal cloud to evaluate the project performance.

(3)

The case study showed that the overall performance level of the project was determined as “High”, whereas the performance level of “operating cost change rate”, “waterproof maintenance of roofs”, and “the renovation of parking lots” were “Low”. Our work shows that the index system is suitable for the performance evaluation of 4P2R. The performance evaluation model can reasonably determine the indicator weights and index values and comprehensively measure the project performance. It can identify the deficiencies of past projects, propose improvement measures, and guide similar projects in the future.

(4)

The performance evaluation index system and performance evaluation model can improve the performance evaluation of 4P2R and potentially be applied to a great extent. It can provide reference for the government to formulate performance evaluation indicators for different types of PPP projects, so as to identify the problems of PPP projects and improve the comprehensive benefits of PPP projects. Because the performance evaluation model integrates an OWA operator and cloud model, the calculation process may be complicated. However, it can be programmed with MATLAB software (MATLAB 7.0 (7.0.0.19920 (R14), Math Works, Natick, MA, USA)) that has strong operability. Concurrently, introducing sustainability as an essential game-changer in the performance evaluation of 4P2R proves itself to be a significant research direction, which is worth studying further.