An Empirical Investigation on Business Analytics in Software and Systems Development Projects

Confirmation is the degree to which the actual use experience confirms their initial expectation (Bhattacherjee, 2001). It means that users’ actual use experience meets initial anticipation or expectation; which leads to their satisfaction. On the other hand, when actual use experience does not correspond or fall below the initial expectation, dissatisfaction occurs because of failing to achieve expectations. Bhattacherjee (2001) elaborated confirmation as the understanding of the anticipated benefits of IS use and dis-confirmation as performance lagging expectancy failure to achieve expectation. Therefore, we hypothesize:

Ratification of fulfilment comes after a meaningful experience, which is a sign of alignment between expectation and confirmation (Bhattacherjee, 2001; Gupta et al., 2020). A range of studies reported a relationship of among confirmation with, perceived usefulness and satisfaction (Bhattacherjee, 2001; Gupta et al., 2020; Huang, 2019; Shang & Wu, 2017; Susanto et al., 2016; Tam et al., 2020; Venkatesh et al., 2011). The same logic applies to BA use. BA users compare the experience of their BA use with prior expectations. If the users expectation is confirmed, then he/she will have a positive continuous intention use with the BA. We hypothesize that confirmation might have a direct connection with continuance intention. Therefore, we hypothesize the following:

Several studies suggested that satisfaction with technology influences users’ intention towards the use of technologies (Bawack & Ahmad, 2021; Cheng, 2020; Gupta et al., 2020). According to Bawack and Ahmad (2021), confirmation has a significant effect on technological compatibility that influence BA continuance. Similarly, we expect that if software and systems professionals find BA tools compatible with their work, they will intend to continue its use in their projects. Therefore, we hypothesize:

Perceived Usefulness has an influence on user intention across temporal stages of information systems use and continuance use intention (Bhattacherjee, 2001). Perceived usefulness has a direct impact on technology continuance intention (Bhattacherjee, 2001; Gupta et al., 2020). The BA users are more likely to perceive usefulness if prescriptive and predictive expectations are met. Therefore, we hypothesize:

Users are using the information systems when it matches their needs, values and expectations (Awa et al., 2017; Bawack & Ahmad, 2021). This is true in the case of analytics as well. According to Chen et al. (2015), organizations are more likely to use big data analytics when their existing values and work practices are compatible. Compatibility is an important driver in analytics acceptance and continuance (Grubljesic & Jaklic, 2015; Gupta et al., 2020). Building on the above reasoning, we hypothesize that:

We collected data from 186 respondents belonging to diverse domains. 41% of the respondents identified themselves as software developer. 27% of the respondents selected telecommunication as their job, whereas 15% belonged to the e-commerce industry. 6% of the respondents are working as IT-Consultants, and 4% are working in a government organization. The remaining 7% worked in various other sectors. 61% of the respondents have a minimum of three years of experience using BA in their work, and the remaining respondents have one to two years of BA experience. The majority of respondents are male (85%). Further demographics of the respondents are provided in Table 1.

To evaluate the measurement model, we focus on assessing the reliability and validity of the constructs and their corresponding items. We consider factor loadings of the items, composite reliability and Cronbach Alpha (α) scores of the constructs. For individual items to be reliable, the factor loading should be at least 0.50. Table 2 presents the summary of the reliability and validity score of the measurement model constructs and items. It is evident from Table 2 that the factor loading for each item is well above the threshold of 0.50. Likewise, for construct reliability, the composite reliability and Cronbach’s alpha scores should be greater than 0.70 (Hair Jr et al., 2016).

We used Average Variance Extracted (AVE) to validate if a construct sufficiently explains the variance of its items. The AVE score should be greater than 0.60 (Fornell & Larcker, 1981). A discriminant validity test is employed to ensure that unrelated constructs are not observed to be related to one another. For this purpose, we used the discriminant validity test of Fornell and Larcker (1981). The Cronbach’s alpha (α), composite reliability and AVE scores are meeting the minimum threshold criterion.

Table 3 presents the discriminant validity score based on Fornell & Larcker method (Fornell & Larcker, 1981). The method compares the square root of each Average Variance Extracted (AVE) in the diagonal with the correlation coefficients of the other constructs. The square root of AVE should be greater than the corresponding correlation coefficients. As in each column (see Table 3), the top value (diagonal value) is the highest among all other values signifying that these constructs are not related.

In the evaluation of the structural model, we perform statistical assessments regarding the predictability of the underlying model as well as the significance of the relationships between various variables of the model. The underlying model should exhibit no collinearity among the constructs to reflect the absence of bias (Fornell & Larcker, 1981; Hair Jr et al., 2016), i.e., the p-value of the predicates should be less than 5%. To gauge the predictive accuracy of the model, we consider the R² score. Threshold scores of 0.25, 0.50 and 0.75 are used to indicate weak, moderate and substantial predictive accuracy of the model. The resultant structural model is illustrated in Figure 2.

For ensuring that the significance of the results is not by chance, we run a bootstrap of our model by considering 5,000 sub-samples. The analysis of the R² values shows that our structural model can explain 45% variance on BA continuance intention, 69% variance on technological compatibility, and 59% variance on perceived usefulness. Analysis of the path coefficients (β) of our structural model reveals that confirmation (β=0.441, p=0.000) significantly affects BA continuance intention. Whereas perceived usefulness (β=0.169, p=0.062) and technological compatibility (β=0.105, p=0.327) do not affect BA continuance intention. We also identified that confirmation affects perceived usefulness (β=0.770, p=0.000) and technological compatibility (β=0.833, p=0.000). The findings of the structural model evaluation are summarized in Table 4.

Our study offers several implications for the existing literature. Firstly, we empirically investigate factors affecting BA continuance in Pakistan and extends the body of knowledge at an organizational level. It is important to investigate the factors that contribute to the BA continuance in software projects. According to Shah et al. (2019), there is a link between rising adoption and failure of BA. The existing literature mostly focused on BA adoption and not on its continuance intention (Daradkeh, 2019). Secondly, this study identified the important enablers of BA adoption based on ECM. We looked into various factors from previous literature related to ECM and adapted them into the BA context of our study. Our findings show that confirmation influence perceived usefulness and technological compatibility as well as confirmation directly influence BA continuous intention. These results are in line with existing studies, which found that such factors are important for BA (Bawack & Ahmad, 2021; Ahmad et al., 2021; Chen et al., 2015; Jaklic et al., 2018; Viaene & Van den Bunder, 2011). On the other hand, technological compatibility and perceived usefulness do not influence the BA continuance.

Finally, this study comes under big data research (Abbasi et al., 2016; Mandal, 2019), in terms of assessing the BA initiatives at the individual and team level in the context of software projects. Our study provides a holistic view of BA continuance in software projects and sheds light on the factors that help to elaborate BA role in the management of software projects (Dennehy et al., 2020). Our amended ECM research model highlighted determinants for BA continuance and can be suitable to use in future studies with a similar goal. The confirmation factor provides a better understanding of the motivation of BA continuance in software projects especially in developing countries such as Pakistan.

To maximize value similar to other technologies, BA continuance requires pervasive use and understanding of their expectations (Wixom et al., 2013). In this study, we reveal factors that influence the software and systems developers’ continuance use of BA in their projects. Our study offers suggestions for the smooth use and adoption of BA in software and system development companies. From a managerial standpoint, the ECM provides useful insights. It is the managers’ responsibility to pay attention to software and systems developers use behaviour to support BA continuance journey. Our study exhibits that perceived usefulness, technological compatibility and continuance intention are directly influenced by confirmation. The software and systems developers are more likely to perceive BA usefulness if their expectations are met. It may be prudent for managers to amplify BA users’ prescriptive and predictive expectations because of its direct impact on BA continuance. For instance, the companies can observe individuals temporal personalities and perceptions to adopt BA tools functions accordingly. In this study, confirmation has a significant contributor to BA continuance. This implies that BA tools should be chosen based on software and systems developers needs and should not be disruptive to the existing way of working. To ensure BA continuance and continue to reap its benefits the managers needs to pay more attention to employees actual use experience and initial expectations. This study also reveals that confirmation influence perceived usefulness. This can be also implied that BA vendors need to carefully analyse the users' expectations and prior experiences. Further, it is important to meet users’ expectations, be compatible with organizational values, and work practices. The individual users' expectations perspective is very important, as BA would affect their work performance. If the BA technology does not meet their expectations and disrupts their existing work, it might lead to undesirable outcomes. By considering users’ expectations the manager could pick the best-fit BA for their organization that is aligned with their values and work practices. The managers need to be aware of the temporal characteristics of business analytics tools as they have a range of complexities. By taking into consideration, the adopted model constructs the software and systems development companies can improve the environment for BA continuance. As a result, software and systems developers would be willing to use the BA in their work and their organizations attain its optimum benefits.

In this study, we have several limitations that could be considered as a future research direction. Our study respondents were general software practitioners from Pakistan using BA in their projects. The study participants’ recruitment was through direct contact of emails to different organizations. We intentionally selected the BA practitioners to obtain an appropriate data sample, as they had better understand BA and its use at work. This might lead to a biased perception regarding BA continuation intention. Nevertheless, such a diverse number of participants working in a range of projects and companies would be considered as a positive aspect of our study. We did not have any control over the type of participants in terms of their titles, job description and so on. Different organizational positions may have divergent views and varying knowledge about BA, factors that could affect the reliability of the results to some degree.

The respondents can be considered as individuals who are from different organizations and cannot represent the whole team or organization. Further, various stakeholders had different needs throughout the development project. For instance, a product owner may want to see different data than a software tester. This highlight the need for diverse information and investigation on stakeholder analysis. One BA tool is not a silver bullet and the consequence is that tool should support different views for product owners, managers, software testers etc. A multi-stakeholder experiences and perceptions investigation is essential. In the same vein, organizations and businesses need to understand a range of capabilities that are required for BA initiatives. These capabilities investigation might be circle around for instance people, processes, technology, organization and its culture. The temporal factor will bring more value to the BA research area if future research investigates it from various dimensions such as individual level, team level, and organization level. These dimensions are worthy of future research.

It is also worth noting that this study only focuses on the BA continuance in a single country i.e. Pakistan. Therefore, this study can be considered as BA use and continuance in companies in terms of Pakistani cultural aspects. This implies that Pakistani culture could have an influence on the outcomes of our study and may be considered as a limitation of this study. In future, it would be interesting to investigate such studies in various companies, business domains and countries. It would be worth investigating the BA under the existence of norms about time in organizations to explore differences in temporal norms among companies, workgroup types (such as managers, developers, designers etc.). This can be examined using Schriber's dimensions. A replication of this study in various contexts is also a direction for future investigations that will help to have more generalized results. The validation and modifying the research model presented in this research to include temporal complexities of the BA tools or analytics capabilities can also be an interesting adjustment worthy of pursuit.

The information and communication technology industry demand the current software engineering or information systems curriculum modification with an emphasis on both quantitative skills as well as software skills. In this regard, many universities around the globe offer analytics related courses in software engineering and information systems education. We believed that analysts using various analytics tools could be trained as part of software engineering or information systems curriculum. It will be good to consider the perspectives of graduates and obtain their opinions at different career stages. The result of such a study would help to obtain evidence of adjustment and offer opportunities for improvement of the current curriculums to meet the industrial demands.