Evaluating AI-Based Code Segmentation for ABAP Programs in an Industrial Use Case

Many maintenance and evolution tasks in software engineering depend on the availability of logical code segments, especially AI- and data-driven approaches rely on segmented source code. In order to obtain logical segments a manual step is typically necessary. However, manual segmentation requires a basic understanding of the source code and delays the application of code analysis and refactoring tools. Automatic code segmentation provides an efficient way of extracting code snippets for further analysis to provide developers with actionable insights on software products and processes. Rule-based approaches rely on syntactic boundaries and lack the applicability of segmentation on multiple languages. In this article, we present our approach to learning logical code snippets using a BiLSTM neural network model. Driven by the requirements of an industrial use case, we train two models, one on the programming languages Go, Java, JavaScript, Python, PHP, and Ruby, the other model is additionally trained on ABAP (“Advanced Business Application Programming”) snippets. To evaluate the performance of the models, we use real-world samples used in the SAP applications of our industry partner. We also compare the predictions of the model, with an accuracy >98%, to the results of human experts for segmenting ABAP code to evaluate whether AI-based code segmentation is perceived as effective by practitioners in our industrial use case. The study shows that only 42–51% of the predicted ABAP code snippets match the manual segmentation of the experts.