Associating Working Memory Capacity and Code Change Ordering with Code Review Performance

Change-based code review is a software quality assurance technique that is widely used in practice. Therefore, better understanding what influences performance in code review and finding ways to improve it can have a large impact. In this study, we examine the association of working memory capacity and cognitive load to code review performance and we test the predictions of a recent theory regarding improved code review efficiency with certain code change part orders. We perform a confirmatory experiment with 50 participants, mostly professional software developers. The participants performed code reviews on one small and two larger code changes from an open source software system to which we had seeded additional defects. We measured their efficiency and effectiveness in defect detection, their working memory capacity, and several potential confounding factors. We find that there is a moderate association between working memory capacity and the effectiveness of finding delocalized defects, influenced by other factors, whereas the association with other defect types is almost non-existing. We also confirm that the effectiveness of reviews is significantly larger for small code changes. We observe a tendency that the order of presenting the code changes influences the efficiency of code review, but cannot conclude this reliably.