TAnnotator: Towards Annotating Programming E-textbooks with Facts and Examples

Use of online resources has increased exponentially with widely available and accessible internet. Novice programmers often rely on various online knowledge sources such as MOOC courses, e-textbooks,¹ interactive and intelligent online tutors, blogs, crowd sourced platforms and so on to learn programming (Keuning et al., 2021; Venigalla & Chimalakonda, 2020; Vinaja, 2014). Many emerging technologies are being used in the classrooms to improve both teaching and learning (Almiyad et al., 2017; Dutta et al., 2022; Kim et al., 2016; Weber & Brusilovsky, 2016). Augmented reality, virtual reality, collaborative environments are being integrated into teaching to improve information retention by learners (Holstein et al., 2018; Kao & Ruan, 2022; Mystakidis et al., 2021).

Among the available online sources for learning, textbooks are observed to comprise of authentic information (Oates, 2014). E-textbooks are observed to be used on a large scale in the field of computer science and engineering (Fischer et al., 2015; Tang, 2021). This extensive use of textbooks also resulted in various innovative ways to be incorporated into the basic e-textbooks (Hori et al., 2015; Rockinson-Szapkiw et al., 2013; Weng et al., 2018). E-textbooks are now being developed to be interactive, including animation features, simulations and emotions (Chang & Chen, 2022; Lee et al., 2013; Sun et al., 2012; Weng et al., 2018). Dictionaries are also being added to the current e-textbooks to reduce the user efforts of browsing through separate platforms for clear understanding of some words in the textbooks (McGowan et al., 2009; Rockinson-Szapkiw et al., 2013). Several other features including quick search, note making and low printing costs have resulted in e-textbooks having an edge over physical textbooks (Davis & Song, 2020; Sun et al., 2012; Weng et al., 2018). Textbooks are also integrated with content from community Q&A forums to provide information that is deficient in the e-textbook (Ghosh, 2022; Venigalla & Chimalakonda, 2020). Other features to support easy navigation across the e-textbook based on various frequent jump-back behaviours of students using e-textbooks are being explored (Ma et al., 2022).

Programming related e-textbooks have also been integrated with video and audio tutorials, visualizations, code executions and so on (Ericson et al., 2015; Miller & Ranum, 2012; Solcova, 2016; Venigalla & Chimalakonda, 2020; Weber & Brusilovsky, 2016). Code execution platforms that show the run time execution of code snippets presented in the textbook are observed to contribute to better and clear understanding (Solcova, 2016; Weber & Brusilovsky, 2016). Researchers have also attempted to enhance existing information platforms by integrating information available on multiple external sources (Venigalla & Chimalakonda, 2020).

Crowd sourced Question and Answer platforms such as StackOverflow are being integrated with code snippets from crowd sourced code sharing platforms such as GitHub and other code hosting platforms such as JExamples (Reinhardt et al., 2018; Venigalla et al., 2019). GitHub is also being integrated with information from StackOverflow platform (Pletea et al., 2014). Several developers post information and observations about various programming concepts on multiple view-sharing platforms such as blogs² and developer forums.³ Integrating e-textbooks with information available on multiple sources could help in learning through different integrated resources, at one place, rather than spending time to explore different resources in finding the desired related information (Almansoori et al., 2021; Seeling, 2016). For example, Almansoori et al. have observed lack of security related discussions in the introductory programming textbooks and suggested augmenting security related information to the contents in the textbook (Almansoori et al., 2021). Patrick Seeling has also observed that integrating online programming textbooks with self-check exercises and feedback has improved the learners’ performance (Seeling, 2016). These integrations help users to gain a wider range of knowledge and exposure. The current e-books, are an authentic source of information, however, they contain only static data. Though e-textbooks are not the only learning materials in a course, they could be used as a reference. Moreover, e-textbooks could support enthusiast learners, though not enrolled in a course, in learning concepts of programming.

The existing research on improving e-textbooks mostly focuses on integration with augmented reality, virtual reality, audio and video tutorials and so on (Rockinson-Szapkiw et al., 2013; Sun et al., 2012; Weber & Brusilovsky, 2016). However, most of these enhancements are static in nature, indicating that they are predefined for a given textbook. Also, the code execution platform augmented with e-textbooks implements only the code presented in the textbook. Only interaction and intelligent tutor-based e-textbooks exhibit a scope to present information apart from that available in the textbook. It has also been observed that existing programming language textbooks do not suffice in well explaining the concepts of programming languages, motivating the need to improve these textbooks (Mazumder et al., 2020). However, to the best of our knowledge, except for the intelligent tutor-based textbooks, we are not aware of any research that aims to dynamically integrate an e-textbook with further conceptual information from other sources.

Though DynamiQue (Venigalla & Chimalakonda, 2020) aims to integrate e-textbook with information from StackOverflow, it is only in the form of question and answers and no programming examples or conceptual information is presented. Researchers have also observed that additional examples help readers to construct better mental models, which could further help in improving the readers understanding on a topic (Gerjets et al., 2006).

We are primarily interested to explore the following two research questions in this work:

Hence, towards integrating external knowledge to e-textbooks, TAnnotator, a web-based portal aims to annotate programming-based textbooks with various facts about the content being discussed in the textbook. It also augments the e-textbook with code-snippet examples, as an add-on to already specified examples on the textbook. Figure 1 presents an example of a fact (Fig. 1B) and code snippet example (Fig. 1A) displayed on TAnnotator for the topic—#include. To analyse the usefulness of this annotation from user perspective, we evaluated the usefulness of TAnnotator through a qualitative user survey based on UTAUT2 model (Rondan-Catalun˜a et al., 2015) with 15 participants. The results of the survey indicated that TAnnotator was found to be useful and easy to use. The questionnaire and results of the user survey are presented here.⁴

Several technologies have been introduced to improve and teaching and learning (Bailey & Zilles, 2019; Benotti et al., 2018; Zavala & Mendoza, 2018). Mumuki has been proposed as a web-based tool, to reduce work load of teachers and to avoid biased evaluations by assessing code snippets submitted by students and providing feedback to teachers about the performance of students (Benotti et al., 2018).

Emotions have been integrated into the e-textbooks to reduce the cognitive load on the students and support better learning achievements. It has been observed that e-textbooks with emotions helped students in learning and retention when compared to e-textbooks without emotions integrated and also paper textbooks (Chang & Chen, 2022).

Content in the e-textbooks is analysed to identify deficient areas that have a scope for inclusion of more information (Ghosh, 2022). These identified deficient content types are explored in the community Q&A forums and relevant information is extracted and integrated to the e-textbook at the deficient area. Such integration was observed to enhance the learning interest among students (Ghosh, 2022).

Zavala et al., have proposed to automatically generate programming assignments based on a specific template mentioned by teachers. This idea has been demonstrated by integrating it with existing programming practice tool (Zavala & Mendoza, 2018). uAssign has been proposed as a software, that can be embedded into any Learning Management System (Bailey & Zilles, 2019). It aims to generate assignments based on unix terminal and also facilitates teachers to understand the skills of students on unix terminal.

Krusche et al., have proposed ArTemis to support automatic assessment of programming solutions and provide feedback to the students based on their performance instantly, that could constructively improve their programming skill (Krusche & Seitz, 2018).

Improving conceptual learning has become the primary goal of learning in the present day. A study conducted on different types of collaborative learning indicated better conceptual skills in students who followed a feedback and discussion based collaborative learning, rather than those who followed collaborative learning with minimal discussions (Harsley et al., 2017).

Integrating information across various sources also supports learners to gain both knowledge and insights of other developers. StackOverflow has been integrated with definitions of APIs, extracted from JDK and examples from a code hosting platform-JExamples (Venigalla et al., 2019). This integration of information from multiple sources improves knowledge of users about various APIs and also supports the users to be aware of multiple ways in which a specific API can be used (Venigalla et al., 2019). Reinhardt et al. have integrated information from multiple platforms to provide insights on API usage patterns to users (Reinhardt et al., 2018). They have integrated source code available in public repositories on GitHub with code snippets present in questions and answers of StackOverflow. Code snippets that misuse APIs are detected from StackOverflow and are presented with multiple examples from GitHub that display the correct usage of APIs (Reinhardt et al., 2018).

Also, linking information present on multiple platforms might provide valuable insights on issues present in code snippets. DynamiQue has been developed to integrate e-textbooks with questions and answers present on StackOverflow (Venigalla & Chimalakonda, 2020). Relevant questions were identified based on topics on the page, extracted using LDA topic modelling technique. However, no other information except question and answers from StackOverflow was augmented with e-textbooks (Venigalla & Chimalakonda, 2020). E-textbooks have also been integrated with programming practice sessions and automatic grading to ease and improve learning in a programming course (Ellis et al., 2019).

Several approaches and tools have aimed to improve learning of programming concepts by making learning interactive, interesting and simple (Berns et al., 2019; Harsley et al., 2017; Krusche & Seitz, 2018; Weber & Brusilovsky, 2016). Various tools have also been augmented to e-textbooks to ease the learning of students (Rockinson-Szapkiw et al., 2013; Sun et al., 2012; Venigalla & Chimalakonda, 2020). Considering the advantages of multiple source integration, linking e-textbooks with information from multiple sources could be of a great advantage to learners (Venigalla & Chimalakonda, 2020). Though there is adequate research in integrating various forms of information present across multiple programming-based platforms, there is hardly any research that attempts to integrate multiple source conceptual information with textbooks, other than integration of question and answers.

TAnnotator augments e-textbook in the domain of programming, with facts and examples from various discussion forums. The current version of TAnnotator is a web portal and augments only one specific e-textbook—The C++ Tutorial, with facts related to some keywords in the textbook content. Each page of the e-book is rendered onto the web portal and facts are displayed adjacent to the text in the e-book, page-by-page, as shown in Fig. 3. Currently, we demonstrate the idea of augmenting e-book with facts and examples, with six pages of the e-book, which could be easily extended to all pages in the e-textbook.

We used Flask⁵ framework to design and implement the web portal of TAnnotator. Implementation of TAnnotator requires identification of useful keywords and rendering of facts and examples for these keywords. Useful keywords can be obtained from identifying frequent words and through topic modelling. In a textbook, the words that occur with more frequency and those that occur with least frequency may not be useful. For example, the word good might have highest frequency in a text that describes usefulness of a data structure, but, this does not imply towards any useful topic. Hence, the words with medium frequencies could be identified as useful keywords, because the words with high frequencies might sometimes be insignificant (Choi & Park, 2019). A plenty of approaches such as Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), and so on can be used to identify topics in a text through topic modelling. Among these approaches, LDA facilitates processing of long texts that occur in textbooks, and hence is used in many studies that involve topic modelling (Onan et al., 2016; Venigalla & Chimalakonda, 2020). For the implementation of TAnnotator, we chose an approach that includes identification of the keywords through LDA and also based on frequency of the keyword, to ensure identification of most useful keywords.

The approach followed in development of TAnnotator is displayed in Fig. 2 and presented below.

This process of development answers RQ1, that while annotations to e-textbooks with additional knowledge could be done, this annotation is limited to availability of information, ease of extraction, associated license guidelines and so on. However, if the information is available for extraction, and is permitted to be extracted, while the content in e-textbook is also permitted to be extracted, then the e-textbook could be annotated with additional information.

Suppose Risha is a computer science student, and is interested in reading the textbook- The C++ Language tutorial. She visits the e-textbook and reads through the book, and wishes to know more about the topics being discussed in the e-book. She learns about TAnnotator and considers that it would aid her learning. Hence, she visits TAnnotator portal and decides to read the e-textbook integrated with tooltips that include facts and examples.

She is displayed with text of The C++ Language tutorial e-textbook, page by page as shown in Fig. 3. The title of the text book is displayed as shown in Fig. 3A, and the text in each page of the textbook is displayed as shown in Fig. 3B. The text of the page displayed is processed using topic modelling technique Latent Dirichlet Allocation (LDA), and the topics such as programming, #include, cout and so on are extracted, as shown in the right panel of Fig. 3. Risha is displayed with facts as tooltips, that are dynamically extracted from preidentified sources based on the keywords of extracted topics. These tooltips are displayed to the right of the web page, as shown in Fig. 3C. A ‘Read More/Less’ icon is presented at every tooltip to enable further reading (as highlighted in Fig. 3D). When Risha clicks on the Read More/Less icon, she is displayed with more elaborated text on the tooltip. When Risha hovers on the image highlighted by in Fig. 3E, she is displayed with example code snippet corresponding to the #include keyword, as shown in Fig. 3F. Further, Risha is facilitated to navigate back and forth through pages using the Previous and Next tabs represented in Fig. 3G.

TAnnotator has been developed with main aim of supporting readers and enhancing user experience with e-textbooks, specifically in the area of programming, keeping in mind, the ever evolving nature of technologies in the domain of computer science and engineering. Hence, our evaluation was based on identifying the extent to which TAnnotator is useful, the complexity associated with using TAnnotator and broadly, the user experience with TAnnotator. Research tools and approaches such as DynamiQue (Venigalla & Chimalakonda, 2020), proposed with similar goals of improving user experience in the existing literature have been evaluated through user surveys, based on multiple models such as TAM (Technology Acceptance Model), UTAUT (Unified Theory of Acceptance & Use of Technology) (Venkatesh et al., 2003, 2012), IDT (Innovation Diffusion Theory), integration of TAM and IDT, and other integrated models (Venigalla & Chimalakonda, 2020).

Considering attributes such as ease of use and usefulness, against which TAnnotator has to be evaluated, and considering the advantages and appropriateness of UTAUT2 model (Venkatesh et al., 2012) over other TAMs (Rondan-Catalun˜a et al., 2015), we evaluate TAnnotator through a user survey based on UTAUT2 model, adapted to the functionalities of TAnnotator. Certain factors of UTAUT2 such as social influence and facilitating conditions that could strongly influence book readers, have made UTAUT2 more appropriate to evaluate TAnnotator, than the other existing TAMs. UTAUT refers to Unified Theory of Acceptance and Use of Technology, that aims to evaluate a novel technological contribution based on multiple factors, with behavioral intention to use and user behavior as the prominent attributes for evaluation.

UTAUT2 (Venkatesh et al., 2012) model evaluates technology based on seven attributes—Performance Expectancy (PE), Effort Expectancy (EE), Social Influence (SI), Facilitating conditions (FC), Hedonic Motivations (HM), Price Value (PV) and Habit (H). These attributes were identified to influence the users’ decision on willing to use the portal. PE describes the usefulness of technology, as perceived by the users, with respect to providing users with the necessary information. EE refers to the effort to be spent by users in order to use the technology, that includes clarity and ease of use. SI indicates the opinion of users’ society towards using the technology, while FC indicates the availability of resources to use the technology and the information and skillset to be learned before using the technology. HM refers to the fun and pleasure obtained by using the technology. PV refers to the users’ opinion on value of the price associated with the technology. H deals with interest of users in using the technology in the users’ daily routine, and BI deals with the tendency of user to use and continue to use the technology. TAnnotator can currently be accessed without any pricing from the source, thus, not requiring to be evaluated against PV attribute. Hence, an adapted UTAUT2 model, with the following six attributes, which are considered for evaluation of TAnnotator, has been used to perform a user survey.

Invites were sent out to 20 under graduate students and 10 industry developers requesting for participation in the evaluation survey. While the invites were sent randomly to 20 undergraduate students from a class of 40, we had a filter that the undergraduate students selected have computer science as their major, in our academic institution. The industry developers were randomly chosen based from different organizations, through their Linkedin¹² India profiles, where the users mention their work profile and details.

We received acceptance responses from 11 students and 4 industry developers. Thus, we considered 15 volunteers, in the age group of 19–44 years. The volunteers included 6 female students, 5 male students and 2 female industry developers and 2 male industry developers. Of all the participants, 12 of them stated that they were very familiar or familiar to using e-textbooks, while three of the participants marked their familiarity as neutral. A 5-point Likert scale based questionnaire of 30 questions, based on the six attributes of UTAUT2, presented in Table 1 and other required demographic information such as age and familiarity, was then circulated among the volunteers. While the first 6 questions of the questionnaire referred to demographic information, the rest 24 questions dealt with attributes of adapted UTAUT2 model, similar to the following questions.

The questionnaire and corresponding results are presented here.¹³

All questions of the questionnaire sent for user survey were classified into corresponding UTAUT2 factors and labelled accordingly. These questions were answered based on a 5-point Likert scale from Strongly Disagree to Strongly Agree. During analysis of the results, the highest score, 5 has been assigned to strongly agree and the lowest score, 1 has been assigned to strongly disagree. Mean and standard deviation of the results have been calculated for each of the questions and the resultant scores are plotted on a graph, that is presented in Fig. 4. The horizontal axis of the graph comprises of question codes for all the questions based on the factors they correspond to, while the vertical axis comprises of mean and standard deviation scores. The mean value of the questions indicates the extent of acceptance, while standard deviation indicates the mutual level of agreement among the participants.

As reported in the Fig. 4 the mean of all the questions is greater than 3 and close to 4 in majority of the cases, indicating better acceptance of TAnnotator among the participants of user survey. Questions corresponding to Effort Expectancy (EE) and Facilitating Conditions (FC) aspects have highest means, close to 4, and reasonably low standard deviations. This indicates that majority of the participants have collectively found learning to use and being skillful at TAnnotator to involve minimal effort from the participants. The mean value 4.1 for question based on correctness indicates that TAnnotator has rendered correct and relevant information on the respective pages of the e-textbook. However, the mean values of questions related to Habit (H), though close to 3, are observed to be lesser than other factors, indicating the need for improvements in TAnnotator towards being regularly used by users, and consequently be put into practice.

This analysis of usefulness through the user survey based evaluation answers RQ2, by indicating that majority of the users consider annotating an e-textbook with additional information to be useful. However, the analysis also indicates that TAnnotator could be further improved to support users in making it a habit to use TAnnotator frequently.

Participants have also suggested improving TAnnotator to support multiple e- textbooks. Some suggestions from users include:

The correlation among the variables from UTAUT2 model considered for evaluation of TAnnotator are presented in Table 2. We calculated the correlation of UTAUT2 variables using the following formula, where r: correlation coefficient, x_i: values of the x-variable in a sample, \(\bar{x}\): mean of the values of the x-variable, y_i: values of the y-variable in a sample and ȳ: mean of the values of the y-variable, and x and y correspond to UTAUT2 variables.

The values of correlation among the variables displayed in Table 2 indicate positive correlation among all the factors. The strongest correlation (0.638) is observed among Social Influence (SI) and Facilitating Conditions (FC), indicating that the users’ facilitating conditions to use TAnnotator are strongly affected by Social Influence on the user. Also, users’ behavioral intention (BI) to use TAnnotator is observed to be strongly influenced by the extent to which TAnnotator contributes to users’ habit (H), than any other factor.

TAnnotator currently demonstrates the idea of annotating e-textbooks with facts and examples through only one e-textbook- The C++ Language Tutorial. However, it could be extended with minimal technical effort to any other e-textbook in the programming domain, based on the e-textbooks’ policies. Extension of TAnnotator to other e-textbooks of any other domains requires identification of active discussion forums that could be used as sources for fetching facts about the concepts being discussed in the textbook.

TAnnotator displays additional information only in extra margin adjacent to the text, and does not disturb the inherent flow of content in the original e-textbook. However, there is need for qualitative assessment towards determining the position of annotations. An expert in the field corresponding to topics of discussion in the e-textbook could be consulted to validate the annotated information and the information that could disturb the didactic concepts could be omitted. This exercise could be conducted for multiple annotations and the actions could be drafted, which could help in training machine learning model that could help in validating the annotations. This could reduce the need for human interference, while facilitating validation of the annotations.

Also, the keywords in a specific page are being identified only once to ensure faster response of the portal. Approaches to dynamically identify keywords every time a page loads, without compromising on the response time of the portal could be explored. This could further improve the knowledge base of users by displaying newer facts frequently. The evaluation performed largely focuses on the user experience and usability of TAnnotator, but does not focus on the learning outcome using TAnnotator. However, as programming courses differ with respect to their goals across various institutions, programmer age groups, educational and social backgrounds, it is difficult to arrive at common learning goals, and hence difficult to assess the learning outcome.

We presented the idea of annotating online textbooks, specifically in the area of computer programming, with facts and examples in this paper. We demonstrated this idea by presenting a prototype version of TAnnotator, a web portal that displays tooltips and corresponding example code snippets adjacent to the text in the e-textbook. These tooltips are displayed based on the context of the topics being discussed in the textbook. The tooltips and examples are extracted from developer forums and blogs. We have evaluated the user experience and usefulness of TAnnotator, through a user survey based on UTAUT2 technology acceptance model. The results of the survey indicate that majority of the users appreciated the idea of annotating e-textbooks in the programming domain and that they were willing to use TAnnotator. It can be used to learn about the latest trends and specific tooltips for multiple topics being discussed in the e-textbooks. It could be useful to the software developers in the industry to be up-to-date about multiple facts in their domain. It could also help students in getting a better idea about the topics being discussed in the textbooks.

We plan to increase the number of sources being used for tooltip and example retrieval based on the content of the textbook. The responses of the survey indicate lesser inclination towards using TAnnotator on a day-to-day basis. The suggestions of the users also indicate need for improvement in the number of facts annotated and the addition of other features. We plan explore multiple sources by considering the participants’ expertise and further explore ways to incorporate the participants’ suggestions. We plan to extend TAnnotator to support more number of freely available e-textbooks in the programming domain. Also, we intend to identify appropriate rich sources of information for each of the textbooks that would be supported by TAnnotator. We also plan to integrate an automated question and answer mechanism, capable of answering questions of the users, to the current version. User activity on the portal could be tracked, with user permissions, and tooltips based on the activity could be provided in the future versions, thus moving towards a personalised textbook annotator.