1 Introduction
-
Class comments are a primary source of documentation in Pharo.
-
As a descendant of Smalltalk-80, Pharo has a long history of class comments being separated from the source code (Goldberg and Robson 1983), and is thus appropriate to analyze the evolution aspect of class comments.
-
Smalltalk supports liveness since more than three decades; therefore, it can present interesting insights into code documentation in live programming environments.
-
Pharo traditionally offers a concise template, consisting of commenting guidelines for class comments, to enter a class comment for newly-created classes, and this template has evolved over the years. Consequently, Pharo is appropriate as a case study to investigate to what extent developers follow the template in writing comments, and what additional information developers embed in them.
-
RQ1: What is the class commenting trend of developers over the Pharo versions, and in particular, do developers change comments of old classes?
-
RQ2: What types of information are present in Pharo class comments?
-
RQ3: To what extent do developer commenting practices adhere to the class comment template over Pharo versions?
2 Background
3 RQ1: Comment Trend Analysis
3.1 Study Setup
Version | Release date | # Classes | # Classes with comments |
---|---|---|---|
1.4 | Apr, 2012 | 2 950 | 1 486 |
2.0 | Mar, 2013 | 3 248 | 1 983 |
3.0 | Apr, 2014 | 4 025 | 3 264 |
4.0 | Apr, 2015 | 4 923 | 3 768 |
5.0 | May, 2016 | 5 670 | 4 493 |
6.1 | Jun, 2017 | 6 484 | 5 181 |
7.0 | Jan, 2019 | 7 863 | 6 324 |
3.2 Methodology
3.3 Result
Version | Added | Removed | Re-documented |
---|---|---|---|
1 | Ring metamodel | Squeak classes | Code simulator, Zinc, Refactoring, Monticello |
2 | QA tools, Spec, Fuel, Native Boost, Nautilius | OmniBrowser, TrueType | Zinc, Refactoring, Monticello |
3 | Versioner, Opal, Athens, Debugger | Kernel tests, Zinc, Monticello, Collection tests | |
4 | GLM, Rubric, TxText, OSWindow, MetaLink | Slot tests | Refactoring, AST, Athens, Zinc, Delay scheduler |
5 | Spur VM, UFFI, Renraku, STON | NativeBoost | Rubric, Refactoring, TxText, Nautilius, Komitter |
6 | Iceberg, Epicea, Tonel, Ombu | Refactoring, AST, UFFI, Spec, Renraku | |
7 | Bootstrapping, Traits2, Refactoring2, Calypso | TxText, Versioner, Nautilius, Kommitter, Traits | UFFI, System tests, Tool, Kernel, STON, System, Iceberg |
-
In Pharo 2, significant effort has been put into refactoring and removing classes from the old version, Pharo 1. The old system browser, OmniBrowser, is replaced with Nautilus.
-
In Pharo 3, a major effort is put into commenting old classes, as shown in Fig. 4.
-
In Pharo 4, developers focus less on commenting old classes but more on adding new classes. New projects added in the version are shown in 2.
-
In Pharo 5, the focus seems more on refactoring classes from old versions, specifically Pharo 1, 2 and 3 but not Pharo 4, as shown in Fig. 6. The ratio of classes with comments to classes without comments is also higher compared to the previous Pharo 4.
-
In Pharo 6, the effort is put into adding new classes and making sure that comments are also added to new classes. One of the main projects added in this version is for git support.
-
In Pharo 7, we find that many new classes are added. After investigating further we found that new versions of Refactoring and Traits, and a new system browser Calypso are added. Refactoring old projects is the primary focus of this version. A substantial number of old class comments are updated, in particular, the projects UFFI, Tool, and System tests.
-
Analyzing Fig. 6, we observe that Pharo 4 classes were rarely refactored in succeeding versions except Pharo 7 as the height of the Pharo 4 magenta bar remains the same through Pharo 6. We believe this is due to the importance of the project GLM (Glamorous toolkit), and the general interest of developers to keep this project in the current, already stable, status.
3.4 Implications
-
Tool support to analyze the co-evolution of code and comments: Understanding software evolution is crucial to ease various software development tasks such as understanding a program, its software elements, finding the actual change that introduced a bug, or detecting change propagation patterns among software artifacts. Our comment evolution results show that developers tend to add class comments to old classes, however, once the ratio of class comments to the total classes reached a particular level (at least 75%), developers do not allocate the same effort, thus indicating the stability of the system. Also, we observe that developers put considerable effort into adding comments to classes newly added to the Pharo core, which is in contrast to previous results involving commenting practices of Java external systems (Fluri et al. 2009). Whether such commenting behaviour is due to the expectation of better commenting practices from core systems compared to the external systems or due to Pharo developer habits requires further analysis. Fluri et al. showed that the Eclipse core system has a better commenting ratio compared to non-core systems such as Eclipse JDT and Eclipse PDE (Fluri et al. 2009). We observe similar behaviour in the Pharo core compared to external projects. Still, these systems lack appropriate tools to analyze the co-evolution of code and comments. We suggest that further research needs to be devoted to developing tools providing co-evolution views of code and comments to monitor better the relative growth and quality of comments over time as well as the actual code comment coverage (Zaidman et al. 2008).
-
More accurate tools to automate the detection of comment changes: Soetens et al. envision that future IDEs will use the notion of changes as first-class entities (AKA change reification approaches). These change-based approaches can help in communicating changes between IDEs and their architectures, and to produce accurate recommendations to boost complex modular and dynamic systems (Soetens et al. 2017). Analyzing and detecting change patterns of comments can enable the vision of Soetens et al. of integrating code comments easily in such change-oriented IDEs. Additionally, detecting which types of information in the comments tend to change more often can help researchers in generating comments automatically. For example, we found a code change due to a class deprecation which triggered a comment change by adding the deprecation notice in the class comment to inform other developers. This effort of updating the class comment whenever a class deprecation code change is detected can be reduced by generating the notice information automatically in the class comment. These comment change patterns are not only helpful for developers to reduce their commenting effort but can also help researchers to improve their bug-prediction models. For instance, Ibrahim et al. showed statistically significant improvements in their bug-prediction models using comment update patterns; similarly, our comment update patterns can be used for future work (Ibrahim et al. 2012).
-
Leveraging change data: Previous studies have leveraged the historical change data in various ways, such as in designing new applications in the IDE (Soetens et al. 2017), evaluating code completion algorithms (Robbes et al. 2010), and recommending future changes in specific code parts (Fluri et al. 2009). In the context of comments, Fluri et al. implemented a tool named ChangeCommander, which recommends comment changes when a new method invocation is introduced in the system, based on the collected code-comment change patterns (Fluri et al. 2009). However, the approach of Fluri et al. to detect comment changes does not work entirely for the Pharo system due to its dynamic nature, and its different comment structure and scope. Based on our code-comment change analysis, we identified patterns of code changes in a class such as deprecating a class, or adding a new method which triggers comment changes more frequently than other code changes. Future tools can utilize these patterns for recommending developers when to update class comments. From a technological point of view, Epicea (a tool to log code changes in Pharo) supports source code changes on the class level. Integrating the type of comment changes we identified in our study, such as formatting changes, typo fixes, instance variable changes, and code-comment change patterns, can help to answer particular developer questions such as “What specific type of the code change led to this comment change? or “Which specific comment changes does a commit consist of?” (Dias et al. 2014).
4 RQ2: Comment Information Types
4.1 Study Setup
stratum | #comments | comment rate | #selected for study |
---|---|---|---|
1-1 | 3 040 | 48% | 175 |
2-2 | 945 | 15% | 54 |
3-6 | 1 224 | 19% | 69 |
7-272 | 1 115 | 18% | 65 |
Total | 6 324 | 100% | 363 |
4.2 Methodology
4.2.1 Taxonomy Study
4.3 Results
Category | Description | Implicitness level | Keywords |
---|---|---|---|
Intent | Describe purpose of the class | Often Implicit | I represent, I am, I’m, This class is, A *Class* is |
Responsibility | List responsibilities of the class | Often Implicit | provide, implement, I do, I know, responsible |
Collaborator | List interactions of the class with other classes | Implicit | use, interact, provide, collaborate |
Public API | List key methods and public APIs of the class | Sometimes Implicit | Key Messages, Public API |
Example | Provide code examples to instantiate the class and to use API of the class | Often Explicit | Usage, Example, For example, code examples |
Implementation Points | Provide internal details referring to the internal representation of the objects, particular implementation logic, conditions about the object state, and settings important to understand the class | Often Implicit | Internal representations, Implementation points: |
Instance Variables | List state variables of the object | Often Explicit | instance variables: |
Class references | Overlaps with Collaborator category but includes extra cases when developers refer to other classes in the class comment to explain the context of the class | Implicit | |
Warnings | Warn readers about using various implementation details of the class | Often Implicit | Note, do not, remarks, should |
Contracts | Inform readers about potential conditions before or after using a class/method/component of the class | Often Implicit | Precondition:, do..when.. |
Dependencies | Describe the dependency of the class on other classes/methods/components | Implicit | used by |
Reference to other resources | Refer reader to extra internal or external resources | Often Explicit | See, Look |
Discourse | Inform the readers about a few class details in an informal manner | Implicit | developers use conversational language |
Recommendation | Recommend the ways to improve the class implementation | Implicit | recommended, see, should be |
Subclasses explanation | Describe details about its subclasses, the intent of creating the subclasses, and when to use which subclass | Implicit | My subclasses |
Observations | Record developer observations while working with the class | Often Implicit | |
License | Store license information of the code | Often Implicit | |
Extension | Describe how to extend the class | Often Implicit | extend, extension |
Naming conventions | Record the different naming convention such as acronyms used in the code | Implicit | |
Coding Guideline | Describe rules to be followed for coding by the developer while writing the class | Often Implicit | |
Link | Refer to a web link for extra or detailed information | Sometimes Implicit | |
TODO comments | Record actions to be done or remarks for developers | Explicit | todo |
Other | Include the comments from other programming languages | Explicit | JavaDoc comments |
Study | Comment types analyzed | System analyzed | Categories proposed | Mapping to our taxonomy (M) |
---|---|---|---|---|
Ying et al. May 2005 | Task comments | [Java]: Eclipse Architect’s Workbench (AWB) project | 7 categories: communication, pointer to a change request, bookmark, current task, future task, location marker, concern tag | 1 category: Task comments |
Padioleau et al. 2009 | Source code comments | [C]: Linux, FreeBSD, OpenSolaris [Java]: Eclipse, [C/C++]: MySQL and Firefox | 6 categories (comment content): type, interface, code relationship, past future, meta, explanation | 5 categories: type (M), code relationship (M), past future (Todo), meta (copyright), explanation (M) |
Haouari et al. 2011 | Source code comments | [Java]: DrJava, SHome3D, jPlayMan | 3 categories (comment type): explanation comments, working comments, commented code, other | 3 categories: explanation comments (M), working comments (Todo), other (M) |
Steidl et al. 2013 | Source code comments | [Java]: CSLessons, EMF, Jung, ConQAT, jBoss, voTUM, mylun, pdfsam, jMol, jEdit, Eclipse, jabref, C++ | 7 categories: Copyright comments, header comments, member comments, inline comments, section comments, code comments (commented code), task comments | 5 categories: copyright comments (license), header comments, member comments (M), section comments (M), and task comments (Todo) |
Pascarella and Bacchelli 2017 | Source code comments | [Java]: Apache (Spark, Hadoop), Google (Guava, Guice), Vaadin, Eclipse | 16 categories: summary, expand, rational (intent), deprecation (warning), usage, exception, TODO, incomplete, commented code, directive, formatter, license, pointer, auto-generated, noise | 9 categories: summary (M), expand (M), rational, deprecation, usage (M), TODO, license, pointer (M), noise |
Zhang et al. 2018 | Source code comments | [Python]: Pandas, Django, Pipenv, Pytorch, Ipython, Mailpile, Requests | 11 categories: metadata, summary, usage, parameters, expand, version, development notes, todo, exception, links, noise | 8 categories: metadata (M), summary (M), usage, expand (M), parameters, development notes(M), todo, links (M), noise (other) |
Shinyama et al. 2018 | Local comments (inside methods) | [Java]: 1 000 projects [Python]: 990 projects | 11 categories: Preconditions, post conditions, value description, instructions, guide, interface, meta information, comment out, directive, visual cue, uncategorized | 7 categories: pre conditions (contracts), post conditions (contracts), value description (instance variables), guide (examples), interface (key message), meta (license), uncategorized (other) |
Hata et al. 2019 | Links in comments | [C], [C++], [Java], [JavaScript], [Python], [PHP], [Ruby]: Projects from GitHub | - | Links |
4.4 Implications
-
Need to analyze class commenting practices in other systems: Previous studies, as shown in Table 5, have focused on classifying code comments, or specific types of information on these comments (e.g., links and task comments). However, we observed that such studies do not classify code comment information according to specific comment types (e.g., package comments, class comments, function comments). According to standard coding style guidelines, different comment types report various kinds of information. For example, the Java Oracle style guideline suggests adding author information to the class comments but not to the method comments. In contrast, Python PEP8 suggest to place this information after the module docstring, and before the relevant statement. On the other hand, in Pharo, the guidelines (and the class comment template) do not mention author information but we found instances of author information in the class comments. This shows that class commenting guidelines vary across languages but to what extent developer class commenting practices vary is still unclear and it requires a systematic investigation.
-
Identify information types automatically: The task of accessing the type of information embedded in comments depends on the kind of information (warning, rationale), level of detail (design level or implementation level) developers seek, the type of development activities they are performing, and the type of audience (user or developers) accessing them. Tools to automatically identify these information types can reduce the effort developers and other stakeholders invest in reading code comments when gathering particular types of information. In addition, on top of these automated tools, visualization strategies could be implemented to highlight and organize the content embedded in the comments, to further ease the process of obtaining the required information. For example, identifying warnings from the comments can help turn them into executable test cases, so developers can automatically check that the mentioned warnings are respected. Similarly, automatically identifying code examples from the comments and executing them can ensure that code examples are up to date. In recent work by Pascarella et al. the authors build a machine learning-based tool to identify information types for Java automatically (Pascarella and Bacchelli 2017). Similarly, Wang et al. developed such an approach for Python (Zhang et al. 2018). However, given the increasing trend of open-source systems written in multiple programming languages, these approaches can be of limited use for developers contributing to these projects (Tomassetti and Torchiano 2014). Our work has the aim to foster the building of language-independent tools based on comprehensive taxonomies for comments analysis of multi-language projects. Future studies can leverage our labelled data as a starting point to build language-independent tools, and verify the correctness of their tools.
-
Designing an annotation language: Annotation languages have proven to improve the reliability of software.18 They can help the community in labelling and organizing a specific type of information, and to convert particular information types into formal specification which can further help in synchronizing comments with the code (Padioleau et al. 2009). Even though Pharo comments do not follow any annotation, they do have hidden patterns for different information types such as instance variables denoted by Instance variables or main methods of a class are indicated by Key Messages. We identified various such patterns in constructing our taxonomy highlighted in Keywords in Table 4. Pharo community can use such patterns in developing an annotation language for Pharo comments. In our study, we find some information types express properties (according to implicitness level in Table 4) which can be described via annotations such as Examples, public APIs, Links. Tool/language designers can utilize the identified patterns to design information headers and annotations.
5 RQ3: Adherence of Commenting Practices to the Template
-
Content adherence: If the comments contain information types as mentioned in the respective template, then we say the comments adhere to the template in the content aspect.
-
Writing style adherence: If the comments follow the writing style conventions of the template, then we say the comments adhere to the template in the writing style aspect. The writing style conventions are composed of various constraints formulated for each template information type. If the comments containing specific information fulfill the corresponding constraints, we say the comments adhere to the writing style.
5.1 Study Setup
5.2 Methodology
5.2.1 Template Evolution
5.2.2 Adherence of Comments to the Template
5.3 Results
5.3.1 Template Evolution
5.3.2 Adherence of Comments to the Template
version | categories |
---|---|
1 | Intent, Collaborator, Instance Variables |
2-3 | Intent, Instance Variables |
4-7 | Intent, Responsibility, Collaborator, Instance Variables, Key Messages, Example, Implementation Points |
5.4 Implications
-
Verifying comments adherence in other languages. To write useful and consistent comments, numerous programming languages such as Java and Python, and communities such as Google and Oracle, provide coding guidelines (Google Style Guidelines 2020; Oracle Documentation Guidelines 2020). For example, Oracle’s guidelines suggest “using third person (descriptive) style and second person (prescriptive)” while writing documentation comments, but it is not known whether developers actually follow this guideline in their comments or not. To ensure developers follow such guidelines, various automated style checkers or linters e.g., Checkstyle,24 Pylint,25 ESLint26 turn such guidelines into rules and then evaluate the rules against comments automatically. However, these style checkers are not available for all programming languages, and for the supported ones, they provide limited rules for addressing code commenting guidelines. The majority are limited to detecting missing comments and verifying formatting guidelines, but not adherences to guidelines concerning the content of comments . Our results for Pharo show that developers embed template-inspired information types in the class comments. Moreover, they also follow various syntactic guidelines to add such information types. Whether developers follow similar commenting guidelines (suggested by the coding guidelines) in other programming languages is not yet explored. Our dataset provides relevant data in which one can observe which commenting guidelines developers frequently follow in their comments and which they do not. Thus, it can help in conducting similar studies for other languages.
-
Comment quality tools: Researchers have provided various heuristics-based approaches to evaluate comment quality (Khamis et al. 2010; Steidl et al. 2013; Scalabrino et al. 2016). However, these approaches provide limited checks, they focus on particular programming languages (mainly Java), and they are not designed to be used for other domains and languages (Khamis et al. 2010; Steidl et al. 2013; Scalabrino et al. 2016). In particular, most approaches are based on language-specific heuristics such as comment syntax, common keywords used in the comments, and the supported annotations for comments (Khamis et al. 2010; Steidl et al. 2013), which cannot be directly applied to other languages. For instance, in Pharo code comments follow a different comment structure and writing style, and do not rely on annotations, which makes these approaches not suitable for this language. In addition, Tan et al. also showed that previous approaches concerning the detection of inconsistencies in the comments require adaptation to new domains and languages (Tan et al. 2007). Hence, our study insights about Pharo commenting practices provide further data to help researchers in designing tools for assessing comment quality across other languages and domains.
-
Template-based comment generation and code summarization approaches: Comment templates not only provide developers with concrete examples on how to write comments, but can also employed by researchers to enable automated generation of code comments for various code entities. In recent work, Moreno et al. proposed a template-based approach to automatically generate comments for Java classes Moreno et al. (2013). Their template includes certain types of information which they deem essential for understanding a Java class. However, the information types included in the template were not derived from class comments written by developers, which could make them potentially out of date with current Java commenting practices. In Pharo, class comments are guided by a default template which includes seven types of information considered important to document a class. We observed in our study that developers write template-inspired information types more often compared to other information types found in comments. We compared the information types included in the class comment template by Moreno et al. and Pharo class comment template. We observed that their template does not include information types such as related classes, algorithmic implementation details, or an example to show the usage of the class. In contrast, the Pharo template includes these information types and Pharo developers frequently refer them and with headers Collaborators, Implementation points, and Example respectively. On the other hand, both templates suggest describing the intent of the class, responsibilities of the class and the main important methods, which are again frequently reported by Pharo developers. Thus, our study insights suggest that further information, typically embedded by developers in code comments developers, need to be included in template-based comment generation or code summarization approaches.