Improving hardware/software interface management in systems of systems through documentation as code

The first five research questions are addressed through the TAR method (Wieringa 2014, Chapter 19), as these require to investigate an experimental treatment (a DaC approach for ICDs management), to help a client with previously identified problems (ASTRON) and learn about its effects in practice. In particular, these questions were addressed through the phases of diagnosing, action planning, action taking, and evaluation of the TAR. In the last phase of the TAR, namely the specifying learning phase, the lessons learned from this study (which reflects on the first cycle of the action research), and their implications in future iterations of the study and the proposed documentation pipeline, were collected and are reported in the Discussion section of the paper.

The following elaborates on how the first four activities of the TAR addressed RQ1 to RQ5 and the participants involved in them.

Diagnosing This phase of TAR is focused on exploring and extending the current understanding of ICD documentation problems identified in Cadavid (2021). Previous to this phase, the general idea of adopting DaC for ICD management was pitched to the potential participants. Subsequently, this phase used a virtual focus group with the ASTRON practitioners; a focus group is a qualitative research method to collect data on a given topic through a group interaction (Kontio et al. 2008). The discussion points for the focus group were derived from the relevant findings of the aforementioned previous study (Cadavid 2021). The actual focus group session was geared toward the collection of more context and details on ICD-management issues experienced in the LOFAR/LOFAR2.0 project in order to answer RQ1.

Action planning This phase was focused on identifying, discussing, and choosing solutions to improve ICD-management related issues experienced by ASTRON practitioners. Exploring and discussing alternative solutions required, in the first place, understanding how the identified ICD-management related problems are manifested within an actual ICD, i.e., what are the symptoms of said problems (RQ2). As a guideline for ASTRON practitioners to properly identify said symptoms, and towards avoiding e.g., open-ended questions without enough context or exhibiting bias towards the particularities of a specific ICD, a generic ICD template with the most representative elements of a hardware/software oriented ICD was created. This template was annotated collaboratively by one of the researchers and a group of practitioners (see Table 1) to highlight specific symptoms of the issues identified in the diagnosing phase, e.g., particular elements of a given type of hardware elements that are often missed despite being key (RQ2). It is worth noting that this template was created by identifying the common elements between the ICDs provided by ASTRON engineers and a curated set of publicly available ICDs (available in the replication package⁷) that were (1) created under a document-centered process, and (2) focused on hardware-software interfacing descriptions. Finally, to address RQ3, researchers, in cooperation with one of the practitioners, defined a set of features that the DaC pipeline should provide in order to address the identified issues, using their particular issue symptoms to gauge their applicability in the process.

Action takingHere the solutions selected to improve the identified issues are implemented. Therefore, in this phase the features identified in the previous phase were turned into an actual design that integrates existing DaC tools with any additional custom artifacts required by said features. This design is the answer to RQ4, and was implemented as a functional proof-of-concept that practitioners can try out and evaluate.

Evaluation In this phase, the effects of the action are captured through different data collection methods (Petersen et al. 2014). This phase addressed RQ5 by exploring the efficacy and fitness for purpose of the proposed documentation management approach to address the identified issues during the diagnosing phase. We chose to perform a single case mechanism experiment, an evaluation approach that involves testing the effectiveness of a treatment on a single case by examining the interaction between a designed artifact (such as a new product or intervention) and the specific problem context that it is intended to address (Wieringa 2014, Chapter 18). That is to say, the experiment involves evaluating the impact of the treatment on a single individual or group, and examining the mechanisms through which it produces its effects. In our case, the experiment was conducted with the group of ASTRON engineers described in Table 1, and was used to evaluate the impact of the proposed ICD-mangement features on the issues identified in RQ1.

Participants and timeline A total of ten people participated on the TAR part of the study, which took place between May 2021 and the beginning of May 2022: two researchers and eight ASTRON practitioners. One of the researchers actively participated in the design and development activities during the action planning and action taking phases; both researchers took part in the research activities, designing the research instruments, and drawing lessons from the action evaluation phase. The eight participants on the ASTRON side, on the other hand, were free to decide in which phase of the study they will participate. As a result, they participated in different phases of the study, as described in Table 1.

The sixth research question (RQ6) was addressed through the expert opinion instrument (Wieringa 2014, Chapter 5). This instrument entails a panel of experts evaluating the design of an artifact by predicting the effects it would have, and how it would interact with potential problematic scenarios imagined by them. In our case, the functional description of the proposed features for the ICD management approach was used as the design to be evaluated. Moreover, the panel was organized by focusing the experts’ predictions on the transferability of the TAR findings to other application domains.

In addition to ensuring that panel members have ample experience with ICDs and are thus fit-for-purpose for this panel, multiple resources were provided to them as the means to allow them to develop a proper understanding of this functional description. More specifically, in addition to making available our previous work (Cadavid et al. 2022), a series of introductory videos for the proposed features were produced; these videos included a series of screencasts of the functional prototype, using the features defined in RQ3 for creating and manipulating a fictional ICD that included hardware/software interfaces. Furthermore, the experts were given access to the ICD management platform developed in the TAR study to further explore the source documents and ICDs generated in the fictional case⁸.

In terms of data collection, we created a questionnaire in order to get the experts’ views about the applicability of the proposed documentation management approach in domains beyond radio-astronomy. The questionnaire included a section for each proposed feature with the aforementioned introductory material referenced from it, two likert-scale questions, and one open-ended one, as follows:

The selection process of potential candidates for the panel was two-fold. First, we sent invitation letters through email and LinkedIn to the authors of the publicly available ICDs used in the Diagnosing Phase of the TAR study (when these were disclosed in said documents). A total of twelve invitations were sent to this group, from which we got four positive responses. Second, we sent invitation letters to the participants of our previous practitioners survey (Cadavid et al. 2020); these participants were already identified as engineers with experience in the architecting of cross-disciplinary Systems-of-Systems. In this case, 50 invitations were sent and we received 8 positive responses; thus we reached an overall total of 12 experts. As illustrated in Table 2 and Fig. 1, the 12 panel participants include staff members of governmental and intergovernmental organizations, research and development centers, and the private sector. Together, they represent significant experience (a mean of 23.4 years) in the domains of space, defense, ground transportation (e.g, railway systems), energy, aeronautics, logistics and business. All of them reported experience with ICDs by creating/maintaining them (11 out of 12) or using them as a reference (8 out of 12), or even reverse-engineering them when no ICDs were available (1 out of 12).

Incomplete designs the physical/logical interfaces sometimes lack elements required by one of the involved parties. A prominent example of this, is the lack of key monitoring points (memory locations where specific hardware values can be monitored) on the hardware side, required on the software for proper control of the system state.

Difficult-to-use design some of the interfacing elements required by one of the parties, particularly to access hardware at a low level, are designed in a way that makes their use difficult. For instance, when the hardware design defines multiple monitoring points covering the same area. This can be particularly difficult to handle on the software side given the potential contradicting values due to e.g., timing issues when checking these points.

The ‘general symptoms’, identified through the ICD template, correspond to those that do not make reference to a particular section of an ICD, but to the overall document or management process. As a symptom of the lack of timely notifications (UPN), it was pointed out that some people held the naive belief that once an ICD has been written, it is done. Therefore, as ICDs are in fact living documents, they need to be kept under document control, amended when references change, amended or modified when referred implementations change and so on. As a symptom of the duplicate efforts (NDE) on ICDs maintenance, it was mentioned that there seems to be a tendency of reiterating information that is written elsewhere. As this is something that is both cumbersome and error-prone, there is the need for a central knowledge system, (e.g., a lexicon shared throughout the entire organization) that can be queried to supply information that could otherwise be redundant or outdated.

The following lists the sections commonly included on ICDs, and the symptoms specific to them, as identified by the participants, and mapped to the issues identified in Section 4.1.1.

The following describe the symptoms identified by the participants along three kinds of hardware/software interfaces that they focused on:

In this kind of interface the software performs operations on the hardware by modifying some of the peripheral’s registers values. Likewise, the status can be evaluated by reading other registers values. These registers, consequently, are known as the control and monitoring points of the hardware. The participants emphasized that the ICD user on the software/firmware side needs a register map describing the bitfields (groups of bits with meaning on the hardware side). For this type of interface, participants enumerated the symptoms described in Table 3.

In this kind of interfaces, hardware and software components are connected through a datalink bus. In this way, the hardware is operated and its status is evaluated by sending and receiving messages encoded in the ‘payload’ segments of the data frames defined by the datalink protocol, as illustrated in Fig. 2. Therefore, the ICD user on the software/firmware side needs to know (1) which values should be given for the datalink protocol parameters to send the payload to the hardware end, (2) the possible values sent from the hardware side, and (3) their meaning. For these elements, the internal structure of the payload should be defined using bitfields or textual values depending on the selected encoding. Prominent examples of datalink protocols include asynchronous ones like UART and Ethernet, and synchronous ones like I2C, SPI, and CAN. Table 4 lists the symptoms pointed out by the participants as common when working with this kind of hardware/software specifications.

In this kind of interfaces, hardware is accessible through protocols defined on top of layers of higher-level network and transport control, such as TCP/IP or UDP/IP. As these base protocols require less custom configuration, and most platforms already support them, the information provided to the ICD user on the software side is limited to the payload structure, as shown in Fig. 3. Examples of this kind of interface include protocols that work on top of TCP/IP like OPC-UA, MQTT, CoAP. Table 5 lists the symptoms as identified by the participants.

There are many tools that can perform a wide variety of quality checks in software source code within a CI/CD platform. However, when it comes to the source code of documentation, that is, markup language documents, quality criteria like the one defined in the Action Planning phase (RQ2) require extending the interpreter of such a markup language so that the criteria are evaluated while the document is being processed. These extensions (e.g., the custom macros (M)) need not only to properly report their status during the building process (e.g., distinguishing between failed quality criteria and syntax errors) but also to work in tandem with the API ((4) in Fig. 4). For this reason, a toolkit for building extensions with such features was created on top of the AsciidoctorJ¹⁰ platform, and used to build the artifacts depicted in Fig. 4 (3). More details are reported in the project repository¹¹.

For the proposed DaC-based ICD pipeline, the generated documents are expected to become the single source of truth for all the development and testing tasks conducted on the interfaces described by it. Therefore, keeping track of the official documents and the sources from which they were generated is key to their proper evolution over time. On top of that, an ICD is often dependent on a specific version of other ICDs it is based on, and hence the validity or consistency of the former could be affected by changes in the latter. Given this, a documentation-status tracking API was implemented for the proposed documentation pipeline. This API, working in tandem with the aforementioned building components, keeps track of the locations of the documents and artifacts generated from each ICD, their dependencies, and their corresponding sources. Furthermore, it updates the status of the documents given the events captured by it as illustrated in Fig. 5, which, in turn, can be accessed through its front-end application ((6) in Fig. 4). More details are reported in the project repository¹².

SystemRDL¹³, a widely adopted hardware description language was chosen to explore the use of a machine-readable formalism embedded in ICDs to describe hardware elements of interfaces (F). This involved the creation of a SystemRDL macro for Asciidoc that performs validations according to the defined completeness quality-gates, and generates a human-readable representation of the hardware along with base software artifacts (as depicted in Fig. 6). With the automatic generation of check-sums of the said generated artifacts implemented in the extension, developers can check whether their local copy is up to date, as previously described.

Additional content generation extensions, based on the results of the Action Planning phase of the study, include: automatic generation/validation of the document log section (based on Git history), references, and glossary. The source of a sample ICD written in Asciidoc, using the above extensions, and its corresponding output, is included in the replication package¹⁴.

The purpose of the single-case mechanism experiment (as described in Section 3.2.1) was to assess to what extent, according to the ASTRON engineers’ viewpoint, the proposed features (RQ3), and the way they were implemented (RQ4) would deal with the ICD-management related issues (RQ1) and their particular symptoms (RQ2). To this end, the functional proof-of-concept of the implemented features was configured to work with an infrastructure similar to the one used by ASTRON engineers: GitLab for version control system, GitLab CI/CD for creating documentation pipelines, and GitLab pages for publishing the generated ICDs. An instance of the platform that keeps track of the published ICDs was deployed and made available on the Internet, while the building/validation tools and extensions were published as a Docker container so these could used by the documentation pipeline anywhere. In order for the participants not only to understand but also to experience the four proposed features, they were asked to carry three hands-on tasks assuming different roles, as they would do in a real setting.

In the first scenario, participants were introduced, hands-on, to the required configuration activities, the editing with a lightweight markup language, and the basics of ICDs versioning. In particular, they were asked to setup a documentation pipeline for a new ICD in the CI/CD platform, and to set up and generate two sequential versions of it: the ‘staging’ version (to be shared and reviewed) and the ‘canonical’ one, which would be indexed by the platform as such.

In the second scenario, the participants were asked to include on the previously registered ICD: (1) the specification of a low-level hardware interface (i.e., a register map) which includes a memory address inconsistency; and (2) a paragraph that violates a hypothetical writing style rule. While dealing with the errors reported by the pipeline (unfulfilled ‘quality gates’) using the details provided by them, the participants experienced: (1) what kind of quality criteria can be enforced for both formal and text-based elements; (2) how these embedded formal elements could reduce redundant and error-prone writing/transcription efforts through content/code generation.

In the third scenario, participants were asked first to: (1) create new ICDs and add them as references (dependencies) to the one created in the previous scenarios; and (2) use the software headers generation feature from such ICDs, and by assuming a developer role to create a basic development environment with it. Afterwards, they were asked to make changes on the hardware specification within the ICD (created in scenario two), and on the new ICDs referenced by it. Then they were asked to deal with the warnings generated by the platform (about the need for revisiting an ICD with outdated dependencies) and in the development environment (about the use of headers generated from an outdated specification). This scenario is aimed at illustrating how the ICDs, by becoming the actual ‘single source of truth’ through the centralized index, could be used to foresee potential issues as interfaces evolve over time. In particular, how keeping track of the documents’ life-cycle and dependencies between them, would enable the timely notification of changes.

Once the practitioners completed the exercise¹⁵, they took an online survey where they evaluated the efficacy and fitness for purpose of the proposed features. The analysis of said evaluations is described in the following sub-sections.

The practitioners’ perception on enforcing minimum quality criteria by means of quality gates was in general positive, as shown by questions Q1 and Q2 in Table 7. However, the Disagreement and Neutral responses in Q1, about the enforcement of the style of the ICD elements described in natural language, were justified with doubts about its applicability in a more realistic setting. More specifically, two of the participants pointed out that (1) some text-based descriptions could be very hard to validate, if such validation is at a technical level, and (2) that writing styles are hard to impose and writers could find workarounds to avoid them. The enforcement of technical elements of the ICDs like forgotten keywords and wrong ranges, on the other hand, was seen in general as useful, although some interfacing parameters could be too difficult to validate automatically, especially the ones related to dynamic behavior, time dependencies and performance limitations. Overall, with the ICDs seen as the single source of truth and reference for engineers to work on an interface, the need for these automated quality gates would decrease in the long run, as will the time spent on resolving inconsistencies during development and testing.

According to the justifications for the grades given in Q3, having macros for content generation would help ICDs to become the single source of truth that can be referenced throughout the process. This would help, in turn, in identifying incorrect assumptions in the early stages of the process. On the other hand, the elements automatically generated through macros tailored for ICDs (content, artifacts, etc.), as long as they are reliable and reproducible, would not only improve their consistency but also free time to work on the content that cannot be automatically generated. However, although the above would improve the overall quality of the documentation, that would not necessarily be the case for its maintainability. In this regard, the analysis of Slack conversations during the evaluation exercise highlighted two areas of improvement. First, as a means to improve the workflow speed, an editing tool tailored to the proposed features, that is to say, that allows technical writers to identify errors or unfulfilled quality criteria before submitting changes to the documentation pipeline was deemed necessary. Second, the pipeline should support allowing locally-defined, document-specific glossary entries, in addition to the ones defined at project/organization level.

Question Q4 (see Table 7) and the corresponding open-ended responses also showed an overall positive perception on the applicability of an embedded formalism within the ICDs as a means to remove ambiguity through the uniformity and standardization of the artifacts automatically derived from it. This is seen as a must-have to prevent issues in different engineering groups. Furthermore, this feature was perceived as something that would save a considerable amount of time normally spent writing boilerplate ICD parts, and reduce the human factor that often causes issues when translating from ICD to source code.

According to the practitioners’ views on Q5 and Q6, a centralized platform makes sense if ICDs are expected to be the single source of truth within a project/organization. Moreover, adopting this systematic approach for managing ICDs implies that now it would be necessary to decide, at a higher level within the project/organization, when to go from one version to another across the whole product. Although this would add complexity to the overall process, it would be a positive change. On the other hand, the generation of artifacts with version matching as supported by such a platform would prevent developers from using an outdated version when implementing parts of an interface. However, as pointed out by the participants, to properly close the loop failed version test results should be reported to the central ICD hub so that action can be taken. Furthermore, the tooling should also allow the interfaces to be round-trip converted from source-code back to the model. On top of these observations, practitioners emphasized that although the documentation pipeline tooling itself would prevent small but annoying mistakes, most of the success of the interfacing process would still lie on the communication and cooperation between engineers. Moreover, it would also be important to ensure that engineering and maintenance teams also keep this documentation up-to-date during operations.

The features of embedded formalisms and automated content generation for ICDs management, were deemed as transferable, albeit to a different extent, ranging from ‘a little’ to ‘a great extent’ (Question 2.1). As seen in the top of Fig. 7, in the applicability scale the opinions are equally divided between ‘a lot’/‘to a great extent’ and ‘a little’/‘to a moderate amount’. This is for the most part also the case when looking at each application domain individually, as seen at the bottom part of Fig. 7.

As described on Fig. 8, experts ranked uniformity as the aspect that would be improved the most, out of these features, followed by completeness (Question 2.2). The foreseen improvements on duplicate efforts were also ranked high, although three respondents selected the ‘do not know’ option as according to the responses to Question 2.3 they had no elements to assess it.

By analyzing the open-ended responses that experts gave to Question 2.3, three themes emerged: (1) arguments for why it would make sense to adopt the two proposed features in the domains they have experience on, (2) what should be considered before adopting these features, regardless of the application domain, and (3) potential limitations upon their adoption. These three themes are elaborated in the following paragraphs.

First, experts pointed out that the one of the premises of the proposed approach, namely the fact that interface descriptions tend to be defective is correct (E4). These defects seem to be prominent when dealing with system-level interfacing elements, as any change always has cascading effects on related subsystems (E1). Specific examples of these cascading effects were given, such as the ICDs with UDP¹⁶ specifications (E1): when the bitfield defined for the payload is modified, it requires updating offset values all over the place (source code, documents, etc.). Moreover, experts already had the perception that versioned databases of interface definitions would improve completeness, reduce human errors, and ease the re-generation of documents upon updates (E3) as well as the system verification (E9).

Second, other experts indicated that the applicability of the features would be subject to a number of factors. These factors include the availability of DSLs for the particular elements considered in the interface, the expertise of the ICD designer on them (E6), and an agreement between the stakeholders involved in the system on adopting this approach (E10). For the latter, it was highlighted that having different documentation approaches, or using them with different levels of effort would lead to further issues in the architecting process. Furthermore, in the end, the applicability of embedded formalism/content generation would also depend on the level of formalism required by the project (E10).

Finally, from the comments focused on the limitations of these two features, the experts pointed out that the scenario used in the evaluation is focused on what is transferred through the interface, but not how it is done (E5), e.g., re-negotiating hand-shakes when a message exchange fails. That is to say, the lack of dynamic behavior specifications becomes an issue here. On the other hand, in many cases, particularly in the avionics and automotive industry, it is common to maintain interface information in databases. Therefore, rather than the proposed embedded text-based formalisms, this approach would be applicable only by enabling references to said databases within the ICDs (E5). It is also worth noting that a number of experts (E8, E11, E12), indicated that these features would improve correctness (by providing formal elements, complementary to the natural language), uniformity, and completeness, but they may not guarantee the latter by default. Human intervention would be needed, for instance, by distilling guidelines or checklists (and feeding them to the platform) from previous experiences creating similar interfaces in the given context/domain.

The feature of enforced quality criteria through quality gates was also seen as transferable to other domains by all the experts (Question 3.1), but to a larger extent (per participant and per area), as seen on Fig. 9. In this case, given the responses to Question 3.2, incomplete specifications is seemingly the problem that this feature would improve the most, as shown in Fig. 10.

The analysis to the open-ended question related to this feature (Question 3.3) also leads to the three themes identified for the previous two features, namely the arguments about why it makes sense to adopt the features, considerations for its applicability, and potential limitations. However, consistently with the results of Question 3.1, the first two themes (which are positive in general) are the most prominent ones. We elaborate on these three themes in the following paragraphs.

First, the experts highlighted the need for quality rules defined up-front for this kind of electronic documents (E4, E5), particularly given the variability on their quality (E2) — across engineers and disciplines — and level of detail (E3). This would be particularly important at the last stage of the ICDs formal update processes (E12). Furthermore, bringing these ‘quality gates’ from the software domain to the other ones, e.g., systems engineering, could help all the disciplines involved to speak the same language (E10). It is also worth noting that a number of experts expressed the same opinion on the importance of the quality gates and artifacts included for ensuring proper acronyms definition and allowing the automatic generation of glossaries. In particular, they mentioned that this terminology-related inconsistencies are common (E3) specially when there is no central authority that manages them (E6); hence, just quality-checking them would improve many of the ICDs they have worked on (E10).

Second, regarding the considerations for the applicability of these documentation-oriented quality gates, interface requirements were mentioned as an ambiguity-prone, commonly used, ICD element in multiple domains. Therefore, the quality of said requirements should also be enforced through quality gates using, for instance, existing rule-based NLP tools that assess their quality (E4). Furthermore, in concordance with one of the comments about the embedded models and content generation features, the quality gates should also be able to check naming consistency within the documents against the parameter-databases commonly used in avionics and automotive industries (E5).

Third, as potential limitations, the experts indicated that adopting this automated validation process in any domain could lead to wrong assumptions about the reliability of the document, which would not be necessarily guaranteed by the enabled gates (E9). To avoid this, manual quality processes should still be in place as part of the documentation life cycle. Finally, one of the experts (E11) was of the opinion that selecting the criteria for the quality gates, could be impossible in complex ICDs, particularly when clarity (beyond terminology consistency) is what needs to be enforced.

According to the experts’ viewpoints on Question 4.1, the feature of Centralized-documents tracking is, in comparison to the previous ones, the more transferable one, as seen in Fig. 11 (with 10 out of 12 scores betwen ‘a lot’ and ‘to a great extent’). According to Fig. 12, however, the experts were almost evenly divided between its applicability as the single source of truth and as a means to reduce duplicate maintenance efforts. Nevertheless, from the analysis of Question 4.3 (Please elaborate on your answer to the questions above) a prominent theme emerged from the comments of seven out of the 12 experts that would explain most of the aforementioned positive views on transferability: the dependency-tracking capabilities this feature enables between documents. In particular, two of them (E5, E8) mentioned that said capabilities are promising for ICDs as similar capabilities have shown their value in the context of requirements management tools like DOORS¹⁷. The other five pointed out that keeping track of specific definitions within the ICDs and not just the whole ICD, and in addition to the quality check (E6), would be of great value to improve traceability and to ensure proper actions are taken when a change is made. These definitions include interface requirements, diagrams and data schemas (E4, E10, E12). Furthermore, tracking the changes on these definitions by the subsystem/capability they belong to would be key in keeping said systems up to date (E1).

Two remaining (and less prominent) themes were identified from the experts’ viewpoints on Question 4.3, namely applicability benefits and applicability considerations. The former corresponds to three viewpoints that highlight the centralized index feature as useful to make sure that changes will not go by unnoticed (E9, E11), particularly in cases where there are different versions of a platform for which the interfaces are defined (E2). The latter includes remarks from two experts: first, in some cases the process of updating an ICD is very formal, and hence there is no actual risk of having multiple versions of ICDs (E12). Second, a centrally managed ICD cannot be, strictly speaking, considered as a single source of truth given the numerous (and independently developed) models or views it puts together, but rather an ‘authoritative’ source of truth.