Challenges in Validating FLOSS Configuration

We study getenv, which is an application programming interface (API) to access environment variables. We chose it because it is the only widely standardized configuration access API (included in C, C++, and POSIX standards) and available in many programming languages. In earlier work [26], we showed that getenv is used at run-time ubiquitously. getenv is often combined with other techniques, for example, overriding configuration file settings. Furthermore, environment variables are not part of configuration settings dialogues, i.e., they are usually not validated before reaching the application.

We carefully selected 16 applications across different domains. We included large applications with a thriving community but also others for diversity. We used the versions of the applications as included in Debian 8 (Jessie) as shown later in Table 1. We downloaded package sources from http://​snapshot.​debian.​org. To determine the code size we used Cloc 1.60 [7].

We manually counted all getenv occurrences for the version specified in Table 1. Then we categorized the resulting 2,683 code snippets around getenv. We looked if getenv occurrences depend on some other configuration. Such situations occur when configuration settings interact; for example, fallback chains of configuration access points depend on each other. Such fallback chains are hints to global configuration access, which we wanted to find. As our last experiment, we searched for places where global validation would be useful, and investigated how helpful the documentation of the getenv parameters is.

We carefully prepared a questionnaire with FLOSS developers in mind. Then we conducted pilot surveys with developers, colleagues and experts for surveys. In the iterations we improved the questions and made the layout more appealing.

In order to reach the target group, we posted requests to fill out the survey in the respective FLOSS communication channels. To obtain a higher quality, we awarded non-anonymous answers with small donations to FLOSS-related projects. We used the non-anonymous answers to cross-check statistics.

We asked some personal questions about age, education, occupation, and FLOSS participation to have the important characteristics of our participants.

We used Limesurvey version 2.50+ for conducting the survey. We will report the percentages relative to the number of persons (n) who answered a particular question. We report means and standard deviations (s) of samples for \(n\ge 95\). We used the Kolmogorov-Smirnov test [15] for smaller samples.

The survey reflects the beliefs of participants. Thus we used other methods to distill facts about the applications. Because opinions help to understand goals and reasons, the survey is an important part of the overall study. It should be considered as supplement to the source-code analysis.

Demographics: The front page of the survey was shown to 672 persons, 286 gave at least one answer, 162 completed the questionnaire, and 116 persons entered their email addresses. The age of the population (\(n=220\)) has a mean of 32 years (\(s=9\)). The degrees in the population (\(n=244\)) are: master (38%), bachelor (25%), student (18%), no degree (13%), or PhD (6%). As their occupation, 56% of the persons selected software developer, 21% system administrator, and 16% researcher (multiple choice question, \(n=287\)). Participants reported work on up to five different FLOSS projects. For the first project, they estimated their participation with a mean of 5.3 years (\(s=5\), \(n=180\)). 60% of them reported a second FLOSS project, 36% a third, 17% a fourth, and 9% a fifth.

Raw data and questions are available at https://​rawdata.​libelektra.​org.

Finding \(\varvec{1}_{a}\) : We observed that getenv is omnipresent with 2,683 occurrences.

The source code of the applications we analyzed has 4,650 textual getenv occurrences. 2,683 of them were actual getenv invocations, 1,967 were occurrences in comments, ChangeLog, build system, or similar. (See Table 1 for details.)

Finding \(\varvec{1}_{b}\) : Three kinds of configuration access are equally popular: Command-line arguments, environment variables, and configuration files. Developers are highly satisfied with them. Others are used less, and are more frustrating to use.

Command-line arguments (92%, \(n=222\)), environment variables (e.g., via getenv) (79%, \(n=218\)), and configuration files (74%, \(n=218\)) are the most popular ways to access configuration. Other systems, such as X/Q/GSettings, KConfig, dconf, plist, or Windows Registry, were used less (\(\le \) 13%, \(n\ge 185\)).

Participants rarely found it (very) frustrating to work with the popular systems: getenv (10%, \(n=198\)), configuration files (6%, \(n=190\)), and command-line options (4%, \(n=210\)). Less-used systems frustrated more (\(\ge \) 14%, \(n\ge 27\)).

Finding \(\varvec{1}_{c}\) : Like other configuration accesses, getenv is used to access configuration settings (57%). Sometimes it bypasses main configuration access.

Of the 2,683 getenv invocations, 1,531, i.e., 57%, relate to run-time configuration settings and not debugging, build-system, or similar. Other investigations in this paper discuss these 1,531 getenv occurrences. (See Table 1 for details.)

Also in the survey we asked about the purpose of getenv (\(n=177\)). The reasons to use it vary: in a multiple choice question 55% say they would use it for debugging/testing, 45% would use getenv to bypass the main configuration access, and 20% would use getenv if configuration were unlikely to be changed.

Finding \(\varvec{1}_{d}\) : In many cases getenv parameters are shared between applications.

In the source code we investigated which parameters were passed to getenv. We found that 716 parameters were shareable parameters such as PATH. In the survey 53% say they use getenv for configuration integration (\(n=177\)).

Finding \(\varvec{1}_{e}\) : Parameters of getenv are often undocumented.

The function parameter passed to getenv invocations tells us which configuration setting is accessed. In an Internet search using the application’s and getenv parameter’s name with https://​startpage.​com, we found documentation for only 283 of the non-shared getenv parameters but not for the 387 others.

The FLOSS projects deal with the missing documentation of getenv parameters in different ways. Most projects simply claim their getenv usage as internal, saying the environment should not be used for configuration by end users, even if there is no other way of achieving some goal. Often we miss a specification describing which parameters are available.

In other projects, the developers invest effort to create lists of available parameters. For example, in LibreOffice developers try to find getenv occurrences automatically with grep, which fails with getenv aliases¹.

Discussion: The getenv API has some severe limitations and is sometimes a second-class citizen. One limitation is that return values of getenv invocations cannot be updated by other processes. For example, getenv("http_proxy") within a running process will still return the old proxy, even if the user changed it elsewhere. Another limitation is that they do not support persistent changes by applications. Configuration files, however, are not easily shareable.

Implication: There is currently no satisfactory solution in FLOSS for global, shareable configuration settings. getenv supports all characteristics of configuration access and can be used to investigate challenges in configuration validation.

Finding \(\varvec{2}_{a}\) : Developers have concerns about adding dependencies for global validation (84%) and reducing configuration (30%) but desire good defaults (80%).

Many persons (30%, \(n=150\)) think that the number of configuration settings should not be reduced. But 43% said it should be reduced to prevent errors.

We got mixed answers (\(n=177\)) to the question “Which effort do you think is worthwhile for providing better configuration experience?” Most persons (80%) agree that proper defaults are important. Most methods exceed the effort considered tolerable by the majority of participants: Only getenv would be used by the majority (53%). System APIs would be used by 44%. Fewer (30%) would use other OS-specific methods, such as reading /proc. Only 21% of the participants would use dedicated libraries, 19% would parse other’s applications configuration files, and 16% would use external APIs that add new dependencies.

Finding \(\varvec{2}_{b}\) : Present configuration validation is encoded in a way unusable for external validation or introspection tools.

In none of the 16 applications was the validation code kept separately, e.g., in a library. Instead it was scattered around like other cross-cutting concerns.

Finding \(\varvec{2}_{c}\) : Developers are unable to support global validation, even if the problem is well-known and they put effort into it. We found out that information essential to check or fix constraints is not available within the applications.

In Table 1 we present the list of applications we analyzed. The column counted getenv lists our manual count of all getenv invocations. The column config getenv shows getenv occurrences used for configuration as described in Finding \(1_c\). The column depend getenv presents manually counted getenv occurrences that depend on, or are used by, other configuration code. The last column lines per getenv shows how often manually counted getenv occurs in code.

Most of these places (1,115, i.e., 73%) were dependent on some other configuration. We found 204 places where some kind of configuration dependencies were forgotten. In 58 cases we found several hints, e.g., fallback chains with missing cases or complaints on the Internet about the not considered dependency.

We give a real-life example from the Eclipse source of how easily dependencies are forgotten. The missing dependencies lead to missing validation, which leads to frustrating user experience. If Eclipse wants to instantiate an internal web browser, some users get an error saying that MOZILLA_FIVE_HOME is not set. On GitHub alone, issues mentioning the variable were reported 71 times. The underlying issue usually is that the software package webkitgtk is missing². The developers even considered the dependency (installation paths) for RPM-based systems by parsing gre.conf. But for other users on non-RPM-based systems the fallback is to query MOZILLA_FIVE_HOME which leads to the misleading error. In Eclipse the workarounds (including parsing gre.conf) needed 213 lines of code. Furthermore, most of the 9006 code snippets we found on GitHub referring to the variable are small wrappers trying to set MOZILLA_FIVE_HOME correctly.

Requirement \(\varvec{1}_{a-c}\) : Developers use different mechanisms for configuration accesses interchangeably or to bypass limitations of others. To avoid the need for bypasses, Elektra bootstraps itself at startup, making it possible for configuration settings to describe the configuration access, for example, which configuration files should be used. To allow administrators to use all popular techniques, Elektra reads from different sources such as configuration files, system settings, and environment variables. Elektra integrates many different configuration file formats such as INI, XML, etc., and it supports notifications to always keep application’s configuration settings in sync with the persistent configuration settings.

Requirement \(\varvec{1}_{d}\) : FLOSS developers demand a way to share configuration settings. We implemented a layer similar to a virtual file system, which enables applications and system administrators to mount configuration files [29]. This technique facilitates applications to access configuration settings of any other application. Using links and transformations [22] one can even configure applications to use other settings without any support from the application itself.

Requirement \(\varvec{1}_{e}\) : There should be a way to document configuration settings. Elektra introduces specifications for configuration settings [24]. These specifications should also include documentation. But even if they do not, users at least know which configuration settings exist, and which values are valid for them.

Requirement \(\varvec{2}_{a}\) : Dependencies exclusively needed for configuration settings should be avoided. Elektra introduces plugins that enable a system-level dependency injection. Developers specify validations in specifications, without the need for their application to depend on additional external libraries. In plugins executed on configuration access, the settings get validated or default settings get calculated. Elektra only uses the C standard library and no other external dependencies [20]. Nevertheless, even the dependency on Elektra itself can be avoided. Elektra supports intercepting of library calls such as getenv [26, 27]. Using this technique, applications think they use environment variables, while in reality they query Elektra.

Requirement \(\varvec{2}_{b}\) : Configuration settings and validations should be open to introspection. Similarly to getenv, Elektra provides an API, but it aims to overcome the limitations of previous abstractions and interfaces. Elektra allows many configuration files to be integrated with a uniform key/value API. Even the specifications of accesses, dependencies, and validations are accessible via the same API. Thus system administrators and applications can use the API to introspect configuration settings. Elektra relies on file system permissions to restrict access to configuration files.

Requirement \(\varvec{2}_{c}\) : Global validation should be supported. Elektra supports global validation through a range of different checker plugins. These plugins do not only check data for consistency but also check if configuration settings conflict with reality. For example, one plugin checks for presence of files or directories, while another plugin checks if a host name can be resolved. Checks are executed whenever Elektra’s API is used for writing. This way also all administrator tools sitting on top of Elektra reject invalid configuration settings. Elektra also allows to integrate system information such as hardware settings via plugins.

Then the goals shifted towards a more technical solution: We avoid marketing campaigns to persuade everyone to use the API with arguments like “it will give benefits once everyone migrates”. Instead it should offer immediate advantages previous APIs did not have. This meant Elektra went into a highly competitive market facing the difficulty of being better than any other configuration library. As a distinctive feature, we started to aim for global validation but without giving up compatibility to current configuration files. We avoid an API that forces our ideology, like our favourite configuration file format or validation, onto users.

These changes made the core more complicated, which led to a recruiting problem. The documentation was for a long time only a master thesis [20], which was a very unsatisfactory situation. The next efforts were to make the project community-friendly again. We started to improve quality by regression tests, fixing reports of code analysis tools, and adding code comments and assertions. Then we started overhauling documentation, writing tutorials, and created a website. Last but not least, we started releasing every two months with fixes and new features in every release. These changes led to more than a dozen contributors, Elektra being packaged for many distributions, and acceptance of a paper on Elektra in “The Journal of Open Source Software” [23].