Systematic visual analysis of groundwater hydrographs: potential benefits and challenges

The following methodological considerations are primarily concerned with similarity analysis and classification, but will be applicable to many other uses of visual analysis as well. To achieve a meaningful visual comparison, plots do need to show differences between time series as clearly as possible, but otherwise they need to be identical (scaling, layout). This may sound obvious but is easier said than done in practice. Many choices exist, yet many of the best ones may not be feasible (for example, the plots required may be too large). The choice of plot has a significant impact on the visual appearance, and thus the analysis results. Choices include, for example, the length of the plotted time interval, temporal resolution, graphical resolution, standardization, and how to deal with gaps (either not plotting or filling with straight lines) among others. Even line thickness, colours and symbols have great impact. The requirement for identical layout requires harsh compromises. As groundwater head data are expressed either in meters above sea level or depth below surface, the subtraction of the mean (or shifting graphs along the y-axis) seems mandatory. Groundwater level fluctuations have very different magnitudes (in this dataset, from a few centimetres up to 15 m within a decade). Z-scores are an option to achieve comparable y-axis scaling, with the disadvantage that differences in magnitude, a significant hydrogeological feature, are lost. The solution taken here was to work with two sets of plots—one with just the mean subtracted, the other showing z-scores. Different plot lengths were also tested: 1, 3, 10 and 30 years. As time series start and end at different dates and have different lengths, plot periods become short if one wants each plot to have the same start and end dates. Longer plot periods with different start and end dates (the chosen option) have the disadvantage that particular events (e.g. a significant drought or flood creating characteristic peaks) may be visible in one plot but not the other. Also, within a short period such as 1 year, signals with long wavelengths are lost, while long periods (30 years) mean that high frequency signals (visually) disappear if plots are not made extremely large. One option tested to overcome this issue was to use EEMD—ensemble empirical model decomposition; Wu and Huang (2009)—plots that show different frequency components of time series in parallel (Fig. 3e).

While different styles of plots emphasize different characteristics of time series, using many different plot styles in parallel has proven unfeasible and confusing, and a consistent classification of time series, taking several different features into account, cannot be achieved. A significant disadvantage of visual techniques is that the human brain, while very good at detecting differences in a small number of objects of the same kind, can only process a rather limited quantity of different information at a time.

In Fig. 3a–d, plots clearly show the impact of standardization on plot appearance: the pairs (a and c; and b and d) show the same hydrograph, one with only the mean subtracted, the other as z-scores. Standardization has the advantage that all plots can be more easily fitted into a chosen plot area with equal size and axis scaling; however, it changes the appearance, drastically in some cases, whereby some characteristics are understated, while others are exaggerated. Again, it depends on the overall objectives of a study as to what is most appropriate. As always, “appropriate” is a relative term: it depends on the research question or hypothesis to be tested. In a study on the impact of long-term climatic changes on groundwater, high frequency signals may be less important, while those might be extremely interesting in a study on shallow groundwater dependent eco-systems. Figure 3e shows an EEMD plot of the same time series shown in Fig. 3b,d. Splitting the time series into different frequency components can be very helpful but, overall, this style of representation was not found to be very helpful in the classification stage of a classification; the additional information provided cannot be “processed” meaningfully. It may prove helpful in a stage where one goes from simple visual characterization to conceptual interpretation in relation to hydrogeological conditions.

In addition to the technical challenges associated with plotting, other challenges arise from the nature and quality of groundwater hydrographs. As described in section “Study area and data”, many hydrographs are irregular, have different length and measurement intervals, gaps, etc. It is thus not always possible to visually compare hydrographs with identical technical descriptions—i.e. identical length, start date or measurement intervals—if one wants to include a large enough number of time series in the comparison. Compromises have to be made. In this context it is worth mentioning that of the more than 5,000 time series available in the projects, only about 1,300 were selected for the visual classification exercise. The rest were removed prior to visual inspection using automatic procedures. Reasons to remove time series from the data set were for example: very short time series, time series with start and end dates far outside the range of the majority of time series, time series with very long gaps, many gaps or very irregular measurement intervals. The ~100 time series classified as “irregular” in the present study, are those which passed the automatic quality control, but were found irregular on closer visual inspection.

The final preprocessing and plotting options chosen were:

One of the most interesting methodological questions in relation to classification of time series is “What makes two time series appear similar or dissimilar?” Looking at the examples shown in Fig. 2a,b, most people will likely agree that the plots are quite similar, while the plot in Fig. 2c is clearly dissimilar from plots of Fig. 2a,b. In this case, where plots show strong similarity, to arrive at this conclusion is straightforward, a simple glance is sufficient. It becomes much more difficult when patterns are mixed or irregularities occur (see Fig. 2 and associated text). Figure 2e, for example, has features that resemble Fig. 2c and also Fig. 2a. How to deal with this is a great challenge and, ultimately, the point where subjectivity dominates the process. Without elaborating further on this, there are different options to visualize time series classification according to similarity. One can:

The first attempts at visual classification in this study were carried out according to option 1. They were started from a set of unique patterns (peak symmetry, upper/lower bounds, flashiness, etc.), after which it was attempted to identify the time series where those patterns occurred. Figure 4 shows four selected examples of “eye-catching” patterns. Some attempts were made to create an inventory of patterns based on verbal descriptions (“symmetric”, “flashy”) and sketches. Attempts were also made to explain potential hydrogeological causes. While visual pattern identification may be a valid and even less subjective approach to visual classification, paving the way for semiautomated or automated classification—see section ‘Comparison with index-based classification’ and Heudorfer et al. (2019)—it was finally decided not to follow the approach as it proved to be extremely tedious. The main challenge is that most time series do not show patterns in pure form, but show rather the superposition of many patterns. In Fig. 4, the uppermost row shows simplified versions of several patterns which frequently occur in the dataset. The middle row of Fig. 4 shows examples of time series that show the same patterns in relatively pure form. The examples in the lower row of Fig. 4 show superpositions of the respective pattern with other patterns. The eye seems well able to distinguish between basic patterns in such a mixture; the challenge remains to decide what pattern, or mixture of patterns, is characteristic for a specific group, subgroup or type. In relation to the attempts to explain patterns using expert hydrogeological knowledge, it should be noted that this may introduce a strong bias and wishful thinking (“this is a shallow unconfined aquifer and thus must go into this group”, even if it looks different from time series from similar locations) and should probably best be avoided. It should also be noted that visual classification essentially struggles with the same issues as formal, algorithm-based classification—in this example, it is the struggle between structural similarity and shape-based similarity as explained in work such as Lin and Li (2009).

The results presented in this paper were reached exclusively using overall visual impression. It should be mentioned that even when one strives to carry out a characterization of the overall visual impression only, one automatically starts to look for patterns, even if those are not systematically described and named.

The classification process was carried out stepwise over several iterations. The main approach used was to display miniature plots on two computer screens in parallel with 20–80 miniature plots on each screen. The entire dataset with unsorted plots was shown on one screen, while a new folder with sorted plots was shown on the other. Plots that appeared similar were selected from the “unsorted” folder displayed on screen 1 and dragged and dropped to the sorted folder on screen 2. First, a very coarse sorting was carried out, leading to a small number (here, 9) of rather large groups. In the next step, subgroups were formed within these large groups, using the same drag and drop procedure as before. Subsequently, subgroups were split into sub-subgroups, called “types, using the same approach. In a final step, each type was compared with the other types and plots were redistributed to achieve maximum similarity in each subgroup. Cross-checks with other plot styles (see previous section) were carried out, and parts of the process were iterated several times. The “type-to-type” comparison sometimes led to splits, joins and removal of types, subgroups and groups.

The process resulted in a hierarchical scheme consisting of groups, subgroups and types. Groups were named accordingly using a a 3 digit scheme (X,Y,Z), where X identifies the group, Y the subgroup and Z the type.

A strictly hierarchical, exclusive classification (meaning each hydrograph belongs to exactly one type) leads to ambiguous results. Many hydrographs could belong to one type, but equally well to another. Visual inspection as used in this study does not provide the means to come to clear decisions due to the lack of objective criteria. For that reason, the authors have experimented with “fuzzy classifications” where a hydrograph can belong to more than one type. The methodology used to achieve the fuzzy classification and the respective results are presented in the electronic supplementary material (ESM).

The index-based classification scheme is described in detail in Heudorfer et al. (2019). The fundamental idea is as follows—for each time series, a set of indices is calculated, each index expressing one characteristic feature (a “pattern”) of groundwater dynamics. Heudorfer et al. (2019) developed a typology of groundwater dynamics with three major categories of features, namely structure, distribution and shape, each having several subcategories. Structure, for example, has the subcategories seasonal magnitude, seasonal timing, interannual variation and flashiness. Within each subcategory, one or more indices are defined—for example, the Richard-Baker index, in the subcategory flashiness is the “sum of absolute values of day‐to‐day changes in head divided by the sum of scaled daily head” (Baker et al. 2004). The index Average seasonal fluctuation, belonging to the category “Seasonality magnitude”, is the “Mean annual difference between the averaged 3 highest monthly groundwater heads per year and the averaged 3 lowest monthly groundwater heads per year” (Martens et al. 2013). Indices were calculated for 10-year time series of weekly and 5-year times series of daily data.

Heudorfer et al. (2019) considered a total of 62 indices, collected from different studies in hydrology and related fields of science. Of those 62, nine indices representing the previously mentioned subcategories were selected for comparison with the visual classification results (Table 1).

To compare the visual classification results with the index-based approach, mean and standard deviation for the selected indices from time series within each type, subgroup and group found through visual classification were calculated. A semiquantitative comparison was carried out with the aim of determining whether similar time series have similar index values.

Within the scope of the wider research framework presented in this article, one main objective of time series classification is to be able to establish and use dependencies between time series characteristics (dynamic behaviour) and hydrogeological conditions. This has the ultimate aim of being able to use the found dependencies to make predictions. Therefore, the question as to whether a found classification scheme has any relation to hydrogeological conditions is a crucial one to be answered. It is assumed that the index-based classification introduced in the previous section shows some degree of relation to hydrogeological conditions, as many of the indices are based on theoretical and empirical considerations of the dynamic behavior of groundwater resources. The visual classification based on “visual appearance” only, however, cannot rely on such assumptions; therefore, the authors found it beneficial to take a closer look at this question. To do so, how selected numerical descriptors from a dataset containing geological and borehole information related to the found classification scheme were analysed. For the purpose of this article, this analysis was only carried out qualitatively. An in-depth quantitative analysis of the same question, based on indices, is described in Haaf (2020) and Giese et al. (2020). Those articles also explain how the descriptors used were developed and why and how selections for the purpose of those analyses were made.

For the German dataset of 841 time series, the procedure described in section ‘Visual comparison and classification of time series’ resulted in 82 visually distinguishable types within 28 different subgroups and nine groups. Up to 31 individual time series were grouped into one type, while a few types had only two or three members. Additionally, there were about 29 time series that did not fit into any of the found types; thus, these were classified as “not fitting anywhere”. Seventy-one hydrographs were put into a group called “irregular”. Note that all time series with obviously peculiar behaviour, outliers, gaps etc. were removed from the dataset before starting the process of visual classification described here. The irregular time series detected here were, therefore, mainly those that were not spotted in the preparation stage. Plots of all the time series and classifications made are available in the ESM (Barthel et al. 2020).

Figure 5 shows the diversity of the time series used in this study, an overview of the used hierarchal classification scheme and a flavour of the achieved classification. The lower panel of Fig. 5 shows examples of time series from each of the nine groups created. The panel in the middle of Fig. 5 shows six representative examples of plots from one subgroup each, each belonging to group 2. The upper panel shows three representative examples from one type that was defined within the subgroup 2.2 of group 2.

Figure 6 shows some of the challenges associated with visual similarity analysis, using examples of plots of time series from two types, one defined within group 8 (type 8.3.1), the other in group 9 (type 9.5.1). Types 8.3.1 and 9.5.1 are clearly dissimilar from each other, while the within-type similarity of type 9.5.1 appears to be higher than the one of type 8.3.1. Although this can partly be explained by the preprocessing and plotting options used, the chosen plots demonstrate one of the challenges of visual similarity analysis—the impression of similarity/dissimilarity is often a relative one, and it depends on the number of time series available. For type 9.5.1, very similar (i.e. similarity of all visible features) time series exist making them easy to group. For type 8.3.1, time series with very similar features exist—e.g. the apparent upper boundness, maxima flashier than minima, overall “noisy” appearance, no strong interannual variations—yet those features are somewhat different in each of the given examples. There is ultimately no way to express degree of similarity clearly or to describe the nature of similarity in visual analysis; it remains subjective and unstructured. To overcome this, further moves towards quantitative analyses have to be made.

Figure 7 (upper two rows) shows examples of time series plots that were classified as being “irregular” and thus excluded from classification. The lower row of Fig. 7 shows examples of time series which are regular, but classified as “not fitting anywhere”—there is no other time series that resembles each of them. As mentioned before, irregularities in time series are of significance, both as “disturbing features” that introduce errors and uncertainty, and as additional information on groundwater systems and system responses—whatever creates strange/irregular and exceptional features may inform us about the influence of properties and processes or changes of these. Much may be learned from analyzing those more closely. It is important to point out that smaller or larger irregularities occur in almost any groundwater hydrograph. Without being able to prove this, the authors claim that many irregularities can be identified much better through visual inspection than by formalized automated approaches. Please note, that the term “noise” in relation to irregularities is avoided here, as identifying signals as noise may imply that they are not the results of the system responses one is interested in. The irregular features, however, may very well be “real” signals, yet created by an unknown and unusual process.

The visual classification of the Swedish dataset was carried out independently (i.e. each observer used their own approach), and the results were compared. The details of this comparison do not add much to the overall conclusions other than confirming the subjectivity of the approach. On a “large scale”, i.e. at the group level, results concur largely, while overlaps are fewer on the subgroup and type levels. In that sense, visual classification does not differ from automatic procedure, where the fine-grained classification results depend on, e.g. the chosen distance measures or clustering algorithms. Figure 8 shows examples from one subgroup created in the joint classification of Swedish and German data. Similarity between Swedish and German data is usually quite high at the subgroup level, but less strong at the type level. One main reason is that the Swedish dataset was created from bi-weekly measurements, usually leading to a smoother appearance. The second reason is that in many of the very shallow and thin aquifers in Sweden, groundwater levels frequently drop below the well bottom or reach the ground surface (wetlands). This creates an appearance of upper and lower bounds. In general, the diversity of types in Sweden is much lower, as there is less variety among hydrogeological settings—for example, hydrographs belonging in some of the subgroups of group 1, associated with deep confined aquifers, are missing in the Swedish dataset.

The results presented so far represent an “exclusive” hierarchical classification—each time series belongs to one type only. However, results remain unsatisfactory—there are many cases where one time series fits in several types, depending on which of its visual features is weighted highest. This leads to an ambiguity of classifications, which can be explained by the mixture of patterns as demonstrated in Fig. 4. In-between-type similarities across groups and subgroups exist. To explore this further, the between-similarities and within-similarities of types, subgroups and groups were determined visually using a similarity matrix (section ‘Visual comparison and classification of time series’). As the results of this supplementary approach are not used to support the conclusions of this paper, they are, together with further explanations, shown in the ESM.

Some of the findings of the visual classifications are summarized thus:

Readers may be interested to learn how much time is needed to perform a meaningful visual inspection and classification of a dataset of around 1,000 hydrographs. Based on the study presented here, this is difficult to answer; to carry out the study and to arrive at the final classification presented, it took several months. However, much of this time was not spent on the actual inspection and classification but on testing different approaches and ideas and the comparison with other approaches. Also, due to the lack of hard and objective criteria, it is hard to determine when one is done, i.e. when a satisfying result is achieved. Nevertheless, to provide some ideas, a rough estimate for a set of 1,000 time series, assuming that preprocessing and plotting decisions have already been made, would be that a reasonable classification at the group level can be made within several hours. From there, a refinement to subgroup level will require about 1 or 2 more days. Type level classification can be achieved in an additional week. A fuzzy classification, involving a type-to-type comparison will require another 2–3 days. For a set of 100 time series, everything, including fuzzy classification could be done in 1 day.

To compare the results of visual classification to index-based classification, Fig. 9 shows the averaged within-subgroup values of nine selected indices. It can be seen that index value distributions show a distinct pattern for most groups and subgroups. Within groups, most indices usually have the same sign and magnitude, while one or two indices can differ significantly between subgroups of one group. There are, however, also exceptions—based on their index values, subgroups 1.4 and 1.5 are much closer to the subgroups in group 2 then they are to the other subgroups in group 1. It would definitely be interesting to look into this more, yet it is outside the scope of this article to discuss this further.

In many cases, index values can be easily explained by the visual appearance (and vice versa)—for example, the pulse.count.h index is low for the rather smooth time series that is dominated by long wavelengths in groups 1, 2 and 3, but has higher values for the rather flashy groups 8 and 9 (compare also with Fig. 5). The pulse.count.h index value for subgroup 9.5 is, on average, 0.67, while the same index value is 1.63 for subgroup 8.3, clearly reflected by the visual appearance of the given examples from these subgroups.

To demonstrate how sets of index values for individual time series relate to the actual visual impression of a time series, Fig. 10 shows the index values of four distinct time series taken from pairs of very close types along with selected plots of those types. The subgroups those types belong to were chosen as they demonstrate strong between-subgroup similarity (2.2 and 2.3, 8.2 and 8.3, respectively), or strong between-subgroup dissimilarity (subgroups from group 2 versus group 8). According to the index profiles, times series ID1550 and ID1369 seem to be very similar, while the visual appearance suggests some differences (thus two different subgroups). ID1825 and 1325, grouped into the same subgroup but different types therein, are also quite similar index-wise (apart from index pulse.count.h). Visually, the different appearance is mainly due to a smoother lower bound in 1325.

Figure 11 shows the averaged, normalized (z-scores) values of six selected, commonly available observation well properties (descriptors) for all subgroups, determined through visual classification. All descriptors are related to thickness and depth, apart from elevation. For more information on the descriptors used in this study, please refer to Giese et al. (2020) and Haaf et al. (2020).

More comparisons between visual classification results and hydrogeological settings are shown in Barthel et al. (2021). Figure 11 indicates that there is a strong, yet not unambiguous relationship between hydrogeological properties and types of groundwater dynamics as determined with visual classification. A deeper analysis of the relationships shown in Fig. 11 would increase the understanding of groundwater system responses to change, but this also is outside the scope of this article. The authors would once again like to draw the reader’s attention to the other studies published by the authors within the framework of this research, as well as the PhD dissertations presented by Haaf (2020) and Heudorfer (2019).