Assisted design of data science pipelines

The general idea of our proposed recommendation process is first to select a subset of DS pipelines that are relevant to the user and then rank them. This way, DORIAN avoids unnecessary computation for ranking DS pipelines that do not match the user’s input. DORIAN supports two types of suggestions—initial recommendation when the end-user does not specify any initial pipeline at the beginning of the User Interaction Cycle and recommendations for incremental improvements when the end-user has already selected an offset pipeline as part of the query. The pseudocode of the entire recommendation process is shown in Algorithm 1. Lines 3–6 concern the initial recommendation and line 8—suggestions for incremental improvements. The rest fits both.

Pipeline Candidate Selection. We distinguish two processes for selecting a pool of pipeline candidates: one for the initial recommendation and one for the recommendations for incremental improvements. Given a query (D, T), the Recommendation Engine provides an initial recommendation by finding pipelines that were previously executed on datasets similar to D and filtering out the ones that do not solve task T (line 11). This selection ensures the exploration element in the recommendation process. For incremental suggestions, the Recommendation Engine selects candidates that are similar to the pipeline chosen in the last user interaction cycle (line 9). This ensures exploitation in the recommendation. In both cases, the Recommendation Engine efficiently retrieves similar datasets and pipelines from the Experiment Store (Sect. 6).

We formulate the problem of candidate ranking as the problem of multi-objective sorting. We map every pipeline candidate to a small numeric vector that contains the values of the identified objectives (described in the following subsection) in a specified order. A pipeline candidate A is said to dominate over pipeline B if and only if, for every objective, the numerical values of A are greater or equal to the values of B, and there exists at least one value (i.e., objective) where its value is strictly greater. Formally, given a set of pipeline candidates \(\mathcal {C}\) and function \(f_o(p)\) that returns an array of objectives for pipeline \(p \in \mathcal {C}\):where \(f_o^k\) denotes the k-th objective returned by \(f_o\).

When comparing two pipeline candidates, not all objectives will be greater than or equal for one candidate to declare domination over another. For this reason, we resort to the problem of non-dominated sorting. Specifically, we utilize the modified Generalized Jensen algorithm [12, 25]. Candidates have rank 0 if they are not dominated by any other candidate and rank i if they are dominated by at least one candidate of rank \(i - 1\). Solutions with equal ranks are considered “equally good”. The algorithm splits candidates into a sequence of Pareto fronts of equal rank. Ranking within one front is based on domination in the first k objectives.

This approach provides a flexible solution where end-users can alter, tune, and extend the list of objectives in a straightforward way. For example: (1) A user wants to “pin” an operator and prioritize suggestions where this exact operator exists in the pipeline. Then, they can simply add the first objective to be a binary metric that denotes the presence of an operator; (2) A user wants to take the concepts of explainability and fairness of the suggested pipeline candidates into account. The corresponding metrics can be added to the list of objectives as well. Such extensions can be easily implemented with DORIAN. The end-user can adjust the ranking objectives by adding or removing an objective, changing their order, or creating a custom one (e.g., prioritizing candidates that demonstrated low average execution time on similar data or contained a particular set of operators), thus tuning the recommendations. The only requirement is that any objective has to be computed with the available data and used in the specified order. Other solutions, such as learning-to-rank algorithms [31], would require re-training of the classifier and preparation of a completely new training set that represents the ground truth in accordance with newly established objectives. They are, therefore, not suitable for our problem.

To provide a flexible user-tailored recommendation engine and support multi-objective ranking, we propose the following interface, which can be used to define a set of numerical objectives:

where candidate is a pipeline candidate to be ranked, query is a user query (i.e., data, DS task, offset pipeline, and evaluation process), store is a reference object that enables access to the content of the Experiment Store, and score is a numerical value returned by the user-defined objective. In the following, we describe the default objectives that DORIAN supports and how we represent them as numerical values for both initial and incremental recommendations (see Algorithm 2). Users can easily add new ranking objectives if they implement the above interface and return a numerical value.

Before defining the objectives, it is crucial to devise a metric of pipeline performance that can be used in cases where quantifiable evaluation metrics are absent, as well as across different evaluation metrics, if available. For this, we define the metric of pipeline preference ratio (PPR) as follows (Algorithm 2, lines 16–22). For a given dataset d and pipeline p, PPR(p, d) is a ratio between the number of times p was preferred by any end-user as a superior candidate during pairwise comparison on d and the number of times p was used with d in total. When the evaluation process specifies a quantifiable performance metric (e.g., ROC AUC score), we compute PPR as a percentile of the candidate’s performance, given the performance distribution that other pipelines exhibit on the same data (Algorithm 2, lines 17–18). In the majority of cases, two ways to define PPR (with or without quantifiable performance metric) are mutually exclusive—we do not need to rely on pairwise comparison when a quantifiable performance metric exists. To support all use cases, we compute the average between the leaderboard percentile (scaled from 0 to 1) and the fractional notation of PPR (each defaulting to 0 when no signals exist to compute the metrics).

Initial Recommendations. Initial recommendations constitute the beginning of the User Interaction Cycle where no offset pipeline is specified in the user query. This case provides the least amount of information to determine candidate ranking. Algorithm 1 (lines 3–6, 10–14) depicts the procedure for initial candidate recommendation. We define two objectives of the initial recommendation:

Incremental Recommendations. The procedure for recommending incremental improvements is depicted in Algorithm 1 (lines 8–18). Note that pipeline candidates that were previously discarded by the end-user (within a single user query, line 11) or the Execution Engine (in case of raised exceptions) are removed from the set of candidates and do not get any rank assigned. When choosing a pipeline candidate from the pool of suggestions, more user input becomes available—pipeline candidates chosen, altered, discarded, executed, and compared to one another. Thus, the ranking objectives shift from exploration to exploitation, prioritizing pipeline candidates that are relevant to the immediate past of the user interaction rather than the “good on average" ones. In that case, additional requirements and, thus, objectives become necessary:

Storing all the datasets used in the experiments would require significant storage capacities. For this reason, we choose to store only a numeric vector of meta-features that serves as a lightweight data footprint for previously used datasets. The benefit of doing so is twofold: data footprints are space-efficient and can be used to efficiently search for similar datasets that past experiments applied to. We compute a small set of meta-features, which are a subset of the ones used in the auto-sklearn library [21]:The rationale behind the selection of these meta-features is twofold. First, they are quantifiable characteristics of the task: They represent the signals of pipeline scalability (number of instances), data imbalance (ratio of the least frequent class to the most frequent class), feature normality and informativeness (Skewness, Kurtosis, Entropy) [50]. Second, their compute time is suitable for real-time user interaction, i.e., \(< 2\)s.

For large-scale datasets, even Landmark- and PCA-based meta-features can take longer time to compute. In this case, we use a partial dataset footprint and update pipeline recommendations when the computation is completed. To get a partial footprint in real time, we compute independent meta-features in parallel and sort them in the order of faster compute times. Upon any updates in a data footprint, we repeat Algorithm 1 in an incremental manner: We take the computed list of pipeline candidates, identify new candidates that were not suggested in the previous version of recommendations, and update the ranking by fitting new pipeline candidates into the existing Pareto fronts instead of re-running from scratch.

Simply storing dataset footprints as numeric vectors is not enough because similarity search would be very inefficient: For every new dataset, we would need to (a) compute n Euclidean distances from this dataset to the n other datasets already in the Experiment Store, (b) sort the distances, and (c) select the first k datasets. Such a strategy has an average complexity of O(nlogn), which can be very impractical for real-time searches. We thus choose to utilize KD-trees [5] as an index structure for dataset similarity search that utilizes the Euclidean distance between a pair of dataset footprints as a similarity score. When the end-user comes with a new dataset, we compute its dataset footprint and insert it into the KD-tree. To retrieve datasets similar to the input data, which is required for the initial recommendations, the Experiment Store queries the KD-tree for k nearest-neighbor (kNN) dataset footprints.

This data structure is optimized for kNN search in multidimensional spaces, making it attractive for similarity search of fixed-length numeric vectors. For k-dimensional data, each tree node of depth d acts as a hyperplane that splits a set of data points into two equal-sized disjoint subsets. That is done by selecting a median data point by means of projection onto the axis \(d \mod k\).

KD-Trees enable us to provide efficient similarity search⁴ without adding storage overhead (i.e., O(n) space complexity) thanks to its O(logn) average complexity for search, insertion, and deletion. Heterogeneity of features (i.e., values of varying magnitudes) can lead to the phenomenon where low-magnitude values have a negligible effect on the distance compared to the high-magnitude values. In other words, when searching for similar datasets, the similarity of high-magnitude values might become “more important” than the similarity of low-magnitude values. To mitigate this, we use Min-Max scaling before computing the Euclidean distance.

To store previously executed DS pipelines and enable efficient search among them, we make the following design decisions. First, we represent each pipeline as a directed acyclic graph (see Sect. 3) where any DS operator consists of three elements: the specification of DS tasks that this operator is designed to perform (e.g., supervised classification), the reference to the physical, black-box implementation of the operator (e.g., the RandomForest), and the values of hyperparameters that the operator can be configured with (e.g., the number of estimators n_estimators=40). Figure 3 illustrates four DS pipelines as graphs. Each operator node can have multiple inputs, parameters, and outputs. Edges between the operator inputs and outputs represent data flow. Edges between the operator and the parameter map the hyperparameter value. Note that the graphs are simplified for illustration purposes. They are conceptually similar to dataflow and task graphs, and depict functions, their dependencies on data, parameters, and semantic annotations. Each function is defined with a code snippet written in a given programming language and, thus, is as expressive as the grammar of the language allows. DORIAN ’s Execution Engine then executes operators by forwarding their code snippets, data, and parameters to an execution runtime of that programming language. Therefore, the expressiveness of DORIAN ’s pipeline specification can accommodate pipelines of varying complexity, including production machine learning applications.

Second, to support efficient similarity search over DS pipelines, we utilize the discrete graph edit distance [2] as a measure of similarity between pipelines and the Burkhard–Keller tree (BK-Tree) [11] as an index to accelerate the search process. The graph edit distance encodes the minimal changes required to transform graph A to graph B. Given a set of DS pipelines, every node of the BK-Tree corresponds to a pipeline, and an edge in the tree depicts the discrete graph edit distance between two pipelines. The root node of the tree can be chosen arbitrarily. Every newly added pipeline is recursively moved down the tree by following the path of matching distances (i.e., edges with the distance value equal to the distance between the newly inserted pipeline and the parent node). A new subtree is created if no edge with a particular distance value exists. Pipelines with distance 0 are equal.

BK-trees require O(n) space complexity and O(logn) average complexity for search, insertion, and deletion. In practice, the time complexity depends significantly on the complexity of the distance measure and the choice of the tolerance value. The tolerance value is applied to each node to limit tree traversal to children with distances within the range \([d - tol, d + tol]\), where d is the distance between the query pipeline and the node pipeline, tol is the tolerance value. This step prunes children nodes that lead to sub-trees with “dissimilar” pipelines. As computing the exact graph edit distance is proven NP-hard [57], we utilize the DF-GED [2] algorithm for computing the exact graph edit distance that dynamically generates a tree-based space of all possible mappings between two given graphs and applies the depth-first search algorithm to find a mapping with the least number of edits. One property of this algorithm is that, depending on the input graphs, it can dynamically yield sub-optimal mappings with non-increasing edit costs. That is, a mapping that corresponds to the exact but not necessarily minimal edit distance. We turn this property to our advantage and return the first, sometimes sub-optimal, edit path with the “not-necessarily minimal” edit distance. This design decision allows us to improve search performance without sacrificing the quality of the search results, as computing the exact minimum graph edit distance is not a critical requirement for our approach.

Figure 3(b) depicts an example BK-tree of the 4 pipelines shown in Fig. 3(a). When G4 is added to the tree \(G1-G3\), it is first compared against the root node G1. Graphs G1 and G4 have graph edit distance \(d=5\) because G4 contains two new operators (min–max scaling and variance threshold feature selection, one new hyperparameter node (threshold is set to 0.2). As an edge with \(d=5\) already exists for G2, the insertion algorithm creates a new node G4 with the depth of 2 and parent G2. The edge \(G2-G4\) gets the graph edit distance \(d=6\) (change to the missing value imputation step, addition of the feature selection step, addition of the \(min\_samples\_split=3\) hyperparameter to the Random Forest estimator).

DORIAN stores user interactions in the Interaction Table (IT). This is used by the Recommendation Engine to compute the ranking objectives for any pipeline candidate that is potentially relevant to the user query. Importantly, the interaction table stores the results of pairwise comparisons between two pipeline candidates that the end-user executes. This is crucial for cases where quantifiable performance metrics are non-existent and for personalizing recommendations. Below, we illustrate the information kept in IT. Importantly, for each pairwise comparison, it records the pipeline candidates being suggested, compared, as well as the decision of which candidate is preferred by the user.

Given the source code of a DS pipeline, we first use a language-specific parser to extract the Abstract Syntax Tree (AST)  . Then, we leverage graph rewrite rules to convert a language-specific AST to a language-agnostic graph that represents the data flow of the DS-related computations and their corresponding hyperparameters (as defined in Sect. 3). Such rewrite rules are, for instance, deleting any language-specific control flow nodes from the tree. Each rewrite rule comprises two parts: the left-hand side that depicts a pattern represented as a DAG structure (being as expressive as the grammar of the underlying programming language), and the right-hand side that depicts a set of procedural commands to alter the graph, e.g., add, remove, update a node, edge, or attribute (i.e., a complete set of operations required to construct any DAG structure). We also implement a regex-based comparator for nodes and edges, enabling a more succinct and reusable rule definition. However, this functionality is used merely for convenience and does not expand nor restrict the generic method of pattern matching and the rewrite rules. The rewrite rules are applied in direct order, as applying one rule changes the graph and, hence, affects the matches of downstream rules. We specify the initial order of the rewrite rules based on their inter-dependencies but enable the end-user to change this order by adding new custom rewrite rules. To support complex multi-line DS scripts, we use a set of “dependency management” rewrite rules that generate edges between variable assignment nodes and variable references. That allows the Pipeline Extractor to parse code snippets without conceptual restriction. For code snippets with repetitive variable assignments (i.e., not in the Static Single-assignment Form) where different content is assigned to variables of the same symbolic representation throughout the source code, we provide a rewrite rule that keeps edges to the most recent assignment in code (based on the order of graph traversal and sequential application of rewrite rules) to prevent the “reference before assignment” edges. Currently, we have 10 python-specific rewrite rules and 4 sklearn-specific rules that process compositional interfaces for pipeline declaration. DORIAN parses data science scripts that consist of up to 25 expressions in under 2 seconds, and more complicated scripts of 100 expressions—under 5 seconds. Note that most rules (apart from “dependency management” and alias resolution for imported functions) are confined to a single expression and executed concurrently.

The DAG structure embraces the Composite design pattern that allows to treat a single DS operator X as a pipeline with one step (i.e., an operator can be treated as a pipeline), as well as a multi-step pipeline can be treated as a complex operator. That allows end-users to compose (i.e., group together) multiple DS operators that, in the context of a pipeline, belong to one atomic DS task, e.g., a definition of a feed-forward neural network where each layer of the network is declared separately (i.e., with a dedicated “Add Layer” operator). In the UI, the Composite principle is represented as a drill-down visualization where a high-level operator is shown, and its detailed representation (i.e., dependent operators) can be shown or hidden on demand.

To support generic pattern matching, these rules also utilize wildcards to “ignore” parts of the subgraph (e.g., variable names or syntactic sugar of a particular programming language). Based on the rewrite rules and a graph pattern-matching mechanism, DORIAN generates the DS pipeline graph  . Specifically, we utilize Sesqui-pushout rewriting [17] as the deterministic approach to graph transformation that supports “precedence of [node] deletion over preservation”—a valuable property when it comes to graph transformation by deletion; and VF2 [16]—a deterministic mechanism for pattern matching and (sub)graph isomorphism testing that supports the graph-subgraph isomorphism problem (in our work, patterns mostly represent subgraphs of a pipeline and rarely involve the pipeline as a whole), is efficient even for large-scale graphs and has reduced memory requirements compared to the related work [15, 33].

Figure 4 (step  ) depicts two example rewrite rules for the scikit-learn ML library. Here, the Pipeline class simply denotes pipeline composition – chaining multiple DS operators together in a sequence. Hence, when the full name sklearn.composition. Pipeline is matched in the AST, rewrite rule R1 simplifies the graph by removing purely syntactic nodes and chaining two operator functions together. When the pipeline has more than two operators, the rewrite is applied recursively, one per match, until all operators are chained. Rewrite rule R2 simplifies the graph by removing the Pipeline node and other purely syntactic nodes. The final pipeline contains two operators: the standard scaling mechanism for normalization of numeric data and the SVM classifier.

Once we get the syntactic DS pipeline graph composed of physical operators and their hyperparameters, we need to annotate each operator with its corresponding DS tasks. This annotation provides semantic information about the type and, thus, capabilities of each operator (e.g., whether it performs a scaling transformation or builds an SVM model). This information is crucial when searching for similar DS pipelines and leads to better recommendations. In contrast to Patterson et al., 2018 [40] who propose an approach for semantically enriching DS programs using dynamic code analysis, we focus on static code analysis only, as it does not require executing a DS script. As most DS pipelines take a significant amount of time to produce the results, it is very inefficient to extract the corresponding graph. Given the scale of thousands of DS programs that one would like to process to populate the Experiment Store from publicly available sites (OpenML, kaggle), using an approach of Patterson et al., 2018 [40] is impractical.

To enable this semantic enrichment, we construct a knowledge graph that provides information on DS operators. The knowledge graph contains a hierarchical categorization of known DS tasks. An excerpt of our constructed knowledge graph is shown in Fig. 4. Each node in the hierarchy (pink nodes) represents a concept of a DS task, while an edge denotes a generalization/specialization relationship. At the leaves of the hierarchy, we place specific classes of algorithms, e.g., SVM. These can be seen as logical operators in database terminology. We then populate the hierarchy with functions of publicly available DS frameworks⁵ (blue nodes in Fig. 4). For instance, sklearn.svm.svc is a specific implementation of an SVM algorithm provided by the scikit-learn library. These operators can be seen as physical operators in database terms. In addition, the knowledge graph contains the implementation-specific information and hyperparameters of the operators’ physical implementations (orange nodes in Fig. 4). By matching the nodes of the intermediate language-agnostic DS graph with the physical operators on the knowledge graph (blue nodes), we get a semantically annotated DS pipeline graph  . The flexibility provided by knowledge graphs allows us to handle cases where certain hyperparameters belong only to a specific implementation and not to the algorithm. We manually curate the knowledge graph, leveraging the involved contributors’ domain expertise and scientific literature. We believe full automation of this task would be impractical and would still require curation via feedback loops or voting. The user interface allows end-users to add new DS tasks locally (on a per-account basis) and extend the centralized ontology globally, pending a review.

Extension of known implementations of DS operators, on the other hand, is done semi-automatically based on web crawling of documentation websites. We implement crawlers per website and reuse existing “building blocks” when possible, as the documentation of data science frameworks and libraries is oftentimes generated automatically by code annotations and comments.

At the time of writing, we crawled sklearn and pandas as two core libraries that handled ML and data management tasks, respectively. When a new DS operator is introduced in the Experiment Store, the minimal necessary input from the end-user is the mapping between the full name of the operator’s implementation (e.g., sklearn [v0.24].preprocessing.MinMaxScaler) and its corresponding DS task (e.g., data preprocessing, normalization of numeric data). This meta-information immediately becomes available to other users, keeping the manual overhead low thanks to the collaborative environment of the Experiment Store.

Hardware, DS tasks, and datasets. We used an Ubuntu workstation with 8 Intel i7-8550U CPU cores (1.80GHz) and 24Gb RAM. As we cannot evaluate DORIAN with domain-specific tasks, operators, or evaluation processes, we evaluate it using three common DS tasks, namely supervised classification, regression, and clustering, and their common evaluation metrics. For supervised classification, we use 63 datasets and 3900 pipelines of the OpenML-CC18 benchmark suite [8] and choose ROC AUC score as the performance metric. For regression, we use 33 datasets from the OpenML AutoML Regression Benchmark (OpenML ID: 269). As OpenML contains only 300 regression pipelines available, we stored the intermediate DS pipelines that the AutoML baselines generated in the Experiment Store resulting in 1000 pipelines in total. We choose the mean absolute error as the performance metric for regression. For clustering, we use 40 datasets from OpenML and 1000 pipelines (also selected from the intermediate results of the AutoML baselines, as no clustering pipelines existed on OpenML at the time of writing). We choose the silhouette score as the performance metric.

For all of the collected pipelines, we utilized the DS Pipeline Extractor to convert DS scripts from different users and sources to the graph-based pipeline representation. For human-in-the-loop baselines, we needed to select a small number of datasets (10) that would make the preliminary user study tangible. To ensure a fair and realistic evaluation, we chose all datasets to be as diverse as possible. We achieved this by running the incremental farthest neighbor search algorithm [41] over the data footprints of all available datasets. As the outcome of this algorithm depends on the initial seed (i.e., the first randomly selected dataset), we iterate over all available datasets and select a sample with the largest mean Euclidean distance. This procedure aims at selecting datasets with the most distinct data footprints and, hence, the highest variety.

Baselines. We use (1) the DABL⁶ library for baseline pipeline generation that applies a static set of pipeline synthesis rules to generate a pipeline for a given task, (2) the popular auto-sklearn [21] AutoML solution using two variants: with the total execution budget of 5min and 1h respectively, (3) the Ray Tune [28] library as a state-of-the-art hyperparameter tuning solution, (4) the RapidMiner Auto Model as the state-of-the-art IDA solution, (5) a synthetic “oracle” baseline that imitates the end-user that knows the optimal path to the best-performing pipeline candidate, and (6) FLAML ⁷—a fast AutoML solution that optimizes the cost of training as well as predictive performance. As Ray Tune comprises a framework with hyperparameter optimization algorithms and not a full-fledged AutoML solution, one has to define the set of algorithms and the domain of hyperparameters that the framework will consider during the search. Therefore, for supervised classification and regression, we define Ray’s search spaces equivalent to that of auto-sklearn. For clustering, we use the common algorithms supported by sklearn⁸ with the values of the hyperparameters suggested in the documentation. Note that not all baselines support all three DS tasks, e.g., DABL, auto-sklearn, and FlaML do not support clustering.

DORIAN . We implemented DORIAN in Python and showcased it in [42]. We run three variants of DORIAN: with simulated user behavior, with a simulated oracle user, and with real users as part of a preliminary user study. Simulated variants are necessary for comparison against baselines on a large set of queries. To simulate user behavior, we apply an exhaustive (i.e., does not “skip” suggestions) breadth-first search strategy: We first execute pipeline candidates from one iteration and then recursively choose the suggestions for incremental improvements in the direct order. We choose breadth-first search because it fits a common decision-making pattern of evaluating all available options first and then selecting an attractive candidate to drill down. Note that the breadth-first search algorithm is exhaustive—e.g., n suggestions in the first iteration lead to n new iterations, n suggestions each, totaling \(n^2\) suggestions in the second iteration, and so on. Parameters k for kNN similarity search are system constants and set to 3 and 20 for the datasets and pipelines, respectively. We study these values in Sect. 9.6.

Evaluation strategy. As DORIAN bases suggestions on the information about past user queries and accumulated DS experiments, we need to synthetically exclude all the meta-information about the user query under evaluation from the Experiment Store to ensure the validity. For each task, we apply a leave-one-out scheme where the experiments related to all but one dataset are kept in the Experiment Store, and the one dataset with all corresponding pipelines specifies a previously unseen user query (i.e., benchmark dataset). We use all pipelines executed on the benchmark dataset and their corresponding predictive performance metrics as ground truth. We choose fivefold cross-validation as the evaluation process eval(C, T, D) for AutoML solutions on benchmark data. When evaluating the quality of DORIAN ’s suggestions, we take the maximal predictive performance across suggested pipelines as the performance of the best-found candidate.

Results. Figure 5 depicts the average predictive performance and total execution time of the baselines on all the datasets for supervised classification, regression, and clustering. DORIAN, on average, performs on the level of automated baselines, outperforming DABL and auto-sklearn 5min. Although the state-of-the-art AutoML baselines can show slightly better predictive performance to DORIAN, they take more time to execute (1 h compared to 0.5\(-\)2.5 min that DORIAN needs). For example, for classification DORIAN achieves an average ROC AUC score of \(97\%\) in few seconds while auto-sklearn requires 1 hour to arrive to an average ROC AUC score of \(99\%\). In addition, AutoML solution require more resources while DORIAN can provide suggestions to the end-user asynchronously to (or even without requiring) the execution of pipeline candidates. DORIAN simulated user by design cannot outperform the oracle user baseline, as both baselines draw suggestions from the same set of pipelines, but DORIAN oracle user simulates the end-user who knows how to find the best performing DS pipeline. However, DORIAN simulated user reaches predictive performance that is comparable to the oracle user baseline (for regression and clustering), which suggests DS pipelines with predictive performance similar or equal to the best performing DS pipeline even under the limited search budget.

Evaluation scenario. As we cannot evaluate human-in-the-loop baselines for all the available datasets due to the manual overhead, we selected six datasets for each DS task. To ensure a fair and realistic evaluation, we chose the datasets to be as diverse as possible by running the incremental farthest neighbor search algorithm [41] over the data footprints of all available datasets. For RapidMiner, we use each of the datasets and the corresponding DS task as a query for the Auto Model component that provides partial automation of the pipeline candidate generation and search process yet requires input from the end-user. For DORIAN, we conducted a preliminary user study with 6 PhD students with varying expertise in DS. We asked them to use DORIAN in order to design a DS pipeline that satisfies a pre-selected user query (i.e., each of the 6 datasets, supervised classification). We report the results of the user study as DORIAN real user, while we also included results for the simulated user. We apply the leave-one-out scheme as described in Sect. 9.1.

Results. Figure 6 depicts the average predictive performance for classification, regression, and clustering, as well as the total execution time of the three baselines—RapidMiner Auto Model, DORIAN simulated user (automated), and DORIAN real user. In this experiment, the total execution time includes delays for user interaction (i.e., real end-user). We observe that, for supervised classification, both DORIAN baselines are one order of magnitude faster than RapidMinder and provide superior performance (\(92-96\%\) against \(81\%\)). DORIAN real user reaches the ROC AUC score of \(92\%\) that is comparable to DORIAN simulated user, requiring, on average, 1 minute longer to compute. The difference in the total time between DORIAN simulated and real is due to the time spent on user interaction and the fact that fewer pipelines were submitted for execution by real end-users compared to the automated DORIAN simulated user baseline. For regression and clustering, the human-in-the-loop baselines require comparable total time, while DORIAN simulated user systematically outperforms RapidMiner in terms of accuracy metrics. This is due to the amount of information that DORIAN stores for decision making and the variety of pipeline configurations that it can recommend. RapidMiner, in turn, has a small fixed number of pipeline configurations that can be reduced for efficiency purposes based on the data features.

Evaluation Scenario. We conducted a preliminary user study with 6 PhD students with varying expertise in DS—they were tasked to use DORIAN to design a DS pipeline that satisfies a pre-selected user query (i.e., 10 datasets and the corresponding DS task). When the user interacted with DORIAN, we recorded the user interaction, such as the rank of selected pipeline candidates in the list of suggestions, the number of iterations that the user chose, the best trailing predictive performance among all the pipelines that the user has chosen for each user query, the total time that the user spent working on each query (including the time for browsing and idle time). We also selected one user query for which the execution engine was disabled. In that case, the end-user could only rely on the visual inspection of the pipeline structure to make decisions about pipeline comparison and not the predictive performance. This baseline task served two purposes: (a) to estimate the level of user expertise in DS, and (b) to answer the question of whether it is reasonable to assume that the users are capable of assessing the pipeline predictive potential by visual inspection only.

Results. For all the users and queries that we evaluate in this study, we observe that: (a) in \(90\%\) of the cases, the users stopped after 4 iterations of recommendations; (b) in \(90\%\) of the cases, the users selected one of the top-5 suggested pipelines, with the lowest selected candidate in the list of recommendations having the rank 9; (c) suggested pipeline candidates follow the power law distribution where 5 candidates were chosen in the majority of cases (account for \(20\%\) of all execution requests) and over 70 pipeline candidates were chosen for execution with the frequency under \(2\%\) (the long tail of infrequent pipeline candidates accounts for \(60\%\) of all execution requests); (d) the users applied a mixture of browsing patterns that resemble the logic of traversal algorithms for search trees, such as breadth-first search and A-star search, and less frequently—random, depth-first or always-top search patterns; (e) more experienced users (the ones that correctly compared a higher fraction of pipelines in the baseline task with the disabled execution engine) tend to be more selective in their search, varying the browsing pattern based on the given pipeline suggestions and making more decisions to select or discard a pipeline candidate without executing it; (f) less experienced users oftentimes employed the execute-discard pattern where they submit a pipeline for execution and immediately discard it from the list of suggestions in case they perceive the reported predictive performance as not satisfiable; (g) the users perceive the tool as useful in scenarios where they want to explore new DS operators or pipeline candidates that might be relevant to their query and where they want to receive a pool of potentially relevant pipeline candidates fast, without writing code.

Interview Protocol. We base the protocol of the qualitative interview on the setup of the preliminary user study and expose another control group of end-users (domain experts and software engineers) with varying expertise in data science to DORIAN ’s user interface with several pre-specified queries. For the qualitative interview, we provide four tasks: (a) an introductory task where end-users explore DORIAN and its UI while selecting a pipeline for the supervised classification task with the execution engine being disabled and the evaluation process set to pairwise comparison—in this scenario, end-users cannot get quantifiable performance metrics and have to make decisions based on their experience and intuition, comparing pipeline only by their structure; and three scenarios where the end-users need to add (b) a new DS task, (c) a DS operator, and (d) a ranking objective. While performing the tasks, the control group can ask questions and provide suggestions that are later aggregated.

Insights. We present the aggregated observations and suggestions from both control groups, marking the commentary of domain experts as [D] and software engineers as [S]. When evaluating an action in terms of intuitiveness and complexity, the individuals identified three categories: “easy” (i.e., “makes sense”, “reasonable”, “fast”, “straightforward”), “requires effort” (i.e., “learning curve”, “commitment”), and “complicated” (i.e., “do not know”, “skip”, “would not spend time”). In summary, the respondents appreciated the practical relevance and timeliness of our approach. Moreover, when discussing the use case of non-quantifiable evaluation processes, most respondents characterized DORIAN ’s fallback pairwise comparison as intuitive. They also identified two challenges to DORIAN ’s practical adoption and extensibility—the learning curve of the UI and the extension of the knowledge graph.

We measure DORIAN ’s relative predictive performance under varying k for the nearest neighbor search for datasets and pipelines, respectively, as well as the execution time that is required to render suggestions.

Evaluation Scenario. For every query in the experimental setup, we compute the leaderboard percentile of the best-found pipeline candidate given the search budget, the value of parameter k for the nearest neighbor search on datasets (from 2 to 10 included), and pipelines (from 10 to 40 included, with the increment step of 5). We also measure the total execution time (in seconds) for rendering the suggestions and evaluate the trade-off between the number of pipeline candidates to evaluate and the time it takes.

Results. Figure 8a depicts the steady relative predictive performance of best-found pipeline candidates across the queries. For \(k \in \{3,4\}\), the standard deviation of the percentile is relatively smaller, and for \(k=8\), the median value is relatively higher than the average across different k. However, there is no systematic evidence to demonstrate that the recommendations with parameter \(k \in \{3,4,8\}\) are inherently better. When it comes to the execution time, on the other hand, we observe a high correlation between the parameter k and the required execution time. Note that the median execution time is under 2 seconds regardless of k, which qualifies as a real-time response. Based on this experiment, we made the decision to use \(k=3\) by default, as it does not lead to significant improvements in the quality of recommendations but is faster.

Figure 8b depicts improvements in the quality of recommendation across different k for the nearest-neighbor search on pipelines, as the median value of the percentiles increases slightly with the increase of k, and the standard deviation of percentile distribution decreases. The execution timeline chart (blue) consists of three distinctive parts—(a) steady growth for \(k \in \{5,10,15\}\), (b) significant decrease for \(k=20\), and (c) a constant-like pattern for \(k \le 25\) with minor variation. The breakdown analysis of the execution time to render suggestions showed the following: for pattern (a), the choice of k was insufficient to render the required number of suggestions after filtering out pipeline candidates that were either previously suggested or explicitly discarded by the end-user. To compensate for the insufficient number of candidates to suggest, we repeat Algorithm 1 with the higher value of k. The fact of re-running the kNN query twice leads to increased overhead in execution time. For pattern (c), the choice of k was high enough to lead to the convergence in the number of distinctive pipeline candidates that were retrieved as a result of the kNN query. For \(k~=20\), we reached a “sweet spot” given the state of the Experiment Store during the evaluation. Thus, for this evaluation, we chose to use \(k=20\) by default, which yielded a sufficient amount of pipeline candidates, even after filtering out the irrelevant ones. Note that, in practice, the optimal value of the parameter k can vary depending on the number of distinctive pipelines that are persisted in the Experiment Store and their pairwise similarity. In the case when DORIAN starts producing an insufficient number of suggestions or re-runs extended kNN queries (with \(2\times k\)) frequently, we can tune k by repeating this experiment on the new state of the Experiment Store.

Evaluation strategy. We use the evaluation strategy on all the datasets for supervised classification and test three baselines: Ray Tune 5 min, Ray Tune 1 h, and a variation of Ray Tune 5 min that is warm-started by using 5 initial suggestions that DORIAN renders for the query. We then compare the predictive performance (ROC AUC) of best-found pipeline candidates and claim that the warm-started variation outperforms the original Ray Tune 5 min baseline when its average score during cross-validation is higher than that of Ray Tune 5 min plus 2 standard deviations of the ROC AUC score recorded during k-fold cross-validation.

Results. We report that, for the search budget of 5 min, the warm-started variation outperformed the original baseline in \(38\%\) of the cases. In 3 datasets (i.e., user queries), the warm-started Ray Tune 5 min also outperformed the Ray Tune 1 h baseline. In absolute terms, the average difference in predictive performance between the original baseline and its warm-started variation is 0.025 (ROC AUC scores are measured from 0 to 1). In the following, we outline several observations regarding DORIAN ’s potential to act as a warm-starting scheme for other AutoML solutions.

First, not all of the AutoML solutions allow for user-defined warm-starting—across the baselines under analysis, only Ray Tune is flexible enough to support warm-starting. Secondly, such a scheme necessitates the implementation of an adaptor that converts DORIAN ’s suggestions (i.e., pipeline DAGs) into an instantiation of the search space that the AutoML solution operates. Furthermore, it requires an assumption that such a conversion is possible, i.e., there are no operators that DORIAN suggests and the AutoML solution “does not know about”. In this experiment, we curated the search space of the Ray Tune baselines and could guarantee one-to-one mapping. However, in practice, the implementation of the adaptor should be responsible for accurate conversion and fallback options for exception handling.