Data integration for materials research

The Materials in Extreme Dynamic Environments (MEDE) Collaborative Research Alliance (CRA) is a multiscale materials research organization consisting of a consortium of major research universities, a national lab, translational institutions, and the Army Research Laboratory and industry, which was established to implement a strategically driven and fundamental science-based paradigm for the development of lightweight protection materials for extreme dynamic environments. The MEDE CRA approach uses advanced experimental techniques, advanced validated modeling and simulation tools, and advanced synthesis, processing, and fabrication capabilities to develop a materials-by-design capability. Our approach is founded on the development of an understanding of the fundamental mechanisms active within extreme dynamic environments at all relevant length and time scales. Using this foundation, our collaborative approach develops a validated modeling and simulation paradigm for the design, optimization, and fabrication of materials for extreme dynamic environments. A critical part of this approach, of course, is the effective and efficient use of data science. We view this program as one of the major testbeds for the integration of data science into materials research, following the Materials Genome Initiative paradigm. This manuscript presents our system design and demonstrates the power of our approach through a specific use-case.

The MEDE research effort is divided by materials class, with groups of faculty, scientists, and students working together on a model metal system, a model ceramic system, a model polymer system, and a model composite system. Our approach is to develop our methods and protocols first for one of these groups (we chose the Ceramics group) and then extend the approach across to the other materials classes once the essentials of the approach have been worked out and demonstrated within the first group. The Ceramics group in MEDE is working on the boron carbide system, using experiments, materials characterization, modeling, large-scale simulations, and synthesis and processing. The core science approach ensures that all of the scientists in the group are working on the same baseline material and that they all work simultaneously on each new generation of materials as they are developed. Thus the timely and efficient sharing of data and metadata is critical, but this is also challenging given the variety of disciplines that are involved in this materials research effort (a sense of the range of disciplines can be obtained from the MEDE website at https://​doi.​org/​hemi.​jhu.​edu/​cmede).

An example of one of the major challenges is provided by the magnitude and variety of the experimental data developed just within the ceramics effort in MEDE. Modern technology provides a dizzying array of high-resolution (in time and in 3D space) experimental information during an impact event. Imaging using high-resolution and ultra-high-speed cameras, computer-aided tomography, and other three-dimensional characterization methods such as serial sectioning provides large datasets on every material sample. Analysis of such datasets provides insights into the complex processes associated with materials undergoing rapid configurational changes (e.g., deformation, failure, phase change) across a wide range of scales. For the most part, these insights are length-scale dependent, but the failure mechanisms are typically multiscale, and information passing and scale bridging are not well-defined within the community in terms of the datasets (as compared to the equations).

As a specific example, the dynamic compressive strength of a centimeter-scale ceramic sample is known to depend on the distribution of micron-scale processing-induced defects in the material. Extensive materials characterization provides the size and orientation distribution of these defects (which are typically carbonaceous inclusions in boron carbide) in the material. This data is provided to both modelers and to those attempting to improve the processing route, and the fidelity of the models is determined by comparing the large datasets generated by simulations using those models with the large datasets obtained from dynamic mechanical experiments using Kolsky bar techniques.

With the objective to provide an online platform that assists and accelerates iterative materials engineering across MEDE, the initial step is to outline current research group collaboration and understand the materials research cycle. A high-level description of the data flow between groups help us see the requirements for a data integration service. For each research group, we collect general information on their procedures and the nature of the data they use and produce as well as the data volumes involved in their research.

An overview of the data dependency in the Ceramics group is illustrated in Fig. 1. They can be sorted into three approximate categories: modeling, processing, and experiments. These groups share a variety of data types including textual simulation results, image-based histories of experiments, processing histories, equation parameters, and recommendations of actors. While currently the heaviest scale of data sharing is imagery on the order of tens to hundreds of gigabytes (GB), several scientists have expressed plans to collaborate with new datasets on the order of terabytes (TB).

With the goal to alleviate materials researchers’ data management burdens while introducing minimal changes to existing workflows, it is good practice to design the collaborative platform to ingest data as close to the source as possible and provide familiar platform-based data processing tools. For example, given a Kolsky Bar physical experiment, the data is produced by sensors monitoring the event. In terms of data reliability, provenance, backup recovery, and collaborator access, it is advantageous to immediately upload the raw sensor data to a storage server. A collaboration platform will track any changes and operations performed on the datasets; good data provenance enumerates the steps taken to reach results and is useful for peer review, the verification of data integrity, and is educational for students learning the art. However, researchers’ current practices include performing laptop- or personal computer-based analyses and aggregations on the raw data before gaining reportable results. If similar analysis tools could be provided to the researchers via the online collaborative platform, not only would a burden be lifted from the individual researcher’s laptop but the workflow would be available for team members to examine and contribute to. We surveyed MEDE scientists and found that data manipulative scripting and plotting tools such as Excel, Matlab, and R are commonly used to process the raw data into results.

A useful data platform should fulfill many requirements that match the needs and constraints of current practices. We envision a completely online, secure single-sign on system with three key connected components: file storage, scalable database services, and a templated scripting environment for plotting and analysis. At each component, interactive access to the data should be browser-based and shareable either privately, with immediate peers, or among research groups.

For the most basic collaboration, there must be an online file storage system where researchers upload and share their data with whomever they choose. This service will handle file sizes ranging from KBs to TBs, and act as the data gateway to the rest of the platform. Beyond simple file storage, the database component will serve as the core engine for processing and aggregating the raw data. The database interface will also provide a history of operations performed on the datasets, allowing students and fellow researchers to learn each others’ methods and track dataset provenance.

Familiar tools used for collaborative materials engineering must also be made available. Commonly performed analysis codes previously executed on researcher’s own computers, such as the derivation of a peak stress versus strain rate plot, should instead be performed on the servers where the data is stored. An online scripting environment, similar to what researchers already use, will provide the main interaction between researchers and their data. Frequently used codes will be encapsulated in pre-written template scripts as an example starting point for new users.

Materials researchers work with multitudes of file formats to store data. While text-based and comma-separated value (CSV) files are frequently used, many instruments that capture experiments produce proprietary data formats that are only officially supported by the manufacturing companies’ software. Initially, the collaboration platform will be able to automatically parse and load commonly used text-based file formats into the database and scripting environments; however, the file storage service is designed to be quickly extensible with custom plugins for parsing other data formats. These plugins are snippets of codes that inform the database on how a specific file format’s data is organized so that the database may appropriately read and load the data into tables. Knowledge of a file format data scheme may not always be readily available, especially with some proprietary data formats. With this worst-case scenario in which developing a parsing plugin is not possible, the researcher is encouraged to both upload the data file to the file storage service for backup and to then use any available external proprietary software to produce a CSV or textual format for database ingestion.

Providing these three services, all customized for use in materials engineering with minimal change to existing workflows, is no small feat. Yet, MEDE may draw from decades of expertise in big data management tools originally developed in astrophysics but now adapted for general scientific use. We have identified and adapted pieces of the SciServer [1] project to fit the requirements of a MEDE data collaboration platform. Materials scientists may use one or more of these components, whether that is to share files with collaborators or interactively develop plots and analyses of data in an online environment.

We began by testing SciServer tools on a subset of research groups; as a starting point, we created a template workflow encompassing the data interaction between the continuum modeling group and physical experiments. The workflow template is a persistent, example use case of the SciServer chain from data ingestion to plot analysis, containing the necessary code and usage directions within each tool’s interface. The examined comparison point between the model and experiment data is a material’s peak stress with respect to strain rate. In this case study, SciServer ingests each group’s datasets, performs aggregation in CasJobs, and produces comparative plots in SciScript.

The experiment sensor data was provided in tab-delimited text files, which CasJobs parses and inserts into a database table by default. Simulation results given by the modeling group are in a Uintah Data Archive (.uda) file format produced by the Uintah software suite [6]. A SciDrive plugin for parsing the UDA compressed files online is currently being tested; however, in the meantime, the simulation files are converted into CasJobs-injestable text files using the external VisIt software suite [7].

Once uploaded to a CasJobs-linked folder on SciDrive, the data is aggregated in the database with SQL queries stored and executed by a SciScript Python notebook. After interviewing modelers, we encoded their common simulation output processing practices into SQL and saved the queries as templates for further use. Each simulated particle’s data is provided as timestamped 3 ×3 matrices representing force and stress. Using our SQL Server extensions which we have installed onto CasJobs, we calculate the strain tensor for each volume element i from the force tensors F _i as

We solve the eigenproblem of these 3 × 3 matrices to obtain the eigenvalues \(\{{\varepsilon ^{1}_{i}}, {\varepsilon ^{2}_{i}}, {\varepsilon ^{3}_{i}}\}\) that can be used to calculate the scalar quantity

The calculation is done for all timesteps in the simulation to map out the changes over time \(\bar {\varepsilon }\,=\,\bar {\varepsilon }(t)\). The stress is calculated similarly in the database, and the aggregation over the eigenvalues provides the empirical determination of the \(\bar {\sigma }(t)\) function.

Next, we plot model-derived strain versus stress with scripts written on SciScript. Figure 5 shows the SciScript-generated plot of five different simulations, each executed with a specified strain rate. From the same scripting environment, the experimental data is queried and compared to the model data. Figure 6 is a comparison of peak stress versus strain rate from both of the modeling and physical realms. While these figures display nothing new in terms of materials science, their point is to illustrate SciServer’s ability to serve as a collaborative platform containing the necessary tools for materials data processing. With the workflow templates created in this case study, researchers may upload their own similarly formatted datasets and simply change the template’s source table name to aggregate their data and produce more meaningful comparative plots all in a matter of seconds.

SciServer hopes to enable faster collaboration between MEDE research groups and advance the materials-by-design approach. As a common file storage, database, and scripting platform for various MEDE researchers, SciServer can bring together data from various sources in order to draw new comparisons. While there are many fundamental benefits to a server-based data workbench, the downside is that researchers will still have to change their existing work habits in order to adapt to the new tools. Much of our effort has gone to observing and learning MEDE research practices so that we may augment SciServer to be as easy as possible to transition to. We hope to make SciServer attractive by streamlining and automating the researcher’s data management; features such as pre-written workflow templates should shorten the time and effort between running a simulation or experiment and obtaining analysis plots, while developing plugins to parse specific file formats will automate database file ingestion. The SciServer approach excels at enabling large-scale data science and online collaboration; however, users will be limited to the tools available within the platform.

Although we have a case study of one collaboration point between two groups, there are still many use cases to examine and consider for SciServer. Striving to provide template scripts, queries, and tools general enough to encourage researchers to code their own analysis within SciServer is a difficult task requiring broad knowledge of how MEDE performs materials-by-design. We can further adapt SciServer to MEDE only through more researcher usage and support experience.