Exploring Correlations Between Properties Using Artificial Neural Networks

The discovery of correlations between datasets has led to many important findings historically[1‐3] but there are two essential prerequisites: reliable data and an inspired guess at where to look for correlations. The increase in data handling capacity and advances in intelligent search methods could, it is claimed,[4] change the way in which some sectors of science proceed. Large databases in materials science could make it possible to search for correlations between properties that would not normally be sought. At present, researchers tend to focus on one set of properties in which they are expert rather than connecting one property with another.

The “high throughput” or “combinatorial” methods are an attempt to increase the pace of materials development in increasingly complex compositional spaces.[5] Combinatorial libraries can be regarded as a capital asset upon which a multitude of properties can be measured to determine structure–property relations of materials behavior.[6] Potentially, these could reveal relationships between different properties of each material experimentally but this strategy has rarely been adopted primarily because each investigator tends to be an expert in a given property regime and to have limited goals in terms of applications which are often set by the research funding source.

Computational chemistry provides a means to model the structure and functional properties of real materials quantitatively and consequently to design and predict novel materials and devices with the improved performance.[7] However, the large number of atoms and many-body interactions place considerable demands on computer resources[8] at higher structural scales.

As pointed out by Ashby,[9] all the properties of materials can be derived ultimately from the structure and bonding, or can be considered to have their ultimate origin in Schrödinger’s equation, so the properties of a material are, to varying degrees, interrelated (Figure 1). Binary correlations among materials properties abound and there is a clear mechanistic, causal interpretation. (i) Specific heat is related to atomic or molecular mass (Dulong and Petit’s law); the heat energy arises partly from the number of atoms or molecules that are vibrating and if a substance has a lower molar mass, then each unit mass has more atoms or molecules available to store heat energy. (ii) The electrical and thermal conductivities in metals were related by the Franz–Wiedemann rule in 1853,[2] which was developed in the electronic theory of Drude[10,11] since both heat and electrical fluxes in metals are strongly influenced by the motion of their electrons. (iii) Melting and boiling temperature can be correlated with the depth of the potential energy well.[12]

Examples of indirect correlations include (i) the specific heat and density in solids are related because the density of a solid is mainly determined by its atomic weight, while to a lesser degree by the atom size and the way in which they are packed.[13] Due to the correlation between density and atomic weight, and between atomic weight and specific heat capacity, there is a strong, inverse correlation between solid density and constant-pressure-specific heat capacity; (ii) the thermal expansion coefficient and melting points of materials with comparable atomic packing vary inversely because the higher melting-point materials have deeper and more symmetrical energy wells; (iii) hardness and melting point are indirectly related because hardness depends on the stress required to separate atoms and initiate dislocation motion. Higher inter-atomic forces imply deeper energy wells so materials with high melting points such as diamond, Al₂O₃, and TiC are the harder materials. The exceptions occur where more than one type of bond is present, such as graphite and polyethylene. For similar reasons, melting point and bulk modulus are related through bond energy.

Many other examples abound with varying degrees of correlation: inverse correlation for toughness and hardness; inverse correlation between dielectric loss and dielectric strength; in porous materials, the mechanical strength and the dielectric strength; in functional ceramics, a dielectric with high loss may show ionic conduction at a higher temperature; in oxides, a change of color may be associated with electrical conduction, both being influenced by point defects.

Ashby points out that some correlations have a simple theoretical basis; others can be found by search routines and empirical methods.[9] Generally, the correlations derived in a direct way from the nature of the atomic bond and structure are strong, such as modulus and melting point, or specific heat and density, while those derived from properties which depend on defects in the structure are less strong, such as strength and toughness, and are further weakened when interaction with the environment is involved, such as corrosion and wear.

A journey into materials science that explores correlations of properties is rather unconventional but the considerable success of Ashby’s property mapping[14‐16] suggests that it could provide a way of identifying compositional zones that are worthy of more detailed exploration and therefore narrow the hugely complex space that confronts the discovery of new materials. Actually, the idea of exploring property–property relationships rather than structure–property relationships seems less unconventional when it is noticed that examples of binary correlations among materials properties abound. In most cases, there is a sound mechanistic connection and the scientific practitioner uses a well-trenched radial path in Figure 1 while being barely conscious of the circumferential relationships. The work described in this paper participates in Ashby’s scientific journey.

The exploration of correlations can be classified into three different types: (I) purely empirical, (II) partly empirical but based on some theoretical concept, and (III) purely theoretical.[17] Data mining, which is defined as a process for extracting useful hidden information directly from data rather than from basic laws of physics, is regarded as a useful tool that could help to probe implicit correlations between different properties empirically. In this work, we employ artificial neural networks, one of several data mining methods, to explore cross-property correlations. It is ideally suited not just for binary, but also for ternary and more complex correlations.

There are several precedents for applying neural networks to explore cross-property correlations. Egolf and Jurs[18] used both regression and neural network techniques to predict boiling points of organic heterocyclic compounds using the molecular weight, dipole moment, 1st order molecule connectivity, and other structure descriptors. Michon and Hanquet[19] used quantitative structure–property relationship methods and neural networks to find non-linear relations between chemical and rheological properties. Homer et al.[20] developed ANN with equilibrium physical properties and structural indicators for prediction of viscosity, density, heat of vaporization, boiling point, and Pitzer’s acentric factor for pure organic liquid hydrocarbons. Boozarjomehry et al.[21] developed a set of ANNs to predict properties such as critical temperature, acentric factor, and molecular weight of pure compounds and petroleum fractions based on their normal boiling point and liquid density at 293 K. Strechan et al.[22] obtained correlations between the enthalpy of vaporization, the surface tension, the molar volume, and the molar mass of a substance using ANNs. Mohammadi and Richon[23] used ANN to predict the enthalpy of vaporization of hydrocarbons, especially heavy hydrocarbons and petroleum fractions from the specific gravity and normal boiling temperatures. Karabulut and Koyuncu[24] developed neural network models to establish correlations of thermal conductivity with temperature and density for propane. Giordani et al.[25] used ANN for correlating a wider range of properties, principally mechanical properties of modified natural rubber. However, all these investigators used prior knowledge to select the properties for causal significance in the prediction.

The two main tasks are data collection and neural network construction. The collected data are used to construct the neural network and then the neural network is used to find cross-property correlations. Believing the elements to have the most reliable data, whole datasets of the physical properties of solid elements were collected from different handbooks, including Chemistry Data Handbook (CDH),[26] The Lange’s Handbook of Chemistry (LHC),[27] The Elements (ELE),[28] Table of Physical and Chemical Constants (TPC),[29] and CRC Handbook of Chemistry and Physics (CRC).[30] The properties used in this work are those recorded under 0.1 MPa, in a small temperature range (293 K to 298 K) and in the solid state in order to minimize the effects from phases, temperatures, and pressures. Sixteen different properties were collected: (i) normal melting point, (ii) normal boiling point, (iii) heat of fusion under normal melting point, (iv) heat of vaporization under normal boiling point, (v) molar heat capacity, (vi) specific heat capacity, (vii) thermal conductivity, (viii) electrical conductivity, (ix) photoelectric work function, (x) linear thermal expansion coefficient, (xi) atomic weight, (xii) density, (xiii) electronegativity (Pauling scale), (xiv) first ionization potential, (xv) polarizability, and (xvi) atomic volume. The elements collected satisfied the phase, temperature, and pressure criteria and had full records of all sixteen properties from the five handbooks. The main criterion for the selection of the 16 properties was that reliable data must be available for all the listed elements if the aim is to show a general, systematic method for exploring correlations between different properties. For this reason, refractive index, for example, which should correlate well with polarizability was excluded, a complete dataset being unavailable. There were 75 elements included in total.

It is worth noting that in our previous work[31,32] the use of ANN revealed some surprising incorrect data in handbooks. In this work, we treat the properties which have close recorded values from the five different sources as reliable. The outliers, treated as incorrect, may have incorrect unit conversions, different reference conditions, or decimal point misplacements.[32] The median values of each property were used here. Provided most of the property data are correct, the general trend of the correlations can be treated as reliable. Certainly, there may be some predictions that do not follow the correlation and we can look back at these data to find the reasons: it may be that the correlation hypothesis is violated for these special elements or the recorded values in handbooks were incorrect, in which case we can use the method developed previously[31] to select the correct ones.

It is well known that materials properties vary over a great range and are generally logarithmically distributed.[9,13,33] Sha[34] points out that when training a neural network with skewed data, it can be misled by a few data far away from average because, unlike linear regression training, neural network training is not based on a definitive starting formula. A logarithmic pre-treatment for properties that are logarithmically distributed is needed. The original property data distributions are shown in Figures 2(i) through (xvi). Observation of these figures shows that (iii) heat of fusion, (vi) specific heat capacity, (vii) thermal conductivity, (viii) electrical conductivity, (x) linear thermal expansion coefficient, (xv) polarizability, and (xvi) atomic volume are skewed and these were logarithmically pre-treated. However, from the data shown in Figure 2(xi), the electrical conductivity is distributed over such a great range that even logarithmic pre-treatment cannot normalize the distribution and this introduces major uncertainties for extracting general correlations and so electrical conductivity was excluded from the trial. Figures 3(i) through (vi) show the distribution of the six properties given logarithmic pre-treatment. From Figures 3(i) through (vi), the values for atomic volume, polarizability, linear thermal expansion coefficient, and heat of fusion become uniformly distributed, while for thermal conductivity and specific heat capacity, the distributions are not totally uniform, but enhanced. Double or even triple logarithms could be used, but it is undesirable to compress the whole range of values into too narrow a region such that most values become nearly the same.

All these property values, appropriately pre-treated, constituted the neural network inputs and each in turn was used as an output. When a property value was used for output, the original values were adopted because the neural network training is based on minimization of the difference between predicted values and experimental values. Small differences in logarithmic value would correspond to a large difference in original value so that the satisfied predictions of logarithmic values may have large differences between predicted and experimental original values.

Back-propagation ANNs were constructed, trained, and simulated by MATLAB 7.4.0.287 (R2007a) software. For most function approximation problems, one hidden layer is sufficient to approximate continuous functions[35,36]; two hidden layers must generally be necessary for learning functions with discontinuities.[37] Also, the neural network user’s guide (MATLAB R2007a) suggested that a two-hidden-layer sigmoid/linear network can represent any function of input/output relationship.[38] As a result, a two-hidden-layer network with tan-sigmoid transfer function in the first hidden layer and a linear transfer function in the second hidden layer was adopted. Bayesian regularization, implemented as trainbr command in MATLAB R2007a, was employed for improving generalization during network training, which updates the weight and bias values according to Levenberg–Marquardt global optimization.[38] Strictly, the Marquardt–Levenberg algorithm searches for local minima; however, in each iteration, it selects a new parameter value (damping factor). Thus, the MATLAB Manual[38] recommends it as generally the best in terms of its performance, memory requirement, and computing efficiency. A loop program was used to redistribute the database in order to make the training set cover the problem domain as recommended by Malinov and Sha.[39] Employing Bayesian regularization and database redistribution can alleviate the overfitting problem. More detail can be found in the author’s previous paper.[40]

Taking one property at a time to be predicted (output of the neural network) and all other properties as inputs, the process was repeated. When property values can be reasonably predicted from groups of other properties, then we can say that part or all of these properties are correlated. The criterion used for highest performance in both training and testing sets was the lowest value of \( \omega \, = \left| {\varphi_{\text{training}}^2 - \varphi_{\text{testing}}^2} \right| \), where \( \varphi = \left| {M - 1} \right| + (1 - R) \), and the smallest value of ω is chosen.

As the range of applications for materials which depend on electric polarizability and hyper-polarizability has expanded dramatically,[41,42] we take the example of the prediction of polarizability from the other 14 properties to illustrate how the method behaves in exploring correlations between properties that appear to stem from different physical principles.

First, using the prediction of polarizability from each of 14 properties individually, those which provide strong predictability for polarizability were selected. The other properties were treated as properties that have weak or no direct correlation with polarizability. However, care is needed in deciding on exclusions. There is a possibility that a combination of properties excluded in this way could have delivered enhanced prediction, compared with the case in which they are omitted.

The square of the correlation coefficient, R², is the proportion of the variation in the values of y that is explained by the least-squares regression of y on x. It ignores the distinction between explanatory and response variables. The correlation between input and output property values was expressed by R²; in this case, the proportion of variation in the experimental values accounted for in a linear relation between predicted and experimental values. Here, we use the criterion of R = 0.9, meaning that about 80 pct of the variation is accounted for, and designate the correlations with R ≥ 0.9 as having significant correlations. Figures 4(a) through (d) show the results of prediction of polarizability with R values greater than 0.9 and Table I lists the statistical analysis for results shown in Figure 4.

Now that we have located four properties that have relatively strong correlations with polarizability and have relegated ten properties with weak or even no correlation, the next step is systematically to introduce other properties to assess improvements in the predictions and hence reveal the ‘effect’ of each property on the prediction of polarizability. Here, it needs to be noted that if the effects of different properties on the prediction of polarizability are combined, the influence of one property cannot be distinguished from the influence of others and it cannot be said how strong the effect of one property on polarizability is. It also means that some properties, which cannot make a strong prediction alone, may have effects or even strong effects on the prediction when combined with other properties.

So this step focuses on the results that show a high degree of correlation (here we take R² = 99 pct, which corresponds to R = 0.995 and the slope M is equal to or greater than 0.99) between polarizability and different combinations of other properties and then from these results, we note the underlying physical principles in an attempt to assess why different combinations of properties can have similar predictive performance. It is found that the prediction of polarizability obtained using the minimum number of other properties involves melting point, heat of vaporization, specific heat capacity, and first ionization potential. The result is shown in Figure 5, and the statistical analysis is shown in Table II.

The discussion of these five results (Figures 4 and 5) that follows comprises (1) exploration of the underlying physical principles for results shown in Figure 4 in order to justify this method for exploring cross-property relationships, (2) analyzing the results shown in Figure 5 and comparing this result with other results to explore the possible confounding effect of different properties, and (3) exploring possible mathematical equations that can formulate these correlations.

The prediction of properties from structures by computational methods is widespread but the interactions between different levels of structure can make these problems very complex. In this work, we apply the principle that all the properties of a material are determined by, or are a common response to, composition and structure, and use this principle to explore the correlations between different properties which should therefore exist by using artificial neural networks. However, interactions between input properties still exist. In neural networks, the nature of the interactions is implicit in the values of the weights. In cases like the one studied in this work, there exist more than just pairwise interactions and, as a result, it is difficult to visualize them from the examination of the weights. As suggested by Bhadeshia,[82] the better method is to use the network to make predictions and to see how these depend on various combinations of inputs. In this work, we made use of underlying physical principles to explain the different results and found that employing neural networks to explore the cross-properties relationships is both reasonable and feasible.

The correlations that exist between different properties are explored by employing artificial neural network methods using the example of prediction of polarizability from combinations of other properties. Through this example, we provide a general, systematic method for exploring correlations between different properties for different types of materials under specified conditions of phase, temperature, and pressure. The method applied in this work depends strongly on the availability of correct data. It is the restrictive availability of such data for compounds that presently limits this novel methodological step.

The advent of e-science has meant that scientific communication can employ media not previously recognized and data can be made accessible globally so that many geographically dispersed groups can analyze raw data according to their own skills. The sharing of data in raw form rather than through the highly processed medium of refereed journals means that the construction of global shared databases is a reality and it follows from that multi-property data can be put up, shared, and processed in novel ways.

Once data sharing is in place, computational processes for mining the relationships are needed. Methods are required for identifying how values of properties p₁, p₂, p₃ … can be used to estimate the likely magnitude of property p_n. This will narrow down considerably the sample space for experimental high-throughput methods for finding materials with a desired range p_n and the computational cost of predicting such materials properties. As the global database grows, this cross-correlation will produce a new type of materials science that allows the scientific world to home in on new materials at a rate previously thought impossible. In the same way that high-throughput methods have compressed laboratory time, multi-property mapping might compress the time taken for new materials discovery. Linked in this way, the mapping would define the compositional space for combinatorial discovery.

The results show how the predictive power of some parameters depends on those with which they are combined and so we begin to see how a much larger ANN analysis could be structured to accommodate large numbers of properties both to predict properties not yet known and to point the direction of compositions not yet made.

However, a prerequisite for all such methods is that the shared databases should be cleansed from unreliable data; this is the basis for getting meaningful and useful information out from them with certainty.