An approach to generate rules from neural networks for regression problems

Abstract

Artificial neural networks have been successfully applied to a variety of business application problems involving classification and regression. They are especially useful for regression problems as they do not require prior knowledge about the data distribution. In many applications, it is desirable to extract knowledge from trained neural networks so that the users can gain a better understanding of the solution. Existing research works have focused primarily on extracting symbolic rules for classification problems with few methods devised for regression problems. In order to fill this gap, we propose an approach to extract rules from neural networks that have been trained to solve regression problems. The extracted rules divide the data samples into groups. For all samples within a group, a linear function of the relevant input attributes of the data approximates the network output. The approach is illustrated with two examples on various application problems. Experimental results show that the proposed approach generates rules that are more accurate than the existing methods based on decision trees and linear regression.

Introduction

Artificial neural networks are powerful tools for business decision making [2], [5], [6], [15], [20], [21], [25], [27]. They work particularly well for problems involving classification and data fitting/regression. Neural networks often predict with higher accuracy than other techniques because of the networks’ capability to fit any continuous functions [3], [8]. One major drawback often associated with neural networks is their lack of explanation power. It is difficult to explain how the networks arrive at their solutions due to the complex non-linear mapping of the input data by the networks. In many applications, it is desirable to extract knowledge from trained neural networks for the users to gain better understanding of the problems at hand. The extracted knowledge is usually expressed as symbolic rules of the form

$$\text{if}\text{condition,}\text{then}\text{consequence.}$$

In order to generate comprehensible and useful rules from neural networks that have been trained to predict continuously valued variables, the rules must be sufficiently simple yet accurate. The conditions of the rules describe a subregion of the input space, while the consequences of the rules are of the form Y=f(X), where f(X) is either a constant or a linear function of X, the attributes of the data. This type of rules is easy to understand because of their similarity to the traditional statistical approach of parametric regression. Since a single rule will not normally approximate the non-linear mapping of the network well, one possible solution is to divide the input space of the data into subregions. Prediction for all samples in the same subregion will be performed by a single linear equation whose coefficients are determined by the weights of the network connections. With finer division of the input space, more rules are produced and each rule can approximate the network output more accurately. However, in general, too many rules––with each rule’s conditions satisfied by a handful of samples––do not provide meaningful or useful knowledge to the user. Hence, a balance must be achieved between rule accuracy and rule simplicity.

Most existing research works have focused on extracting symbolic rules for solving classification problems where the network outputs are discrete. A regression problem, on the other hand, has continuous output. Few methods have been devised to extract rules from trained neural networks for regression [22]. Among these methods is a recently proposed algorithm by Setiono et al. [19] where piece-wise linear regression rules are extracted from pruned neural networks. The piece-wise linear equations are obtained as the linear combinations of the linearized hidden unit activation function. The non-linear hidden unit activation function is approximated by either a three-piece or a five-piece linear function that would minimize the approximation errors as represented by the area bounded by the non-linear function and the approximating linear function. Test results on a wide range of problems show that the method could outperform another method that generates decision trees for regression. In order to reduce the number of regression rules, clustering of the hidden unit activations is proposed prior to rule generation [17]. For samples in each cluster, a regression equation is obtained by applying statistical multiple regression technique.

For applications where one is only interested in obtaining accurate predictions, the trained neural networks will suffice. In other applications, one may want to know more about the relationships between the input variables and the continuous output variable. One feasible approach is to replace the prediction of the trained neural network by a set of multiple linear regression equations without compromising the accuracy of the prediction. Our approach works on a network with a single hidden layer and one linear output unit. We impose the restriction on the number of hidden layers to reduce the errors in approximating the hidden unit activation function by a piece-wise linear function as these errors will propagate from the input layer to the output layer through the units in the hidden layers. Experimental evidence indicates that neural networks with just one hidden layer perform as well as those with more than one hidden layer and that the former are less prone to be trapped at a local minimum of the error function while being trained [26].

The hidden unit activation function for our neural networks is the hyperbolic tangent function. This function is used because it can be approximated easily by piece-wise linear functions with relatively good accuracy. We attempt to reduce the number of rules by pruning the redundant network units. The continuous activation function of each hidden unit is then approximated locally by a three-piece linear function. Unlike in our previous method where this linear function is obtained by minimizing the area bounded by the hyperbolic tangent function and the linear approximating function [19], we compute the coefficients of the piece-wise linear approximating function such that the sum of squared errors computed from the training data samples is minimized. By minimizing the sum of squared errors, a piece-wise linear approximating function that better fits the training data can be obtained. The various combinations of the approximating linear functions divide the input space into subregions such that the function values for all inputs in the same subregion can be computed by a predicting linear function of the inputs.

This paper is organized as follows. The next section describes our neural network training and pruning. In Section 3 we describe how the hidden unit activation function of the network is approximated locally by a three-piece linear function such that the sum of the squared errors of the approximation is minimized. In Section 4 we present our approach which generates a set of regression rules from a neural network. Two examples illustrate how the decision rules can be extracted from the pruned networks in Section 5. In Section 6 we present our results and compare them with those from other methods for regression. Finally, in Section 7 we conclude the paper.