Robustness of texture-based roundwood tracking

The first two datasets Forest Lumix (FL) and Forest Huawei (FH) were acquired at the forest (see Fig. 2a) using a Lumix camera and a Huawei smartphone, respectively. Both datasets consist of 4 images per log end. After two images the camera was rotated by approximately 45 degrees and two more images were captured. In that way the logs are captured under different rotations as can be seen in Fig. 3.

The log ends were cut with a chain saw (no fresh cut) and are subjected to scratches and dirt due to the transport in the forest and color changes due to the storage. The annual ring pattern is only poorly visible.

The next dataset, denoted as Sawmill dataset (SM), was captured at the sawmill after cutting off a thin disc from each log end (see Fig. 2b). Three images of each freshly cut log end (chain saw cut) were taken at different rotations using the Huawei smartphone. The visibility of the annual ring pattern is better than for the forest datasets but still quite poor. The CS-images of the two forest and the Sawmill dataset were taken without tripod which causes slight variations in the perspective toward the CS and the positions of the CS in the image.

The Sanded (S) dataset consists of images from the 200 discs that were cut off at the sawmill from both sides of each log (see Fig. 2b). The images were acquired using a Canon EOS 70D with a tripod and lighting in the laboratory after the discs got sanded (in order to offer a better visibility of the annual ring pattern). The CS-images are taken only from one side of the discs (the side with the fresh cut at the sawmill) and so are basically mirrored versions of the CSs in the Sawmill dataset. Six CS images with different rotations were recorded for the Sanded data set. We denote the Forest, Sawmill and Sanded datasets from now on as RGB datasets.

Finally, the 100 logs with the removed discs from both ends of the logs are once again recorded using a CT scanner. About all 5 mm along the length of a log a CT image is taken from the CS, resulting in about 890 CT images per log. If not mentioned otherwise, we only employ the first and the last 15 images for log recognition, those that are most close to one of the ends of the log. In that way, this dataset is acquired in a similar way as the RGB datasets. The CT images of a log all have the exactly same rotation, scale and perspective. Since the images were taken at slightly different longitudinal positions of the log, images of the same log can show different knots and hence there are clear differences even between the images of one side of a log. We denote this dataset as CT dataset. CT images offer a nearly perfect visibility of the annual ring pattern in the heart wood but hardly any information in the sapwood.

The number of images per log end and the total number of images of all datasets are presented in Table 1. In our log tracking experiments, the two ends of a log are considered as different classes (the logs are 4.5 m long and so there is hardly any similarity between the two log ends of the same log).

Figure 4 includes a schematic illustration of the acquisition of the 5 different log datasets and exemplary images of one log from each of the datasets. It can be observed that the CS-Images of the two forest datasets (FH and FL) look quite similar since the images were taken with the same surrounding. The CSs share the same saw cut pattern and there is hardly any time shift between the taking of the images of the two datasets. The CS-Images captured at the sawmill yard (SM) look completely different from the images of the forest datasets because of the fresh cut that results in a totally different saw cut pattern and wood coloration. The CS-Images of the sanded discs (S) are captured under idealistic conditions and show an undisturbed annual ring pattern. The CT images look completely different from the RGB images because of the different imaging modality.

Prior to any feature extraction, the log CS area in the CS-Image is localized and segmented from the background. The segmentation is done in the same way as in our previous work on CNN based log tracing (Wimmer et al 2021a). We apply the Mask R-CNN framework (He et al 2017) to get a segmentation mask. As net architecture we employ the ResNet-50 architecture using a model pretrained on the COCO dataset. The segmentation net is then fine-tuned on the MVA log CS dataset (see Wimmer et al (2021a)), a dataset similar to our employed HLDB RGB datasets for which manually segmented masks are available. The segmentation net is trained for 30 epochs in order to differentiate the log CS from the background. Then, the fine-tuned segmentation net is applied to the RGB HLDB datasets to segment the CS from the background.

The obtained segmentation mask of the CS is further used to set the background to black. Finally, each CS-Image is reduced to the smallest possible square shaped image section so that the segmented CS is still completely included in the image together with a five pixel thick black border on each side of the image. The schematic representation of the segmentation and the extraction of the square shaped image patch containing the CS is displayed in Fig. 5. In Fig. 6, we present exemplar outcomes of the CNN segmentation and patch extraction process for the RGB CS datasets.

For the CT images, CNN-based segmentation is not feasible because of the lack of manually segmented CT log images to train the segmentation net and the too big difference between the RGB images of the manually segmented MVA dataset and the CT images.

However, the CT images are easy to segment since they have a mostly black background outside of the CS. Like in our previous publication on log tracing using RGB and CT images (Wimmer et al 2021b), we apply the active contour (AC) method (Chan and Vese 2001) for the segmentation of the CT images. For this we employ the Matlab function ’activecontour.m’. As initialization, a 223x223 centered square (Fig. 7b) is used and AC is applied for 500 iterations. The AC convergence process after 50, 250 and 500 iterations can be viewed in Fig. 7c–e. Finally, three morphological operations are applied to the AC output (Fig. 7f). First, all connected components with less than 1000 pixels are removed, then morphological closing is applied using a disc shaped structuring element with radius = 3, and finally holes are filled in the segmentation mask.

The segmentation outcomes on the Sawmill, Sanded and CT dataset all look perfectly fine based on the authors’ visual impression. For the two forest datasets (FH and FL), most images are well segmented, but on some images, parts of the log CS are predicted as background (see Fig. 6d) or bits of background surrounding the log CS are predicted as log CS (see Fig. 6f).

The advantage of our proposed segmentation and square image patch extraction approach for log recognition is that the background of a log CS image does not influence the log recognition. For CNN based recognition systems, where the images usually have to be resized to a fixed size before feeding them through the network, an additional advantage is the reduced loss of image quality. The segmented square shaped image patches are clearly smaller than the original CS-Image and so less information of the log is lost by reducing the image resolution to fit the required CNN input size. The disadvantage of the segmentation is that resizing the segmented images to the required CNN input size destroys information on the size of the log, since based on the segmented and resized image it is not possible to deduce how big the log CS was in the original image.

To counter the problem of the practically not existing visibility of the sapwood in CT images (see CT image in Fig. 4), we additionally extract smaller square shaped image patches with a side length of 40% of the segmented images that are centered at the pith. Hence, the image patches only include areas of the heartwood from the log CS but not the sapwood as can be seen in Fig. 5. This also means that the HW patches do not include information on the shape of the log CS. We employ a method from Schraml and Uhl (2013) to detect the pith using the tree ring pattern. We further denote those image patches as Heartwood (HW) images and employ those HW images for all experiments using CT images (additionally to the experiments using the segmented images with black background). The HW images of the RGB image datasets are transformed to grayscale to reduce the domain shift between CT and RGB images.

In biometric applications, the problem with CNN loss functions that learn the network to classify images (e.g. the cross-entropy loss) is that CNNs are only able to identify those subjects which have been used for the training of the neural network. If new subjects are added in a biometric application system, then the CNN needs to be trained again or else a new subject can only be classified as one of the subjects that were used for training. This makes the practical application of CNNs trained with these loss functions impossible for any biometric application including log tracking. This problem does not apply for CNNs that learn an embedding (feature vector output) instead of a class assignment, like e.g. CNNs trained with the triplet loss function. The triplet loss (Schroff et al 2015) does not learn the CNN to classify images, but to produce feature vectors that are similar for images of identical classes and different for images of different classes. The triplet loss has already been used successfully in two of our previous works on wood log tracking (Wimmer et al 2021a, b). The triplet loss is applied to three training images (a so called triplet) at once, where two images belong to the same class (the so called Anchor image and a sample image from the same class, further denoted as Positive) and the third image belongs to a different class (further denoted as Negative). The triplet loss using the squared Euclidean distance is defined as follows:

where A is the Anchor, P the Positive, N the Negative and \(\Vert f(A)-f(P) \Vert ^{2}\) is the squared Euclidean distance between f(A) and f(P). \(\alpha\) is a margin that is enforced between positive and negative pairs and is set to \(\alpha =1\) (same as in Wimmer et al (2021a, 2021b)). f(x) is an embedding (the CNN output) of an input image x (Fig. 8).

The optimization goal of the triplet loss is to minimize L(A, P, N), that means making the distance between CNN outputs from images of the same class (\(\Vert f(A)-f(P) \Vert\)) smaller than distances of CNN outputs from images of different classes (\(\Vert f(A)-f(N) \Vert\)). In that way the images of same classes (log ends) are clustered together in the CNN output space and images of different classes are apart.

The max operator in Eq. (1) is used because a loss function cannot be smaller than zero. That means if \(\Vert f(A)-f(P) \Vert ^{2} +\alpha < \Vert f(A)-f(N)\Vert ^{2}\), then L(A,P,N) is set to zero. Hence, also the gradients for back-propagation are zero, which means the CNN is not changed based on this triplet (which is perfectly fine because for this triplet the images of the same class (A, P) are closer together in the CNN output space than the images of different classes (A, N)).

To avoid the problem of selecting triplets for training that cannot improve the CNN, we employ hard triplet selection (Schroff et al 2015). For hard triplet selection, only those triplets are chosen for training that actively contribute to improving the model (triplets with \(L(A,P,N)>0\)). Given a batch of images from the combined training data of two different training datasets, the Anchor, Positive and Negative are chosen using hard triplet selection from the batch (no matter which dataset the images are from). So a triplet (A, P, N) can consist from images of the same training dataset or from images from the two different training datasets. However, since CS images of the same log end and the same dataset are quite similar (see Fig. 3), the hard triplet selection mostly selects triplets where the Anchor and Positive are from different datasets in order to fullfil the triplet selector’s criteria (\(L(A,P,N)>0\)).

A similar approach was already applied in Wimmer et al. (2021b) for log tracking using images of the CT and Sanded dataset. However, in Wimmer et al, (2021b) the Anchor image had to be selected from the dataset to be identified (CT), whereas the Positive and the Negative had to be selected from the other dataset (Sanded) in order to specifically train the net for cross-dataset (CD) recognition. In this work on the other hand, there are no constraints from which dataset the Anchor, Positive and Negative have to be chosen. This achieved better results for all applied experiments in this work except for log tracking using images of the CT and Sanded dataset.

Same as in Wimmer et al (2021b), we employ the Squeeze-Net (Sq) (Iandola et al 2016) architecture. Sq is a small neural network that is specifically created to have few parameters and only small memory requirements. This allows big batch sizes which is important in the later stages of training, where it is already hard for the triplet selector to still find triplets that can improve the net (triplets with loss >0). Sq is pre-trained on the ImageNet database (http://​www.​image-net.​org/​) and fine-tuned on the respective HLDB datasets. The size of the CNN’s last layer convolutional filter is adapted so that a 256-dimensional output vector (embedding) is produced. To make the CNN more invariant to shifts and rotations and simultaneously increase the amount of training data, we employ data augmentation for CNN training. The images are randomly rotated in the range of 0-360\(^{\circ }\) and random shifts in horizontal and vertical directions are applied by first resizing the input images to a size of \(234\times 234\) and then extracting a patch of size \(224\times 224\) (the best working input size using the Sq for log recognition) at a random position of the resized image (\(\pm 5\) pixels in each direction). The implementation of the network was realized in PyTorch. A batch size of 150 images is used for training (5 images per class from 30 different classes). The CNN is trained for 400 epochs with the Adam optimizer, starting with a learning rate of 0.001, which is divided by 10 every 120 epochs. We further define this CNN approach as ’Sq-T’ (’Sq-T\(_{CD}\)’ using the slightly different approach from Wimmer et al (2021b)).

A more common approach than using the triplet loss is to train a net with the cross-entropy (CE) loss function (L\(_{CE}\)):where n is the number of classes, \(t_{i}\) is the truth label, \(p_{i}\) is the Softmax probability and \(s_{i}\) is the CNN output for the \(i^{th}\) class. As network architecture we employ the DenseNet161 (Huang et al 2017), which connects each layer to every other layer in a feed-forward fashion.

Same as for Sq-T, the DenseNet is pretrained on the ImageNet database and fine-tuned on the respective HLDB datasets. For each experiment, the CNNs are fine-tuned using the combined training data from two different HLDB datasets (same as Sq-T). As already mentioned in the previous section, CNNs with cross-entropy loss can only classify images of those classes the CNN was trained with. To avoid this problem, we turn the CNN to a feature extractor after the CNN got trained and fine-tuned. A common approach for that is to extract the CNN output of intermediate layers (e.g. Razavian et al (2014)). During training, the classification layer of DenseNet is learned to map the output of the penultimate layer (a 2208 dimensional vector) to the number of classes (n) in the training data, so that each class is assigned an CNN output (\(s_{i}\) for the \(i{\text {th}}\) class as in equation 2). So based on the output of the penultimate layer, the CNN classification layer learns to assign the images to classes (log ends). So, the output of the penultimate layer needs to include all necessary information to classify a log image. By removing the classification layer (the last CNN layer) of the trained and fine-tuned DenseNet, the CNN works as feature extractor and produces an image descriptor (a 2208 dimensional feature vector) instead of a class prediction. For evaluation, the images are fed to the CNN and thus a feature vector output is produced for each evaluation image (like for Sq-T). Contrary to class assignments, the feature vectors of images from classes that were not present in the training data can be employed for log recognition.

We use the same training parameters and data augmentations as for the triplet loss trained CNN, except of a batch size of 12 and a learning rate of 0.005, which is repeatedly divided by 10 after 200, 280 and 340 epochs. We further denote this CNN approach as ’Dense-CE’.

Additional to the CNN method, we employ two methods describing the shape of the log CS. The shape features are extracted from the binary segmentation masks (see Fig. 5) of the original images. The advantage of shape features is that the extracted shape of the log is the same for log images taken under different imaging modalities, different lighting conditions and for images taken with different cameras. The disadvantages of shape features is that there are slight variations of the shape of the log CS if the log images are taken from different longitudinal positions of the log. Furthermore, different viewpoints and scales at the image acquisition change the shape and size of the log CS, which can render the use of shape features pointless. Hence, shape features only make sense if the images of the log CSs are taken from identical viewpoints and scales, which is only the case for the Sanded and CT dataset. Hence, the shape features are only applied for the cross-dataset experiment with the Sanded and CT dataset.

The first shape feature is based on Zernike Moments and was already utilized in Schraml et al (2015c) for wood log recognition, but only for experiments on one single log dataset and not for cross-dataset experiments as in this work. We further denote this feature descriptor as ’Zernike’.

The second shape feature, further denoted as ’LogShape’, was already successfully employed in Wimmer et al (2021b) for wood log recognition for a cross-dataset experiment with images of the CT and Sanded dataset. The LogShape feature consists of three different features, the length of the mayor and minor axis of the ellipse that has the same normalized second central moment as the region of the log CS in the log mask, and the distance between the center of mass of the log CS and the pith (the middle point of the log).

Since the images of different datasets can have different resolutions, we need to normalize the shape feature vectors from both methods. Given a feature vector f of a log l, each element of each shape feature vector is normalized on each dataset separately as follows:where \(\overline{f}\) is the mean and \(\sigma (f)\) the standard deviation over all feature vectors from the considered shape descriptor of a dataset. In that way, we balance the different resolutions and can directly compare shape features from different datasets.

In an additional experiment, the shape features are combined with CNN features. This is done by concatenating a CNN feature vector with a shape feature vector, where the CNN feature vector is from the segmented Heartwood image and the shape feature vector from the segmentation mask, both extracted from the same original log image.

To balance the clearly higher dimensionality of the CNN feature vectors (Sq-T:256, Dense-CE:2208, Zernike:36, LogShape:3), the shape features are multiplied by a factor of 5 (each element of the shape feature vector is multiplied by 5) same as in Wimmer et al (2021b). The shape feature descriptor is combined with the CNN descriptor applied to the Heartwood images. This is a perfect combination since the Heartwood images contain neither direct information on the shape of the log (since they are extracted from the middle of the log cross section) nor about the size of the log (since the Heartwood images get resized to the CNN input size), which is both supplied by the shape features. We further denote this combination of features as ’CNN+LogShape’ or ’CNN+Zernike’ (where CNN is either Sq-T or Dense-CE).

Additionally, we apply two descriptors that come from the biometric fields of fingerprint and iris recognition that were applied for log tracking on single datasets in previous works but not for cross-dataset experiments. Both, the iris recognition method (Schraml et al 2016b) and the fingerprint recognition method (Schraml et al 2020), are based on Gabor filters.

Unfortunately, it turned out that both the iris and the fingerprint based method are completely unsuited for all cross-dataset log tracking experiments. While the methods worked for experiments on single datasets, they achieved terribly poor results that are equal to random feature descriptors in all cross-dataset experiments applied in this work. Hence, we refrain from showing the results of the two methods in all coming experiments.

In this work we apply three different cross-dataset experiments. In the first experiment (Forest-Forest), we employ images of the two Forest datasets and the aim is to identify images of the Forest Huawei dataset by means of images of the Forest Lumix dataset. In the second experiment, we identify images of the Sawmill dataset using images from the forest (Forest Huawei dataset). In the final experiment, we aim to identify CT images acquired at the sawmill using either RGB images from the Sawmill dataset, the Forest Huawei dataset, or the Sanded dataset.

We further define the dataset with the images to be identified as evaluation dataset. The other dataset in our cross-dataset is denoted as gallery dataset. For the gallery dataset, each of the two sides of each log is assigned a unique identity. The images of the evaluation set are identified by comparing them to the ones of the gallery dataset using our image descriptors.

In our experiments we employ a fourfold cross validation for all employed methods. For each combination of two different datasets, the CNN is trained four times, each time using three of the folds for training and evaluation is applied on the remaining fold. Each fold consists of the images from 25 logs (a quarter of the 100 logs) from both datasets. That means the images of one log (no matter which dataset) are all in the same fold.

We have to consider that each of the four trained CNNs per experiment (one per fold) has a different mapping of the images to the CNN output feature space. Thus, feature vectors of different folds cannot be compared and therefor the performance measures have to be computed for each fold separately using only similarity scores between images of the same fold.

As already mentioned before, the datasets consist of images from both log ends, which show no obvious visible similarities. To employ the maximum number of images for CNN training, both log ends are considered as different classes, thus resulting in 200 classes in total. To avoid any bias by assigning different classes to the two sides of the same log for Sq-T, we exclude those triplets during training where the Anchor and the Negative (which is selected from the images of a class other than the Anchor) are from the same log but different sides. For log recognition/identification on the evaluation fold we proceed in a similar way and omit all comparisons between images from different sides of the same log for both CNN approaches.

For a better comparability to the other employed methods, also the non-CNN methods are applied using fourfold cross validation in the same way as for the CNNs (similarity scores only between images of the same fold and not between images from different sides of the same log).

As performance measures we compute the equal error rate (EER) as well as the Rank-1 recognition rate.

The EER is the error rate of a verification system when the operating threshold for the accept/reject decision is adjusted such that the probability of false acceptance and that of false rejection become equal. So actually, computing the EER is an artificial scenario since we aim to identify the logs and not to verify their identity. However, the EER is widely used in biometrics and a quick way to compare the accuracy of devices with different receiver operating characteristic (ROC) curves. In general, the device with the lowest EER is the most accurate. For the EER computation, we employ the genuine and imposter scores of all genuine (image pairs from the same log and log side) and imposter comparisons (image pairs from different logs), where the image pairs come from different datasets (one image from the gallery dataset and one from the evaluation dataset). For all employed methods, the similarity score between two images is computed using the Euclidean distance between the feature vectors of the images. To transform the Euclidean distance to a similarity metric, the Euclidean distances are inverted (\(d \rightarrow 1/d\)) and normalized (for each fold separately) so that the resulting similarity scores range from zero to one. We report the mean EER over the EERs of the four folds.

The computation of identification results (rank-1 recognition rate) is applied using two different protocols and is also applied for each fold separately.For both scenarios, the recognition rate is defined as the number of correct class assignments (from all four folds) divided by the total number of class assignments (which corresponds to the number of images of the evaluation dataset for the first scenario and m times the number of images of the evaluation dataset for the second scenario).

In Table 2 we present the results of the Forest-Forest cross-dataset experiment, where we identify the images of the FH dataset using the images of the FL dataset.

As we can observe, Dense-CE achieves better results than the triplet loss trained CNN (Sq-TD). For Dense-CE, the EER is 1.3 % and the recognition rates are over 99%. Sq-T performs slightly worse with an EER of 2.5 % and recognition rates between about 97% (SRR) and 99% (MRR).

In Table 3 we present the results of the Forest-Sawmill cross-dataset experiment where we identify the images of the Sawmill dataset using the images of the FH dataset.

We can clearly observe that the outcomes of this experiment are really bad. For Dense-CT, the EER is 36.9% with recognition rates of about 12%. Sq-T is even worse with an EER of 38.3% and recognition rates of about 4.5%.

In Table 4 we present the results of the three RGB-CT cross-dataset experiments where we identify the images of the CT dataset using either the images of the FH dataset, the Sawmill dataset or the Sanded dataset. Shape features were only employed for the CT-Sanded experiment, since the shape features do not work for datasets where the images were taken under different scales and viewpoints. The best results for each of the three RGB-CT cross-dataset experiments are marked in bold letters.

As we can observe, the results of the Forest-CT and Sawmill-CT experiments are really bad for both CNN approaches. Sq-T achieves EERs of about 40% for the Forest-CT and Sawmill-CT experiments for Heartwood images (HW) as well as images of the whole log CS. For the whole log CS images, Dense-CE achieves EERs of slightly under 40% and recognition rates between 9 and 12% for the Forest-CT and Sawmill-CT experiments. Using the Heartwood image patches, Dense-CE does not work at all with EERs of about 50%.

The results for the Sanded-CT experiment are distinctly better than for the two other CT-RGB experiments. For the CNN approaches, results are clearly better using the filtered image patches (filt.). Sq-T performs better using only image information from the heartwood (HW) whereas Dense-CE performs slightly better using images of the whole CS. The triplet trained CNNs clearly perform better than Dense-CE. The best CNN results achieve an EER of 13.9% (Sq-T \(_{CD}\)(HW, filt.)) and 15.3% ( Sq-T (HW, filt.)) with recognition rates between about 50 and 58%. The two shape features perform worse than the triplet loss trained CNNs with EERs of 16.7% (LogShape) and 20.5% (Zernike) and recognition rates of around 35–40% for LogShape and around 15% for Zernike. The overall best result is achieved by combining Sq-T and LogShape with an EER of 11.8% and recognition rates of about 72% (Sq-T (HW,filt) + LogShape).

We have shown in our experiments that log tracking works well in the case of the Forest-Forest cross-dataset experiment, very poorly in the case of the Forest-Sawmill, Forest-CT, and Sawmill-CT experiments, and at least partially in the case of the Sanded-CT experiment. Now we aim to methodically find out the reasons why some of the experiments work better and some worse. For this, we analyze the different challenges in wood log cross-dataset recognition separately to find out which challenges do pose severe problems and which are rather negligible. This is done by applying new experiments but also by using findings from the previously shown experiments and previous publications.