Human identification through panoramic dental radiographs: a novel matching approach

Biometric systems are designed to identify individuals based on unique personal characteristics, such as fingerprints, face, voice, eyes, and text. These features are highly successful for person identification. However, in cases of severe damage to the body, such as in fires or earthquakes, these biometric features may not be usable for person identification after death. Teeth are considered to be more resilient than other organs used for biometric identification, which makes them a more reliable feature in cases of severe accidents or injuries. Thus, dental features are preferred when other biometric features are not usable. While DNA detection technology is also a method of identification, it can be time-consuming and costly [1]. When other biometric features are not available, forensic dental scientists examine the oral and dental features of the individual to make identifications [2]. Forensic dentists search a specific database that includes information about the person's teeth and jaws. The use of computers in these procedures can aid forensic dentists and accelerate the identification process. It can also allow for searches of larger databases [3]. Panoramic radiography images are X-rays in which the entire mouth is viewed panoramically. These images are used to observe various diseases such as dental and bone anomalies, cysts, tumors and infections [4‐7]. The main stages of person recognition, after preprocessing and segmentation, are feature extraction and matching. In dental image recognition, distinctive differences are identified from the segmented images to facilitate the identification of an individual. These differences are then extracted as features, and the person's information that best matches these features is obtained through matching techniques. These stages are crucial for accurately identifying an individual based on their dental features. During the segmentation, feature extraction, and matching stages of dental image recognition, there are several challenges that need to be addressed. These include variations in dental radiography images taken at different times, inadequate segmentation success, use of images from different types of machines, dental procedures performed on the teeth and jaws of individuals, and images with high noise. These challenges can lead to inaccuracies in identifying an individual based on their dental features, and it is crucial to account for them in the recognition process. In Fig. 1, it can be seen that the noise causes the brightness of the teeth to change. Figure 2 illustrates the changes in a person's teeth after getting braces, and Fig. 3 shows the variations in X-ray images obtained with different machines.

Segmentation is a critical step in the dental X-ray analysis process, as it allows for the extraction of features and information from dental images [8‐10]. Segmentation is usually the first part of the dental human identification process [11‐13]. Despite the fact that dental human identification studies are limited, a number of methods for dental X-ray segmentation have been proposed. The segmentation method to be used can be chosen according to the type of X-ray used, the purpose of the application, and the amount of data available. Different methods established for panoramic X-rays [8, 14‐17] periapical X-rays [18, 19] and bitewing X-rays [20, 21], which are common among dental imaging types. On the other hand, segmentation can be applied not only for dental human identification [17, 22, 33, 34] but also for dental work identification [23, 24], tooth extraction or classification [18, 23] or dental disease detection [25, 26]. Depending on the approach of the dental person recognition process, different methods, such as dental work or tooth segmentation can be applied. Deterministic methods [27, 28] can be used for segmentation, as well as training-based methods [26, 29] such as neural networks. Similarly, the human-matching approach to be applied is important for the choosing the type of segmentation approach to be used. However, another important point here is the amount of available data. Deep learning-based segmentation approaches, such as CNN or Mask R-CNN, require a substantial amount of data, high computing capability, and training time [29, 30].

In the literature, various methods are available for extracting features from dental radiographs to recognize individuals. Studies have utilized techniques such as keypoint-based methods [5, 6], PHOG [33], Hu moments [34], Zernike moments [35, 36], GLCM [37, 38], and Fourier descriptors [13, 37]. When matching source images with a database, distance measures are commonly used. Popular methods include sum of difference squares (SSD) [33], Euclidean distance [34, 39‐41] Hausdorff distance [13, 22, 42], and Mahalanobis distance [38]. Although the proposed algorithms have demonstrated success, their performance varies across different databases due to the lack of a standardized database. Moreover, the desired level of success has not been consistently achieved, and most studies have not been tailored specifically to panoramic X-rays.

This study contributes to the literature by presenting a new method for the identification of individuals with panoramic dental X-ray images. In the literature, the identification of individuals using dental radiography images can be made through two approaches: i) teeth-based [3, 33] and ii) jaw-based [32, 35]. Teeth-based approaches involve segmenting the boundaries of each tooth and extracting features from each tooth, which are then compared to the features of all teeth in the database to identify a match. Meanwhile, jaw-based approaches extract features from the entire jaw in both the target and source X-ray images, which are then compared to the jaw features in the database to identify a match.

The most significant methods developed for dental human identification with panoramic dental X-ray images in recent years are [12, 24, 29, 31, 33, 50]. In [31], the authors used curvature scale space (CSS) transform as the feature extraction method and used Euclidean distance to investigate similarities. In [24], instead of using teeth or jaws as the source of features, they used dental works on the jaw or teeth. In a more recent study, [33] proposed a teeth-based method, using pyramid histogram of oriented gradients (PHOG) on teeth. They used support vector machine (SVM) and sum of squared differences to calculate similarities. In [12], Gurses and Oktay proposed using a deep learning-based segmentation method and Speeded Up Robust Features (SURF) for the features. In both [29] and [50], the authors proposed a deep learning architecture for dental human identification.

When the literature is examined, it is understood that dental person identification studies are still limited today. Especially, it can be said that person recognition systems with panoramic dental X-rays are much more limited [29]. The proposed work contributes to the literature by presenting a novel approach that can be used for person identification with panoramic X-ray images. The highlights of the study are as follows: it can be used even with small datasets without the need for an additional training process, it offers a high matching performance comparable to neural network-based studies, it can be used with different segmentation methods and is less affected by segmentation errors, and it is less impacted by the missing tooth problem due to teeth-jaw matching.

In the article, the X-ray image of the unknown person is referred to as the target image, and the X-ray image database and images of potential matching persons are referred to as the source database and source image, respectively.

Teeth-based approaches have the advantage of being able to examine teeth features independently, which means that even if many of the teeth in the jaw are missing, the person can still be identified in the target database. However, extracting clear tooth boundary information from dental X-ray images can be challenging due to overlapping teeth, machine noise, and dental work-related issues. Additionally, if there are no teeth in the jaw or all existing teeth are prosthetic, these methods may not be effective.

The teeth matching approach is based on measuring the sequential similarity of teeth or tooth regions extracted from two X-ray images. In this perspective:

An example of such a tooth matching approach is given in [5]. An illustration flow of a teeth-based approach can be seen in Fig. 4.

The jaw-based approach uses information from the entire jaw. Since each tooth is not analyzed separately, the processing time is shorter compared to the tooth matching method. In this approach, the jaw may be identified even if no teeth are present on the jaw. It is also possible to identify a person in images consisting of dental repairs and prosthetic teeth.

The disadvantage of the jaw-based approach is that individual teeth cannot be analyzed separately, which means that if some of the teeth are damaged, the chances of identifying the person in the database would decrease. For instance, in cases of accidental death, if some of the teeth are lost or damaged, it may be more difficult to identify the person. An example structure for jaw-based matching can be seen in Fig. 5.

In this paper, a novel matching approach for person identification in panoramic X-ray images is proposed. To the best of the author's knowledge, this is the first time that this approach has been presented for person identification in panoramic radiography images. In the proposed approach, the target image is segmented into regions of teeth, and then keypoint features of these teeth are extracted. As the segmentation method, we used the meta-heuristic supported semi-automatic segmentation method [16] and the other segmentation method we proposed, the fully automatic segmentation method [17]. Segmentation was performed using both methods, and the results were analyzed and compared. In the proposed approach, keypoint features are extracted from the segmented regions of teeth in the target image, and similarly, keypoints are extracted from the jaw images in the database. It should be noted that the database images are not divided into tooth regions. The method then matches the tooth features of the target image with the jaw features of the database. An example structure of the teeth-jaw approach is shown in Fig. 6.

The advantage of this approach is that it allows searching with a limited number of teeth and reduces the number of searches to be performed. In this way, the proposed method is a solution to the problems of excessive number of searches and insufficient utilization of non-dental regions in the teeth-based approach. A balanced solution is presented in terms of speed and matching performance.

In this study, local keypoint-based algorithms are used for feature extraction. A comparative analysis of local keypoint-based feature extraction methods is presented. The algorithms were evaluated in terms of matching success and execution time. The matching performances of each jaw are considered separately. In this way, the effect of jaw selection on performance is observed. Two different segmentation methods are used, and the effect of segmentation on performance is analyzed. The experimental results show that the proposed approach can be useful in person identification. It is observed that the performance rate of matching the target image in the first 10 images is quite high. The experimental results are promising in this respect.

Section 2 of the paper describes the materials and methods used in the proposed work. The dataset used in the proposed work, i.e., X-ray images, is summarized and illustrated in Sect. 2.1. Section 2.2. describes the method of the proposed work, and the steps of the proposed approach are given graphically. In Sect. 2.2., first, the proposed teeth-jaw-based matching approach (TJMA) is explained. Then, the process of the keypoint-based methods utilized in TJMA is described. In the Method section, the Detection of matching jaw subsection describes the feature matching approach of the proposed method in detail. In addition, a flow diagram summarizes the operation of the system. Section 3 describes the experiments and results of the proposed work in different perspectives. The experimental results and comparisons with similar works in the literature are discussed in Sect. 4. Future directions are explained in Sect. 5. Finally, the conclusion of the study is presented in Sect. 6.

In the tooth-jaw matching approach, various keypoint-based features are utilized for person detection. Feature extraction algorithms are applied to the regions obtained in the segmentation phase. Next, feature extraction is performed on the images in the database. Since the features of the entire jaw are extracted during database feature extraction, segmentation is not necessary. The extracted feature points are then matched, and it is expected that the number of matches will be higher in images with a high similarity ratio. Based on this, the images with the highest number of matches are determined to be the matching images. SIFT, SURF, MSER, and KAZE features are all used for feature extraction.

The matching results for each of these features are presented in Table 1 (SIFT), Table 2 (SURF), Table 3 (MSER), and Table 4 (KAZE), respectively. The rows of the tables contain the matching results for different segmentation methods and jaw regions, while the columns show the ratios of matching images found in the first n images (where n = 1, 2, 3, 4, 5, 10, 15).

For each method, two different segmentation methods were tested. These are Semi Auto Segmentation and Auto Segmentation, which we have previously presented in [16] and [17], respectively.

Comparisons of the matching results obtained are given in Fig. 17, Fig. 18, Fig. 19, and Fig. 20. Figure 17 and Fig. 19 show the comparison of SIFT, SURF, MSER and KAZE features for mandibular jaw.

Figure 18 and Fig. 20 show the comparison of SIFT, SURF, MSER and KAZE features for maxillary jaw (Fig. 21).

In addition to the matching success, the proposed feature extraction methods are compared in terms of time. Figure 22 shows the average time needed to identify a person in the given database.

The proposed study evaluates two methods, one semi-automatic and one fully automatic, for identifying tooth regions. In the semi-automatic method, separation curves are determined after jaw separation, and the user selects the potential positions between consecutive teeth on the upper separation curve. Separation lines are then created perpendicular to the curve from the points selected by the user, and these lines are automatically oriented in accordance with the orientation of the teeth. This method offers the advantage of generating no missing or excessive separation lines. However, it also has a disadvantage in that it is semi-automatic, and the user's involvement in the segmentation process increases the user's workload and sensitivity to errors.

The fully automated method, on the other hand, automatically detects the separation lines with high accuracy. However, there is often an overestimation of accuracy, which is addressed by checking the rejects and automatically eliminating unnecessary ones. This method is preferred because it is fully automatic, reduces user workload, and is resistant to user mistakes.

In the next stage of the study, the person matching process is applied using a novel approach. Various experimental studies were conducted to evaluate this approach, and the resulting performances were analyzed.

When examining teeth-person matching studies in the literature, it becomes clear that the studies generally attempt to extract individual tooth boundaries and then detect the most similar tooth in the database. To the best of the author's knowledge, the teeth-jaw matching approach used in this study has not been previously used for person detection from panoramic dental x-ray images. This method offers several advantages over individual tooth matching.

The first advantage is that tooth boundary segmentation can be challenging, and incomplete or inaccurate determination of tooth boundaries can decrease the performance of the algorithm. In the proposed approach, it is not necessary to determine the tooth boundaries; instead, it is sufficient to determine the tooth region. Additionally, even if there are excess or missing segmentation regions, matching can still be performed using the obtained regions.

Another advantage of the teeth-jaw matching approach is the reduced number of searches required. Since the tooth region is matched with an entire jaw, the number of searches is significantly reduced. Let m denote the number of jaws and n denote the number of teeth. In this case, the search complexity is reduced from O(m²n²) to O(m²n).

Teeth-based approach, teeth-jaw-based approach and jaw-based approach are compared in Table 5. In Table 5, the pseudocodes for each approach are given and the complexity analysis is expressed.

For instance, suppose it is needed to search for 100 images with an average of 32 teeth within a database of 100 images. This means that it is needed to search for 320 target images over 320 database images, requiring 10,240,000 (3200 × 3200) searches. However, in the proposed method, since teeth and jaws are matched, only 320,000 (3200 × 100) searches are needed. The workload is significantly reduced, particularly since the number of jaws (m) is usually much lower than the number of teeth (n).

Jaw-to-jaw matching is the lowest approach in terms of computational load. Its complexity corresponds to O(m²). According to the example above, the number of searches required for jaw-jaw matching is 10,000 (100 × 100). On the other hand, in the jaw-to-jaw matching approach, the target jaw must be very intact. Also, individual teeth cannot be evaluated.

It is challenging to conduct a fair comparison since there is no publicly available dental panoramic dataset for comparing the human identification techniques. [33]. To the best of the author's knowledge, the most notable methods developed for panoramic dental images in recent years are [12, 24, 29, 31, 33, 50]. The method described in reference [31] was evaluated on a dataset consisting of 60 images, each belonging to one of 30 individuals. The results showed that the method achieved a rank-1 accuracy of 67%. However, this study does not present rank-5 or rank-10 accuracy rates. The relatively low rank-1 accuracy performance is a weakness of this study. The method described in reference [24] was evaluated on a dataset of 30 images of 20 different individuals. It achieved a success rate of 90%. However, it is important to note that the effectiveness of this approach directly depends on the presence of dental work in the images. Therefore, in datasets with a limited number of dental work images or in cases where dental work is absent, the performance of the method can be significantly degraded. Furthermore, as Oktay emphasized in [33], the performance of the method was also observed to degrade when dealing with different types of dental work.

In [33], it was found that about 81% of the images were successfully matched in the first order, indicating a high level of accuracy. Furthermore, 89% of the images were matched in the first or second rank, further emphasizing the efficiency of the system. It is worth noting that the 5th rank accuracy is 92%. This shows that a significant portion of the images is correctly matched within the first five ranks. Furthermore, the rank-10 accuracy reaches 94%. It is understood that the changes between rank-1, 2, 5 and 10 do not change very sharply. This implies a more stable performance progression. The downside of this study is that it employs a teeth-based technique, which results in increased computing complexity.

A novel deep learning approach for dental human identification was proposed in [29], where the authors developed and trained a CNN-based model using dental images. This was the first attempt to apply CNNs to this problem domain, and the results were promising. The model achieved 85.16% accuracy for rank-1 identification and 97.74% for rank-5 identification. However, a limitation of this method is that it is requires a training process and needs a large and diverse dataset for training, as CNN performance is highly influenced by the quality and quantity of data [29, 30].

A similar work is proposed in [50]. In the study, a novel deep neural network is proposed. It is provided 81.90% rank-1 and 91.22 rank-5 performance. As it seen in [29], this study also investigates jaw similarity.

Another study utilizing deep learning was presented in [12]. In this study, deep learning was used for tooth segmentation, and SURF features were used for matching. Rank-1 performance is 80.39% and rank-5 performance is 96.08%. This method and [50] have similar downside with [29] and [33]. First, it is a teeth-based method and has higher computer complexity. Second, it is a deep learning-based method and requires many dental X-ray images to train.

These deep learning methods focus on jaw-to-jaw similarity. This has the downside that individual tooth matching is difficult. In post-mortem x-rays, some of the teeth may be deteriorated or missing. In this case, the person needs to be matched with the remaining teeth. While this deterioration or loss complicates all approaches, it becomes a more significant problem for methods that primarily search for similarity between jaws. However, in tooth-to-tooth matching or tooth-jaw matching approaches, as in the proposed study, it is possible to examine the matching pattern of individual teeth.

Characteristics of other person matching studies in the literature are summarized in Table 6.

In the Table 6, the tooth-jaw matching approach proposed in this study is abbreviated as TJMA.

In the proposed work, deterministic approaches are used for both tooth segmentation and tooth matching that do not require a training process. Still, it is seen that high performances can be achieved comparable to the approaches utilizing deep learning.

The proposed approach is evaluated with multiple methods, and it is observed how well the methods provide suitable results for dental matching. In this respect, it gives an insight into the performance of different keypoint-based approaches for dental human identification. Another contribution of the proposed study, different from other studies, is that the stability of the method is tested by using different segmentation methods. This study examines whether the proposed method is robust to segmentation errors. As seen in Tables 1, 2, 3 and 4, the method provides consistent results for both semi-automatic and automatic segmentation.

In the proposed work, a teeth-jaw-based matching approach is introduced as a novel method for dental person recognition. The study focuses on keypoint matching, necessitating the evolution and enhancement of various keypoint-based feature extractors. Significant progress can be achieved in this field by creating or improving keypoint approaches tailored for dental X-ray images. Preprocessing, i.e., image enhancement, of dental images is also important in this respect. Image enhancement methods focused on dental images will help to better detect keypoints. Finally, it is planned to advance studies on matching without certain parts of dental images.

This study presents a novel matching approach to use teeth as a reliable biometric feature to identify individuals following serious injuries caused by accidents or disasters. It discusses the performance and potential of this method. The current process of identification through dental records performed by forensic dentists is known to be time-consuming and laborious. Therefore, this research is aimed to present a computer-aided methodology to streamline and improve the identification process performed by forensic dentists.

Identification of individuals using dental features can be achieved by comparing each tooth of the individual with records in the database or by comparing the features of the source jaw with those in the database. Previous research has predominantly focused on jaw-to-jaw (jaw-based) or tooth-to-tooth (tooth-based) matching approaches. However, this paper proposes a new tooth-jaw-based approach. This method involves extracting keypoint-based features for each tooth and then matching these features with jaw keypoints in the database. Notably, this study marks the first time that such an approach has been applied for human identification using panoramic X-ray images. The proposed methodology combines the advantages of using jaw keypoints for matching in the database while simultaneously allowing the extraction of dental features.