Image-based search and retrieval for biface artefacts using features capturing archaeologically significant characteristics

The image analysis and pattern recognition literature contain several works related to feature extraction, similarity grouping, and classification of archaeological artefacts.

As part of Graphically Oriented Archaeological Database project, the authors in [21] presented an automatic method for shape extraction from raster images of line drawings to facilitate matching and retrieval. Smith et al. [32] developed a scheme to classify thin-shell ceramics based on colour and texture descriptors in order to aid in vessel reconstructions, using a new feature based on total variation geometry along with SIFT (scale-invariant feature transform). Abadi et al. [2] present a system for automatic texture characterization and classification of ceramic pastes, fabrics, and surfaces. They use Gabor filter along with linear discriminant analysis and k-nearest neighbour techniques in order to achieve the desired objectives. In [5], the authors analyse surface properties in pottery and lithic artefacts. They define texture with attributes such as coarseness, contrast, directionality, line-likeness, regularity, and roughness. Durham et al. [13] use generalized Hough transform as a practicable and robust tool for matching whole and partial artefact shapes.

Few examples of image-based identification systems for archaeological artefacts also exist in the literature. The work in [34] illustrates the use of computer vision techniques for the development of content-based image retrieval system for historic glass and an automatic system for mediaeval coin classification. In another work by [36] a prototype is presented which allows the end-user to search for similar digital library objects based on the image content. An integrated content and metadata-based retrieval system for art images in the domain of museum and gallery image collections is presented in [22]. Image retrieval methods are proposed to perform query on the basis of subimages. It also presents methods of querying by very low-quality images. IBISA [26] is a software tool that allows the user to perform image-based searches on the database of digital images of archaeological objects such as ancient coins. This system performs image segmentation with a method based on active contours and image registration with Fourier–Mellin Transform and computes similarity with classic intercorrelation factor. An extension of this system which is robust to lighting conditions [25] has also been proposed recently. CLAROS [10] allows image-based searches for sculpture and pottery images based on scale-, illumination-, and viewpoint-invariant image patches encoded using a bag-of-words scheme.

While the use of images, either hand drawings or photographs, remains the most common visual record of found lithics artefacts, there are opportunities to use 3D scanning technology to improve various aspects of archaeological work. For example, Grosman et al. [15] proposed the use of 3D scanning for purposes of documenting the essential characteristics of lithics in a standardized way that is less open to interpretation and error. Lin et al. [23] proposed the use of 3D scanning for analysis of lithic artefacts by using precise measurements of the cortex recorded in scans to make approximations of the size of artefacts. Finally, there are examples of people using 3D scanning for more specific types of analysis, such as Evans and Donahue [14] who looked at microwear on lithic artefacts.

Even with these advances in 3D scanning, there are currently still many thousands more photographs taken of artefacts than there are of 3D scans. Financial and time resource costs are one reason for this, as is the need to capture artefacts at different stages, including on site where scanners are unlikely to be available. In other cases, it is that the 3D scans provide some information, while images can provide different perspectives on the objects, such as the original patination on the surfaces of flint bifaces, and thus both are likely to be in ongoing use [8]. Combining this with the thousands of images that are already in archives worldwide, it is the case that archaeologists are going to need to find and work with images for the foreseeable future. The purposes for working with these images are varied; however, often it is to compare and identify objects to understand their function in society. These, and other purposes for using images in archaeology, are discussed in the contextual inquiry study described in Sect. 3.

With our work, we have expanded the corpus of image-based search of archaeological images to biface artefacts using image features based on established archaeological methodology for identifying similarities between the artefacts. Expert-verified metadata is used to validate the retrieval performance.

Our data set is a public database from the British Archaeology Data Service consisting of 3501 images, each \(500\times 375\) pixels in size, of 1167 biface artefacts in three views: front, rear, and side [27]. Figure 1 contains several examples of bifaces of varying size, shape, and texture, with different views of the same artefact shown in the same row. The images were captured at varying resolutions indicated by the scale bar on the top-right corner of each image.

The data set also consists of 52 metadata fields for each artefact including the site and country from where the biface was excavated, the artefact’s raw material, type, and its physical dimensions. The metadata was verified by biface experts and is of high quality, with little to no missing information. We utilized this information to define and measure relevance of results retrieved by our proposed system.

Image-based features suitable for representing the characteristics of bifaces that are important for identification and classification were selected after conducting a contextual inquiry (CI) study in which research archaeologists with relevant experience were interviewed. The CI study and its outcomes are described in Sect. 3. The specific features that were selected based on the result of CI study are described in Sects. 4.2 and 4.3.

An algorithm for identifying biface images similar to that of a query image based on the selected image features was developed. The details of this algorithm are presented in Sect. 5.

The biface search algorithm was validated in several ways by comparing the metadata of the images returned by the search algorithm to the metadata of the query image and by looking at patterns of disagreements in the retrieval results for different image descriptors. The evaluation methodology and results of the evaluation are described in Sect. 6.

Four participants were recruited through mailing lists and personal contacts available to the Archaeological Data Service (ADS) at the University of York. Each participant had over 5 years of experience as a research archaeologist, made extensive use of picture archives in their work, and previously worked on prehistory sites with artefacts that included biface artefacts.

Interviewees were provided with an information sheet that discussed how the interview would proceed and were provided with an informed consent form. Interviews were recorded on a Panasonic HD video camera, and the interviewer took notes to support analysis of the recording. Of particular interest to the interviewer was the documentation of explicit task flows and tacit knowledge the participant was using to make decisions. After the interview was completed, participants were provided with a demographic questionnaire through the online questionnaire tool Qualtrics.

During the interview, participants were shown some preliminary results of the image processing algorithms to give them some insight into how their data would be used. These images were printed in full colour at 150 dots per inch (dpi) to ensure clarity in the presentation.

Participants were met in a comfortable, quiet setting with a computer connected to the Internet. Participants were asked to bring with them any physical or digital materials that they use in typical, day-to-day work when trying to identify or classify artefacts.

Participants were asked about their own background in archaeology, their training, and their area of expertise. They were also asked what they considered to be the most important tasks that they undertook with online archaeological archives.

The CI study consisted of an initial semi-structured interview regarding how and when participants interacted with image archives and what they felt the most important aspects of the archives were. Further, the interview explored issues around favourite and least favourite aspects of working with image archives. This initial interview provided some initial context regarding what users do with their data.

Participants were asked about a recent time that they had used image archives to solve a research question and were asked to demonstrate how they used the archive to solve that problem. The participants demonstrated the tasks, describing how and why they used the archive. The interviewer would periodically stop the participant to discuss particular interesting aspects of the interaction, to elaborate on particular challenges or problems highlighted by the participant, or to clarify the purpose of particular actions. In this way, the interviewer was able to gain a better understanding of tacit aspects of the interaction that would not necessarily be raised during a traditional interview.

Finally, users were shown a set of images of bifaces and asked about different features that were used for classification and identification. This part of the interview led to a list of common features, as well as a set of research publications and secondary resources that would elaborate these features.

All participants were trying to find information about artefacts that related to the ones they were already working with. However, the final uses of the images were varied. A few researchers were working on presentations and displays for public engagement, while others worked to find images for inclusion in publications or lectures. However, the majority of the participants were trying to collect together sets of similar artefacts in order to identify an artefact they had in hand or, in some cases, to pass off that identification task to another specialist team member.

In these cases, there were often several types of analysis that would be conducted by other team members in parallel, with individuals contributing analyses from a number of perspectives. For example, if the artefact is longer than they are wide, it indicates that the artefact was, perhaps, the end product, whereas other lithics in an assemblage were potentially flakes from the manufacturing process. If the blade is heavily worked, meaning much of the stone has had the cortex removed, it gives indication of the advancement of the civilization that created the artefact. Finally, morphology will often indicate the function of the tool, with the visual characteristics providing a means of classifying the tools by its function in society [3, 28].

The recordings of the contextual inquiries were reviewed and partially transcribed to identify the tasks undertaken by participants in image archives, with a particular focus on the types of information they used to formulate and refine their queries. For purposes of this paper, we will focus on the characteristics of bifaces that emerged from this analysis. Experts reported that the following characteristics of biface tools are used to classify artefacts, judge the similarity of artefacts, and to formulate new queries for image search systems:In the next section we describe how different characteristics are mapped to image descriptors to match images automatically.

Biface images were preprocessed to isolate the object of interest from the rest of the image. A sample unprocessed image is shown in Fig. 2. Images were smoothed by using a \(5\times 5\) Gaussian filter with \(\sigma =0.5\) pixels. Images were then converted to the HSV colour space, and the biface object is segmented by selecting all pixels that have neither minimum nor maximum saturation. This gives accurate segmentation results for almost all bifaces, except for a very few which have very dark or very bright areas on their surface. To address this problem, all holes in the detected foreground regions were filled. The bounding box of the segmented artefacts was determined from their segmentations as in Fig. 2. Finally the subimages defined by the bounding boxes were converted from RGB colour to greyscale (luminance as defined by BT.601 standard). All features comprising the biface descriptor are extracted from these greyscale images.

In this section we describe the size (geometric) and shape features that contribute to the biface descriptor.

The CI study indicated that texture of a biface’s surface appearance was an important indicator of the biface’s raw material type. In this section we describe 13 candidate texture descriptors. Some are existing features taken directly from existing literature, and others are new or are variations of existing features. The relative effectiveness of these 13 descriptors is evaluated in Sect. 6.

The shape descriptor f, the geometric features \(g=\{g_1,\ldots , g_7\}\), and all thirteen texture descriptors described in Sect. 4.3 were computed for each image.

The distance between each geometric feature of a query image Q and a database image I is defined as a simple absolute difference:where \(g_{i}^Q\) and \(g_{i}^I\) represent the geometric features of objects in images Q and I, respectively.

The distance between shape descriptors of Q and I is defined as the \(L_2\) norm:where \(f_{i}^Q\) and \(f_{i}^I\) represent the shape features of images Q and I, respectively, and \(n_\mathrm {{shape}}\) is the number of shape features, which, as previously noted, is 39.

The distance between texture descriptors of images Q and I is defined as the Chi-square distance:where \(H_{i}^Q\) and \(H_{i}^I\) represent the features of the texture descriptors of images Q and I, respectively, and \(n_\mathrm {{tex}}\) is the length of the texture descriptor.

The min–max normalization technique [18] is employed to separately normalize each of the geometric feature distances:where the index k is over all images in the database. Shape and texture distances, \(D_\mathrm {{shape}}\) and \(D_\mathrm {{texture}}\), are normalized in the same way.

Given a query image, the steps mentioned in Algorithm 1 are performed to retrieve similar images from the database. In order to avoid online computation, the geometric, shape, and texture descriptors of the objects in the database images are precomputed. The algorithm can be used with any one of the texture descriptors described in Sect. 4.3 together with the geometric features g and the shape descriptor f.

Section 2.1 briefly described the metadata associated with each image. Certain fields of this metadata were used to analyse the performance of Algorithm 1 by comparing the metadata of the retrieved images to the metadata of the query image. Many of the metadata fields contain information that is not useful for image-based matching because they do not directly describe appearance of the objects, for example, fields that detail the date and location of the find. The metadata fields chosen for the use in the analysis were motivated by the findings of the CI study detailed in Sect. 3. The biface type (e.g. hand axe) field was used due to its relationship with biface morphology and the blade vs. flake categorization. The raw material type field (e.g. flint or chert) was used because this is related to geographic location and availability of material at a site. Several metadata fields describing physical measurements of the artefacts were used since they can be used to judge how well the physical measurements of the bifaces are extracted from the images. These fields were: Area, Aspect Ratio, Length, Breadth, Breadth at 20% of the length (Breadth20, the breadth of the artefact at 20% of the distance between the wider end of the artefact to the narrower end of the biface), Breadth at 80% of the length (Breadth80), and the ratio of Breadth80 to Breadth20.

As a first look at how the different texture descriptors performed, for each texture descriptor t and metadata field m, we computed the percentage of retrieved images over all queries where field m of the retrieved image matched field m of the query image. The results are shown in Table 2 for the 10, 20, and 99 most similar images retrieved by Algorithm 1. The biface type field was matched about 93% of the time for all texture descriptors; the lack of variability observed coincides with the expectation that texture descriptors do not capture information about biface type.

Different texture descriptors exhibited greater differences in ability to match raw material type ranging from about 71% for ULBP, OCLBP, and GWF to about 76% for L–A–S for the top 99 retrievals. Generally, the various descriptors consisting of descriptors fused with LWGF resulted in the most raw material type matches, with SFTA and ARPCH also performing well on their own. For 10 and 20 retrievals the same trends are present, though the spread increases to a low of about 72% to a high of about 81% for 10 retrievals. This indicates that the descriptors which are fusions with LWGF are more likely to return bifaces of the same raw material type with higher ranks.

Even though the texture descriptors are not capturing information about a biface’s physical dimensions, the trend was that ULBP, and OCLBP resulted in more matches for the physical measurement metadata fields, while the fusions of descriptors with LGWF performed the worst, though in all cases the difference between best and worst was only about four percentage points. This suggests that texture descriptors that are better at matching raw material type do so at the expense of finding bifaces of a more similar size, likely because bifaces of the same raw material type but with greater differences in physical dimensions are selected over bifaces of more similar physical dimensions. The last two columns of the table where we have aggregated the results for metadata features meant to be, respectively, captured by shape and size image features, and we can see the trade-off clearly. Texture descriptors that are better able to extract bifaces with the correct raw material type do so at the expense of matching objects of more similar shape and size.

The relevance score, \(rel_{{\mathrm {score}}}\), which characterizes the relevance of an image D retrieved by Algorithm 1 to the given query image Q is defined as:where \(meta_{\mathrm {score}}^{i}(Q,D)\) is the binary match score functionwhere \(meta^{i}(Q)\) and \(meta^{i}(D)\) refer to the value of the \(i^{th}\) metadata feature of Q and D, respectively. Thus, if the two images exactly match in all of their metadata features, the relevance score would be 9 and the image D is considered highly relevant to the query image Q.

The maximum possible total relevance score, denoted as \(max_\mathrm {score}(Q)\), for a query image Q and N retrievals is determined by computing \(rel_{\mathrm {score}}(Q,D)\) for every database image D and finding images \(E_1, \ldots , E_N\) with the N largest relevance scores. Then \(max_\mathrm {score}(Q) = \sum _{i=1}^{N} rel_{\mathrm {score}}(Q, E_i)\). Similarly, the actual total relevance score achieved by Algorithm 1 for query image Q is \(query_{\mathrm {score}}(Q) = \sum _{i=1}^N rel_{\mathrm {score}}(Q, D_i)\), where \(D_1, \ldots , D_N\) are the N images that were retrieved. Finally, the normalized accuracy for a given query Q is:

For each texture descriptor, the normalized accuracy was computed for each query using the leave-one-out methodology described in Sect. 6.1 at \(N=99\) retrievals. The mean normalized accuracy for each texture descriptor over all queries is shown in Fig. 4. A Wilcoxon signed-rank test was performed pairwise on the texture descriptors to test the null hypothesis that the 1167 paired normalized accuracy scores came from the same distribution. In nearly all cases the null hypothesis was rejected at the \(\alpha = 0.05\) level with \(p < 0.0069\) with the exceptions of ULBP/GPCH (\(p=0.0733\)) and OPCH/BTF (\(p=0.9236\)). Results for \(N=10\) and \(N=20\) retrievals were extremely similar, both in the magnitude of the means for each descriptor and the results of the Wilcoxon signed-rank tests. These results indicate that all of the texture descriptors (along with the shape and geometric features) are doing very well overall in matching the most relevant images. The differences in performance, while small in magnitude, are statistically significant and indicate that ULBP, OCLBP, GPCH, and BTF are the best texture descriptors on average.

However, given that this image retrieval system is intended to support user search and reasoning over archives, looking only at differences in mean performance does not provide a sufficiently nuanced understanding of the differences between the algorithms and what the consequences of those differences might be for the search experience. In the next section, we propose an alternative analysis that examines the data sets from a differing perspective.

For the remainder of our analysis we will be interested in the behaviour of the ULBP, OLCBP, GPCH, and BTF as the group of descriptors that had the “best” overall performance in Table 4 which will refer to as the SO group (superior overall group) and the fused descriptors L–A, L–B, L–A–B, and L–A–S as the IO group (inferior overall group) which had the “worst” performance in Fig. 4.

All of the texture descriptors resulted in similar overall accuracy (Fig. 4). When the overall accuracy score is within two to three percentage points, it is tempting to say that any particular algorithm would be suitable. However, because this algorithm is intended to support user search, these aggregate measures of accuracy do not give us any indication of what result sets look like which could be very important in choosing a particular algorithm. There could be distinct differences in the sets of images that are returned because when comparing any two descriptors across the nearly 1200 query images, there are a number of different distributions of results that could produce similar overall accuracy scores. If there are situations where one descriptor performs particularly well or particularly poorly this may skew the average up or down. Similarly, if the result sets retrieved by two different descriptors consistently share a number of results, this will obscure the actual differences between the algorithms.

To better elucidate the previously observed trade-offs apparent in the set of texture descriptors we performed an analysis in which, for a given query image Q and pair of texture descriptors \(D_1\) and \(D_2\), we only considered the symmetric difference of the images retrieved by Algorithm 1 using \(D_1\) and \(D_2\), that is, images not in common to both retrievals.

For a pair of descriptors \(D_1\) and \(D_2\), let \(R^1_q\) and \(R^2_q\) be the set of images retrieved for a given query image q by Algorithm 1 using \(D_1\) and \(D_2\), respectively. Obtain the symmetric difference of \(R^1_q\) and \(R^2_q\): \(\varDelta _q(D_1, D_2) = R^1_q\cup R^2_q \setminus R^1_q\cap R^2_q\). Let \(\delta ^1_q(D_1,D_2)\) be the elements of \(\varDelta _q(D_1, D_2)\) in \(R^1_1\), and let \(\delta ^2_q(D_1,D_2)\) be the elements of \(\varDelta _q(D_1, D_2)\) in \(R^2_q\).

Our results have built a robust picture of the strengths and weaknesses of the different descriptors. Considering overall accuracy, it does appear that the descriptors perform largely the same in the way that they match images, with less than a percentage point of accuracy between the top 5 methods. This means that if we were worried strictly about finding as close to perfect matches as we can, then any of these descriptors would likely be adequate. This is particularly true when there are small numbers of images in the sets that have exact matches against the ground truth metadata. For example, if there are only 7 objects that perfectly match in the ground truth to the query image, we can be relatively certain each of the algorithms will return those 7 reliably.

However, given that we are looking to support online search, we are interested in more than just exact matches. Researchers working with image archives rely on comparing and contrasting different, yet related, objects together while making their decisions. As a result, it was important to understand whether each descriptor was just returning the same sets as every other descriptor or whether there were distinct differences in their performance. There was distinct difference between the best and worst overall performing descriptors (the SO group and the IO group). First, it is interesting that in the SO group there is almost no variation in the sets of images they retrieve. Even at their most varied, we are only seeing about 13% variation in the images that are returned from the standard descriptors, with most hovering around only 8% difference. This tends to imply that if you choose any one of these algorithms, you are going to get largely the same set of images being returned.

In comparison, the IO group of descriptors retrieves image sets that are 20–30% different from the SO group. It appears that the IO descriptors are better at detecting differences in material while trading off shape matching accuracy, whereas the SO descriptors are doing the opposite. If we are working with a single collection of flint, where most pieces come from the same source stone, then the SO descriptors are likely to perform better. However, for a more varied collection of bifaces, comprised of flint, glass, etc., we would expect the IO descriptors to be superior.

Considering the dominance between the different descriptors, it appears that one could choose one of the SO descriptors at random and produce largely the same results over a series of query images. However, choosing one of the IO descriptors would result in a system that returns slightly more varied results. This may be important if search system designers want to encourage serendipity in their search, something which is often valued by researchers.

In future work, we will pursue the evaluation of these different descriptors with users in the archaeological archives. It would be particularly interesting to know which, if any, descriptor is preferred by users in supporting their search.

Overall, the new fused descriptors proposed in this paper have a set of advantages and disadvantages over the descriptors in the SO group. In general, the new descriptors appear to work well in finding relevant and related images in an image search. However, they have the advantage of producing more varied sets of results which may add value to search systems, in particular in heterogeneous image archives where there will be large variation in material qualities of artefacts.