Development and validation of a longitudinal soft-tissue metastatic lesion matching algorithm

Metastasis is the leading cause of cancer-related mortality (Chaffer and Weinberg 2011). Although cancer patients have treatment options, the treatment response among lesions can be substantially different. This heterogeneous response poses a great challenge to cancer management in metastatic patients because the behaviour of one lesion (or even a few) can be unrepresentative of the global response to treatment. If response can be assessed at the lesion-level, inter-lesion heterogeneity can be quantified, providing valuable information for improved cancer management via early prediction of outcomes (Gayed et al 2004, Schwarz et al 2005) or via identification and subsequent selective targeting of resistant lesions (Lubner et al 2013, Roth et al 2020).

Molecular imaging is an essential tool for detecting metastatic lesions and monitoring their progression and therapeutic response (Weissleder 2006, Baum et al 2010, Vanderhoek et al 2013, Scarpelli et al 2018). Imaging can capture the heterogeneity in response because it is able to spatially resolve different lesion sites and to capture different quantitative and qualitative characteristics of each disease site. However, late-stage cancer patients can present with tens of metastatic lesions spread throughout their whole-body. Therefore, even though imaging can resolve different lesions, lesion-level disease analysis is impractical in daily clinical practice because it is overwhelming to manually assess individual lesion response. Therefore, response assessment employed by physicians often reduces the rich heterogeneous lesion-level image information to patient-level information either by summarizing all the lesions into a single score (Larson et al 1999, Gallamini et al 2014) or by evaluating a subset of lesions (Eisenhauer et al 2009, Wahl et al 2009) (e.g. RECIST or PERCIST). One way to overcome this information reduction is to achieve automatic assessment of individual lesion response via lesion matching.

Matching individual lesions across multiple time points has been investigated via rigid registration for identification of new cancerous lesions (Moltz et al 2009) and for treatment response assessment of multiple sclerosis brain lesions (Bosc et al 2003, Shahar and Greenspan 2005, Köhler et al 2019). Going beyond rigid registration, Yip et al introduced a hierarchical articulated registration that enabled treatment response assessment of late-stage prostate cancer bone lesions (Yip et al 2014, Yip and Jeraj 2014). Finally, two dimensional (2D) deformable registration was previously explored as a means of refining the rigid registration-based lesion matching of lymphoma lesions (Xu et al 2011).

The aforementioned studies demonstrated that rigid, articulated, and deformable image registration enable lesion matching in specific organs (e.g. skeleton, brain, liver). However, a comprehensive algorithm that enables matching of metastatic lesions spread across the entire patient anatomy remains unavailable. Rigid registration applicability is limited to certain body-parts (e.g. brain) and fails to account for skeletal movement and soft-tissue deformations. Deformable registration showed promising results to aid in lesion matching, but the 2D nature of its previous application fell short on exploring the full potential of the deformations.

In this work we introduce a novel algorithm for lesion matching of metastatic lesions spread through the whole-body that is based on several distinctive steps, including 3D deformable registration, lesion dilation, clustering, and linear assignment of lesions. The lesion matching algorithm was tested in a population of cancer patients that received longitudinal PET/CT scans as part of their treatment response assessment.

We pose the lesion matching as a linear assignment problem. Following image registration (Step 1), the matching algorithm includes lesion dilation (Step 2), clustering (Step 3), and linear assignment (Step 4). A schematic description of the algorithm's structure is presented in figure 1.

Zoom In Zoom Out Reset image size

The algorithm acts on a scan-pair of anatomical medical images, each one having an associated binary lesion mask. The goal of the algorithm is to automatically perform lesion matching decisions based on lesion superposition. A matching decision consists of identifying if a lesion present in one scan has an associated lesion (homologous lesion) in the paired scan; if so, the longitudinal pair of lesions is called single corresponding lesion (CL). Lesions that have no CL in the paired scan are either new lesions (NL) or disappearing lesion (DL). In each scan-pair, one matching decision had to be made for each unique lesion and the number of unique lesions per scan-pair was defined as the sum of CLs, NLs, and DLs. The concepts of scan-pairs, unique lesions, matching decisions, CLs, NL, and DLs are illustrated in figure 2.

Zoom In Zoom Out Reset image size

In the first step, 3D deformable image registration is used to superimpose anatomically homologous regions of a scan-pair. The deformation vector field is estimated by minimization of the sum of square differences between voxel intensities in the moving and in the fixed anatomical medical images, respectively the later and the earlier scans. Deformable registration is done using free form deformation with 3rd order B-splines interpolation and hierarchical control grid (Rueckert et al 1999). This technique was chosen for its wide acceptance in the medical image registration field, its local support, and for requiring relatively simple and fast calculations (Sotiras et al 2014). The hierarchical control grid approach is also well established and is known to help in preserving the smoothness of the registration (Lee et al 1996, Rueckert et al 1999, Mattes et al 2003).

The calculated transformation field T is then applied to the binary lesion mask of the later scan in the scan-pair.

$$\begin{eqnarray}&&{{L}}_{{T}}=\,{T}({L}),\end{eqnarray} \tag{ 1 }$$

where L is the set of coordinates of the lesions in the binary mask of the later scan and L_T is this set of coordinates after transformation.

In the second step, a morphologic conformal lesion dilation operation (D) is applied to the binary lesion masks (on both time points in the scan-pair) to increase the lesion superposition probabilities (figure 3).

$$\begin{eqnarray}&&{L}_{T,D}=D\left({L}_{T}\right)\end{eqnarray} \tag{ 2 }$$

$$\begin{eqnarray}&&{E}_{D}=D\left(E\right),\end{eqnarray} \tag{ 3 }$$

where E is the set of coordinates of the lesions in the binary mask of the earlier scan, ${E}_{D}$ is the conformal dilation of the set E, such that $E\subset {E}_{D},$ and ${L}_{T,D}$ is the conformal dilation of the set L_T , such that ${L}_{T}\subset {L}_{T,D}.$ Subsequently, the individual lesion components are numbered based on connected components analysis, the superposition of the lesions is evaluated and a matrix M of intersections is automatically constructed as

$$\begin{eqnarray}&&{M}_{{i}{j}}=\left|{E}_{D{i}}\cap {L}_{T,D{j}}\right|,\end{eqnarray} \tag{ 4 }$$

where ${L}_{T,{D}_{j}}\subset {L}_{T,D}$ is the subset of the coordinates of the jth lesion in the later scan, ${E}_{{D}_{i}}\subset {E}_{D}$ is the subset of the coordinates of the ith lesion in the earlier scan, and ${M}_{ij}$ is the cardinality of the intersection of these subsets. The matrix M is a square matrix with N² elements where N is the maximal integer between ${N}_{E}$ and ${N}_{L},$ which are the number of lesions identified in the earlier and in the later scans.

Zoom In Zoom Out Reset image size

The algorithm's third step is clustering of lesions, an important consideration for treatment response assessment, which can occur due to merging or splitting of multiple lesions from one to another imaging time point (Shafiei et al 2021). Merging is usually a result of disease progression in adjacent lesions. Conversely, lesion splitting is usually caused by one lesion responding to treatment and separating into two or more smaller disjoint lesion components. After the second step, a lesion may intersect multiple lesions on the paired scan. This may be a spurious intersection with a neighboring lesion (figure 3(C)), or may be due to merging or splitting (figure 3(D)). We developed a novel lesion clustering algorithm designed specifically to distinguish between these two instances, so that the overall matching algorithm considers sets of components originating from splitting, and sets of components that eventually merge, as individual lesions. A set of lesion components is clustered when:

• i.  
  The components intersect to at least one common lesion on the other scan (figure 4(B)), considering the dilation introduced on the second step.
• ii.  
  The distance $\vec{u}$ between the centroids of the components (figure 4(C)) is smaller than a distance $\vec{d}$ (figure 4(D)).

The distance $\vec{d}$ is defined as the longest chord in the common matching lesion that makes a smaller than 5° angle with the vector $\vec{u}$ connecting the centroids of the components to be clustered (figure 4), the angular constraint is expressed as

$$\begin{eqnarray}&&\arccos \left(\displaystyle \frac{\vec{u}\cdot \vec{d}}{\left|\vec{u}\right|\cdot \left|\vec{d}\right|}\right)\leqslant 5^\circ .\end{eqnarray} \tag{ 5 }$$

Zoom In Zoom Out Reset image size

Assuming perfect registration, if a set of components originate from a homologous lesion in the paired scan (either merging or splitting), the components should all be spatially contained in the homologous lesion, in which case the condition $\left|\vec{d}\right|\gt \left|\vec{u}\right|$ will be satisfied. If registration is not perfect, but the dilation step guarantees that all the components overlap to the homologous lesion, the relationship between the vectors $\vec{u}$ and $\vec{d}\,\,$will still hold. The angular constraint of equation (5) is imposed to consider that a chord $\vec{d}$ in a direction in which the homologous lesion is long might be an overly lenient threshold for evaluating a vector $\vec{u}$ connecting components separated along a different direction, in which the homologous lesion is short. The necessity for an angular constraint is illustrated in figure 5. The 5° tolerance in equation (5) is used to consider that $\vec{d}$ and $\vec{u}$ need not be exactly parallel, but approximately so. The 5° value is an algorithm hyperparameter and different values could be explored depending on the application.

Zoom In Zoom Out Reset image size

In the fourth step, the lesions are renumbered and voxels that pertain to a given lesion are reevaluated based on the clustering decision. A new intersection matrix $M^{\prime}$ is calculated under the new conditions. A cost matrix C is then defined as ${C}_{ij}=1/{M}_{ij}^{{\prime} }.$ The lesion matching problem is thus reduced to a linear assignment problem (Jaqaman et al 2008), which consists of finding an optimal permutation matrix A

$$\begin{eqnarray}&&A=\mathop{\text{arg}\,\min }\limits_{A}\left(\displaystyle \sum _{i=1}^{N}\displaystyle \sum _{j=1}^{N}{A}_{ij}{C}_{ij}\right)\end{eqnarray} \tag{ 6 }$$

within the constraints to the permutation matrix A

$$\begin{eqnarray}&&\displaystyle \sum _{i=1}^{N}{A}_{ij}=0\,{\rm{o}}{\rm{r}}\,1\,{\rm{and}}\,\displaystyle \sum _{j=1}^{N}{A}_{ij}=0\,{\rm{o}}{\rm{r}}\,1,\end{eqnarray} \tag{ 7 }$$

where summations might be zero if a lesion is superimposed to no other (NLs or DLs). The Munkres assignment algorithm (Munkres 1957) is used to solve the linear assignment problem. One of the possible mathematical interpretations of the Munkres algorithm is that it finds the optimal permutation matrix A that minimizes the global sum of costs defined by equation (6) within the constraints imposed by equation (7). Briefly, the Munkres algorithm performs a non-greedy minimization of C, which is equivalent to a maximization of the volume of intersection between all lesions in a scan-pair.

The developed algorithm was evaluated in 172 PET/CT images of 32 patients with metastatic cancer, all images were acquired under the institutional standard procedures. PET/CT images provide anatomical structure and functional information of the location of diseased tissue, which makes them suitable for validating the algorithm. The PET/CT images were retrospectively acquired from two independent clinical trials (table 1). One of the trials contained 19 metastatic melanoma patients, for whom lesion-level disease assessment is particularly challenging because of the plurality of metastatic sites and the elevated number of lesions, which in the most extreme case was of 64 unique lesions spanning from the legs to the head. A second dataset containing 10 different types of cancers was included to evaluate the algorithm for its robustness regarding disease type.

In the validation dataset, a scan-pair was defined as two longitudinal and sequential PET/CT scans of the same patient. Therefore, if n is the number of scans that a patient received, $n-1$ is the number of scan-pairs; e.g. a patient that received four scans (${s}_{1},\,{s}_{2},\,{s}_{3},\,{s}_{4}$), will have three scan-pairs ${s}_{1}$ to ${s}_{2},$ ${s}_{2}$ to ${s}_{3},$ and ${s}_{3}$ to ${s}_{4}.$ The dataset contained 140 scan-pairs.

In the validation, CT images served as the anatomical information, while the binary lesion masks were derived from the PET image. Using radiology reports as guidance, cancerous lesions, indicated by elevated PET SUV uptake relative to background, were manually contoured by an imaging scientist, defining thus the binary lesion masks associated to each PET/CT image. The manual contouring started by using an active-contours algorithm to automatically conform the contours to the lesions (Yang et al 2009), then the contours were reviewed by a nuclear medicine physician and manually corrected when necessary. Reference matching decisions were made by visually analyzing the scan-pairs' images for each patient and preparing a reference lesion matching decision matrix for each scan-pair. The merging or splitting of lesions over time was accounted for in the reference matching decisions. Clustering decisions were made when two or more lesions occupied the same anatomical position within the patient body as a larger lesion in the corresponding scan (figure 4(B)).

The main metric used to assess the algorithm performance on the validation dataset was the overall accuracy of lesion matching, determined as the number of correct lesion matching decisions made divided by the total number of matching decisions. Additionally, the accuracy of matching CLs and the accuracy of identifying vanishing lesions and NL were estimated separately.

3.1. Image registration sensitivity analysis

To fully understand the impact of image registration on lesion matching, we studied the algorithm performance under four distinct registration methods. Two deformable registration approaches were investigated: (1) articulated deformable registration (ADR) which was preceded by whole-body rigid registration (WRR) and articulated rigid registration (ARR) and (2) whole-body deformable registration (WDR), which was preceded only by WRR (figure 6). Both deformable registration approaches (WDR and ADR) were done using free form deformation with 3rd order B-splines interpolation and hierarchical control grid (Rueckert et al 1999). Additionally, lesion matching was assessed after WRR and ARR alone.

Zoom In Zoom Out Reset image size

The centroids of CLs were used as landmarks to analyze the performance of image registration, only lesions whose volume varied less than 50% between scans were used because high volume variation can lead to displacements of the lesion centroids which are unrelated to registration performance. The Wilcoxon signed-rank test was used to compare the centroid distance distributions. Additionally, the impact of the registration methods on lesion matching performance was assessed using the McNemar test to intercompare all the registration methods (0.05 significance level).

3.2. Lesion dilation sensitivity analysis

3.3. Anatomic distribution sensitivity analysis

3.4. Lesion burden sensitivity analysis

In this study, 736 matching decisions were necessary to analyze all 140 scan-pairs. Based on the reference matching decisions, 298 of the 736 matching decisions (40%) were CLs and 438 (60%) were new or DLs. The accuracy of lesion matching is shown in table 2. These results represent the accuracy with no dilation.

4.1. Image registration sensitivity analysis

The centroids of 186 CLs were used as landmark. The landmark distance distributions are shown in figure 7. All distributions were significantly different (P < 0.001). A visual understanding of how registration influences landmark distance, and thus proximity of CLs, is shown in figure 8.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The McNemar test was used to intercompare the deformable registration approaches' impact on the lesion matching performance. Lesion matching performance using WDR was better than ADR, nevertheless with marginal statistical significance (P < 0.04). We also used the McNemar test to compare both deformable approaches to the rigid approaches, WRR and ARR. Comparison of both approaches with WRR yielded statistically significant differences (P < 0.001). WDR significantly improved lesion matching when compared to ARR (P < 0.001). Improvement observed with ADR was close on the verge of significance (P < 0.06).

4.2. Lesion dilation sensitivity analysis

An increase in matching accuracy was observed for CLs across the dilation range (figure 9(A)). By contrast, the accuracy for new or DLs decreased with lesion dilation, (figure 9(B)). The overall accuracy was observed to have an initial increase associated with the CL accuracy trend, followed by a decrease associated with the NL/DL accuracy trend (figure 9(C)). The optimal dilation magnitude was determined by the peaks in the overall accuracy curves. The optimal dilations were of 25 mm for all registration methods besides WRR, which had an optimal dilation of 30 mm.

Zoom In Zoom Out Reset image size

4.3. Anatomic distribution sensitivity analysis

The data in table 3 are derived from our observation of the 736 lesions present in the patients in our datasets, the location of all unique lesions are shown in table 3, most of which were the chest + abdomen (52%–380/736) and in the pelvis (21%–156/736). The thighs (13%–99/736) and the legs (8%–60/736) also presented with relatively high disease densities.

Table 3. Number of lesions per body-part.

Body-part	# of lesions	%	Body-part	# of lesions	%
Chest+abdomen	380	52%	Feet	9	1.2%
Pelvis	156	21%	Forearms	5	0.7%
Thighs	99	13%	Arms	5	0.7%
Legs	60	8%	Hands	3	0.4%
Head and neck	19	3%

For the four body-parts with highest number of lesions (chest+abdomen, pelvis, thighs, and legs), assessment of the overall accuracy was performed separately. The overall accuracies per body-part, evaluated at the optimal dilations (figure 10) saw minimal variation across registration methods and body-parts. No accuracy result was smaller than 0.90 (WRR in the Thigs).

Zoom In Zoom Out Reset image size

4.4. Lesion burden sensitivity analysis

In the complete patient population, containing all 140 scan-pairs, the median number of lesion matching decisions was 3, with minimum of 1 and maximum of 68. In the high lesion burden subset, the median number of lesion matching decisions was 19, minimum of 11 and maximum of 68. The high lesion burden subset accounted for 345 of the 736 matching decisions (47% of the total) and for 12 of the 140 scan-pairs (8.5% of the total). Accuracy results were evaluated for the optimal dilation magnitudes (figure 11).

Zoom In Zoom Out Reset image size

In this work we presented a lesion matching algorithm based on 4 steps: image registration, lesion dilation, lesion clustering, and linear assignment (figure 1). The algorithm was validated on two independent patient populations and the lesion matching accuracy obtained was excellent (0.98 overall accuracy). No trend in accuracy was observed across patient populations, overall accuracies were 0.98 for both. We explored the lesion matching accuracy sensitivity regarding image registration methods, lesion dilation magnitude, anatomic distribution, and lesion burden. The deformable registration techniques (WDR and ADR) showed superior registration results, when compared to rigid techniques (WRR and ARR) as shown by the landmark distance (figure 7), which led to a significant improvement in the lesion matching accuracy. Furthermore, for WDR and ADR, residual registration error was in accordance with literature for image intensity based methods (Yin et al 2011). Use of lesion dilation magnitude was an important factor in achieving the excellent accuracy level. For the best performing registration approach (WDR), the use of dilation caused overall accuracy to increase by 0.08, from 0.90 with no dilation to 0.98 with the optimal dilation of 25 mm (table 2, figure 9). In addition, the excellent accuracy levels were minimally dependent on lesion body-part location (figure 10) and were sustained on the analysis of high burden scan-pairs (figure 11). Lastly, the matching criteria used by our methodology are positional and make no assumption regarding lesion shape, instead, the shape is taken into consideration by the methodology via conformal dilation (step 2) and via the use of lesion derived distance metrics for clustering (step 3).

Previous works have explored rigid registration-based lesion matching (Bosc et al 2003, Shahar and Greenspan 2005, Moltz et al 2009, Xu et al 2011, Köhler et al 2019). Moltz et al suggested that rigid registration suffices for lesion matching applications; however, we hypothesize that this conclusion was based on their focus on liver and lung cancer lesions. These organs are located in the chest + abdomen body-part, where we observed WRR-based matching to have its best performance (0.96 overall accuracy—figure 10). However, for the purpose of whole-body soft tissue lesion matching, the rigid registration method (WRR) resulted in the worst overall matching accuracy; matching accuracy of CLs was of only 0.58 without dilation (table 2). When compared the ARR method (Yip et al 2014), use of 3D deformable registration resulted in a remarkable lesion matching improvement, 29%, from 0.58 to 0.75 (table 2). Deformable registration has been previously employed for the task of lesion matching (Xu et al 2011). However, it was limited to 2D deformation of a few image slices. The use of 3D deformable registration for lesion matching purposes is one of the novelties presented in this work. Other novelties of our work are the use of conformal dilation for lesion matching, the development of a novel lesion clustering methodology that is specific for longitudinal monitoring of merging and splitting lesions, and the use of a non-greedy optimizer to solve a linear assignment problem in the setting of whole-body longitudinal disease matching using in-vivo imaging of human patients. Most importantly, the main novelty of our work consists in combining several image analysis techniques to solve the problem of whole-body soft tissue lesion matching, which had not been done previously.

One challenge of our work was defining the reference dataset for the lesion matching decisions, which is subject to human error and bias in its nature. However, these same limitations would be encountered in manual lesion matching performed for clinical decisions. In other words, some arbitrariness is unavoidably associated with visual interpretation of medical images. We observed, however, that this limitation is likely to have little impact in the validation of the algorithm because the correspondence between the vast majority of lesion pairs was conspicuous.

The 12 incorrect matching decisions made by the algorithm for the best performer parameters (WDR with dilation of 25 mm) had three causes. First, lesion dilation caused 2 incorrect decisions where a new and a DL were registered close enough that dilation led to incorrect superposition. Second, misregistration caused 4 incorrect decisions, all in the same scan-pair and located in the patient's legs. Largely different legs positioning resulted in a large deformation vector field that failed to accurately approximate the CLs; use of ADR for matching lesions in the legs is one way of overcoming this limitation. Finally, Incorrect clustering caused 6 incorrect decisions (figure 12), where two or more lesions (blue) that should not cluster were superimposed to a common lesion (orange). Due to the particular geometric positioning, the threshold distance $\vec{d}$ (figure 4(D)) and the angle restriction failed to prevent the incorrect clustering. This limitation could be overcome by use of an adaptive dilation magnitude parameterized by local lesion density, which will be pursued in future developments. The impact of incorrect lesion matching decisions on possible applications of the herein presented algorithm, such as automated treatment response assessment, will be subject of future investigations.

Zoom In Zoom Out Reset image size

While the herein presented algorithm was validated using PET/CT images, we hypothesize that it is generalizable to other types of medical imaging modalities. The requirements for application of the algorithm are anatomical information that is used to guide the registration step and the existence of lesion conspicuity, which in our validation dataset was obtained through PET scans. We note, however, that the implementation of the method to other imaging modalities would require sensible adaptation of hyperparameters involved in all four steps of the algorithm.

In this work we introduced an algorithm that enables automated matching of metastatic lesion. We validated the algorithm for whole-body longitudinal PET/CT scans of metastatic cancer patients and showed it has high overall accuracy. One novelty of our algorithm is the leveraging of 3D deformable image registration, which outperformed the existing methods for every studied scenario and was a significant improvement, when compared to previously established methods. Our study suggests that whole-body lesion matching can be done automatically, possibly facilitating lesion-level disease assessment of metastatic cancer patients in clinical practice and in clinical trials, ultimately leading to enriched information for treatment response assessment and facilitating the development of novel treatment combinations.