Automatic Bone Surface Restoration for Markerless Computer-Assisted Orthopaedic Surgery

With the degradation of articular cartilage caused by the overload or cyclical episodes on the knee joint, the subchondral bone may collapse, causing a lesion to form in the smooth joint surface [11]. OCD mainly affects the femoral condyle and/or patellofemoral articulation of the knee [12]. During OCD treatment, the affected area needs to be removed and processed in a regular shape (e.g., ellipse) to be compatible with an optimal implant chosen from the library built through the statistical modelling of real OCD dataset [2].

With the fast development of deep learning and the availability of large 3D datasets, many generative models have been proposed to reconstruct geometries with the implicitly learned structural features [13]. Most of the proposed networks are based on a vanilla architecture [14]: an encoder extract the latent features of the input data first. Then, the encoded features are depicted by a decoder to generate desired outputs [15‐17]. The encoder usually consists of fully convolutional layers, followed by optional pooling, activation and/or fully connected layers. The decoder either contains deconvolutional layers or fully connected layers for upsampling.

The 3D geometry can be recorded in volumetric voxels, surface meshes, point cloud or multi-views. Among them, the point cloud-based learning offers high flexibility and efficiency in computation and memory. PointNet, based on the same encoder-decoder design, is regarded as the most popular backbone for point cloud-based learning [18]. It adopts three unique designs: a symmetry function to deal with unstructured input, a local and global information aggregation layer and a joint alignment network. Inspired by PointNet, many new extensions and variants exist. An example is PointNetLK, a network trained for the point cloud registration [19]. By integrating a modified Lucas&Kanade (LK) algorithm in the PointNet framework, the spatial transformation can be depicted by a differentiable learning framework.

We collected a dataset \(\mathcal {D}\) containing depth captures of a cadaver leg with an intact femur surface. A commercial depth camera, Realsense D415, was used to be consistent with the original work [8]. The camera can acquire depth data with less than 2% error at up to 90 frames per second, which is competent for real-time applications [20] The dataset includes 9334 depth captures of lab scenes. Each depth image is represented by a \(160\times 160\times 4\) matrix. The first three channels in the last dimension are (x, y, z) coordinates of a sampled pixel, and the last channel is the binary label that denotes whether the point belongs to the femur surface.

Sawbones were used to generate realistic femur cutting patterns. Figure 1 shows the overall workflow. The original femur surface \(f_o\) was first captured and manually segmented from the depth frame. After cutting the femur around the condal following a conventional procedure for OCD removal, the modified femur surface \(f_m\) was again captured. Due to the possible change in spatial pose, \(f_o\) was registered to \(f_m\), and the depth value \(\tilde{z}\) of points fall in an annotated cutting A area was interpolated. \(\tilde{z}\) was subtracted by the depth value of \(f_m\) in the cutting area (i.e., \(\Delta z_A = z - \tilde{z}\)) and normalised. To ensure the smooth connection between modified and unmodified surface, zeros were padded around the edge of \(\Delta z_A\). The padded 3D variation was fitted by Clough Tocher 2D interpolation for a cutting pattern f. The same procedure was repeated 20 times to generate enough patterns with different cutting shapes and depth variations.

The intact femur points p were segmented from \(\mathcal {D}\) according to their binary labels. K (\(=1\)-3 in our trial) rectangular area were selected on the femur surface with the arbitrary size and location, to which the collected deformation patterns f were mapped and scaled by the arbitrary maximum intrusion depth (i.e., 0–15 mm). The original and deformed point clouds were separately resampled in a \(N\times 3\) point cloud and concatenated into a \(N\times 6\) array. Since the time of later reconstruction is proportional to the number of points N, we chose \(N=2500\) as a compromise between speed and surface representation quality.

The surface reconstruction network is shown in Figure 2. The encoder borrows the PointNet design of sequential multilayer perceptrons (MLPs) and a max pooling layer to ensure the same loss expression regardless the order of input points. Note that for our need, no transformation (based on T-Net) is included for input and features, since it is designed to increase classification accuracy [19], and we want to keep the spatial consistency between input and output. After encoding the deformed points \(p'\), the most critical point among N points is selected for every latent dimension. The obtained latent feature vector is then passed to the decoder that consists of three fully convolutional layers to recover the \(N\times 3\) points \(\hat{p}\). The reconstruction loss is defined as the chamfer distance (CD) between the output \(\hat{p}\) and the ground truth (GT) intact points p:To ensure the spatial consistency, the reconstructed \(\hat{p}\) and GT p are fed to a pretrained PointNetLK to obtain a relative transformation T. We define a spatial regularisation loss \(\mathcal {S}\) as the mean-squared error (MSE) between T and an identity matrix. The overall cost function is defined as (where \(\lambda\) is a weight factor):

The performance of the proposed network was evaluated on the test dataset first. The points of reference geometry were scanned by a highly precise commercial scanner (HDI 3D scanner, LMI Technologies Inc.). After the RANSAC global alignment [21], the femur pose \(T_{def}\) was computed by the standard ICP registration (Open3D library [22]) between reference points and the deformed points with or without reconstruction. The threshold corresponding distance was set to 5 mm. We evaluated the registration reliability in terms of fitness (i.e., the number of inlier correspondences divided by the total number of target points) and the root-mean-squared error (RMSE) between the registered inlier correspondences. The spatial tracking error induced by geometric changes was defined as the difference between registered poses \(T_{def}\) and the GT poses \(T_{undef}\) obtained between the undeformed frames and reference. As shown in Figure 3, by reconstruction, the registration fitness is improved from 76.29±6.50% to 87.55±5.06%, and the RMSE is reduced from 2.26±0.17 mm to 1.96±0.11 mm. Both the 3D positional and rotational accuracy are significantly (p-values<0.05 by a Kruskal-Wallis one-way analysis of variance test).

We additionally ran a spatial consistency check by registering the reconstructed point clouds from multiple single-view captures to the same pre-scanned reference. As shown in Figure 4, the merged frames are consistent with the model, indicating that the 3D geometric features of the input were learned by the network and used for the surface restoration.

The combined markerless segmentation, reconstruction and registration workflow were then tested on a model knee (i.e., a drilled femur sawbones held by a metal leg). An optical marker was pinned into the leg and tracked by an optical tracker (Fusiontrack 500, Atracsys LLC) to provide the GT pose \(T_{gt}\) for evaluation. The difference between the markerless femur pose (T) registered from the restored surface and the \(T_{gt}\) obtained by marker-based tracking was defined as the tracking error:The pose registered from the raw segmented points without reconstruction was also evaluated as a reference.

The Python inference takes 0.05 s per frame for the real-time reconstruction. As shown in Figure 5, the reconstruction improves the registration fitness from 75.56±6.36% to 81.69±10.97%, and reduces the registration RMSE from 2.40±0.10 mm to 2.07±0.30 mm. However, the improvement in 3D spatial pose is not obvious: the Kruskal-Wallis test shows no significant difference between results obtained with and without reconstruction (p-value=0.83 for translational error and 0.28 for rotational error).