DisguisOR: holistic face anonymization for the operating room

With the advent of public face detection benchmark datasets like WIDERFACE [4], numerous deep learning-based face detectors were introduced in recent years [11‐13]. Such methods typically regress a bounding box onto the region where a face can be successfully identified in the image. As WIDERFACE consists of annotated images from everyday scenarios, face detectors trained on this dataset can suffer in complex and crowded OR environments [5]. Occlusions and obstructions from medical equipment or personnel in close quarters, masks, and skull caps can lead to missed predictions and, ultimately, incomplete anonymizations. While we also use 3D data for anonymization, our work diverges from 3D face recognition [14], where a scan of a 3D face is matched to a catalogue of face scans.

Identity scrubbing can be achieved by removing the sensitive area, blurring, or pixelization [3]. In the OR, standardized scrubs and gloves already obscure many possible landmarks, leaving the face as the primary identifier that could be used for re-identification, as previously established [5, 6]. A recent line of work has proposed to replace faces with artificially generated faces using GANs [10, 15] or parametric face models [16]. Such replacement methods tend to yield a more realistic-looking output, and the resulting anonymized area resembles the input more closely, which can positively affect downstream applications [15]. However, these methods typically contain a separate branch to handle face detection [10] and thus suffer similarly in OR environments due to partial obstructions.

Using human pose estimation as an additional context to localize faces has been demonstrated as valuable [5]. The torso, shoulders, and arms provide useful cues for localizing faces occluded under a surgical mask and skull cap. Beyond mere 2D human keypoint detection, a significant emphasis has also been placed on regressing keypoints from multiple camera views in a shared 3D space [17, 18]. 3D human pose detection can be especially beneficial for multi-person scenarios such as surgical ORs, where ubiquitous occlusions can lead to poor performance in individual camera views [19, 20]. Regressing a 3D human shape from a single input image is also an active area of research [21]. However, such methods would suffer similarly to partial occlusions. To avoid this shortcoming, we leverage the 3D nature of multi-view OR acquisitions.

We adopt an unsupervised 3-stage approach to fit a 3D mesh [22] for each person in the scene. 2D human keypoints are first detected [23] in each camera view, and regressed in a global coordinate frame with VoxelPose [18]. As neither 2D nor 3D human poses are available as ground truth, we use an existing detector [23] trained on COCO to estimate 2D human keypoints from an image. In order to combine poses from each view in a robust manner, Voxelpose must first be trained to learn how multiple 2D poses from each camera can be optimally combined in 3D. To achieve this, we follow the procedure described in [18] and synthetically generate ground truth by sampling existing 3D human poses from the Panoptic dataset [24] and placing them at random locations in the 3D space. These poses are then projected back into each 2D image-plane and used as input to guide VoxelPose through the 2D-to-3D multi-person regression task. The trained model ultimately combines 2D human poses from multiple views into one joint 3D human pose for each person in the scene. We then perform an additional temporal smoothing on each 3D human pose sequence to interpolate missing poses and reduce noise (for details, see suppl.).

In order to adequately represent the face of each individual in the scene, we propose to use the statistical parametric human mesh model SMPL [22], which we regress onto each 3D human pose obtained as output from Voxelpose. While temporal smoothing yielded less noisy keypoint estimates, we noticed that the 3D mesh model did not always align with the 3D point cloud of an individual, resulting in inaccurate face localizations. To resolve this issue, we perform a rigid registration between the head segment of the SMPL model and the point cloud. More specifically, we crop the point cloud around the estimated head of the SMPL model and align the model with the point cloud using the probabilistic point-set registration method FilterReg [25], and subsequently fine-tune using iterative closest point (ICP) [26]. As a final step, we extract the face from the SMPL mesh, which should now be aligned with the 3D location of an individual’s face.

Thus far, our pipeline estimates a global 3D face mesh that overlaps with each person in the scene. In order to yield a 2D anonymization, these meshes can now be projected back into all camera views, replacing the face of each individual with a unique template (see Fig. 5). However, a 3D face might be occluded in a particular view, for example, due to an OR light and, therefore, not visible. To mitigate false-positive predictions, we check whether a 3D face is visible in 2D by looking for a disparity between the camera’s depth map and the 3D face mesh (for details, see suppl.). We then utilize the Poisson Image Editing technique [27] to harmonize the face template and the background image for a more natural appearing face replacement. The template can also be changed for each individual to influence factors such as age, sex, or ethnicity.

We identified three distinct scenarios of varying difficulties from the complete dataset [7]. We then annotated each visible face manually in all camera views for a total of 4913 face bounding boxes. The annotation criteria were adopted to closely match the style of the WIDERFACE dataset [4]. The three scenarios are chosen to specifically represent the varying characteristics in the OR. They differ in the number of individuals present, their attire, and the degree of obstructions (Fig. 3).

We compare the proposed method’s face localization performance with that of DSFD [11], a state-of-the-art detector also used in DeepPrivacy [10]. We use the model pre-trained on WIDERFACE [4] provided by the authors. We additionally evaluate the self-supervised domain adaption (SSDA) strategy proposed by Issenhuth et al. [5]. Here we also use DSFD as the face detection backbone, fine-tuning it on 20k unlabeled images as proposed, with the suggested hyperparameters.

In addition to recall, we propose to evaluate multi-view OR anonymization with what we coin holistic recall. The holistic recall considers a face as detected only if it was identified in all camera views where it is at least partially visible. We argue that this is more suitable than image-wise evaluation, as a missed detection of a face in a single view results in a breach of anonymization for that individual.

We calculate the smallest rectangle outlining the rendered mesh to generate face predictions for evaluation. As the proposed method does not rank the output detections with a confidence score, the commonly used average precision (AP) score is not defined. Therefore, we additionally report precision and F1-score for all three methods in the supplementary materials. Furthermore, the four cameras are categorized as either a surgical camera (SC) or workflow camera (WFC), depending on the perspective of the camera (see Fig. 3). The images and angle of acquisition in WFCs are more similar to what might be found in public face detection datasets [4], while SCs may acquire the scene from above, and individuals are more frequently obscured by OR equipment.

We compare the images anonymized by our approach to those altered by several conventional anonymization methods, such as blurring (\(61\times 61\) kernel), pixelization (\(8\times 8\) pixels), blackening, as well as the established GAN-based model DeepPrivacy [10] (see 2D anonymization Fig. 1). To disentangle image quality and face detection, we only evaluate image quality on faces detected by both our method and Deep Privacy, totaling 3786 faces. We evaluate the effectiveness of our face replacements on the cropped ground-truth bounding boxes with three established image quality metrics. The fréchet inception distance (FID) [28] measures the overall realism by calculating the distribution distance of the original and generated set of images. Learned perceptual image patch similarity (LPIPS) [29] reflects the human perception of an image’s realism by computing the difference between activations of two image patches for a standard neural network. The structural similarity index measure (SSIM) [30] calculates the quality of an image pixel-wise based on luminance, contrast, and structure.

Finally, we conduct additional experiments on the downstream behavior of off-the-shelf methods on our anonymized faces (see suppl.)

Figure 4 depicts the performance of our proposed method in comparison with two existing baselines. In the easy evaluation scenario, both DSFD and DisguisOR perform comparably, while the SSDA achieves a 9% higher holistic recall. In the medium and hard scenarios, DisguisOR outperforms DSFD and SSDA in holistic recall by 10% and 3%, and 11% and 16%, respectively.

These disparities are largely due to a poor detection rate in the surgical cameras, which are acquired from unusual camera angles and contain frequent obstructions (Fig. 3). DisguisOR is able to better cope with the increased occlusions and the number of individuals present in these scenarios, highlighting the proposed method’s robustness under partial visibility. By combining information from multiple cameras, DisguisOR yields a geometrically consistent detection—if an individual face has been accurately localized in the 3D scene, it can be more consistently identified in each individual image. While DSFD achieves a significantly higher accuracy in the easy scenario when refined via SSDA, the human pose backbone of DisguisOR underwent no such refinement and would likely also see some performance improvements.

SSDA underperforms the baseline DSFD model in the hard scenario, as well as DisguisOR in both the medium and hard scenarios. This could be because these more challenging detection candidates are less represented in the training data distribution or not detected with a high confidence score, and thus not pseudo-labeled frequently enough.

The recall rates over individual cameras reflect the characterizations of surgical and workflow camera views. While DSFD generally achieves slightly higher recall rates on workflow camera views (WFC1, WFC2), DisguisOR achieves much higher recall rates in the surgical camera views (SC1, SC2), see Fig. 3. SSDA improves recall for DSFD in surgical cameras, although it still falls short of DisguisOR in medium and hard scenarios. The surgical cameras in the hard scenario are especially challenging for face detectors, as severe occlusions, unusual camera angles, and surgical scrubs drastically impair the face detectors’ detection rate. In the case of faces in SC1 of the hard scenario (see person 1 in Fig. 1), DSFD achieves a recall rate of 16.9%. Using SSDA increases this recall rate to 52.8%, which DisguisOR still outperforms with a recall rate of 97.8%.

Our method is somewhat limited by the field-of-view (FOV) of the depth sensors during an acquisition. This partially explains the comparable performance with DSFD in the easy scenario, as individuals frequently move along the edge of the scene where depth coverage is limited. The 3D reconstruction we use to triangulate faces could also be performed without the use of the slightly more costly depth sensors, albeit less accurately.

In Table 1, we measure the quality of images altered by baseline approaches and our proposed method. As expected, conventional obfuscations like blackening, pixelization, and blurring achieve inferior results across all three metrics. DeepPrivacy [10] is designed to generate synthetic faces instead of applying conventional privacy filters, explaining the improved results on all image quality metrics compared to the conventional methods. Our method further improves upon these results as the replacement of the face information can be precisely controlled, even enabling the replacement of people wearing masks without creating corrupted or unnatural faces. In Fig. 5, we illustrate examples where DeepPrivacy replaces the face mask of a person with an unnatural mouth (e), while our method manages to blend the template and original image (f) more effectively.