Resource-Efficient Medical Image Analysis

Vascular network analysis is crucial to define the tumoral architecture and then diagnose the cancer subtype. However, automatic vascular network segmentation from Hematoxylin and Eosin (H &E) staining histopathological images is still a challenge due to the background complexity. Moreover, there is a lack of large manually annotated vascular network databases. In this paper, we propose a method that reduces reliance on labeled data through semi-supervised learning (SSL). Additionally, considering the correlation between tumor classification and vascular segmentation, we propose a multi-task learning (MTL) model that can simultaneously segment the vascular network using SSL and predict the tumor class in a supervised context. This multi-task learning procedure offers an end-to-end machine learning solution to joint vascular network segmentation and tumor classification. Experiments were carried out on a database of histopathological images of renal cell carcinoma (RCC) and then tested on both own RCC and open-source TCGA datasets. The results show that the proposed MTL-SSL model outperforms the conventional supervised-learning segmentation approach.

Multiplexed pathology imaging techniques allow spatially resolved analysis of cell phenotypes for interrogating disease biology. Existing methods for cell phenotyping in multiplex images require extensive annotation workload due to the need for fully supervised training. To overcome this challenge, we develop SANDI, a self-supervised-based pipeline that learns intrinsic similarities in unlabeled cell images to mitigate the requirement for expert supervision. The capability of SANDI to efficiently classify cells with minimal manual annotations is demonstrated through the analysis of 3 different multiplexed immunohistochemistry datasets. We show that in coupled with representations learnt by SANDI from unlabeled cell images, a linear Support Vector Machine classifier trained on 10 annotations per cell type yields a higher or comparable weighted F1-score to the supervised classifier trained on an average of about 300–1000 annotations per cell type. By striking a fine balance between minimal expert guidance and the power of deep learning to learn similarity within abundant data, SANDI presents new opportunities for efficient, large-scale learning for multiplexed imaging data.

Automated analysis of chest radiography using deep learning has tremendous potential to enhance the clinical diagnosis of diseases in patients. However, deep learning models typically require large amounts of annotated data to achieve high performance – often an obstacle to medical domain adaptation. In this paper, we build a data-efficient learning framework that utilizes radiology reports to improve medical image classification performance with limited labeled data (fewer than 1000 examples). Specifically, we examine image-captioning pretraining to learn high-quality medical image representations that train on fewer examples. Following joint pretraining of a convolutional encoder and transformer decoder, we transfer the learned encoder to various classification tasks. Averaged over 9 pathologies, we find that our model achieves higher classification performance than ImageNet-supervised and in-domain supervised pretraining when labeled training data is limited.

Image attacking has been studied for a long time. However, in reality, the number of research on defending against the attacks on segmentation models is still limited especially for medical imaging. To fill this research gap, we propose a novel defending mechanism against adversarial attacks for the segmentation models. We focus on segmentation as robustness improvement on segmentation is much more challenging due to its dense nature, and segmentation is at the center of medical imaging tasks. In this paper, we are the first time to employ Transformer as a technique to protect the segmentation models from attacks. Our result on several medical well-known benchmark datasets shows that the proposed defending mechanism to enhance the segmentation models is effective with high scores and better compared to other strong methods.

The large-scale pretrained models from terabyte-level (TB) data are now broadly used in feature extraction, model initialization, and transfer learning in pathological image analyses. Most existing studies have focused on developing more powerful pretrained models, which are increasingly unscalable for academic institutes. Very few, if any, studies have investigated how to take advantage of existing, yet heterogeneous, pretrained models for downstream tasks. As an example, our experiments elucidated that self-supervised models (e.g., contrastive learning on the entire The Cancer Genome Atlas (TCGA) dataset) achieved a superior performance compared with supervised models (e.g., ImageNet pretraining) on a classification cohort. Surprisingly, it yielded an inferior performance when it was translated to a cancer prognosis task. Such a phenomenon inspired us to explore how to leverage the already trained supervised and self-supervised models for pathological survival analysis. In this paper, we present a simple and low-cost joint representation tuning (JRT) to aggregate task-agnostic vision representation (supervised ImageNet pretrained models) and pathological specific feature representation (self-supervised TCGA pretrained models) for downstream tasks. Our contribution is in three-fold: (1) we adapt and aggregate classification-based supervised and self-supervised representation to survival prediction via joint representation tuning, (2) comprehensive analyses on prevalent strategies of pretrained models are conducted, (3) the joint representation tuning provides a simple, yet computationally efficient, perspective to leverage large-scale pretrained models for both cancer diagnosis and prognosis. The proposed JRT method improved the c-index from 0.705 to 0.731 on the TCGA brain cancer survival dataset. The feature-direct JRT (f-JRT) method achieved \(60\times \) training speedup while maintaining 0.707 c-index score.

Self-supervised learning methods have been receiving wide attentions in recent years, where contrastive learning starts to show encouraging performance in many tasks in the field of computer vision. Contrastive learning methods build pre-training weight parameters by crafting positive/negative samples and optimizing their distance in the feature space. It is easy to construct positive/negative samples on natural images, but the methods cannot directly apply to histopathological images because of the unique characteristics of the images such as staining invariance and vertical flip invariance. This paper proposes a general method for constructing clinical-equivalent positive sample pairs on histopathological images for applying contrastive learning on histopathological images. Results on the PatchCamelyon benchmark show that our method can improve model accuracy up to 6% while reducing the training costs, as well as reducing reliance on labeled data.

One challenge of training deep neural networks with gigapixel whole-slide images (WSIs) is the lack of annotation at pixel level or patch (instance) level due to the high cost and time-consuming labeling effort. Multiple instance learning (MIL) as a typical weakly supervised learning method aimed to resolve this challenge by using only the slide-level label without needing patch labels. Not all patches/instances are predictive of the outcome. The attention-based MIL method leverages this fact to enhance the performance by weighting the instances based on their contribution to predicting the outcome. A WSI typically contains hundreds of thousands of image patches. Training a deep neural network with thousands of image patches per slide is computationally expensive and requires a lot of time for convergence. One way to alleviate this issue is to sample a subset of instances/patches from the available instances within each bag for training. While the benefit of sampling strategies for decreasing computing time might be evident, there is a lack of effort to investigate their performances. This project proposes and compares an adaptive sampling strategy with other sampling strategies. Although all sampling strategies substantially reduce computation time, their performance is influenced by the number of selected instances. We show that if we are limited to only select a few instances (e.g., in order of 1\(\sim \)10 instances), the adaptive sampling outperforms other sampling strategies. However, if we are allowed to select more instances (e.g., in order of 100\(\sim \)1000 instances), the random sampling outperforms other sampling strategies.

Breast cancer is one of the leading causes of cancer deaths in women. As the primary output of breast screening, breast ultrasound (US) video contains exclusive dynamic information for cancer diagnosis. However, training models for video analysis is non-trivial as it requires a voluminous dataset which is also expensive to annotate. Furthermore, the diagnosis of breast lesion faces unique challenges such as inter-class similarity and intra-class variation. In this paper, we propose a pioneering approach that directly utilizes US videos in computer-aided breast cancer diagnosis. It leverages masked video modeling as pretraning to reduce reliance on dataset size and detailed annotations. Moreover, a correlation-aware contrastive loss is developed to facilitate the identifying of the internal and external relationship between benign and malignant lesions. Experimental results show that our proposed approach achieved promising classification performance and can outperform other state-of-the-art methods.

In this paper, we propose a few-shot medical images classification framework for low prevalence disease detection. The proposed method leans to transfer medical knowledge from common diseases to low prevalence cases using meta-learning. Indeed, compared to natural images, medical images vary less diversely from one image to another, with complex patterns and less semantic information. Hence, extracting clinically relevant features and learning a disease-specific signature from few images is challenging. Inspired by clinician’s cognitive and visual diagnosis, we integrate an attention mechanism in the meta-learning process and we revised the meta-loss function to learn clinically disease-specific features over tasks. The proposed approach has been evaluated on two image-based diagnosis problems namely low prevalence skin and low prevalence thorax diseases diagnosis. We obtained respectively for those two use-cases \(84.3\%\) and \(73.4\%\) of average AUC in 2-way 5-shot classification setting. Obtained results demonstrate the effectiveness of the proposed framework compared to baselines and state-of the-art few-shot disease detection methods.

The retinal layer segmentation from OCT images is a fundamental and important task in the diagnosis and monitoring of eye-related diseases. The quest for improved accuracy is driving the use of increasingly large dataset with fully pixel-level layer annotations. But the manual annotation process is expensive and tedious, further, the annotators also need sufficient medical knowledge which brings a great burden on the doctors. We observe that there exist a large number of repetitive texture patterns in the flatten OCT images. More surprisingly, by significantly reducing the annotation from 100% to 10%, even to 1%, the performance of a segmentation model only drops a little, i.e., error from \(2.53\, \upmu \text {m}\) to \(2.76\,\upmu \text {m}\), and to \(3.27\,\upmu \text {m}\) on a validation set, respectively. Such observation motivates us to deeply investigate the redundancies of the annotation in the feature space which would definitely facilitate the annotation for medical images. To greatly reduce the expensive annotation costs, we propose a new annotation-efficient learning paradigm by annotating a fixed and limited number of pixels for each layer in each image. Considering the redundancies in the repetitive patterns in each layer of OCT images, we employ a VQ memory bank to store the extracted features on the whole datasets to augment the visual representation. The experimental results on two public datasets validate the effectiveness of our model. With only 10 annotated pixels for each layer in an image, our performance is very close to the previous methods trained with the whole fully annotated dataset.