Medical Image Understanding and Analysis

In oncology, Positron Emission Tomography (PET) is frequently performed for cancer staging and treatment monitoring. Metabolic active tumor volume (MATV) as well as total MATV (TMATV - including primary tumor, lymph nodes and metastasis) derived from PET images have been identified as prognostic factor or for evaluating treatment efficacy in cancer patients. To this end a segmentation approach with high precision and repeatability is important. Moreover, to derive TMATV, a reliable segmentation of the primary tumor as well as all metastasis is essential. However, the implementation of a repeatable and accurate segmentation algorithm remains a challenge. In this work, we propose an artificial intelligence based segmentation method based on textural features (TF) extracted from the PET image. From a large number of textural features, the most important features for the segmentation task were selected. The selected features are used for training a random forest classifier to identify voxels as tumor or background. The algorithm is trained, validated and tested using a lung cancer PET/CT dataset and, additionally, applied on a fully independent test-retest dataset. The approach is especially designed for accurate and repeatable segmentation of primary tumors and metastasis in order to derive TMATV. The segmentation results are compared with conventional segmentation approaches in terms of accuracy and repeatability. In summary, the TF segmentation proposed in this study provided better repeatability and accuracy than conventional segmentation approaches. Moreover, segmentations were accurate for both primary tumors and metastasis and the proposed algorithm is therefore a good candidate for PET tumor segmentation.

The combination of datasets is vital for providing increased statistical power, and is especially important for neurological conditions where limited data is available. However, our ability to combine datasets is limited by the addition of variance caused by factors such as differences in acquisition protocol and hardware. We aim to create scanner-invariant features using an iterative training scheme based on domain adaptation techniques, whilst simultaneously completing the desired segmentation task. We demonstrate the technique using an encoder-decoder architecture similar to the U-Net but expect that the proposed training scheme would be applicable to any feedforward network and task. We show that the network can be used to harmonise two datasets and also show that the network is applicable in the common scenario of limited available training data, meaning that the network should be applicable for real-world segmentation problems.

Mycosis fungoides (MF) is a rare, potentially life threatening skin disease, which in early stages clinically and histologically strongly resembles Eczema, a very common and benign skin condition. In order to increase the survival rate, one needs to provide the appropriate treatment early on. To this end, one crucial step for specialists is the evaluation of histopathological slides (glass slides), or Whole Slide Images (WSI), of the patients’ skin tissue. We introduce a deep learning aided diagnostics tool that brings a two-fold value to the decision process of pathologists. First, our algorithm accurately segments WSI into regions that are relevant for an accurate diagnosis, achieving a Mean-IoU of $$69\%$$ and a Matthews Correlation score of $$83\%$$ on a novel dataset. Additionally, we also show that our model is competitive with the state of the art on a reference dataset. Second, using the segmentation map and the original image, we are able to predict if a patient has MF or Eczema. We created two models that can be applied in different stages of the diagnostic pipeline, potentially eliminating life-threatening mistakes. The classification outcome is considerably more interpretable than using only the WSI as the input, since it is also based on the segmentation map. Our segmentation model, which we call EU-Net, extends a classical U-Net with an EfficientNet-B7 encoder which was pre-trained on the Imagenet dataset.

Several approaches based on convolutional neural networks (CNNs) are only able to process 2D images while most brain data consists of 3D volumes. Recent network architectures which have demonstrated promising results are able to process 3D images. In this work, we propose an adapted approach based on a CNN to process 3D contextual information in brain MRI scans for the challenging task of brain tumour segmentation. Our CNN is trained end-to-end on multi-modal MRI volumes and is able to predict segmentation for the binary case, which segments the whole tumour, and multi-class case, which segments the whole tumour (WT), tumour core (TC) and enhancing tumour (ET). Our network includes multiple layers of dilated convolutions and autofocus convolutions with residual connections to improve segmentation performance. Autofocus layers consist of multiple parallel convolutions each with a different dilation rate. We replaced standard convolutional layers with autofocus layers to adaptively change the size of the effective receptive field to generate more powerful features. Experiments with our autofocus settings on the BraTS 2018 glioma dataset show that the proposed method achieved average Dice scores of 83.92 for WT in the binary case and 66.88, 55.16, 64.13 for WT, TC and ET, respectively, in the multi-class case. We introduce the first publicly and freely available NiftyNet-based implementation of the autofocus convolutional layer for semantic image segmentation.

As the fetal brain develops, its surface undergoes rapid changes in shape and morphology. Variations in the emergence of the sulci on the brain surface have commonly been associated with diseased or at-risk pregnancies. Therefore, the process of surface folding is an important biomarker to characterise. Previous work has studied such changes by automatically delineating the cortical plate from MRI images. However, this has not been demonstrated from ultrasound, which is more commonly used for antenatal care. In this work we propose a novel method for segmenting the cortical plate from 3D ultrasound images using three varieties of convolutional neural networks (CNNs). Recent work has found improvements in medical image segmentations using multi-task learning with a distance transform regularizer. Here we implemented a similar method but found it was outperformed by the U-Net, which was able to segment the cortical plate with a Dice score of $$0.81\,\pm \,0.06$$ .

Deterministic deep learning methods for image segmentation require very precise ground-truth labels. However, obtaining perfect segmentations for medical image analysis is highly time-consuming and usually not feasible. In ultrasound imaging this problem is especially pronounced, as ultrasound scans are challenged by low contrast, speckle and shadow artifacts, all contributing to imperfect manual labelling. To overcome the problem of imperfect labels, we propose a label correction step which can correct the imperfect ground-truth labels in the training set by applying active contours. This forces the ground-truth segmentations towards regions which coincide with edges in the original volume (and thus object boundaries). We demonstrated the proposed active contour correction with a standard U-Net on the boundary segmentation of the cavum septum pellucidum in 3D fetal brain ultrasound and on the segmentation of the left ventricle in 2D ultrasound scans. The active contour label correction yielded more precise boundary predictions, suggesting that this simple correction step can improve boundary segmentation with imperfect labels.

Liver cancer diagnosis and treatment response assessment typically rely on the inspection of multi-phase contrast-enhanced computed tomography (CT) or magnetic resonance (MR) images. To date, various methods were proposed to automatically segment liver lesions in single time-step CT; but limited research addressed image analysis of multiple contrast phases. In this paper, we propose a multi-encoder 3D U-Net which, inspired by radiological practice, combines complementary tumour characteristics from both the arterial phase (AP) and portal venous phase (PVP) CT images. We demonstrate that encoder-decoder networks with disentangled feature extraction in two encoder streams outperform the baseline U-Nets that process single-phase data alone or apply input-level fusion for stacks of multi-phase data as channel input. Finally, we make use of a public single-phase CT liver tumour dataset for the pre-training of network parameters to improve the generalisation capabilities of our networks.

Selective segmentation is an important aspect of image processing. Being able to reliably segment a particular object in an image has important applications particularly in medical imaging. Robust methods can aid clinicians with diagnosis, surgical planning, etc. Many selective segmentation algorithms use geometric constraints such as information from the edges in order to determine where an object lies. It is still a challenge where there is low contrast present between two objects, and an edge is difficult to detect. Relying on purely edge constraints in this case will fail. We aim to make use of area constraints in addition to edge information in a segmentation model which is robustly capable of segmenting regions in an image even in the presence of low contrast, when given suitable user input. In addition, we implement a deep learning algorithm based on this model, allowing for a supervised, semi-supervised or unsupervised approach, depending on data availability.

Magnetic resonance cholangiopancreatography (MRCP), an MRI-based technique for imaging the bile and pancreatic ducts, plays a vital role in the investigation of pancreatobiliary diseases. In current clinical practice, MRCP image interpretation remains primarily qualitative, though there is growing interest in using quantitative biomarkers, computed from segmentations of the biliary tree, to provide more objective assessments. The variable image quality and duct contrasts in MRCP images, as well as the presence of bifurcations, tortuous bile ducts and bright gastrointestinal structures, makes segmenting the biliary tree from MRCP images a challenging task. We propose a method, based on the monogenic signal, for detecting the biliary tree in MRCP images. Using both phantom and clinical data we show that by tuning the monogenic signal to detect symmetric features we can successfully detect bile ducts and obtain accurate duct diameter measurements. Compared to the Hessian-based Frangi vesselness filter, we show that our method gives superior background noise suppression and performs better at duct bifurcations, where the model assumptions underlying vesselness fail.

Manual segmentations of anatomical regions in the brain are time consuming and costly to acquire. In a clinical trial setting, this is prohibitive and automated methods are needed for routine application. We propose a deep-learning architecture that automatically delineates sub-cortical regions in the brain (example biomarkers for monitoring the development of Huntington’s disease). Neural networks, despite typically reaching state-of-the-art performance, are sensitive to differing scanner protocols and pre-processing methods. To address this challenge, one can pre-train a model on an existing data set and then fine-tune this model using a small amount of labelled data from the target domain. This work investigates the impact of the pre-training task and the amount of data required via a systematic study. We show that use of just a few samples from the same task (but a different domain) can achieve state-of-the-art performance. Further, this pre-training task utilises automated labels, meaning the pipeline requires very few manually segmented data points. On the other hand, using a different task for pre-training is shown to be less successful. We then conclude, by showing that, whilst fine-tuning is very powerful for a specific data distribution, models developed in this fashion are considerably more fragile when used on completely unseen data.

Quantitative imaging biomarkers derived from magnetic resonance imaging of the pancreas could reveal changes in pancreas organ volume and shape manifest in chronic disease. Recent developments in machine learning facilitate pancreas segmentation and volume extraction. Machine learning methods could also help in designing a data-driven approach to pancreas shape characterization. We present an automated pipeline for pancreas volume and shape characterization. We start off with deep learning-based segmentation; we show the impact of choice of loss function in pancreas segmentation by comparing a 3D U-Net model trained using soft Dice over cross-entropy loss. Then, a diffeomorphic algorithm for group-wise registration as well as manifold learning are used to extract prominent shape features from the segmentation masks. The technique shows potential in a subset (N = 3,909) of the UK Biobank imaging sub-study for (1) automated quality control, e.g. suboptimal pancreas coverage acquisitions; and (2) determining abnormal pancreas morphology, that might reflect different patterns of fat infiltration. To our knowledge, this work is the first to attempt learning pancreas shape features.

Automatic detection and identification of the intervertebral discs on the spine MR images is a challenging task due to similarity of the discs on the same image, size and shape differences between subjects, and poor resolution. Many deep learning-based methods have been proposed recently to achieve automated detection and identification of human intervertebral discs. However, since there is usually only a small amount of labeled vertebral images available, employing an end-to-end deep learning system is not easily achievable. In this paper, we use a multi-stage deep learning system to detect and identify human lumbar discs from MRI data. We first use a Faster Region based Convolutional Neural Network (FRCNN) method to detect candidate disc positions. Each candidate from the FRCNN becomes a node in a weighted graph structure. The edge weights between the nodes are calculated using the FRCNN scores and the scores from a Binary Classifier Network (BCN) that tests compatibility of the nodes of the edge. A novel application of Dijkstra’s shortest path algorithm in this network produces both localizations and identifications of the lumbar discs in a globally optimal manner. Experiments on our dataset of 80 MRI scans from 80 patients achieved very promising results as they exceeded the state of the art alternatives on similar datasets.

Accurate segmentation of cellular structures is critical for automating the analysis of microscopy data. Advances in deep learning have facilitated extensive improvements in semantic image segmentation. In particular, U-Net, a model specifically developed for biomedical image data, performs multi-instance segmentation through pixel-based classification. However, approaches based on U-Net tend to merge touching cells in dense cell cultures, resulting in under-segmentation. To address this issue, we propose DeepSplit; a multi-task convolutional neural network architecture where one encoding path splits into two decoding branches. DeepSplit first learns segmentation masks, then explicitly learns the more challenging cell-cell contact regions. We test our approach on a challenging dataset of cells that are highly variable in terms of shape and intensity. DeepSplit achieves 90% cell detection coefficient and 90% Dice Similarity Coefficient (DSC) which is a significant improvement on the state-of-the-art U-Net that scored 70% and 84% respectively.

Perivascular spaces are fluid-filled tubular spaces that follow the course of cerebral penetrating vessels, thought to be a key part in the brain’s circulation and glymphatic drainage system. Their enlargement and abundance have been found associated with cerebral small vessel disease. Thus, their quantification is essential for establishing their relationship with neurological diseases. Previous works in the field have designed visual rating scales for assessing the presence of perivascular spaces and proposed segmentation techniques to reduce flooring and ceiling effects of qualitative visual scales, processing times, and inter-observer variability. Nonetheless, their application depends on the acquisition quality. In this paper, we propose a framework for improving perivascular spaces quantification using both texture analysis and total variation filtering. Texture features were considered for evaluating the image quality and determining automatically whether filtering was needed. We tested our work using data from a cohort of patients with mild stroke ( $$n=60$$ ) with different extents of small vessel disease features and image quality. Our results demonstrate the potential of our proposal for improving perivascular spaces assessments.

In medical imaging it is common practice to acquire a wide range of modalities (MRI, CT, PET, etc.), to highlight different structures or pathologies. As patient movement between scans or scanning session is unavoidable, registration is often an essential step before any subsequent image analysis. In this paper, we introduce a cost function based on joint total variation for such multimodal image registration. This cost function has the advantage of enabling principled, groupwise alignment of multiple images, whilst being insensitive to strong intensity non-uniformities. We evaluate our algorithm on rigidly aligning both simulated and real 3D brain scans. This validation shows robustness to strong intensity non-uniformities and low registration errors for CT/PET to MRI alignment. Our implementation is publicly available at https://github.com/brudfors/coregistration-njtv .

This paper reports on a new CT volume registration method, using 3D Convolutional Neural Networks (CNN). The proposed method uses the Least Square Generative Adversarial Network (LSGAN) model consisting of the Contraction-Expansion registration network as the LSGAN’s generator and a deep 3D CNN classification network as the LSGAN’s discriminator. The training of the generator is performed first on its own, using Charbonnier and smoothness loss functions, with progressive weights update moving from lower to higher resolution layers of the Expander. Subsequently, the complete network (Contraction-Expansion with the Discriminator) is trained as a LSGAN network. For the training, CREATIS and COPDgene datasets have been used in a self-supervised paradigm, using 3D warping of the moving volume to estimate the error with respect to the reference volume. The input to the network has 256 × 256 × 128 × 2 voxels and the output is displacement field of 128 × 128 × 64 × 3 voxels. The Contraction-Expansion registration network, on its own, achieves mean error of 1.30 mm with 1.70 standard deviation (SD) on the DIR-LAB dataset. When the whole proposed LSGAN network is used, the mean error is further reduced to 1.13 mm with 0.67 (SD). Therefore, the use of the GAN paradigm reduces the mean error by approximately 15%, providing the state-of-the-art performance.

Late gadolinium enhanced (LGE) cardiac magnetic resonance (CMR) imaging is the current gold standard for assessing myocardium viability for patients diagnosed with myocardial infarction, myocarditis or cardiomyopathy. This imaging method enables the identification and quantification of myocardial tissue regions that appear hyper-enhanced. However, the delineation of the myocardium is hampered by the reduced contrast between the myocardium and the left ventricle (LV) blood-pool due to the gadolinium-based contrast agent. The balanced-Steady State Free Precession (bSSFP) cine CMR imaging provides high resolution images with superior contrast between the myocardium and the LV blood-pool. Hence, the registration of the LGE CMR images and the bSSFP cine CMR images is a vital step for accurate localization and quantification of the compromised myocardial tissue. Here, we propose a Spatial Transformer Network (STN) inspired convolutional neural network (CNN) architecture to perform supervised registration of bSSFP cine CMR and LGE CMR images. We evaluate our proposed method on the 2019 Multi-Sequence Cardiac Magnetic Resonance Segmentation Challenge (MS-CMRSeg) dataset and use several evaluation metrics, including the center-to-center LV and right ventricle (RV) blood-pool distance, and the contour-to-contour blood-pool and myocardium distance between the LGE and bSSFP CMR images. Specifically, we showed that our registration method reduced the bSSFP to LGE LV blood-pool center distance from 3.28 mm before registration to 2.27 mm post registration and RV blood-pool center distance from 4.35 mm before registration to 2.52 mm post registration. We also show that the average surface distance (ASD) between bSSFP and LGE is reduced from 2.53 mm to 2.09 mm, 1.78 mm to 1.40 mm and 2.42 mm to 1.73 mm for LV blood-pool, LV myocardium and RV blood-pool, respectively.

Stain separation is an important pre-processing technique used to aid automated analysis of histopathology images. In this paper, we propose a novel, unsupervised deep learning method for stain separation (Hematoxylin and Eosin). This approach is inspired by Non-Negative Matrix Factorisation (NMF) and decomposes an input image into a stain colour matrix and a stain concentration matrix. In contrast to existing approaches, our method predicts stain colour matrices at the pixel level rather than the image level, thus enabling implicit modelling of tissue-dependant interactions between stains. We demonstrate an 8.81% reduction in mean-squared error on a stain separation task measuring the similarity between predicted and actual hematoxylin images from a publicly available dataset of digitised tissue images. We also present a novel approach to artifact detection in histological images based on a constrained generative adversarial network which we demonstrate is able to detect a variety of artifact types without the use of labels.

Serial-sectioning block-facing (SSBF) imaging is an attractive method to overcome the depth limitations of optical imaging and slice alignment challenges of traditional serial sectioning histology. Despite these advantages, SSBF modalities suffer from reduced axial resolution caused by out of focus sub-surface signals at the block face. In order to restore axial resolution, the sub-surface signal must be removed. In this work, we describe a methodology for restoring the axial resolution through a combination of sample preparation and deconvolution in post-processing. An opacifying agent used during sample preparation, decreases the subsurface signal by absorbing excitation light. From these image stacks we estimate parameters to generate a highly asymmetric point-spread function (PSF), which is then used in a Richardson-Lucy deconvolution algorithm. Whilst our methodology can be widely applied to any SSBF technique, we show its application in multi-fluorescent high resolution episcopic microscopy (MF-HREM), which is a simple and cost-effective alternative to optical sectioning techniques such as two-photon microscopy.

One of the main reasons for a half-sided facial paralysis is a dysfunction of the facial nerve. Physicians have to assess such a unilateral facial palsy with the help of standardized grading scales to evaluate the treatment. However, such assessments are usually very subjective and they are prone to variance and inconsistency between physicians due to their varying experience. We propose an automatic non-biased method using deep features combined with a linear regression method for facial palsy grading index prediction. With an extension of the free software tool Auto-eFace we annotated images of facial palsy patients and healthy subjects according to a common facial palsy grading scale. In our experiments, we obtained an average grading error of 11%.

Recent randomised clinical trials have shown that patients with ischaemic stroke due to occlusion of a large intracranial blood vessel benefit from endovascular thrombectomy. However, predicting outcome of treatment in an individual patient remains a challenge. We propose a novel deep learning approach to directly exploit multimodal data (clinical metadata information, imaging data, and imaging biomarkers extracted from images) to estimate the success of endovascular treatment. We incorporate an attention mechanism in our architecture to model global feature inter-dependencies, both channel-wise and spatially. We perform comparative experiments using unimodal and multimodal data, to predict functional outcome (modified Rankin Scale score, mRS) and achieve 0.75 AUC for dichotomised mRS scores and 0.35 classification accuracy for individual mRS scores.

‘Radiomics’ is utilized to improve the prediction of patient overall survival and/or outcome. Target segmentation, feature extraction, feature selection, and classification model are the fundamental blocks of a radiomics workflow. Nevertheless, these blocks can be affected by several issues, i.e. high inter- and intra-observer variability. To overcome these issues obtaining reproducible results, we propose a novel radiomics workflow to identify a relevant prognostic model concerning a real clinical problem. In the specific, we propose an operator-independent segmentation system with the consequent automatic extraction of radiomics features, and a novel feature selection approach to create a relevant predictive model in 46 patients with prostate lesion underwent magnetic resonance imaging.In the specific, using an operator-independent method of target segmentation based on an active contour, ad-hoc automated high-throughput analysis tool capable of calculating a total of 290 radiomics features for each imaging sequence, a novel statistical system for feature reduction and selection, and the discriminant analysis as a method for feature classification, we propose a performant and replicable radiomics workflow for the diagnosis of prostate cancer.The proposed workflow revealed three and five relevant features on T2-weighted and apparent diffusion coefficient (ADC) maps images, respectively, that were significantly correlated with the histopathological results. In the specific, good performance in lesion discrimination was obtained using the combination of the selected features (accuracy 76.76% and 75.20%, for T2-weighted and ADC maps images, respectively) in an operator-independent and automatic way.

This paper shows that deep modelling of subtle changes of cardiac motion can help in automated diagnosis of early onset of cardiac disease. In this paper, we model left ventricular (LV) cardiac motion in MRI sequences, based on a hybrid spatio-temporal network. Temporal data over long time periods is used as inputs to the model and delivers a dense displacement field (DDF) for regional analysis of LV function. A segmentation mask of the end-diastole (ED) frame is deformed by the predicted DDF from which regional analysis of LV function endocardial radius, thickness, circumferential strain (Ecc) and radial strain (Err) are estimated. Cardiac motion is estimated over MR cine loops. We compare the proposed technique to two other deep learning-based approaches and show that the proposed approach achieves promising predicted DDFs. Predicted DDFs are estimated on imaging data from healthy volunteers and patients with primary pulmonary hypertension from the UK Biobank. Experiments demonstrate that the proposed methods perform well in obtaining estimates of endocardial radii as cardiac motion-characteristic features for regional LV analysis.

The purpose of this study is to develop a new automated system to classify susceptibility weighted images (SWI) obtained to evaluate neonatal hypoxic-ischaemic injury, by detecting and analyzing ridges within these images. SW images can depict abnormal cerebral venous contrast as a consequence of abnormal blood flow, perfusion and thus oxygenation in babies with HIE. In this research, a dataset of SWI-MRI images, acquired from 42 infants with HIE during the neonatal period, features are obtained based on ridge analysis of SW images including the width of blood vessels, the change in intensity of the veins’ pixels in comparison with neighboring pixels, the length of blood vessels and Hessian eigenvalues for ridges are extracted. Normalized histogram parameters in the single or combined features are used to classify SWIs by $$ kNN $$ and random forest classifiers. The mean and standard deviation of the classification accuracies are derived by randomly selecting 11 datasets ten times from those with normal neurological outcome (n = 31) at age 24 months and those with abnormal neurological outcome (n = 11), to avoids classification biases due to any imbalanced data. The feature vectors containing width, intensity, length and eigenvalue show a promising classification accuracy of 78.67% $$ \pm $$ 2.58%. The features derived from the ridges of the blood vessels have a good discriminative power for prediction of neurological outcome in infants with neonatal HIE. We also employ Support Vector Regression (SVR) to predict the scores of motor and cognitive outcomes assessed 24 months after the birth. Our mean relative errors for cognitive and motor outcome scores are 0.113 $$ \pm $$ 0.13 and 0.109 $$ \pm $$ 0.067 respectively.

Wrist fractures (e.g. Colles’ fracture) are the most common injuries in the upper extremity treated in Emergency Departments. Treatment for most patients is an intervention called Manipulation under Anaesthesia (MUA). Surgical treatment would be needed for complex fractures or if the wrist stability is not restored. In addition, an unsuccessful treatment via MUA may also require subsequent surgical operation causing inefficiency in constrained medical resources and patients’ inconvenience. Previous geometric measurements in X-ray images [21] were found to provide statistical differences between healthy controls and patients with fractures, as well as pre- and post-intervention images. The most discriminating measurements were associated with the texture analysis of the radial bone. This work presents further analysis of these measurements and applying them as features to identify an appropriate machine learning model for Colles’ fracture treatment diagnosis. Random forest was evaluated to be the best model based on classification accuracy among the selected models commonly used in similar research. The non-linearity of the measurement features has attributed to the superior performance of an ensembled tree-based model. It is also interesting that the most important features (i.e. texture and swelling) required in the optimised random forest model are consistent with previous findings [21].

Knee osteoarthritis (OA) is very common progressive and degenerative musculoskeletal disease worldwide creates a heavy burden on patients with reduced quality of life and also on society due to financial impact. Therefore, any attempt to reduce the burden of the disease could help both patients and society. In this study, we propose a fully automated novel method, based on combination of joint shape and convolutional neural network (CNN) based bone texture features, to distinguish between the knee radiographs with and without radiographic osteoarthritis. Moreover, we report the first attempt at describing the bone texture using CNN. Knee radiographs from Osteoarthritis Initiative (OAI) and Multicenter Osteoarthritis (MOST) studies were used in the experiments. Our models were trained on 8953 knee radiographs from OAI and evaluated on 3445 knee radiographs from MOST. Our results demonstrate that fusing the proposed shape and texture parameters achieves the state-of-the art performance in radiographic OA detection yielding area under the ROC curve (AUC) of $$95.21\%$$ .

Deep neural network (DNN) classifiers have attained remarkable performance in diagnosing known diseases when the models are trained on a large amount of data from known diseases. However, DNN classifiers trained on known diseases usually fail when they confront new diseases such as COVID-19. In this paper, we propose a new deep learning framework and pipeline for explainable medical imaging that can classify known diseases as well as detect new/unknown diseases when the models are only trained on known disease images. We first provide in-depth mathematical analysis to explain the overconfidence phenomena and present the calibrated confidence that can mitigate the overconfidence. Using calibrated confidence, we design a decision engine to determine if a medical image belongs to some known diseases or a new disease. At last, we introduce a new visual explanation to further reveal the suspected region inside each image. Using both Skin Lesion and Chest X-Ray datasets, we validate that our framework significantly improves the accuracy of new disease discovery, i.e., distinguish COVID-19 from pneumonia without seeing any COVID-19 data during training. We also qualitatively show that our visual explanations are highly consistent with doctors’ ground truth. While our work was not designed to target COVID-19, our experimental validation using the real world COVID-19 cases/data demonstrates the general applicability of our pipeline for different diseases based on medical imaging.

We train a convolutional neural network (CNN) to classify the sex of mice from x-ray images in order to develop a tool that can be used not only for quality control of high throughput image data, but also to identify sex-related phenotype alterations. The method achieved 98% accuracy, recall of .98 and precision of .98 and identified the chest and pelvis as the areas relevant for sex classification. We identified four knockout lines (Duoxa2, Tmem189, Dusp3 and Il10rb) potentially affected by phenotype changes related to sex.This study demonstrates that CNNs can be trained for the purpose of quality control of images and can aid the discovery of novel genotype-phenotype associations. In addition to facilitating quality control, the method presented (1) allows the creation of a tool that will help phenotypers flag images of mice that should be inspected in more detail, (2) has highlighted areas of the mouse that are of particular interest in phenotype changes related to sex, and (3) has the potential to identify genes that may be causing sex related phenotype changes and/or are involved in sexual dimorphism.

We present a deep learning semantic segmentation algorithm for processing images acquired by a novel ophthalmic device, the NIDEK GS-1. The proposed model can sophisticate the current reference exam, called gonioscopy, for evaluating the risk of developing glaucoma, a severe eye pathology with a considerable worldwide impact in terms of costs and negative effects on affected people’s quality of life, and for inferring its categorization. The target eye region of gonioscopy is the interface between the iris and the cornea, and the anatomical structures that are located there. Our approach exploits a dense U-net architecture and is the first automatic system segmenting irido-corneal interface images from the novel device. Results show promising performance, providing about 88% of mean pixel-wise classification accuracy in a 5-fold cross-validation experiment on a very limited size dataset of annotated images.

Corneal cross-linking (CXL) is a surgical intervention to treat the progression of an eye disease called keratoconus that may lead to significant loss of visual acuity. Manually detecting the presence and the depth of a stromal demarcation line in optical coherence tomography (OCT) images is a standard procedure used by ophthalmologists to check the success of CXL. In this paper, we propose a deep learning model trained in a semi-weakly supervised fashion to segment the area between the top boundary of the cornea and the demarcation line that is later used by our extraction algorithm to obtain the demarcation line automatically. We report an improvement in performance compared to the fully supervised learning approaches in terms of the dice coefficient.

The retinal and cerebral microvasculatures share many morphological and physiological properties. In this pilot we study the strength of the associations between morphological measurements of the retinal vasculature, obtained from fundus camera images, and of features of Small Vessel Disease (SVD), as white matter hyperintensities (WMH) and perivascular spaces (PVS), obtained from MRI brain scans. We performed a 500-trial bootstrap analysis with Regularized Gaussian linear regression on a cohort of older community-dwelling subjects (Lothian Birth Cohort 1936, N = 866) in their eighth decade. Arteriolar bifurcation coefficients, vessel tortuosity and fractal dimension predicted WMH volume in 23% of the trials. Arteriolar widths, venular bifurcation coefficients, and venular tortuosity predicted PVS in up to 99.6% of the trials.

The conventional adaptive optics scanning laser ophthalmoscopy (AOSLO) arrangement is specifically designed to capture the confocal (directly backscattered) light by placing a physical pinhole conjugate to a chosen layer in the retina. This arrangement can be used to generate high contrast images of the photoreceptor mosaic by limiting the light from other retinal layers, such as the retinal pigment epithelium. However, there is growing demand for the study of different retinal features that has led to the development of different off-axis techniques to collect the non-confocal (multiply scattered) light. In this paper, we replace the physical pinhole of the AOSLO with a reconfigurable aperture to simultaneously collect the directly backscattered light, generating confocal images, as well as the multiply scattered light, generating non-confocal images. The reconfigurable aperture pattern is implemented with a digital micromirror device (DMD) and is optimised based on the information collected from Shack Hartmann wavefront sensor data. We present preliminary experimental results with a human eye to illustrate our findings.

We present a computational method for real-time, patient-specific simulation of 2D ultrasound (US) images. The method uses a large number of tracked ultrasound images to learn a function that maps position and orientation of the transducer to ultrasound images. This is a first step towards realistic patient-specific simulations that will enable improved training and retrospective examination of complex cases. Our models can simulate a 2D image in under 4 ms (well within real-time constraints), and produce simulated images that preserve the content (anatomical structures and artefacts) of real ultrasound images.

Fibres are present in many biological tissues and their geometric properties can be a useful indication of their role. Hence, imaging of nano-fibre volumes is useful for a number of different biomedical applications. It is possible to image nano-fibres with a variety of imaging modalities such as 2D Scanning Electron Microscopy (SEM) or 3D X-ray Computed Tomography (XCT). The 3D XCT has some advantages over conventional SEM. The principal ability is to gain an understanding of the 3D structure of objects. However, XCT has limited resolution compared to SEM. This means SEM can be useful to provide more detailed specific estimates of the sizes of structures such as estimates of the diameters of fibres. Image processing of these images has resulted in the need for a gold standard to help demonstrate the correct functioning and validation of designed algorithms. Simulation can play an important part in the validation of algorithms. However, previous works have performed limited simulations. Some methods simulate fibres as straight vectors. The approach taken here is more realistic, allowing for curving, overlapping and other more realistic generation of fibre volumes with the use of splines. The limited resolution in the imaging processes are also considered here, another important factor. Simulation results are compared with real world imaging data from both SEM and XCT. The generated results appear to show similar properties and could potentially be used as gold standards for the validation of image processing algorithms.