Data structure

The input to the software is either a multipage TIFF or a folder of TIFF files. Each page in the multipage TIFF or each image in the folder, respectively, depicts the signal for a pre-specified mass range, which corresponds to a single conjugated antibody. Each processing step with MAUI will add an additional folder of TIFFs to the dataset, using as input the results of the previous processing step.

Channel crosstalk

Channel crosstalk refers to scenarios where the signal of a source channel contaminates a target channel or is correlated with such contamination. Channel crosstalk in MBI can occur for a variety of reasons, some of which we will detail below. The first predominant cause for channel crosstalk is ionic contamination. In addition to the predominating monoatomic ion (X⁺, where X = m/z), metal reporters used for antibody labeling can also produce relatively small amounts of metal hydrides (XH⁺), oxides (XO⁺), and hydroxides (XOH⁺) [17] with masses of X+1, X+16, and X+17 respectively (Fig 3A). These polyatomic metal adducts can potentially contaminate heavier mass reporters. For example, 149Sm typically produces a metal oxide at 165 amu (149+16) with 4% the intensity of the monoatomic peak that can contaminate signal arising from an antibody tagged with 165Ho [17]. Alternatively, in early versions of the instrument we observed that bare regions of the slide often resulted in high-intensity signal across multiple mass channels. Since this signal is derived from the slide, it has a near-perfect overlap with components of the slide coating, e.g. gold (197Au), tantalum (181Ta) and silicon (28Si) channels (Fig 3B). The current generation of instruments does not suffer from this form of signal contamination (S3A Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Removal of cross-channel contamination.

(A-C) Shown are examples for different sources of cross-channel contamination. In all cases a target channel (top row) is contaminated by another channel (middle row) or is correlated with such contamination. MAUI allows to remove this contamination from the target image (bottom row). (D) The first step for removal of cross-channel contamination is to cap, blur and threshold the contaminating channel to generate a binary mask of the contamination. The second is to remove signal where the mask is positive. (E-H) Examples of the resulting image when varying the algorithm parameters, including the Contaminant Cap (E), Gaussian Radius (F), Threshold (G), and the Remove Value (H). (I) The GUI enables to rapidly evaluate these parameters across many points and channels.

https://doi.org/10.1371/journal.pcbi.1008887.g003

Channel crosstalk can also occur from isotopic impurities in the metal stocks used for conjugation. Many elements naturally exist as a mixture of isotopes, and while these have been enriched to high purity for conjugation, they still have varying amounts of contaminating isotopes of the same element. For example, the element samarium has 7 naturally occurring isotopes with an abundance ranging from 3.08% (¹⁴⁴Sm) to 26.74% (¹⁵²Sm). ¹⁵²Sm can be purified to >98% by electromagnetic enrichment, but will still contain small amounts of the other samarium isotopes. Finally, channel crosstalk can result from systematic, off-target staining, resulting from the antibodies or metals used. This often presents as dim staining of biological structures that the antibody is not expected to bind to. For example, indium and aluminum preferentially localize to nuclei, and therefore these channels may present nonspecific nuclear staining (Figs 3C and S3B). In all these instances the contaminating signal in one channel mirrors real signal in another channel. Thus, one can use the latter to clean the signal in a target channel.

We designed MAUI to facilitate cleaning of a target channel using the signal of a contaminating source channel. The pipeline is divided into three parts, easily visualized in the user interface (UI). First, the contaminating channel is processed in several steps to generate a binary mask, whereby pixels that are positive for the signal will have a value of one, and pixels that are negative for the signal will have a value of zero. In this step, the user can tune several parameters to influence the resulting mask. Once users are satisfied with the mask, they can evaluate its performance in cleaning different target channels, a step that also involves user-defined parameters. These two steps can be iterated until the user is satisfied with the result. Finally, the user can apply the selected channels and parameters to remove the contamination in multiple images.

A detailed mathematical description of the contamination-removal algorithm is presented in the Methods section. Briefly, the first step of the algorithm is to generate a binary mask of the contaminating channel. Due to the sparse nature of mass spectrometry data, it is necessary to preprocess the contaminating channel before using it to clean the target channel. This includes capping the signal, smoothing it and binarizing it to generate a mask (Fig 3D). Each of these steps requires the user to input parameters that can be easily tuned and evaluated for their effect on the resulting mask. In the capping step, the user inputs a cap parameter, which sets the maximum value of the contaminating channel. Counts greater than the cap value will be set to the cap. This parameter allows for the homogenization of the signal intensity of the contaminating channel, by controlling overly bright signal (Fig 3E). The cap should be proportional to the expected intensity of the contaminating channel. Reducing its value will increase the number of pixels that are classified as positive for contaminant signal in the final mask.

Next, the capped contaminant signal is blurred using a radial Gaussian filter. The user inputs a Gaussian Radius parameter to set the standard deviation of the filter. This step allows for the homogenization and smoothing of the signal in neighboring pixels (Fig 3F). Increasing the radius increases the blurring and will most likely result in more positive pixels in the mask. We have found that a radius of one pixel was suitable in most cases.

Then, the capped and blurred image is normalized between zero and one. The Threshold parameter, which also takes values between zero and one, is then applied to generate a binary mask of the contaminating signal. Any pixel in the normalized contaminant channel above the threshold is set to 1, and any value below the threshold is set to 0. A higher threshold will reduce the number of positive pixels in the mask (Fig 3G). MAUI allows the user to view the original contaminant signal, while easily changing the algorithm parameters and evaluating the resulting mask. It also plots a histogram depicting the frequency of intensity values in the normalized contaminant image to facilitate the user in picking a threshold that adequately captures the signal in the contaminating channel.

Once a binary mask of the contaminating channel has been generated, the user can now evaluate how this mask performs in cleaning any other channel using the Evaluation Parameters UI box. Cleaning is performed by subtracting a fixed value from all pixels in the target channel that were positive in the contaminant channel mask (Fig 3H). The value to remove is indicated by the user-defined parameter Remove Value. After removal, any value in the target channel that is below zero is set to zero. The user is given the option to remove a specific value, rather than, for example, zeroing out these pixels, because sometimes these pixels also have real signal superimposed with the contaminant. It is possible to zero out these pixels by choosing an arbitrarily large Remove Val. MAUI gives the user the option to evaluate all the parameters in cleaning different target channels and in different images.

Finally, once the users are satisfied with the chosen channels and parameters, they can use them to remove contamination in multiple images (Fig 3I). MAUI will create a new folder and populate it with the contamination-removed versions of the loaded data. These can be used as input for further preprocessing, including additional rounds of contamination removal. The software will also save a log file documenting the parameters used, to allow reproducibility of the analysis.

Noise removal

Image noise is traditionally defined as unwanted variation in image information, which deteriorates image quality. In antibody-based visualization of tissues it can result from various factors, ranging from instrumentation to tissue quality to nonspecific binding of antibodies, which generate weak signal that may follow histological patterns. For our purposes, we define noise as any signal outside of the biological structures the stain is expected to mark, and that does not specifically overlap with any of the other channels in the panel. Therefore, it is not possible to remove it by the contamination-removal technique described above.

Here, we apply a K-Nearest-Neighbor (KNN) approach to noise filtering [18]. The approach takes advantage of the relatively low density of noise counts as compared to signal counts (high signal to noise ratio). It is based on the assumption that the presence of the intended biological target at a location will increase the odds of the antibody binding at that location, and that higher ion counts in and near a pixel indicate a higher probability of the associated biological signal being present at that location. We find this approach particularly suitable for MIBI data due to the sparse nature of the data.

The denoising algorithm consists of two stages: first, the count density near each pixel is estimated using an intensity-normalized version of KNN, and second, the signal from low-density pixels is filtered. KNN estimates density by calculating the distances from a specific pixel x_(i,j) in the image to the closest k non-zero pixels. The shorter these distances are, the higher the density in that location. The intensity-normalized version performs similarly, but takes into account the count value in each positive pixel. Thus, instead of measuring the distances to the k closest positive pixels, it will calculate the distance to the k closest individual ion counts, whereby several counts can be found in the same pixel.

For example, in Fig 4A we demonstrate the calculation of the 5-nearest-neighbors of the central pixel of the image, with a value if x_(i,j) = 2. We think about this value as constituting two separate events. Thus, the first neighbor is inside of the same pixel, with a distance of zero. The next nearest neighbor is one pixel to the right, with a distance of one. The next three nearest-neighbors are two pixels away above, each of these with a distance of two (Fig 4B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Removal of noise using a K-Nearest-Neighbor (KNN) algorithm.

(A-B) Signal density at each pixel is determined using its K nearest neighbor counts. Shown is an example of the 5 nearest counts (A) and their distances (B) to the central pixel. (C) An example of staining of Histone H3 with noise surrounding the nuclear signal. (D) Shown is the frequency of the average distance to the 25 nearest counts for all pixels in (C). Signal is characterized by short distances (high density), whereas noise is characterized by long distances (low density). The red line is the chosen threshold for denoising, and pixels with large distances to their neighbors will be set to zero. (E) Histone H3 signal after denoising. (F) The average KNN-distribution for Histone H3, same as in (D), after denoising. (G-H) Examples of the resulting image of Histone H3 when varying the algorithm parameters, including the K-value (G) and Threshold (H). (I-J) Examples of the resulting image of a membranous protein, CD8, when varying the algorithm parameters, including the K-value (I) and Threshold (J). (K) The GUI enables to rapidly evaluate these parameters across many points and channels.

https://doi.org/10.1371/journal.pcbi.1008887.g004

We define the ADK as the average distance to the k nearest neighbors. The average distance to the 5-nearest-neighbors in the example above is (0+1+2+2+2)/5 = 1.4. For a given value of k, we then calculate the ADK for every pixel x_(i,j) in the image. Fig 4C depicts an image of Histone H3 (HH3), while Fig 4D depicts the corresponding distribution of ADK values for all positive pixels in the image, using a k value of 25. This is a bimodal distribution in which low ADK values correspond to high density of counts and are therefore attributable to signal, whereas high ADK values correspond to low density of counts, and therefore are considered as noise. The actual denoising involves picking a threshold used to separate “noise” from “signal”. Pixels with higher count densities (and therefore lower ADK values) are considered to be true signal and are left unchanged, while pixels with lower count densities (and therefore higher ADK values) are considered to be noise and their count value is set to zero. For example, using a threshold of 1.5 on the HH3 image shown in Fig 4C, any pixel with an ADK above 1.5 is set to zero, resulting in the image and corresponding distribution of ADK values shown in Fig 4E and 4F, correspondingly. A full mathematical derivation of the process is detailed in the Methods section.

MAUI is designed to facilitate, streamline and expedite the process of denoising using the intensity-normalized KNN approach for multiple images at a time (Fig 4G–4K). The k-value and threshold are user-defined parameters, which can be easily adjusted for each marker at the discretion of the domain-expert. MAUI generates real-time feedback as the k-value and threshold are adjusted, making it fast and easy to intuit which value ranges are most effective for denoising a particular dataset. When a given marker is selected on a given image, the ADK distribution for the k-value is plotted, along with an indication of the current threshold value. Easy scrolling between loaded images allows rapid evaluation of these properties on different images. The raw data is plotted, along with the denoised data and the difference between the two. Because the dynamic range of MIBI images can be quite skewed (significant biological information can be contained within ion-count differences of 1 or 2 counts, while outliers of 10 or 20 counts are possible), a display cap parameter is included for easy visual display of the data. We find that both the ADK histogram and the images are useful tools in guiding users’ choice of parameters.

Notably, current experience with this approach has demonstrated that the algorithm is quite robust to the specific value of k and for the most part does not need to be changed given the same imaging conditions. On the other hand, the threshold is dependent on the distributions of signal and noise for each target and is usually defined for each channel separately. Once appropriate k and threshold values have been selected for each channel, the entire dataset can be easily denoised using these parameters. MAUI will create a new folder and populate it with the noise-removed versions of the loaded data. These can be used as input for further processing. The software will also save a log file documenting the parameters used, to increase reproducibility.

Aggregate removal

Aggregates are concentrated areas of high counts that have been reported in both MIBI and IMC data [16,19]. These spots are uncorrelated with any biological structures and are thought to result from conglomerations of antibodies. Aggregates can interfere with downstream analysis, particularly of weak channels, where they may be falsely interpreted as positive staining (Fig 5A). A more detailed discussion of aggregate identification is presented in the Methods section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Removal of aggregates.

(A) An overview of the aggregate-removal process. Left: An image of denoised Tryptase signal. Aggregates are evident in the top right corner. The image is blurred with a Gaussian filter and then binarized. Objects are extracted from the binary image, and any object small enough is considered an aggregate and removed. (B-C) Examples of the resulting image of Tryptase when varying the algorithm parameters, including the Threshold (B,C) and Blur radius (D-E). (F) The GUI enables to rapidly evaluate these parameters across many points and channels.

https://doi.org/10.1371/journal.pcbi.1008887.g005

The procedure for removing aggregates takes advantage of their small size, much smaller than real signal of biological structures, and involves finding small continuous areas of positive counts. First, the image is blurred using a Gaussian filter, and then the blurred image is thresholded using a cutoff of zero to create a binary mask. The blurring has the effect of merging patches of signal that are near each other, usually indicative of them being part of a larger structure, such as a cell or vessel. Then, once a mask has been generated, all connected objects in the binary mask and their sizes are identified. Any object below a size threshold selected by the user is considered to be an aggregate, and the values of the counts in its constituent pixels are set to zero (Fig 5A).

MAUI allows the user to pick for each channel in the data the radius of the Gaussian blur as well as the size threshold for objects to be considered aggregates, listing the chosen parameters for each channel in a list (Fig 5B and 5C). MAUI will graphically display in real time the raw data and the aggregate-removed data, resulting from the currently chosen parameters. Empirically, we find that mild blurring (e.g. radius = 1) and a single threshold are often adequate across channels and datasets. Once an appropriate set of parameters has been chosen, MAUI allows to easily apply these parameters to all images in the dataset. Just as with the contamination removal and noise removal steps, MAUI will create a new folder and populate it with the aggregate-removed versions of the loaded data, as well as a log file recording the parameters.

Availability and future directions

High-dimensional imaging technologies are poised to revolutionize tissue histology. These new modalities need specialized computational pipelines to facilitate pre-processing such that clean, high-quality data will be the input for subsequent stages of analysis. Here, we surveyed some of the common issues associated with data collected with multiplexed immunohistochemistry imaging platforms. We have described the simple computational solutions developed in our lab to address these issues and make MBI data suitable for subsequent analysis. MAUI, the graphical user interface that wraps these solutions, has substantially improved the speed and ease in which biologists and clinicians in our lab interact with the data and perform these labor and computationally-intensive steps in a reproducible and efficient manner. MAUI can be downloaded from https://github.com/angelolab/MAUI. We expect future developments in assays, instrumentation and computation to further enhance the quality of multiplexed-imaging data, making it more accessible to the large scientific community.

Contaminant removal

For a given channel of MBI data f(x,y) (1) the user finds an associated contaminating channel h(x,y) (2). The user then picks a cap value c which is used to create a capped version of the contaminating channel h_capped(x,y) (3). The user then picks a standard deviation σ for a Gaussian blur g(x,y) (4), which is convolved with h_capped to create a blurred version h_blurred (5). This image is then rescaled to have a maximum of 1, producing an image h_rescaled (6). The user then picks a threshold value t which is used to binarize h_rescaled, producing a binary mask that can be used for contaminant removal, h_mask (7). Finally, the user selects a removal value v. At any pixel location (x,y) where h_mask(x,y) is 1, we remove v from f(x,y), producing f_removed (8). Anywhere f_removed<0 is set to zero, since negative values of counts are nonsensical. This produced the final contaminant-removed MBI data, f_clean (9).

1

2

3

4

5

6

7

8

9

Noise removal

We let f(x,y) be the function that maps pixels in an image to ion counts on a particular channel of MBI data (10). We define a function C(x,y) which takes in a single pixel (x,y) and outputs a list with f(x,y)copies of f(x,y) (11). We let this function be defined on lists of pixels (a,b,c,…) such that the output is the concatenation of the outputs of the function on each of the individual pixels in the input list (12). We then define another function N(x,y) which outputs a list of all pixels in the image ordered by their Euclidean distance (denoted by ‖∙‖) from (x,y) (13). We then let (14) and take only the first K entries, defining , which we call the K-nearest neighbors of (x,y) (15). We then calculate the average distance to the K-nearest neighbors of each pixel (x,y) denoted by A_K(x,y) (16). To clean the data, the user picks a threshold t_n, which is used to separate “noise” counts from “signal” counts. Any pixel with an average distance to its K-nearest neighbors more than the threshold is treated as noise and zeroed-out, otherwise its value is left unchanged (17).

10

11

12

13

14

15

16

17

Aggregate removal

We let f(x,y) be MBI data, as in (10). The user picks a standard deviation σ for a Gaussian filter g (18), which is convolved with our data to produce f_blurred (19). We then binarize this image to create f_mask (20).

We use a built-in function in MATLAB called regionprops, here denoted as f_object, which takes in a binary mask and outputs all connected sets of pixels with a value of 1 (thus any set of pixels in the input with a value of 1 which are connected to each other will be considered a single object). All such objects are returned as outputs of the function (21). The user defines a threshold t_a for the size of aggregate objects, where objects with fewer pixels than t_a are counted as aggregates (22). Finally, all pixels in the data the user is cleaning that are part of an aggregate object are zeroed-out (23).

18

19

20

21

22

23

Judging whether large clumps of signal is true signal vs aggregation can range from easy to non-trivial depending on the antibody, tissue and aggregation state. Currently, this process depends to a large extent on experts with domain knowledge. We commonly employ the following guidelines to decide if a signal is real or an aggregate: (A) Familiarity with the expected staining pattern of the antibody. Whereas for some antibodies we expect small, punctate staining, which is difficult to differentiate from aggregates (e.g. Granzyme B), most of the antibodies that we use stain larger cellular structures. We use knowledge on the expected staining patterns, both intra-cellularly (e.g. nuclear vs. membrane), cellularly (e.g. expected coexpression of markers) and histologically (e.g. expected staining in some anatomical locations and not others) to decide whether a “clump of signal” is real or not. While there may be ambiguous cases, for the most part it is quite clear. (B) Comparison between different tissues. In our experience, once an antibody vial begins to aggregate, aggregates will be observed in all tissues stained with that vial. As such, if one suspects that signal is actually an aggregate, it is useful to see whether aggregates are present in this channel across additional images. (C) Design of specific experiments to check for aggregates. When one is uncertain whether an antibody is aggregated, it is possible to design specific staining experiments to address this. For example, staining a negative control tissue that is not expected to express the target protein. Alternatively, one can stain serial sections and see whether the signal repeats between sections.