Fluorescence microscopy is a key driver of discoveries in the life sciences, with observable phenomena being limited by the optics of the microscope, the chemistry of the fluorophores, and the maximum photon exposure tolerated by the sample. These limits necessitate trade-offs between imaging speed, spatial resolution, light exposure, and imaging depth. In this work we show how content-aware image restoration based on deep learning extends the range of biological phenomena observable by microscopy. We demonstrate on eight concrete examples how microscopy images can be restored even if 60-fold fewer photons are used during acquisition, how near isotropic resolution can be achieved with up to tenfold under-sampling along the axial direction, and how tubular and granular structures smaller than the diffraction limit can be resolved at 20-times-higher frame rates compared to state-of-the-art methods. All developed image restoration methods are freely available as open source software in Python, FIJI, and KNIME.

Fluorescence microscopy is an indispensable tool in the life sciences for investigating the spatio-temporal dynamics of cells, tissues, and developing organisms. Recent advances such as light-sheet microscopy^1,2,3, structured illumination microscopy^4,5, and super-resolution microscopy^6,7,8 enable time-resolved volumetric imaging of biological processes within cells at high resolution. The quality at which these processes can be faithfully recorded, however, is determined not only by the spatial resolution of the optical device used, but also by the desired temporal resolution, the total duration of an experiment, the required imaging depth, the achievable fluorophore density, bleaching, and photo-toxicity^9,10. These aspects cannot all be optimized at the same time—trade-offs must be made, for example, by sacrificing signal-to-noise ratio (SNR) by reducing exposure time to gain imaging speed. Such trade-offs are often depicted by a design space that has resolution, speed, light exposure, and imaging depth as its dimensions (Fig. 1a), with the volume being limited by the maximal photon budget compatible with sample health^11,12.

a, Trade-offs between imaging speed, spatial resolution, and light exposure need to be made owing to the constraints of the maximal photon budget a sample permits. Image restoration enlarges this design space. b, Overview of the proposed pipeline for image denoising. Pairs (x_i,y_i) of registered high- and low-SNR volumes are acquired at the microscope. A convolutional neural network is trained to restore y_i from x_i. The trained CARE network can then be applied to previously unseen low-SNR images \(\tilde {x}\), yielding \(\tilde {y}\). c, Input data and restorations for nucleus-stained (RedDot1) flatworm (S. mediterranea). Shown are a single image-plane of a raw input stack (top row), the output of NLM denoising²² (second row), the network prediction (third row), and the high-SNR gold standard/ground truth (bottom row). d, Prediction error for data from c at three imaging conditions C1–C3. Box-dot plots (n = 20 per condition) show NRMSE and SSIM (higher is better) for the input, for the denoising baseline (NLM), and for network restorations. Boxes show interquartile range (IQR), lines signify medians, and whiskers extend to 1.5 times the IQR. e, Input data and restorations for a nucleus-labeled (EFA::nGFP) red flour beetle (Tribolium castaneum) embryo. Figure structure as in c.

Full size image

These trade-offs can be addressed through optimization of the microscopy hardware, yet there are physical limits that cannot easily be overcome. Therefore, computational procedures to improve the quality of acquired microscopy images are becoming increasingly important. Super-resolution microscopy^{4,13,14,15,16}, deconvolution^17,18,19, surface projection algorithms^20,21, and denoising methods^22,23,24 are examples of sophisticated image restoration algorithms that can push the limit of the design space, and thus allow the recovery of important biological information that would be inaccessible by imaging alone. However, most common image restoration problems have multiple possible solutions, and require additional assumptions to select one solution as the final restoration. These assumptions are typically general, for example, requiring a certain level of smoothness of the restored image, and therefore are not dependent on the specific content of the images to be restored. Intuitively, a method that leverages available knowledge about the data at hand ought to yield superior restoration results.

Deep learning is such a method, because it can learn to perform complex tasks on specific data by employing multilayered artificial neural networks trained on a large body of adequately annotated example data^25,26. In biology, deep learning methods have, for instance, been applied to the automatic extraction of connectomes from large electron microscopy data²⁷, for classification of image-based high-content screens²⁸, fluorescence signal prediction from label-free images^29,30, resolution enhancement in histopathology³¹, or for single-molecule localization in super-resolution microscopy^32,33. However, the direct application of deep learning methods to image restoration tasks in fluorescence microscopy is complicated by the absence of adequate training data and the fact that it is impossible to generate them manually.

We present a solution to the problem of missing training data for deep learning in fluorescence microscopy by developing strategies to generate such data. This enables us to apply common convolutional neural network architectures (U-Nets³⁴) to image restoration tasks, such as image denoising, surface projection, recovery of isotropic resolution, and the restoration of sub-diffraction structures. We show, in a variety of imaging scenarios, that trained content-aware image restoration (CARE) networks produce results that were previously unobtainable. This means that the application of CARE to biological images transcends the limitations of the design space (Fig. 1a), pushing the limits of the possible in fluorescence microscopy through machine-learned image computation.

Images with a low SNR are difficult to analyze in fluorescence microscopy. One way to improve SNR is to increase laser power or exposure times, which is usually detrimental to the sample, limiting the possible duration of the recording and introducing artifacts due to photo-damage. An alternative solution is to image at low SNR, and later to computationally restore acquired images. Classical approaches, such as non-local-means denoising²², can in principle achieve this, but without leveraging available knowledge about the data at hand.

Image restoration with physically acquired training data

To demonstrate the power of machine learning in biology, we developed CARE. We first demonstrate the utility of CARE on microscopy acquisitions of the flatworm Schmidtea mediterranea, a model organism for studying tissue regeneration. This organism is exceptionally sensitive to even moderate amounts of laser light³⁵, exhibiting muscle flinching at desirable illumination levels even when anesthetized (Supplementary Video 1). Using a laser power that reduces flinching to an acceptable level results in images with such low SNR that they are impossible to interpret directly. Consequently, live imaging of S. mediterranea has thus far been intractable.

To address this problem with CARE, we imaged fixed worm samples at several laser intensities. We acquired well-registered pairs of images, a low-SNR image at laser power compatible with live imaging, and a high-SNR image, serving as a ground truth (Fig. 1b). We then trained a convolutional neural network and applied the trained network to previously unseen live-imaging data of S. mediterranea (Supplementary Notes 1 and 2). We used networks of moderate size (∼10⁶ parameters) based on the U-Net architecture^34,36, together with a per-pixel similarity loss, for example absolute error (Supplementary Fig. 1, Supplementary Note 2, and Supplementary Table 3). We consistently obtained high-quality restorations, even if the SNR of the images was very low, for example, being acquired with a 60-fold reduced light dosage (Fig. 1c,d, Supplementary Video 2, and Supplementary Figs. 2–4). To quantify this observation, we measured the restoration error between prediction and ground-truth images for three different exposure and laser-power conditions. Both the normalized root-mean-square error (NRMSE) and the structural similarity index (SSIM; a measurement of the perceived similarity between two images³⁷) improved considerably compared with results obtained by several potent classical denoising methods (Fig. 1d, Supplementary Figs. 3 and 5, and Supplementary Table 1). We further observed that even a small number of training images (for example, 200 patches of size 64 × 64 × 16) led to an acceptable image restoration quality (Supplementary Fig. 6). Moreover, while training a CARE network can take several hours, the restoration time for a volume of size 1,024 × 1,024 × 100 was less than 20 s on a single graphics processing unit. (We used a common consumer graphics processing unit (Nvidia GeForce GTX 1080 or Titan X) for all presented experiments.) In this case, CARE networks are able to take input data that are unusable for biological investigations and turn them into high-quality time-lapse data, providing a practical framework for live-cell imaging of S. mediterranea.

We next asked whether CARE improves common downstream analysis tasks in live-cell imaging, such as nuclei segmentation. We used confocal microscopy recordings of developing Tribolium castaneum (red flour beetle) embryos, and as before trained a network on image pairs of samples acquired at high and low laser powers (Fig. 1e). The resulting CARE network performed well even on extremely noisy, previously unseen live-imaging data (Supplementary Note 4, Supplementary Video 3, and Supplementary Fig. 7). To test the benefits of CARE for segmentation, we applied a simple nuclei segmentation pipeline to raw and restored image stacks of T. castaneum. The results show that, compared to manual expert segmentation, the segmentation accuracy (as measured with the standard SEG score³⁸) improved from SEG = 0.47 on the classically denoised raw stacks to SEG = 0.65 on the CARE restored volumes (Supplementary Fig. 8). Since this segmentation performance is achieved at substantially reduced laser power, the gained photon budget can now be spent on the imaging speed and light-exposure dimensions of the design space. This means that Tribolium embryos, when restored with CARE, can be imaged for longer and at higher frame rates, thus enabling improved tracking of cell lineages.

Encouraged by the performance of CARE on two independent denoising tasks, we asked whether such networks can also solve more complex, composite tasks. In biology it is often useful to image a three-dimensional (3D) volume and project it to a two-dimensional (2D) surface for analysis, such as when studying cell behavior in developing epithelia of the fruit fly Drosophila melanogaster^39,40. Also, in this context, it is beneficial to optimize the trade-off between laser power and imaging speed, usually resulting in rather low-SNR images. Thus, this restoration problem is composed of projection and denoising, presenting the opportunity to test whether CARE networks can deal with such composite tasks. For training, we again acquired pairs of low- and high-SNR 3D image stacks, and further generated 2D projection images from the high-SNR stacks²⁰ that serve as ground truth (Fig. 2a). We developed a task-specific network architecture that consists of two jointly trained parts: a network for surface projection, followed by a network for image denoising (Fig. 2b, Supplementary Fig. 9, and Supplementary Note 2). The results show that with CARE, reducing light dosage up to tenfold has virtually no adverse effect on the quality of segmentation and tracking results obtained on the projected 2D images with an established analysis pipeline⁴¹ (Fig. 2c,d, Supplementary Video 4, and Supplementary Figs. 10–12). Even for this complex task, the gained photon budget can be used to move beyond the design space, for example, by increasing temporal resolution, and consequently improving the precision of tracking of cell behaviors during wing morphogenesis⁴¹.

a, Schematic of the composite task at hand. A single cell layer of interest of a Drosophila wing is embedded in an imaged 3D volume. The desired pipeline extracts and denoises the 2D tissue layer from low-SNR input volumes. b, The proposed CARE network first projects the data and then performs a 2D denoising step. c, Restoration results on E-cadherin-labeled fly wing data acquired with a spinning disk microscope. Shown are a max-projection of the raw input data (top row), a surface projection baseline obtained by state-of-the-art method PreMosa²⁰ (second row), CARE network results (third row), and ground-truth projections obtained by applying PreMosa on a very high-laser-power acquisition of the same sample (bottom row). d, Prediction error for three imaging conditions (C1–C3; see Methods). Box-dot plots (n = 26 per conditions) show NRMSE and SSIM (higher is better) for results obtained using PreMosa (blue), PreMosa with additional denoising (NLM²², green), and CARE network results (orange). Boxes show IQR, lines signify medians, and whiskers extend to 1.5 times the IQR. Comparison to additional baselines can be found in Supplementary Fig. 11.

Full size image

Image restoration with semi-synthetic training data

A common problem in fluorescence microscopy is that the axial resolution of volumetric acquisitions is substantially lower than the lateral resolution (some advanced modalities allow for isotropic acquisitions, such as multiview light-sheet microscopy^19,42). This anisotropy compromises the ability to accurately measure properties such as the shapes or volumes of cells. Anisotropy is caused by the inherent axial elongation of the optical point spread function (PSF), and the often low axial sampling rate of volumetric acquisitions required for fast imaging. For the restoration of isotropic image resolution, adequate pairs of training data cannot directly be acquired at the microscope. Rather, we took well-resolved lateral slices as ground truth, and computationally modified them (applying a realistic imaging model; Supplementary Note 2) to resemble anisotropic axial slices of the same image stack. In this way, we generated matching pairs of images showing the same content at axial and lateral resolutions. These semi-synthetically generated pairs are suitable to train a CARE network that then restores previously unseen axial slices to nearly isotropic resolution (Fig. 3a, Supplementary Fig. 13, Supplementary Note 2, and refs ^43,44). To restore entire anisotropic volumes, we applied the trained network to all lateral image slices, taken in two orthogonal directions, averaged to a single isotropic restoration (Supplementary Note 2).

a, Schematic of the semi-synthetic generation of training data. Lateral slices of the raw data (green) are used as ground truth, which are then synthetically downsampled and convolved with the rotated PSF of the microscope used. This results in corresponding anisotropic slices (black) with similar resolution as the raw axial slices (orange). b, Application to time-lapse acquisitions of D. melanogaster⁴⁵. Shown are three areas of the raw axial input data (top row) and their respective isotropic restorations (bottom row). Additionally, the Fourier spectrum of raw and restored images illustrates how missing spectral components are recovered. c, An axial slice through a developing zebrafish (Danio rerio) eye. Shown are the anisotropic raw data (top row) and isotropic restoration (bottom row). Nuclei are stained with DRAQ5 (magenta), and the nuclear envelope is labeled with GFP-LAP2b (green). d, An axial slice through mouse liver tissue. Shown are the anisotropic raw data (with a subsampling of σ = 8, top row) and the isotropic restoration by the network (middle row). Nuclei and membranes of hepatocytes are labeled with DAPI and phalloidin, respectively, and imaged in a single channel. Plots show the effect of increasing levels of axial subsampling of raw (blue) and isotropically restored (orange) volumes. We plot the NRMSE and SSIM (higher is better) with respect to our restorations of the shown isotropically sampled raw data. Details on data and training can be found in Supplementary Table 3.

Full size image

We applied this strategy to increase axial resolution of acquired volumes of fruit fly embryos⁴⁵, zebrafish retina⁴⁶, and mouse liver, imaged with different fluorescence imaging techniques. The results show that CARE improved the axial resolution in all three cases considerably (Fig. 3b–d, Supplementary Videos 5 and 6, and Supplementary Figs. 14 and 15). To quantify this, we performed Fourier spectrum analysis of Drosophila volumes before and after restoration, and showed that the frequencies along the axial dimension are fully restored, while frequencies along the lateral dimensions remain unchanged (Fig. 3b and Supplementary Fig. 16). Since the purpose of the fruit fly data is to segment and track nuclei, we applied a common segmentation pipeline⁴⁷ to the raw and restored images, and observed that the fraction of incorrectly identified nuclei was reduced from 1.7% to 0.2% (Supplementary Note 2 and Supplementary Figs. 17 and 18). Thus, restoring anisotropic volumetric embryo images to effectively isotropic stacks leads to improved segmentation, and will enable more reliable extraction of developmental lineages.

While isotropic images facilitate segmentation and subsequent quantification of shapes and volumes of cells, vessels, or other biological objects of interest, higher imaging speed enables imaging of larger volumes and their tracking over time. Indeed, respective CARE networks deliver the desired axial resolution with up to tenfold fewer axial slices (Fig. 3c,d; see Supplementary Fig. 19 for comparison with classical deconvolution), allowing one to reach comparable results ten times faster. We quantified the effect of subsampling on raw and restored volumes with respect to restorations of isotropically sampled volumes for the case of the liver data (Fig. 3d and Supplementary Fig. 20). Finally, we observed that for two-channel datasets such as the zebrafish, networks learned to exploit correlations between channels, leading to a better overall restoration quality compared to results based on individual channels (Supplementary Fig. 15).

Image restoration with synthetic training data

Having seen the potential of using semi-synthetic training data for CARE, we next investigated whether reasonable restorations can be achieved even from synthetic image data alone, that is, without involving real microscopy data during training.

In most of the previous applications, one of the main benefits of CARE networks was improved imaging speed. Many biological applications additionally require the resolution of sub-diffraction structures in the context of live-cell imaging. Super-resolution imaging modalities achieve the necessary resolution, but suffer from low acquisition rates. In contrast, widefield imaging offers the necessary speed, but lacks the required resolution. We therefore tested whether CARE can computationally resolve sub-diffraction structures using only widefield images as input. Note that this is a fundamentally different approach compared to recently proposed methods for single-molecule localization microscopy that reconstruct a single super-resolved image from multiple diffraction-limited input frames using deep learning^32,33. To this end, we developed synthetic generative models of tubular and point-like structures that are commonly studied in biology. To obtain synthetic image pairs for training, we used these generated structures as ground truth, and computationally modified them to resemble actual microscopy data (Supplementary Note 2 and Supplementary Fig. 21). Specifically, we created synthetic ground-truth images of tubular meshes resembling microtubules, and point-like structures of various sizes mimicking secretory granules. Then we computed synthetic input images by simulating the image degradation process by applying a PSF, camera noise, and background auto-fluorescence (Fig. 4a, Supplementary Note 2, and Supplementary Fig. 21). Finally, we trained a CARE network on these generated image pairs, and applied it to two-channel widefield time-lapse images of rat INS-1 cells where the secretory granules and the microtubules were labeled (Fig. 4b). We observed that the restoration of both microtubules and secretory granules exhibited a dramatically improved resolution, revealing structures imperceptible in the widefield images (Supplementary Video 7 and Supplementary Fig. 22). To substantiate this observation, we compared the CARE restoration to the results obtained by deconvolution, which is commonly used to enhance widefield images (Fig. 4b). Line profiles through the data show the improved performance of the CARE network over deconvolution (Fig. 4b). We additionally compared results obtained by CARE with those from super-resolution radial fluctuations (SRRF¹⁴), a state-of-the-art method for reconstructing super-resolution images from widefield time-lapse data. We applied both methods on time-lapse widefield images of GFP-tagged microtubules in HeLa cells. The results show that both CARE and SRRF are able to resolve qualitatively similar microtubular structures (Fig. 4c and Supplementary Video 8). However, CARE reconstructions enable imaging to be carried out at least 20 times faster, since they are computed from a single average of up to 10 consecutive raw images while SRRF required about 200 consecutive widefield frames. We also used SQUIRREL⁴⁸ to quantify the error for both methods and observed that CARE generally produced better results, especially in image regions containing overlapping structures of interest (Fig. 4d and Supplementary Fig. 23).

a, Schematic of the fully synthetic training pipeline. b, Raw widefield images of rat secretory granules (pEG-hIns-SNAP; magenta) and microtubules (SiR-tubulin; green) in insulin-secreting INS-1 cells (top row), the corresponding network restorations (second row), and a deconvolution result of the raw image as a baseline (bottom row). Line plots show image intensities along the dashed lines in the top panels. c, GFP-tagged microtubules in HeLa cells. Raw input image (top row), network restorations (second row), super-resolution images created by the state-of-the-art method SRRF¹⁴ (bottom row). Line plots show image intensities along the dashed lines in the top panels. d, Error quantification via SQUIRREL⁴⁸ for network results and the results obtained by SRRF. Shown are error maps corresponding to the dashed box in c and the resolution scaled error (RSE) for 20 consecutive frames. The data shown in c correspond to frame 1.

Full size image

Taken together, these results suggest that CARE networks can enhance widefield images to a resolution usually obtainable only with super-resolution microscopy, yet at considerably higher frame rates.

Reliability of image restoration

We have shown that CARE networks perform well on a wide range of image restoration tasks, opening up new avenues for biological observations (Supplementary Table 2). However, as for any image processing method, the issue of reliability of results needs to be addressed.

CARE networks are trained for a specific biological organism, fluorescent marker, and microscope setting. When a network is applied to data it was not trained for, results are likely to suffer in quality, as is the case for any (supervised) method based on machine learning. Nevertheless, we observed only minimal ‘hallucination’ effects, where structures seen in the training data erroneously appear in restored images (Supplementary Figs. 24 and 25). In Supplementary Fig. 25a we show the two strongest errors across the entire body of available image data.

Nevertheless, it is essential to identify cases where the above-mentioned problems occur. To enable this, we changed the last network layer so that it predicts a probability distribution for each pixel (Fig. 5a, Methods, and Supplementary Note 3). We chose a Laplace distribution for simplicity and robustness (Supplementary Note 3). For probabilistic CARE networks, the mean of the distribution is used as the restored pixel value, while the width (variance) of each pixel distribution encodes the uncertainty of pixel predictions. Intuitively, narrow distributions signify high confidence, whereas broad distributions indicate low-confidence pixel predictions. This allows us to provide per-pixel confidence intervals of the restored image (Fig. 5a, Supplementary Figs. 26 and 27). We observed that variances tend to increase with restored pixel intensities. This makes it hard to intuitively understand which areas of a restored image are reliable or unreliable from a static image of per-pixel variances. Therefore, we visualize the uncertainty in short video sequences, where pixel intensities are randomly sampled from their respective distributions (Supplementary Video 9). We additionally reasoned that by analyzing the consistency of predictions from several trained models we can assess their reliability. To that end, we train ensembles (Fig. 5b) of about five CARE networks on randomized sequences of the same training data. We introduced a measure D that quantifies the probabilistic ensemble disagreement per pixel (Methods, Supplementary Note 3). D takes values between 0 and 1, with higher values signifying larger disagreement, that is, smaller overlap among the distributions predicted by the networks in the ensemble. Using fly wing denoising as an example, we observed that in areas where different networks in an ensemble predicted very similar structures, the disagreement measure D was low (Fig. 5c, top row), whereas in areas where the same networks predicted obviously dissimilar solutions, the corresponding values of D were large (Fig. 5c, bottom row). Therefore, training ensembles of CARE networks is useful for detecting problematic image areas that cannot reliably be restored. Another example for the utility of ensemble disagreement can be found in Supplementary Fig. 28.

a, For every pixel of a restored image, CARE networks can predict a (Laplace) distribution parameterized by its mean μ and scale σ (top). These distributions provide pixel-wise confidence intervals (bottom), here shown for a surface projection and denoising network (see Fig. 2). The line plot shows the predicted mean (blue), the 90% confidence interval (light blue), and corresponding ground-truth intensities (dashed red) along the yellow dashed line in the image on the left. b, Multiple independently trained CARE networks are combined to form an ensemble, resulting in an ensemble distribution and an ensemble disagreement measure D ∈ [0, 1]. c, Ensemble predictions can vary, especially on challenging image regions. Shown are two examples for a surface projection and denoising ensemble of four networks (rows). From left to right we show the maximum projection of input data, predictions of the four networks of the ensemble, the pixel-wise ensemble mean, and the ensemble disagreement measure. While the top row shows an image region with low ensemble disagreement, the bottom row shows a region where individual network predictions differ, resulting in a high disagreement score in respective image areas.

Full size image

We have introduced CARE networks designed to restore fluorescence microscopy data. A key feature of our approach is that the generation of training data does not require laborious manual training data generation. With CARE, flatworms can be imaged without unwanted muscle contractions, beetle embryos can be imaged much more gently and therefore for longer and much faster, large tiled scans of entire Drosophila wings can be imaged and simultaneously projected at dramatically increased temporal resolution, isotropic restorations of embryos and large organs can be computed from existing anisotropic data, and sub-diffraction structures can be restored from widefield systems at high frame rates. In all these examples, CARE allows the photon budget saved during imaging to be invested into improvement of acquisition parameters relevant for a given biological problem, such as speed of imaging, photo-toxicity, isotropy, and resolution.

Whether experimentalists are willing to make the above-mentioned investment depends on their trust that a CARE network is accurately restoring the image. This is a valid concern that applies to every image restoration approach. What sets CARE apart is the availability of additional readouts, that is, per-pixel confidence intervals and ensemble disagreement scores, which allow users to identify image regions where restorations might not be accurate.

We have shown multiple examples where image restoration with CARE networks positively impacts downstream image analysis, such as segmentation and tracking of cells needed to extract developmental lineages. Interestingly, in the case of Tribolium, CARE improved segmentation by efficient denoising, whereas in the case of Drosophila, the segmentation was improved by an increase in the isotropy of volumetric acquisitions. These two benefits are not mutually exclusive and could very well be combined. In fact, we have shown on data from developing Drosophila wings that composite tasks can be jointly trained. Future explorations of joint training of composite networks will further broaden the applicability of CARE to complex biological imaging problems (see ref. ⁴⁹).

However, CARE networks cannot be applied to all existing image restoration problems. For instance, the proposed isotropic restoration relies on the implicit assumption that structures of interest do appear in arbitrary orientations and that the PSF is constant throughout the image volume. This assumption is only approximately true, and becomes increasingly worse as the imaging depth in the sample tissue increases. Additionally, because of the nonlinear nature of neural network predictions, CARE must not be used for intensity-based quantifications such as, for example, fluorophore counting. Furthermore, the disagreement score we introduced may be useful to additionally identify instances where training and test data are incompatible, that is, when a CARE network is applied on data that contain biological structures absent from the training set.

Overall, our results show that fluorescence microscopes can, in combination with CARE, operate at higher frame rates, shorter exposures, and lower light intensities, while reaching higher resolution, and thereby improving downstream analysis. The technology described here is readily accessible to the scientific community through the open source tools we provide. We predict that the current explosion of image data diversity and the ability of CARE networks to automatically adapt to various image contents will make such learning approaches prevalent for biological image restoration and will open up new windows into the inner workings of biological systems across scales.

For each of the described experiments and restoration modalities, we (1) imaged or generated suitable training data, (2) trained a neural network (or ensemble of networks), and (3) applied the trained network and quantified/reported the results.

Network architecture and training

For all experiments (except fly wing projection) we used residual versions of the U-Net architecture³⁴ in 3D or 2D (Supplementary Fig. 1 and Supplementary Fig. 13). For the fly wing projection task, we used a two-stage architecture combining a projection and a denoising sub-network (Supplementary Fig. 9). All restoration experiments were performed in Python using Keras⁵⁰ and TensorFlow⁵¹. Source code for training and prediction, example applications, and documentation can be found at http://csbdeep.bioimagecomputing.com/doc/. The training details for each restoration experiment (e.g., number of used images, network hyper-parameters) are listed in Supplementary Table 3 and are described in Supplementary Note 2.

Image normalization

For training, prediction, and evaluation, it is important to normalize the input images to a common intensity range. We used percentile-based normalization, typically using percentile ranks p_low ∈ (1, 3) for determining the lowest value and p_high ∈ (99.5, 99.9) for the highest value. All image pixels are then affinely scaled, such that the lowest and highest values are converted to 0 and 1, respectively. For a given image y, the percentile-normalized image will be denoted by N(y, p_low, p_high).

Quantification of restoration errors

Since the image y predicted by any restoration method (CARE or any compared baseline) and the corresponding ground-truth image y₀ typically differ in the dynamic range of their pixel values, they have to be normalized to a common range first. To that end, we first percentile-normalize the ground-truth image y₀ as described before with p_low = 0.1 and p_high = 99.9. Second, we use a transformation φ(y) = αy + β that affinely scales and translates every pixel of the restored image based on parameters

with

That is, α and β are chosen such that the mean squared error (MSE) between φ(y) and N(y₀, 0.1, 99.9) is minimal (note that α, β can be easily computed in closed form). All final error metrics, such as NRMSE and SSIM³⁷, were computed on images normalized in this way. More details can be found in Supplementary Note 2.

Planaria denoising

Planaria (S. mediterranea) were cultured at 20 °C in planarian water⁵² and fed with organic bovine liver paste. To label nuclei, S. mediterranea samples were stained for 15 h in planarian water supplemented with 2× RedDot1 and 1% (v/v) dimethylsulfoxide (DMSO). For training data acquisition, planaria were euthanized with 5% (w/v) N-acetyl-l-cysteine in PBS and subsequently fixed in 4% (w/v) paraformaldehyde in PBS. For time-lapse recordings, RedDot1-stained planaria were anesthetized for 1 h with 0.019% (w/v) Linalool prior to mounting, which was maintained throughout the course of the live-imaging experiments. A 5-min incubation in 0.5% (w/v) pH-neutralized N-acetyl-l-cysteine was used to remove the animal’s mucus before mounting. For imaging, fixed or live animals were mounted in refractive-index-matched 1.5% agarose (50% (w/v) iodixanol) to enhance signal quality at higher imaging depths as described in ref. ⁵². For imaging, a spinning disc confocal microscope with a 30×/1.05-NA (numerical aperture) silicon oil-immersion objective and 640-nm excitation wavelength was used. We used four different laser-power/exposure-time imaging conditions: GT (ground truth) and C1–C3, specifically 2.31 mW/30 ms (GT), 0.12 mW/20 ms (C1), 0.12 mW/10 ms (C2), and 0.05 mW/10 ms (C3). To ensure that corresponding image stacks were well aligned, we interleaved all four different imaging conditions as different channels during acquisition. In total, we acquired 96 stacks of average size 1,024 × 1,024 × 400. From these data we sampled around 17,000 randomly positioned sub-volume pairs of size 64 × 64 × 16 voxels. We evaluated our results on 20 previously unseen volumes of S. mediterranea imaged at various developmental stages. As competing denoising methods, we chose lowpass filter, median filter, bilateral filter⁵³, non-local-means denoising (NLM)²², total variation denoising⁵⁴, BM3D²³, and BM4D⁵⁵. Please see Supplementary Table 1 and Supplementary Note 2 for more details.

Tribolium denoising and segmentation

An EFA::nGFP transgenic line of Tribolium castaneum was used for imaging of embryonic development with labeled nuclei⁵⁶. The beetles were reared and embryos were collected according to standard protocols⁵⁷. Imaging was done on a Zeiss 710 multiphoton laser-scanning microscope using a 25× multi-immersion objective. Similar to the planaria dataset, we used four different laser-power imaging conditions: GT and C1–C3, specifically 20 mW (GT), 0.1 mW (C1), 0.2 mW (C2), and 0.5 mW (C3). For each condition we acquired 26 training stacks (of size ∼700 × 700 × 50) using different samples at different developmental stages. From that, we randomly sampled around 15,000 patches of size 64 × 64 × 16 and trained a 3D network as before. For testing, we used six additional volumes per condition, again acquired at different developmental stages. As a denoising baseline we again used NLM²². Nuclei segmentation was performed using a thresholding-based segmentation workflow. To create the segmentation ground truth, we used ilastik to train a pixel-wise random forest classifier to distinguish nuclei and background pixels in the high-SNR (GT) image, whose output was curated using a combination of segmentation tools from SciPy⁵⁸, the 3D volume rendering software spimagine (https://github.com/maweigert/spimagine) and manual, pixel-wise corrections. To create segmentations for restorations (NLM or CARE) of the low-SNR images (C2), we thresholded their intensities and labeled connected components of pixels above the threshold as individual nuclei. The segmentation accuracy was computed as the SEG score⁵⁹, which corresponds to the average overlap of segmented regions with matched ground-truth regions (0 ≤ SEG ≤ 1). More details can be found in Supplementary Note 2.

Flywing projection, segmentation, and tracking

D. melanogaster expressing the membrane marker Ecad::GFP were raised under standard conditions at 25 °C. Pupae were collected and prepared for imaging as described in ref. ⁶⁰. The dorsal side of the pupal wing was imaged with a Yokogawa CSU-X1 spinning disk microscope using a Zeiss LCI Plan-Neofluar 63×/1.3-NA Imm Corr objective. We acquired image stacks at four different conditions: GT and C1–C3, with camera exposure/laser power of 240 ms/20% (GT), 120 ms/2% (C1), 120 ms/3% (C2), and 120 ms/5% (C3), where we again interleaved all conditions during imaging. For each condition, we acquired 180 different 3D stacks (of size ∼700 × 700 × 50). As a prediction target we used the surface-projected 2D ground-truth signal obtained via PreMosa²⁰ computed on data acquired with GT settings. For training we sampled around 17,000 random 3D patches of size 64 × 64 × 50 from the acquired stacks. For the composite task of joint projection and denoising, we designed a stacked network architecture consisting of a projection and a denoising sub-network (see Supplementary Fig. 9). We evaluated the restoration quality on 26 previously unseen volumes, and compared results obtained with CARE against maximum projection (MIP), smooth 2D manifold extraction (SME)²¹, minimum cost surface projection (GraphCut)^61,62, and PreMosa²⁰. For all competing methods (except CARE), we additionally applied NLM denoising²² to the respective output (see Supplementary Fig. 11). To evaluate segmentation and tracking results on restored stacks, we used a time-lapsed acquisition of 26 time points imaged with the GT and C2 settings. To create a binary segmentation of membrane and background regions, we used a random forest classifier (Trainable Weka Segmentation⁶³ plugin in Fiji⁶⁴) that was trained on images with 30 manually labeled cells (membrane contour and corresponding non-membrane region inside) for both imaging settings. The probability maps generated by the classifier were processed with Tissue Analyzer⁶⁵, a tool for tracking and segmentation of cells in 2D epithelia, yielding a joint segmentation and tracking of cells over all frames. For each frame we computed the SEG score based on the raw and restored images with respect to the ground truth (see Supplementary Fig. 12). For more details, see Supplementary Note 2.

Drosophila isotropic restoration and segmentation

All input stacks were provided by the authors of ref. ⁴⁵, where histone-labeled D. melanogaster embryos were imaged using a light-sheet microscope. Note that this dataset was already processed, but still exhibited an anisotropic PSF and a fivefold axial subsampling that translated into a combined 4–6-fold decrease in axial resolution. We used the training data strategy as described in Supplementary Note 2 and ref. ⁴³, where the 2D lateral slices are used as ground truth and are synthetically subsampled by σ = 5 and blurred with the theoretical PSF of the light-sheet microscope. We used 15 volumes from equally spaced time points during development (between embryo cellularization and germband retraction), resulting in around 10,000 training patches of size 128 × 128. As network architecture we used a 2D U-Net (Supplementary Fig. 13). To quantify the restoration quality, we computed the spectral isotropy ratio Φ as the ratio of spectral energy of the signal in the Fourier domain along the axial and lateral dimension. To evaluate a nuclei segmentation task, we used a crop of a densely populated center region containing approximately 470 nuclei from an unseen test volume and generated ground-truth segmentation masks with ilastik employing extensive manual curation. We compared the segmentability of network-restored images with bicubically upsampled images by training a random forest classifier on both images using the GT masks as a target and generated instance segmentation via connected components of the thresholded probability maps. To evaluate the segmentation, we computed a bipartite matching between proposed and ground-truth nuclei instances (intersection over union ≥ 0.5) and used the fraction of unmatched nuclei as a measure of segmentation error.

Zebrafish retinal tissue isotropic restoration

Zebrafish (Danio rerio) imaging experiments were performed with a transgenic zebrafish line Tg(bactin:eGFP-LAP2b) that labels the nuclear envelope. Embryos were raised in E3 medium at 28.5 °C and treated with 0.2 mM 1-phenyl-2-thiourea at 8 hours post-fertilization (hpf) onward to delay pigmentation. Embryos were fixed at 24 hpf in 4% paraformaldehyde, permeabilized with 0.25% trypsin, and incubated with a far-red DNA stain (DRAQ5) for 2 d at 4 °C. Imaging of agarose-mounted embryos was performed on a spinning disk confocal microscope (Andor Revolution WD) with a 60×/1.3-NA objective, using excitation wavelengths of λ = 638 nm (DRAQ5) and λ = 488 nm (eGFP-LAP2b). Stacks were acquired with 2-μm steps, resulting in an axial subsampling factor of σ = 10.2. For generating training data, we acquired five multichannel volumes from which we extracted around 25,000 lateral patches of size 128 × 128 × 2, applied the corresponding theoretical PSF and subsampling model, yet always keeping the information of both image channels. Network training was done as before. To compare the restoration quality with classical deconvolution, we ran Huygens (Scientific Volume Imaging, http://svi.nl) on the bicubic upsampled raw stacks once with the actual PSF and once with a σ-fold down- and upsampled PSF (to account for the additional blur related to upsampling). We used the following parameters from Huygens: method, MLE; number iteration, 70; SNR parameter, 15; quality threshold, 0.05.

Mouse liver isotropic restoration

Mouse livers were fixed through transcardial perfusion with 4% paraformaldehyde and post-fixed overnight at 4 °C with the same solution. Tissue slices were optically cleared by a modified version of SeeDB⁶⁶ and stained with 4′,6-diamidino-2-phenylindole (DAPI) (nuclei) and phalloidin (membrane). The samples were imaged using a Zeiss LSM 780 NLO multiphoton laser-scanning microscope with a 63×/1.3-NA glycerol-immersion objective (Zeiss) using 780-nm two-photon excitation and an isotropic voxel size of 0.3 μm. We acquired eight stacks of mouse liver each of size 752 × 752 × 300. For the range of subsampling factors σ = 2,…,16, we created respective axial anisotropic stacks by retaining only every σth axial slice from the original volumes to be restored later. For each σ, we extracted around 15,000 patches of size 128 × 128 from the given body of data and trained a network as described before. Refer to Supplementary Note 2 for more details.

INS-1 cell tubular/granule restoration

Rat insulin-secreting beta cells (INS-1 cells) were cultured and transiently transfected with pEG-hIns-SNAP as previously described⁶⁷. The cells were labeled with 1 μM SNAP-Cell 505-Star (secretory granules) and with 1 μM SiR-tubulin (microtubules) for 1 h. Imaging was done with the DeltaVision OMX (GE) microscope using an Olympus Plan-Apochromat 60×/1.43-NA objective, yielding dual channel images. Time-lapse movies were acquired in widefield mode with 50-ms exposure time and 10% fluorescence intensity for each channel resulting in a final speed of 2 frames per second (fps). Deconvolution was done with the SoftWorkx software package running on-board the OMX. We created synthetic ground-truth images of tubular networks and granular structures by simulating 2D trajectories and granular points on a pixel grid, respecting the known physical properties (for example, microtubule width and persistence length). We generated the corresponding synthetic widefield input images by adding low-frequency Perlin noise mimicking auto-fluorescence, convolving the result with the theoretical PSF of the microscope, and adding Poisson and Gaussian noise mimicking camera noise. In total, we created around 8,000 synthetic patch-pairs of size 128 × 128. For both secretory granules and microtubules, we trained a 2D network (as before) to invert this degradation process and applied it on the respective channel of the widefield images (Supplementary Fig. 13). More details can be found in Supplementary Note 2.

HeLa cell microtubule restoration and error map calculation

HeLa cells stably expressing H2B-mCherry/mEGFP-α-tubulin⁶⁸ were grown in DMEM containing 10% FBS, 100 U ml^–1 penicillin and 100 mg ml^–1 streptomycin at 37 °C with 5% CO₂ in a humidified incubator. Before imaging cells were seeded onto a #1.5 glass-bottom 35-mm u-Dish. Imaging was performed on a Zeiss Elyra PS.1 inverted microscope at 37 °C and 5% CO₂ in total internal reflection fluorescence mode with a Plan-Apochromat 100×/1.46-NA oil-immersion objective (Zeiss) and additional 1.6× magnification with 488-nm laser illumination at an on-sample intensity of <10 W cm^–2. We created synthetic microtubule training data as described before, resulting in around 5,000 patch-pairs of size 128 × 128, and trained a 2D network as described before. Super-resolution images were reconstructed via SRRF¹⁴. Error maps for both SRRF and CARE restoration were computed with SQUIRREL⁴⁸ against the widefield reference frames.

Reliability measures and calibration

To model the inherent (aleatoric) uncertainty of intensity predictions, we adapted the final layers of the network to output a custom probability distribution for every pixel of the restored image, instead of just a scalar value. Specifically, the network predicted the parameters μ and σ of a Laplace distribution,

for intensity value z. To represent the (epistemic) model uncertainty for a specific experiment, we trained an ensemble of M networks (for example, M = 5) and averaged their results (as a mixture model; see ref. ⁶⁹). We validated our probabilistic approach by adapting the concept of a calibrated classifier⁷⁰ to the case of regression, which allows computation of the accuracy/confidence curves and definition of an expected calibration error of a regression model (see Supplementary Note 3). Furthermore, we quantified the normalized per-pixel disagreement of a network ensemble via the average Kullback–Leibler divergence between the individual network distributions and the ensemble mixture distribution. This allowed us to highlight image regions with elevated disagreement scores that may indicate unreliable network predictions (for example, for very challenging low-SNR input; see Fig. 5 and Supplementary Fig. 28). For an extensive and detailed discussion including all derivations, see Supplementary Note 3.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

Training and test data for all experiments presented can be found at https://publications.mpi-cbg.de/publications-sites/7207. The code for network training and prediction (in Python/TensorFlow) is publicly available at https://github.com/CSBDeep/CSBDeep. Furthermore, to make our restoration models readily available, we developed user-friendly FIJI plugins and KNIME workflows (Supplementary Figs. 29 and 30).

The authors thank P. Keller (Janelia) who provided Drosophila data. We thank S. Eaton (MPI-CBG), F. Gruber and R. Piscitello for sharing their expertise in fly imaging and providing fly lines. We thank A. Sönmetz for cell culture work. We thank M. Matejcic (MPI-CBG) for generating and sharing the LAP2b transgenic line Tg(bactin:eGFP-LAP2b). We thank B. Lombardot from the Scientific Computing Facility (MPI-CBG) for technical support. We thank the following Services and Facilities of the MPI-CBG for their support: Computer Department, Light Microscopy Facility and Fish Facility. This work was supported by the German Federal Ministry of Research and Education (BMBF) under the codes 031L0102 (de.NBI) and 031L0044 (Sysbio II) and the Deutsche Forschungsgemeinschaft (DFG) under the code JU 3110/1-1. M.S. was supported by the German Center for Diabetes Research (DZD e.V.). T.B. was supported by an ELBE postdoctoral fellowship and an Add-on Fellowship for Interdisciplinary Life Sciences awarded by the Joachim Herz Stiftung. R.H. and S.C. were supported by the following grants: UK BBSRC (grant nos. BB/M022374/1, BB/P027431/1, and BB/R000697/1), UK MRC (grant no. MR/K015826/1) and Wellcome Trust (grant no. 203276/Z/16/Z).

Affiliations

1. Center for Systems Biology Dresden, Dresden, Germany
   • Martin Weigert
   • , Uwe Schmidt
   • , Tobias Boothe
   • , Alexandr Dibrov
   • , Benjamin Wilhelm
   • , Deborah Schmidt
   • , Coleman Broaddus
   • , Mauricio Rocha-Martins
   • , Loic Royer
   • , Florian Jug
   •  & Eugene W. Myers
2. Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany
   • Martin Weigert
   • , Uwe Schmidt
   • , Tobias Boothe
   • , Alexandr Dibrov
   • , Akanksha Jain
   • , Coleman Broaddus
   • , Mauricio Rocha-Martins
   • , Fabián Segovia-Miranda
   • , Caren Norden
   • , Marino Zerial
   • , Michele Solimena
   • , Jochen Rink
   • , Pavel Tomancak
   • , Loic Royer
   • , Florian Jug
   •  & Eugene W. Myers
3. Molecular Diabetology, University Hospital and Faculty of Medicine Carl Gustav Carus, TU Dresden, Dresden, Germany
   • Andreas Müller
   •  & Michele Solimena
4. Paul Langerhans Institute Dresden of the Helmholtz Center Munich at the University Hospital Carl Gustav Carus and Faculty of Medicine of the TU Dresden, Dresden, Germany
   • Andreas Müller
   •  & Michele Solimena
5. German Center for Diabetes Research (DZD e.V.), Neuherberg, Germany
   • Andreas Müller
   •  & Michele Solimena
6. University of Konstanz, Konstanz, Germany
   • Benjamin Wilhelm
7. MRC Laboratory for Molecular Cell Biology, University College London, London, UK
   • Siân Culley
   •  & Ricardo Henriques
8. The Francis Crick Institute, London, UK
   • Siân Culley
   •  & Ricardo Henriques
9. CZ Biohub, San Francisco, CA, USA
   • Loic Royer
10. Department of Computer Science, Technical University Dresden, Dresden, Germany
    • Eugene W. Myers

Contributions

F.J. and E.W.M. shared last-authorship. M.W. and L.R. initiated the research. M.W. and U.S. designed and implemented the training and validation methods. U.S., M.W., and F.J. designed and implemented the uncertainty readouts. T.B., A.M., A.D., S.C., F.S.M., R.H., M.R.M., and A.J. collected experimental data. A.D., C.B., and F.J. performed cell segmentation analysis. T.B. performed analysis on flatworm data. U.S. and M.W. designed and developed the Python package. F.J., B.W., and D.S. designed and developed the FIJI and KNIME integration. E.W.M. supervised the project. F.J., M.W., P.T., L.R., U.S., and E.W.M wrote the manuscript, with input from all authors.

Corresponding authors

Correspondence to Martin Weigert or Loic Royer or Florian Jug.

Competing interests

The authors declare no competing interests.

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Reprints and Permissions