Elsevier

Medical Image Analysis

Volume 12, Issue 6, December 2008, Pages 764-777
Medical Image Analysis

Multiple object tracking in molecular bioimaging by Rao-Blackwellized marginal particle filtering

https://doi.org/10.1016/j.media.2008.03.004Get rights and content

Abstract

Time-lapse fluorescence microscopy imaging has rapidly evolved in the past decade and has opened new avenues for studying intracellular processes in vivo. Such studies generate vast amounts of noisy image data that cannot be analyzed efficiently and reliably by means of manual processing. Many popular tracking techniques exist but often fail to yield satisfactory results in the case of high object densities, high noise levels, and complex motion patterns. Probabilistic tracking algorithms, based on Bayesian estimation, have recently been shown to offer several improvements over classical approaches, by better integration of spatial and temporal information, and the possibility to more effectively incorporate prior knowledge about object dynamics and image formation. In this paper, we extend our previous work in this area and propose an improved, fully automated particle filtering algorithm for the tracking of many subresolution objects in fluorescence microscopy image sequences. It involves a new track management procedure and allows the use of multiple dynamics models. The accuracy and reliability of the algorithm are further improved by applying marginalization concepts. Experiments on synthetic as well as real image data from three different biological applications clearly demonstrate the superiority of the algorithm compared to previous particle filtering solutions.

Introduction

Advances in imaging technology for studying molecular processes in living cells continue to encourage biologists to conduct more and more challenging experiments and to collect large amounts of image data. Fluorescent labeling combined with time-lapse microscopy imaging enables visualizing the dynamic behavior of virtually any intracellular structure at high spatial and temporal resolution (Tsien, 2003, Gerlich and Ellenberg, 2003, Vonesch et al., 2006) (see Fig. 1 for example images). Quantitative analyses of this behavior requires the detection and tracking of large and time-varying numbers of nanoscale objects in the image sequences. Existing software tools (commercial and freeware) for this purpose are often not robust enough to yield satisfactory results when facing poor imaging conditions (very low-signal and high-noise levels are common in live cell imaging to minimize photodamage) and large numbers of objects with complex motion patterns (objects may interact or exhibit different modes of motion at different times). As a result, such analyses are still largely performed manually, by expert human observers. This is extremely labor intensive and very likely leads to user bias. Also, as only a part of the data can be analyzed this way, it may lead to the loss of important information. Therefore, the development of reliable automated algorithms, which allow the tracking of all individual objects moving along variable and unpredictable trajectories, constitutes an important first step in improving our understanding of the mechanisms controlling intracellular processes (Thomann et al., 2002, Gerlich et al., 2003, Genovesio et al., 2006, Meijering et al., 2006).

The majority of approaches that have been proposed so far for tracking small objects in bioimaging applications consist of two stages. In the first stage, objects are detected separately in each frame of the image sequence, and in the subsequent second stage, an attempt is made to solve the interframe correspondence problem in linking detected objects between frames. Since the two stages are usually completely separated, without the possibility of feedback from linking to detection and vice versa, the tracking performance of such approaches is often suboptimal and extremely sensitive to failures in either stage. Moreover, as most of these approaches are designed to be generally applicable, they are often based on rudimentary detection algorithms (thresholding or template matching) and linking strategies (nearest neighbor or smooth motion). Recently, several popular approaches were quantitatively evaluated, and were found to break down below signal-to-noise ratios (SNRs) around 4–5 (Cheezum et al., 2001, Carter et al., 2005), which are not uncommon in practice. More integrated, spatiotemporal segmentation approaches have also been proposed (Sage et al., 2005, Bonneau et al., 2005), but current implementations of this idea have been demonstrated to work well only for single or very limited numbers of well-separated objects. More robust tracking approaches that can deal with larger numbers of objects have been developed for tracking of migrating cells using phase-contrast video microscopy (Li et al., 2006, Debeir et al., 2005). The robustness is achieved by exploiting the cell shape/appearance information, which cannot be utilized to the same extent for tracking of subresolution objects in fluorescence microscopy.

Most recently, probabilistic tracking approaches have been developed (Genovesio et al., 2006, Smal et al., 2007a, Smal et al., 2007b), which overcome the shortcomings of previous approaches by improved interaction between object detection and linking, and the possibility to more effectively incorporate prior knowledge about object dynamics and image formation. For example, for the tracking of growing microtubule plus-ends, whose dynamic behavior can be described accurately by a nearly constant velocity model, we have shown previously (Smal et al., 2007b) that a Bayesian estimation approach, in our case implemented by a sequential Monte Carlo (SMC) technique known as particle filtering (PF), makes better use of all available spatiotemporal information, yielding more accurate and more consistent tracking results (for more information about the success of the PF approach in other applications, and especially for tracking of multiple interacting objects, we refer to Doucet et al., 2001, Vermaak et al., 2003, Khan et al., 2005). However, that approach required a great deal of tailoring to the specific motion type to be analyzed, and was not able to directly deal with multiple motion types concurrently, nor with switching between them. It has also been shown (Genovesio et al., 2006) that an interacting multiple models (IMM) filter, which is capable of self-adapting to different motion types as well as to switching between them, can achieve more reliable tracking results than a Kalman filter (KF) using only one of the dynamics models. However, that approach did not optimally exploit all available spatiotemporal data, as the detection was implemented as a separate stage, completely decoupled from the linking stage.

In this special issue paper, we extend our previous work on the topic, and present an improved, fully automated algorithm for the tracking of many subresolution objects in time-lapse fluorescence microscopy images. Specifically, we take the successful particle filtering framework (Smal et al., 2007a) as a starting point and propose five fundamental changes that make the algorithm more flexible, more robust, and more accurate. First, instead of using a single, dedicated dynamics model, multiple models are incorporated to be able to use the algorithm for different biological applications without the need for careful fine-tuning to each application. Second, a new detection scheme is integrated into the tracking framework, which is based on mean-shift clustering and performs better than the previously described classification approach. Third, a new likelihood evaluation strategy is proposed, which does not require the previously described “hierarchical searching” and reduces the computational cost. Fourth, we propose marginalization of the previously described filter, which increases the accuracy by reducing the variance of the track estimations. Finally, Rao-Blackwellization is applied to one of the state variables, which further improves the accuracy and reduces the computational cost, as it allows an analytical solution in the form of a Kalman filter. In addition to these methodological improvements, we extend our previous work by exploring two new biological applications, which could not be analyzed by our original algorithm (Smal et al., in press) without careful tuning to each of these specific applications. By contrast, the algorithm proposed here can handle all of these applications without changing the parameter settings, as it naturally handles multiple and changing motion patterns.

The paper is organized as follows. First, in Section 2, we recap the main ingredients of the particle filtering framework for multiple object tracking, and propose multiple dynamics models and a novel track management strategy. The subsequent two sections focus on the main novelties of the tracking approach compared to our previous work. In Section 3, we explain how multiple dynamics models can be conveniently incorporated into the particle filtering framework. Next, in Section 4, we show how to apply marginalization concepts to improve the performance of the framework. An overview of the algorithm and its parameters is given in Section 5. The results of experiments on synthetic as well as on real image data from three different biological applications are presented and discussed in Section 6. The evaluation includes a comparison with our previous algorithm (Smal et al., in press) and with manual tracking, confirming the theoretically claimed improvements. Finally, in Section 7, we summarize the main findings of the present work.

Section snippets

Probabilistic tracking framework

The tracking approach proposed in this paper is based on the principle of Bayesian estimation. In this section we first recap the Bayesian estimation framework and its implementation by means of particle filtering. Then we discuss two different ways of extending the framework to allow tracking of multiple objects. This is followed by a presentation of the dynamics and observation models that we propose for the biological imaging applications considered in this paper. Finally we explain how we

Incorporating multiple dynamics

The objects of interest in our applications exhibit quite different and complicated motion patterns that cannot be accommodated by the transition prior p(xtxt−1) in a simple form. For accurate estimation and robust tracking, it is better to model each of the (sub)patterns by a separate transition prior, as described in Section 2.3. There are, however, no straightforward solutions to incorporating multiple dynamics models into the PF framework. In order to deal with different motion patterns,

Applying marginalization concepts

In the previous sections, we have presented the general PF framework and the specific choices that we have made to tailor this framework to the problem of detecting and tracking multiple nanoscale objects exhibiting complex dynamics in biological imaging applications. Here we propose to further improve the framework by marginalization of the filtering distribution, data-dependent importance sampling, and Rao-Blackwellization. In the sequel, when we speak of the standard PF approach, we mean the

Algorithm overview

Having described all aspects of the proposed tracking approach in the previous sections, we now give a step-by-step overview of our algorithm, which also summarizes the parameters involved. Apart from parameter setting (steps 1 and 2), which needs to be done by the user depending on the applications, the algorithm is fully automatic.

  • 1.

    Given image sequences from time-lapse microscopy imaging experiments, specify prior knowledge about object features (Section 2.3): σmax and σmin (shape), Vmin and V

Experimental results

The proposed algorithm was thoroughly tested using synthetic image data, for which ground truth was available, as well as real biological image data from several time-lapse microscopy studies. In both cases, the dynamics of three different types of intracellular objects were considered, which are representative of the dynamics encountered in practice.

Conclusions

In this paper we have proposed a novel algorithm for simultaneous tracking of many nanoscale objects in time-lapse fluorescence microscopy image data sets. The algorithm, which is built within a Bayesian tracking framework, shows several important improvements compared to our previous work (Smal et al., 2007b, Smal et al., in press). Tracking accuracy is improved by using marginalization of the filtering distribution and one of the state variables, for which the optimal solution (the Kalman

References (47)

  • Y. Boers et al.

    A particle-filter-based detection scheme

    IEEE Signal. Proc. Lett.

    (2003)
  • S. Bonneau et al.

    Single quantum dot tracking based on perceptual grouping using minimal paths in a spatiotemporal volume

    IEEE Trans. Image Process.

    (2005)
  • B.C. Carter et al.

    Tracking single particles: a user-friendly quantitative evaluation

    Phys. Biol.

    (2005)
  • D. Comaniciu et al.

    Mean shift: a robust approach toward feature space analysis

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2002)
  • O. Debeir et al.

    Tracking of migrating cells under phase-contrast video microscopy with combined mean-shift processes

    IEEE Trans. Med. Imaging

    (2005)
  • A. Desai et al.

    Microtubule polymerization dynamics

    Annu. Rev. Cell Dev. Biol.

    (1997)
  • A. Doucet et al.

    Sequential Monte Carlo Methods in Practice

    (2001)
  • A. Doucet et al.

    On sequential Monte Carlo sampling methods for Bayesian filtering

    Stat. Comput.

    (2000)
  • Driessen, H., Boers, Y., 2004. An efficient particle filter for jump Markov nonlinear systems. In: IEE Target Tracking:...
  • P. Farla et al.

    Antiandrogens prevent stable DNA-binding of the androgen receptor

    J. Cell Sci.

    (2005)
  • H. Flyvbjerg et al.

    Microtubule dynamics: caps, catastrophes, and coupled hydrolysis

    Phys. Rev. E

    (1996)
  • A. Genovesio et al.

    Multiple particle tracking in 3-D+t microscopy: Method and application to the tracking of endocytosed quantum dots

    IEEE Trans. Image Process.

    (2006)
  • D. Gerlich et al.

    4D imaging to assay complex dynamics in live specimens

    Nature Cell Biol.

    (2003)
  • Cited by (78)

    • Biomolecule trafficking and network tomography-based simulations

      2022, Biomedical Image Synthesis and Simulation: Methods and Applications
    • Computerized cell tracking: Current methods, tools and challenges

      2021, Visual Informatics
      Citation Excerpt :

      In most recent particle filtering methods, human communication or other sort of initialization is essential to explain the prior explanation of object states in the initial frame. Moreover, it is necessary that the deformation of the tracked object remains small, because large deformations preclude their application to individual cell tracking (Nketia et al., 2017; Masuzzo et al., 2016; Smal et al., 2008b). Filtering tracking approaches can be categorized into deterministic and statistical methods (Meunier et al., 2010):

    • High density cell tracking with accurate centroid detections and active area-based tracklet clustering

      2018, Neurocomputing
      Citation Excerpt :

      So many algorithms in this category need post-processing to correct tracking errors or manual initializations. One typical algorithm is particle filtering method within Bayesian framework proposed in [22], which is able to track multiple cells. In [23], the authors proposed to combine Kalman filtering with the modified mean shift algorithm to predict the position and shift in current frame.

    View all citing articles on Scopus
    View full text