Standardized evaluation methodology and reference database for evaluating IVUS image segmentation

Abstract

This paper describes an evaluation framework that allows a standardized and quantitative comparison of IVUS lumen and media segmentation algorithms. This framework has been introduced at the MICCAI 2011 Computing and Visualization for (Intra)Vascular Imaging (CVII) workshop, comparing the results of eight teams that participated.

We describe the available data-base comprising of multi-center, multi-vendor and multi-frequency IVUS datasets, their acquisition, the creation of the reference standard and the evaluation measures. The approaches address segmentation of the lumen, the media, or both borders; semi- or fully-automatic operation; and 2-D vs. 3-D methodology. Three performance measures for quantitative analysis have been proposed. The results of the evaluation indicate that segmentation of the vessel lumen and media is possible with an accuracy that is comparable to manual annotation when semi-automatic methods are used, as well as encouraging results can be obtained also in case of fully-automatic segmentation. The analysis performed in this paper also highlights the challenges in IVUS segmentation that remains to be solved.

Introduction

Cardiovascular diseases account for 30% of all deaths worldwide [1]. Atherosclerosis, a disease of the vessel wall, is the major cause of cardiovascular diseases such as heart attack or stroke [2].

Early atherosclerosis results in remodeling, thus retaining the lumen despite plaque accumulation [3]. In later stages, luminal narrowing occurs, either diffuse over a vessel segment [4] or as localized stenosis. Only the latter case is usually directly visible using X-ray angiography. Intravascular Ultrasound (IVUS) allows monitoring and quantifying the state of the vessel wall and lumen. In addition, IVUS is an intra-operative imaging tool for the quantification and characterization of coronary plaque, used for diagnostic purposes and for guiding Percutaneous Coronary Intervention (PCI), enabling the visualization of high resolution images of internal vascular structures. Finally, IVUS is a fundamental tool for stent deployment because it allows assessment of the intervention and the correct placement of the device.

The acquisition of an IVUS sequence consists of inserting an ultrasound emitter, carried by a catheter, into the arterial vessel and pulling the probe from the distal to the proximal position (pullback). The standard IVUS image is a 360-degree tomographic cross-sectional view of the vessel walls, denoted as short-axis view (Fig. 1a), which allows an accurate assessment of vessel morphology. Given an angular position on the short-axis view (indicated in Fig. 1b by a dotted line), the correspondinglongitudinal view can be generated by considering the gray-level values of the sequence along the diameter at the chosen angle. This longitudinal image depicts the morphology of the vessel section according to the selected orientation. Compared with other angiographic imaging modalities, (e.g. X-ray, MRA and CT), IVUS enables the visualization of both vessel morphology and plaque, and it provides extremely high image resolution (up to 113 μm). Such characteristics are essential for the clinical diagnosis since to date it is the only modality enabling the accurate morphological segmentation of both vessel membranes (lumen and media) and the assessment of the plaque type in vivo [5], [6]. Additionally, when IVUS is fused with X-ray projections, three dimensional plaque quantification and reconstruction can be obtained [7], [8] (Fig. 2).

Since a typical pullback contains more than 3000 IVUS frames, an accurate (semi-)automatic assessment of lumen and media contours is highly desirable to reduce the workload of the physician, and to speed up the diagnosis. In particular, the manual evaluation needs to be performed by expert physicians because the images are affected by speckle noise, and the textural appearance may significantly vary according to the echograph brand and by the type of probe used in the acquisition.

Manual lumen and media segmentation is a laborious task that suffers from inter- and intra-observer variabilities due to the high amount of noise and artifacts present on the IVUS images. Consequently, much work has been performed on (semi-)automated IVUS image processing as illustrated by Katouzian in a recent review [9]. After a brief summary of the most recent lumen and media segmentation algorithms, we present a comparison of several segmentation algorithms performed using the proposed evaluation framework. It has to be noted that our state of the art description partially overlaps with the one presented in [9]; hence, the reader interested in an exhaustive review is referred to [9].

Automated lumen segmentation of IVUS sequences has been a topic of interest since the early 1990s. Many of the early approaches were based on the use of local properties of the image such as pixel intensity and gradient information (edges) combined with computational methods including graph search [10], [11], active surfaces [12], active contours [13], and neural networks [14]. In later approaches, segmentation was accomplished by the use of gray level variances to model ultrasound speckle [15], contrast of regions [16], statistical properties of the image [17], [18], spatio-temporal information (3D segmentation) [19], and discrete wavelet decomposition [20]. Recently, a shape-driven method for lumen and media-adventitia segmentation was introduced by Unal et al. [21]. In this work, the lumen and media-adventitia contours were constrained to a smooth, closed geometry. Then, a shape space was built using training data and principal component analysis (PCA). Finally, segmentation was performed on this shape space by minimizing an energy function using nonparametric probability densities with global measurements. Taki et al. [22] proposed a method for the identification of the vessel borders. This method consisted of a preprocessing step followed by the geometric deformation of parametric models using edge information. Downe et al. [23] introduced a method where principal component analysis was used for pre-processing, while active contour models were used to provide an initial segmentation for a 3D graph search method. Multilevel discrete wavelet frames decomposition was used by Papadogiorgaki et al. [24] to generate texture information that was used along with the intensity information for contour initialization.

Similarly, Katouzian et al. [25] presented a method where texture information was extracted using a discrete wavelet packet transform. Then, pixels were classified as lumen or non-lumen using k-means clustering. Finally, the contour was parameterized using a spline curve. Mendizabal-Ruiz et al. [26] presented a probabilistic segmentation method based on the minimization of a cost function which deformed a contour parameterized represented by a one dimensional periodic function. In this method, the likelihood of each pixel to belong to lumen are computed using samples of the regions of interest on a number of frames of the sequence to be segmented. This method is capable of segmenting the lumen employing either the B-mode reconstruction images or the radio frequency (RF) IVUS data [27]. Ciompi et al. [28] presented a method in which segmentation was tackled as a classification problem and solved using an error correcting output code technique. In that work, contextual information was exploited by means of conditional random fields computed from training data. Wennogle and Hoff [29] proposed improvements over the method presented in [19] which include a preprocessing step to remove motion artifacts, a new directional gradient velocity term, and a post-processing level-set method. Roy Cardinal et al. [30] presented a multiple interface 3D fast-marching method that was based on a combination of gray level probability density functions and the intensity gradient. The segmentation method included an interactive initialization procedure of the external vessel wall border. Moraes et al. [31] proposed a method on which preprocessing and feature extraction over the IVUS images is performed, and then binary morphological object reconstruction is used to find the contours. Zhua et al. [32] presented a method based on the use of a linear-filtered gradient vector flow which drives the deformation of a balloon snake. Sun and Liu [33] presented a two step method which first detects the contours of interest on a number of L-mode cuts of the sequence and then evolve contours on the B-mode images until they reach the target points given by the first step. Finally, Balocco et al. [34] proposed an approach to automatically segment the vessel lumen, which combines model-based temporal information extracted from successive frames of the sequence, with spatial classification using the Growcut algorithm [35].

As for the lumen border detection, several techniques have been proposed for the automatic and semi-automatic detection of the media border in IVUS. Most of the approaches rely on the idea that the most useful information for the media assessment are: (1) the local appearance of the vessel in proximity of the media layer and (2) the vessel shape.

Regarding the hypothesis on tissue appearance (1), the most exploited assumption relies on the echogenicity of media and adventitia tissues. The common pattern describing a gray level transition dark-bright in the media-adventitia interface has been used by several authors as initial approximation of the media. For this purpose, gradient-based operators as well as edge detectors have been used [11], [14], [36], [21], [22], [37], [31], [38], [24]. In some approaches some kind of user interaction is required [39], [11], [14]. Sonka et al. [11] proposed a method that uses knowledge on the local appearance of the media-adventitia interface while guiding a graph search. Papadogiorgaki et al. [24]employ an algorithm where the media-adventitia border is initialized based on gray level intensity. Additionally, techniques of median filtering [36], [21], despeckle [22], [37], [40] or edge detection by means of the Canny operator [22], [38] are used during border initialization.

The second common hypothesis (2) for media definition relies on the smoothness and continuity of the vessel shape. To this aim, given an initial approximation of the media, several approaches use a deformable model to compute the final vessel border [41], [14], [18], [36], [42], [22], [37], [19], [43], [12], [30]. The evolution of the model is commonly guided by an energy function, embedding information on the vessel morphology, gray level distribution [19], [30], edge information by gradient [36], [38], [21], [30], image intensity [42] and image contrast [41].

In some approaches, the information on the vessel shape and tissue properties is obtained by means of a learning process rather than using local per-image observation. Unal et al. [21] propose a strategy where the shape of the media is learned from training examples to create a shape space that is used to find the borders. Gil et al. [18] present an approach based on a learning task applied to tissue characteristics by defining the class calcification and vessel border and discriminated by a Fisher classifier. Mojsilovic et al. [44] propose an algorithm that describe plaque, lumen and adventitia regions by means of two textural features, and then classify them in an unsupervised fashion. An unsupervised classification is also proposed by Moraes and Furuie [40], where the Otsu thresholding [45] is applied to wavelet-based features to separate the adventitia from the plaque region.

Given the implicit ambiguity of the media appearance, the idea of reproducing the human reasoning while detecting the media has been exploited as well. Bovenkamp et al. [46] aim at encoding the relationships between the parts constituting the vessel morphology by means of a set of agents. Using a similar idea, a two-steps approach was presented by Olszewski et al. [47]. Regions that are most likely to belong to the media are first described at low resolution by means of an operator learned by training examples. Then, dynamic programming defines the final border at a higher resolution. In the algorithm of Mojsilovic et al. [44], a priori assumptions on vessel morphology are used to guide the media definition while refining classification results by morphological filters. Papagodiorgaki et al. [24] present a method that exploits only the relationship between lumen and media-adventitia border during initialization.

The ad hoc detection of regions hindering the vessel continuity, as calcifications or side-branches (bifurcations) has also been exploited. The detection of such artifacts allows, in some cases, to avoid an inaccurate border detection where the vessel appearance is partially occluded [21]. Plissiti et al. [14] makes use of a neural network based on a priori knowledge about the geometry of the vessel to overcome the effect due to calcifications. Dijkstra et al. [48], take profit of the presence of a stent manually detected by the user in order to define the lumen border within a framework that detects both lumen area and media in IVUS. Taki et al. [22], make use of a Bayesian classifier and a thresholding procedure on grey level to detect calcifications. Finally, in some approaches, the extension to 3D computation is proposed, or even the combination of cross-sectional and long-axis view [48], in order to improve the media detection [12], [19].

As explicitly indicated in the review paper [9], direct performance comparisons of different approaches is difficult due to the lack of standard databases and validation criteria for most vascular segmentation applications. Moreover, many of the reviewed algorithms are not publicly available; this fact hampers an objective and fair comparison by third parties.

Nowadays, in medical imaging and computer vision communities, there is a growing number of initiatives that set up a publicly available data repository and standardized evaluation framework. This demonstrates an increasing interest in standardized evaluation and the possibility to compare methods to each other. In this section, we provide a few examples and we briefly discuss some published vascular segmentation algorithms. A standardized evaluation framework for carotid bifurcation lumen segmentation and stenosis grading, and for evaluating coronary artery centerline has been recently published [49], [50]. Several workshops involving the setup of an evaluation framework in the field of medical imaging, have been organized at the MICCAI, ISBI and SPIE conferences. Reports on some of these frameworks have appeared recently [49], [51], [52], [53]. More initiatives can be found at the website http://www.grand-challenge.org/.

The evaluation of lumen and media segmentation in IVUS has been introduced at the Computing and Visualization for (Intra)Vascular Imaging (CVII) workshop during the MICCAI 2011 conference held in Toronto [54], [55], and this paper is an extended version of the challenge result evaluation presented at the workshop. Researchers interested in comparing the performance of new IVUS segmentation algorithms are encouraged to register on the site [56].

Finally the goals of this paper are (i) to suggest algorithms and databases for benchmarking, (ii) to propose three performance measures for quantitative analysis and (iii) to highlight the current challenges in IVUS segmentation.