Precision of manual two-dimensional segmentations of lung and liver metastases and its impact on tumour response assessment using RECIST 1.1

Tumour progression and response to treatment are currently evaluated according to response criteria based on morphologic imaging such as those firstly proposed by the World Health Organization (WHO) or by the more widely used Response evaluation criteria in solid tumours (RECIST) [1, 2]. Developed to simplify the assessment of tumour response, these two evaluation systems are based on the measurement of a given tumour along the greatest axes, corresponding to an assessment of anatomical tumour burden and changes in the measure over time, with the ultimate goal of categorizing adequate tumour response [3].

RECIST has been demonstrated to be useful in clinical trials where objective response was the primary study endpoint as well as in trials where assessment of stable disease, tumour progression or time-to-progression analyses were undertaken [2, 4]. In RECIST, measurable disease is defined by the presence of at least one measurable lesion [5]. Target tumours should be selected on the basis of their size and be representative of all involved organs, but in addition should be those that lend themselves to reproducible repeated measurements. Thus, it is possible to define a partial response (PR), corresponding to at least a 30% decrease in the sum of diameters of target tumours, taking as reference the baseline sum diameters [2]. Progressive disease (PD) is an increase of at least 20% in the sum of diameters of target tumours, taking as reference the smallest sum.

However, RECIST has significant limitations in terms of variability and reproducibility, which may not be independent [6–9]. In practice, the maximal size mensuration or segmentation (in two or three dimensions) are performed manually and concerns remain about the accuracy of such segmentation as a result of interobserver and intraobserver variability [10]. A recent meta-analysis has shown that interobserver relative measurement difference in measuring single tumour burden and calculating the interval change may exceed the 20% cut-off for progression [11]. However, variability decreased when tumour burden was measured by a single observer or assessed by the sum of multiple tumours [11].

The aim of our study was: firstly to evaluate the precision of manual two-dimensional (2D) segmentations depending on organ, reader experience, and segmentation method; and secondly to quantify the uncertainty in tumour response assessment (PR, PD or stable disease) depending on the segmentation precision.

This retrospective study was approved by the institutional research ethics board. The requirement for patient informed consent was waived. The authors had full control of the data and the information submitted for publication.

Among the 520 segmentations performed, only 2.5% of segmented ROIs were removed due to consistent errors of segmentation. No significant difference between radiologists and scientists was observed. Moreover, while considerable interobserver and intraobserver variability has been reported thus far for radiological tumour response evaluation according to RECIST and WHO criteria [11, 14], inter-observer agreements were excellent (ICC >0.90) for diameter and area assessment in both organs.

These observations may be related to the method used in this study. While not representative of typical radiologic practice, the maximum diameter was calculated from an evaluation of the perimeter of the tumour. This analysis was performed to simplify the experimental design and limit the bias. Moreover, for patients with multiple tumours, an approximate tumour location was given by a range of slices where the tumour could be located. While this method appeared effective and confirmed the call for computer-aided detection software for tumour response assessment [15], it remains uncertain how the results are generalizable in clinical practice. Further evaluations are now mandatory.

Based on the regression models of SD performed in this study, the level of uncertainty increased with tumour size in the liver while it decreased in the lung. In liver, therefore, 2D segmentation findings have to be carefully interpreted due to these increasing 95% CIs. A potential impact on tumour response assessment may be observed either for area or for diameter. For area, 95% CIs systematically overlapped. This finding suggests the limited interest of area calculation for therapeutic assessment. For diameter, cut-off values were identified at the intersection of these overlaps, above which it may be difficult to assess confidently therapeutic response. For tumours above these thresholds the impression of progression or partial response may only be related to the uncertainty of the measures. These size limits have to be taken into account in further evaluations of RECIST [2, 16]. This justifies the development of alternatives for liver, such as the recently proposed modified RECIST (mRECIST) [7] or the introduction of functional imaging in the current evaluation of liver metastases after treatment [17, 18]. In the lung, greater uncertainty was observed for small tumours. This finding is consistent with the introduction of a minimum lesion size in RECIST of 10 mm in the lung, which improved reproducibility between WHO and RECIST [19].

This study showed that there were no significant differences in terms of uncertainty between segmentations made by a group of radiologists aware of RECIST and those of a group of scientists with only basic knowledge of RECIST. The SD remained similar for both groups. Moreover, no significant differences were observed between the groups, or after combining the groups, when comparing mensuration obtained on an imposed slice or after the selection of the slice. Therefore, 2D segmentation, even manual, seems not to be affected by a slight variation in slice selection. These findings justify the RECIST 1.1 recommendations to perform mensuration using the same plane of evaluation with the maximum diameter of each target lesion always being measured at subsequent follow-up time points, even if this results in measuring the lesion at a different slice level or in a different orientation or vector compared with the baseline study [2, 16].

This study has some limitations. The series is retrospective and may have selection bias. The number of tumours evaluated is limited and tumours were chosen by two independent investigators, which may have caused selection bias. No comparison of the findings was performed with the results of a single observer or after summing multiple tumours [11]. The segmentation was performed manually but the diameters were extracted automatically. Further studies may compare the results of manual versus automatic segmentations [20, 21]. Volumetric assessment of the entire tumour has not been performed, as recently proposed [22]. However, volumetric assessment and RECIST have been shown not to be interchangeable, neither technique demonstrating clinical superiority [23, 24].

To summarize, the results of our study highlight the concerns remaining for manual segmentation, although accuracy of manual 2D segmentation does not appear to be limited by the experience of operator. For liver but not for lung metastases, segmentation in 2D for response assessment remains uncertain for large tumours. We established thresholds above which the impression of tumour progression or response may be related only to the uncertainty of 2D segmentation. While a prospective validation of these findings on a larger scale is now needed before drawing definitive conclusions regarding their true impact from a clinical perspective, these results could be easily incorporated in daily clinical practice. Moreover, it may justify the development of alternative quantitative assessment of tumour response using multiparametric or functional imaging tools.