Consensus Expert Committees

The most common method to assess medical image quality is the use of Receiver Operating Characteristic (ROC) studies. In these studies doctors rate their confidence about the presence of a given abnormaility in a set of patient images.  The ROC curve plots the hit rate as a function of the false alarm rate.  From the ROC curve a bias-free measure of performance known as the area under the curve (AOC) is derived. Many  ROC studies rely on establishing “truth” (the gold standard) about lesion absence/presence on the agreement of a panel of experts (consensus expert committees).  In addition, in the consensus committee methodology, images where the members of the committee did not reach any agreement about the lesion absence/presence are discarded from the ROC study. But how reliable are “gold standards” established by these expert committees?  And does discarding images where no agreement was reached bias the spectrum of difficulty of the test image set for the ROC study?  Computer simulated lesions (filling defects) of different strengths (signal contrasts) were embedded in real x-ray coronary angiogram backgrounds in order to measure the agreement among the decisions of members of the committee as a function of signal strength, to establish the accuracy of the decisions of the consensus expert committee and to compare it to more inexperienced readers.

Publications:

Eckstein, M.P., Wickens, T.D., Aharonov G., Ruan G., Morioka C.A., Whiting, J.S., Quantifying the limitations of the use of consensus expert committees in ROC studies, Proceedings SPIE Image Perception, 3340, 128-134, (1998)