Effect of picture archiving and communication system image manipulation on the agreement of chest radiograph interpretation in the neonatal intensive care unit
Denise A Castro1, Asad Ahmed Naqvi2, Elizabeth Vandenkerkhof3, Michael P Flavin4, David Manson1, Donald Soboleski2
1 Department of Diagnostic Imaging, University of Toronto, Toronto, Canada
2 Department of Diagnostic Radiology, Queen's University, Kingston, Ontario, Canada
3 Department of Anesthesiology and Perioperative Medicine, School of Nursing, Queen's University, Kingston, Ontario, Canada
4 Department of Pediatrics and Diagnostic Radiology, Queen's University, Kingston, Ontario, Canada
|Date of Submission||10-Mar-2016|
|Date of Acceptance||22-Apr-2016|
|Date of Web Publication||19-May-2016|
Department of Diagnostic Radiology, Queen's University, Kingston, ON K7L 2V7
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Objective: Variability in image interpretation has been attributed to differences in the interpreters' knowledge base, experience level, and access to the clinical scenario. Picture archiving and communication system (PACS) has allowed the user to manipulate the images while developing their impression of the radiograph. The aim of this study was to determine the agreement of chest radiograph (CXR) impressions among radiologists and neonatologists and help determine the effect of image manipulation with PACS on report impression. Materials and Methods: Prospective cohort study included 60 patients from the Neonatal Intensive Care Unit undergoing CXRs. Three radiologists and three neonatologists reviewed two consecutive frontal CXRs of each patient. Each physician was allowed manipulation of images as needed to provide a decision of "improved," "unchanged," or "disease progression" lung disease for each patient. Each physician repeated the process once more; this time, they were not allowed to individually manipulate the images, but an independent radiologist presets the image brightness and contrast to best optimize the CXR appearance. Percent agreement and opposing reporting views were calculated between all six physicians for each of the two methods (allowing and not allowing image manipulation). Results: One hundred percent agreement in image impression between all six observers was only seen in 5% of cases when allowing image manipulation; 100% agreement was seen in 13% of the cases when there was no manipulation of the images. Conclusion: Agreement in CXR interpretation is poor; the ability to manipulate the images on PACS results in a decrease in agreement in the interpretation of these studies. New methods to standardize image appearance and allow improved comparison with previous studies should be sought to improve clinician agreement in interpretation consistency and advance patient care.
Keywords: Agreement and consistency in interpretation, chest radiograph, manipulation of images, Neonatal Intensive Care Unit, picture archiving and communication system
|How to cite this article:|
Castro DA, Naqvi AA, Vandenkerkhof E, Flavin MP, Manson D, Soboleski D. Effect of picture archiving and communication system image manipulation on the agreement of chest radiograph interpretation in the neonatal intensive care unit. J Clin Imaging Sci 2016;6:19
|How to cite this URL:|
Castro DA, Naqvi AA, Vandenkerkhof E, Flavin MP, Manson D, Soboleski D. Effect of picture archiving and communication system image manipulation on the agreement of chest radiograph interpretation in the neonatal intensive care unit. J Clin Imaging Sci [serial online] 2016 [cited 2017 May 23];6:19. Available from: http://www.clinicalimagingscience.org/text.asp?2016/6/1/19/182730
| Introduction|| |
Chest radiographs (CXRs) are routinely utilized to access changes in status of newborns and infants in the neonatal intensive care setting and often lead to changes in management. , Differences in image interpretation have been attributed to multiple factors including differences in the interpreters' knowledge base, experience level, and access to the clinical scenario. Inconsistent radiograph image quality due to patient movement, exposure setting alterations, and low lung volumes are a few of the many variables limiting assessment, and many researchers have demonstrated significant inter-observer variability in interpretation. ,,,, The advent of picture archiving and communication system (PACS) has allowed the user to manipulate the images by altering the brightness (window level) and/or contrast (window width) of each radiograph while developing their impression. This manipulation will vary the appearance of the radiograph, and in effect, each user will be basing their impression on their own uniquely obtained image. This resulting image will be different from the image that their colleagues will generate to help them come to their impression. The purpose of this study is to determine the agreement of CXR impressions among radiologists and neonatologists and help determine the effect of image manipulation with PACS on reported diagnosis.
| Materials and Methods|| |
The study was approved by the university and affiliated teaching hospitals research ethics board (REB), and patient informed consent was waived by the REB.
This prospective cohort study included 60 nonconsecutive patients from the Neonatal Intensive Care Unit (NICU) (age range: 1 day to 3 months; 34 females, 26 males; gestational age range: 26-32 weeks) who had undergone CXRs as part of their routine care during a 3-month period in 2011. Patients included in the study had CXRs performed on variable days during their stay in the NICU. All patients had an underlying history of surfactant deficiency disease with follow-up radiographs ordered due to a concern of disease progression respiratory status. Exclusion criteria included infants who did not undergo two CXRs during their NICU stay. Images in which the CXR was combined with the abdomen were excluded from the study as well. The radiographs were obtained using 60 kVp and 1.5 mAs, the standard technique in the department.
The sixty sets of frontal CXRs, each consisting of one previous and one recent examination, totaling 120 CXRs, were randomly placed as acquired into two identical viewers on PACS (GE Centricity 3.2, GE Healthcare IT, Barrington, Illinois, USA) used in the imaging department. The normally scanned requisition for the study was not included and each set of radiographs was sequentially numbered from1A/B to 60A/B. The previous or earlier study was placed to the right of the most recent study as per the department's usual routine. Three radiologists with a median length of expertise of 14 years (range: 5-25 years) along with three Neonatal Intensive Care Unit (NICU) physicians with a median length of expertise of 16 years (range: 2-25 years), each reviewed all 60 sets of frontal chest images in each of the two viewers. Each interpreter was instructed to provide a solitary impression of overall "improved," "no change," or "disease progression" lung disease for each of the 60 cases in each viewer. Each interpreter reviewed the 60 sets of cases in one viewer in their usual manner with manual windowing/manipulation as judged needed. Each interpreter (neonatologist or radiologist) reviewed the 60 sets of cases within the second viewer after an independent person (a radiologist) performed image windowing and contrast adjustment to best optimize the CXR appearance and to best match appearance with the previous study performed on that patient. The interpreters were not allowed to independently manipulate the appearance of the CXRs in this second viewer to come to their impression of whether the recent examination was improved, disease progression, or unchanged. Three interpreters began their review of the radiographs in the first viewer and the other three interpreters began with the second viewer. The time span between full review of the radiographs in one viewer and the completion of assessment of the radiographs in the other viewer ranged from 1 to 12 days.
Percent agreement was calculated between all six observers and separately for radiologists and neonatologists, for each of the two methods used in determining their impressions (allowing image manipulation/windowing and not allowing image manipulation/windowing). Similarly, the percent reporting opposing views, i. e., improved versus disease progression and vice versa, was calculated. Statistical significance was assessed using Chi-square test; the magnitude of the difference in agreement and opposition between manipulated and nonmanipulated impressions was assessed using relative risk (RR) and 95% confidence intervals (CI). The within observer kappa and the percent agreement, opposing, and unchanged impressions between manipulated and nonmanipulated methods were calculated. Statistical significance was set at <0.05. SPSS version 21 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.) and Microsoft Excel were used to conduct the analysis.
| Results|| |
Overall, there was more agreement in CXR interpretations when images could not be manipulated. One hundred percent agreement in image impression (whether thought to be improved, disease progression or unchanged) between all six observers (three radiologists, three neonatologists) was only seen in 3 of the 60 cases (5%) when allowing image manipulation while 100% agreement was seen in 8 of the 60 cases (13%) when there was no manipulation of the images allowed (Chi-square = 2.5, P = 0.11) [Table 1]. Hence, the clinicians were 2.9 times more likely (95% CI 0.7, 14.8) to agree on image impressions when no manipulation of the images was permitted. There was 100% agreement between all three radiologists in 21 of the 60 cases (35%) when there was independent manipulation and 25 of the 60 cases (42%) without image manipulation (RR = 1.3, 95% CI 0.6-3.0). There was 100% agreement between all three NICU physicians in 8/60 (13%) of cases with independent manipulation and 18/60 (30%) of the cases when no manipulation was allowed (RR = 2.8, 95% CI 1.0-7.8). Overall, there were fewer opposing radiograph impressions when image manipulation was not allowed as compared to when manipulation/windowing was permitted [Table 1]. There were opposing image impressions (improved versus disease progression, vice versa) between all six interpreters in 37/60 (62%) of manipulated images and 29/60 (48%) of cases with no manipulation of images, resulting in a 40% decrease in opposing impressions when images could not be manipulated (RR = 0.6, 95% CI 0.3-1.3). Radiologists were less likely to report opposing impressions between the two methods compared to the neonatologists. Among the three radiologists, there were opposing impressions in 17/60 (28%) with manipulation and 12/60 (20%) with no manipulation of images (RR = 0.6, 95% CI 0.3-1.6). With the NICU physicians, there were opposing impressions in 24/60 (40%) with manipulation and 11/60 (18%) when no manipulation was allowed (RR = 0.3, 95% CI 0.1-0.8).
|Table 1: Assessment of sixty sets of frontal chest radiographs without versus with the ability of to manually manipulate the images|
Click here to view
Radiologists were 3.5 times more likely than neonatologists to have 100% agreement on radiograph impressions when they had the ability to manipulate the image (95% CI 1.3-9.7) [Table 2]. Radiologists were also 1.7 times more likely to have 100% agreement on image interpretation when manipulation was not allowed (RR = 1.7, 95% CI 0.7-3.8) and 0.6 times less likely to have opposing impressions when manipulation was permitted (RR = 0.6, 95% CI 0.3-1.4). However, in some instances, the results were not statistically significant, primarily due to the small number of clinicians involved.
|Table 2: Assessment of 60 sets of frontal chest radiographs with and without the ability to manually manipulate the images: Radiologists versus neonatologists|
Click here to view
| Discussion|| |
CXR interpretation is a routine essential requirement in the daily care of patients, particularly important in the NICU setting. Changes in the radiographic appearance result in a change in management in a significant percentage of patients. ,,,,, Multiple studies have demonstrated high inter-observer variability in the interpretation of radiographs. This has led to potential solutions such as double and triple reading of studies which are both costly and time-consuming. ,,,, More recent research has shown even less inter-observer agreement on digital compared to analog studies.  Numerous factors can contribute to poor image quality, such as patient and cassette positioning, exposure techniques, body habitus, and motion, and the radiographic images obtained are often suboptimal. , These variations in image quality and appearance often result in the interpreting clinician attempting to optimize the appearance by manipulating/windowing the image on PACS. The effect of this independent, individualized ability to manipulate the radiographic images has not been well documented.
Our study demonstrates an adverse effect in regards to the agreement in image interpretation when manipulation on PACS system is performed. The poorer inter-rater agreement was more significant among the NICU physicians, possibly due to less experience in image manipulation as compared to the radiologists who perform these maneuvers on a daily basis. The study also demonstrates overall poor agreement in image interpretation which has been shown in the medical literature. ,,,,, Particularly, concerning is the percentage of opposing radiograph impressions which could result in significant changes to patient care and outcome. The ability to manipulate the appearance of the CXR images on PACS led to an increase in opposing interpretations in our study and raised the question of whether clinicians' access to image windowing/manipulation is of benefit to the patient. Access to PACS and the ability to manipulate radiographic images independently by health care providers have increased in most hospital and patient care systems lately. This has been found to be very beneficial in regards to access data and patient radiographs; however, the potential disadvantages have not been elucidated. Further studies to demonstrate the impact on patient care and hospital costs are needed to help guide future endeavors by researchers, hospitals, and industry to improve our consistency in image interpretation. Novel and improved methods are needed to optimize image quality and allow for more reliable and consistent interpretation and comparison of radiographs.
A pitfall of the study is the lack of a gold standard in the diagnosis of the radiograph abnormality. The study was designed to assess agreement between interpreters; therefore, accuracy per se was not deemed pertinent. All interpreters were specialists in their fields who routinely accessed the PACS as part of their daily routine of caring for patients. The inter-observer variability was thus regarded as a limitation of the test itself and not related to radiologist/neonatologist expertise. The dictating of the 60 sets of CXRs may have allowed for some patient recognition, and potential bias, although the lack of patient identification, the similar appearance of these types of studies and the alternating use of viewers are thought to have minimized this problem.
| Conclusion|| |
Agreement in CXR interpretation is poor, with conflicting impressions common with both neonatologists and radiologists. The ability to manipulate the images on PACS has resulted in a decrease in agreement in the interpretation of these studies. New methods to standardize image appearance and allow improved comparison with previous studies should be sought to improve clinician agreement in interpretation consistency and advance patient care.
We would like to acknowledge Dr. J. Flood, M. Clarke, A. Acker and S. Salahudeen, who provided insight and expertise that greatly assisted our study. We would also like to acknowledge Wilma Hopman for her statistical and editorial expertise.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Lakhal K, Serveaux-Delous M, Lefrant JY, Capdevila X, Jaber S; AzuRéa Network for the RadioDay Study Group. Chest radiographs in 104 French ICUs: Current prescription strategies and clinical value (the RadioDay study). Intensive Care Med 2012;38:1787-99.
Graat ME, Choi G, Wolthuis EK, Korevaar JC, Spronk PE, Stoker J, et al. The clinical value of daily routine chest radiographs in a mixed medical-surgical intensive care unit is low. Crit Care 2006;10:R11.
Meade MO, Cook RJ, Guyatt GH, Groll R, Kachura JR, Bedard M, et al. Interobserver variation in interpreting chest radiographs for the diagnosis of acute respiratory distress syndrome. Am J Respir Crit Care Med 2000;161:85-90.
Rubenfeld GD, Caldwell E, Granton J, Hudson LD, Matthay MA. Interobserver variability in applying a radiographic definition for ARDS. Chest 1999;116:1347-53.
Tudor GR, Finlay D, Taub N. An assessment of inter-observer agreement and accuracy when reporting plain radiographs. Clin Radiol 1997;52:235-8.
Cleveland RH, Schluchter M, Wood BP, Berdon WE, Boechat MI, Easley KA, et al. Chest radiographic data acquisition and quality assurance in multicenter studies. Pediatr Radiol 1997;27:880-7.
Bloomfield FH, Teele RL, Voss M, Knight DB, Harding JE. Inter- and intra-observer variability in the assessment of atelectasis and consolidation in neonatal chest radiographs. Pediatr Radiol 1999;29:459-62.
Hauser GJ, Pollack MM, Sivit CJ, Taylor GA, Bulas DI, Guion CJ. Routine chest radiographs in pediatric intensive care: A prospective study. Pediatrics 1989;83:465-70.
Chahine-Malus N, Stewart T, Lapinsky SE, Marras T, Dancey D, Leung R, et al. Utility of routine chest radiographs in a medical-surgical intensive care unit: A quality assurance survey. Crit Care 2001;5:271-5.
Horst HM, Fagan B, Beute GH. Chest radiographs in surgical intensive care patients: A valuable "routine". Henry Ford Hosp Med J 1986;34:84-6.
Greenbaum DM, Marschall KE. The value of routine daily chest x-rays in intubated patients in the medical intensive care unit. Crit Care Med 1982;10:29-30.
Henschke CI, Pasternack GS, Schroeder S, Hart KK, Herman PG. Bedside chest radiography: Diagnostic efficacy. Radiology 1983;149:23-6.
Bekemeyer WB, Crapo RO, Calhoon S, Cannon CY, Clayton PD. Efficacy of chest radiography in a respiratory intensive care unit. A prospective study. Chest 1985;88:691-6.
Ioos V, Galbois A, Chalumeau-Lemoine L, Guidet B, Maury E, Hejblum G. An integrated approach for prescribing fewer chest x-rays in the ICU. Ann Intensive Care 2011;1:4.
Bhagwanjee S, Muckart DJ. Routine daily chest radiography is not indicated for ventilated patients in a surgical ICU. Intensive Care Med 1996;22:1335-8.
Lichtenstein D, Goldstein I, Mourgeon E, Cluzel P, Grenier P, Rouby JJ. Comparative diagnostic performances of auscultation, chest radiography, and lung ultrasonography in acute respiratory distress syndrome. Anesthesiology 2004;100:9-15.
| Authors|| |
[Table 1], [Table 2]