An Intervention to Tag Findings Suspicious for Lung Cancer on Chest Computed Tomography Has Good Sensitivity and Number Needed to Diagnose


Jennifer R Dusendang, MPH1; Lori C Sakoda, PhD1; Thomas H Urbania, MD2; Sora Ely, MD3; Todd Osinski, MD2; Ashish Patel, MD3; Lisa J Herrinton, PhD1

Perm J 2021;25:20.155
E-pub: 05/12/2021


Background: In 2015, Kaiser Permanente Northern California implemented an intervention to improve follow-up for pulmonary findings on diagnostic chest computed tomography (CT). The intervention includes tagging CT reports with the prefix “#PUL” followed by a character (0–6 or X) to track specific findings. #PUL5, indicating “suspicious for malignancy,” triggers automatic referral for multidisciplinary care review.

Methods: Among patients who obtained an index chest CT exam from August 2015 to July 2017 without an exam in the previous 2 years, we computed the frequency of lung cancer diagnosis within 120 days of CT in relation to each #PUL tag. For #PUL5, we computed sensitivity, specificity, positive and negative predictive values, and number needed to diagnose. We also performed a chart review to assess why some patients diagnosed with lung cancer were not tagged #PUL5.

Results: Of the 39,409 patients with a tagged CT report, 1105 (2.8%) had a new primary lung cancer diagnosis within 120 days. Among the 2255 patients tagged #PUL5, 821 were diagnosed with lung cancer, with a sensitivity of 74% (95% confidence interval, 72%–77%). The positive predictive value was 36% (35%–38%), number needed to diagnosis was 2.7 (2.6–2.9), and specificity and negative predictive values were > 95%. Chart review identified opportunities to improve system defaults and clarify concepts.

Conclusion: The intervention performed well but needed improvement. Automating CT reports was simple and generalizable, and enabled reduction of care gaps and system improvement.


Each year, more than 200,000 US residents are diagnosed with lung cancer.1 Chest computed tomography (CT) is a key modality for diagnosing lung cancer. However, follow-up of findings suspicious for lung cancer may be hindered if the free-text radiology report is not clear or specific about recommended follow-up.2

In 2015, Kaiser Permanente Northern California implemented a standardized reporting system3 to tag findings suspicious for lung cancer seen on diagnostic chest CT in accordance with Fleischner guidelines.4-6 The intervention included a tagging system to automate forwarding of chest CT scans highly suspicious for lung cancer (#PUL5) combined with a review by a multidisciplinary review group. In a quasi-experimental evaluation, implementation of the tagging system combined with case management and multidisciplinary review of cases suspicious for malignancy was effective for increasing the diagnosis of early-stage lung cancer within 120 days (odds ratio, 1.24; 95% confidence interval [CI] 1.09–1.41).3

In this analysis, we assessed the frequency of lung cancer diagnosis associated with each #PUL category. In addition, for #PUL5 (suspicious for malignancy), we computed the sensitivity, specificity, and other performance characteristics in relation to lung cancer diagnosis within 120 days. Last, we conducted chart reviews to assess why some newly diagnosed lung cancer cases were assigned tags other than #PUL5.


Kaiser Permanente Northern California is an integrated, community-based health-care system that provides comprehensive care to approximately 4.3 million members and currently performs 75,000 chest CT procedures each year that are read by > 300 diagnostic radiologists. All medical information is documented in an electronic medical record. The chest CT tagging intervention was built into the radiology dictation software. Tags use the prefix “#PUL” followed by a character (0–6 or X) that encodes specific pulmonary findings (Table 1).

Table 1. Tags, findings, and recommendations used to tag computed tomographic scans

Code and finding Report phrase Recommendationa
#PUL0. No pulmonary nodule; Lungs: normal; or Pulmonary nodules/masses: none No pulmonary nodule No further action
#PUL1. Benign. Complete, concentric, central, popcorn; definite internal fat; solid nodules stable for 2 y or more (unless part solid/ground glass, see #PUL3) Pulmonary nodule(s) presumed benign based on imaging criteria. No further action
#PUL2. Likely infectious or inflammatory. Small centrilobular nodules in presumed small airways infection; nodules related to septic emboli, fungal, or mycobacterial infection; inflammatory nodules associated with conditions such as rheumatoid arthritis or vasculitis; diffuse nodular lung disease, including but not limited to sarcoidosis, hypersensitivity pneumonitis, and amyloidosis Likely infectious or inflammatory pulmonary nodule(s). The need for any follow-up imaging depends on the broader clinical context. The need for any follow-up imaging depends on the broader clinical context.
#PUL3. Subsolid. Subsolid nodules include pure ground-glass and part ground-glass/part-solid nodules. These nodules may represent preneoplastic or indolent neoplastic primary lung lesions and often require long-term follow-up. The presence of a solid component correlates with the presence of invasive malignancy. Upgrade to #PUL5 if a solid component is initially 4 mm or larger. Subsolid pulmonary nodule(s); follow-up according to KP guidelines to document stability Follow-up CT according to size and ground-glass/subsolid characteristics to document stability
#PUL4. Indeterminate solid nodule. Solid nodules that measure up to 8 mm Indeterminate solid pulmonary nodule(s); follow-up CT according to Fleischner Society guidelines based on patient risk factors for lung cancer Follow-up CT scheduled by the ordering provider according to Fleischner Society guidelines based on nodule size and patient risk factors for lung cancer
#PUL5. Suspicious for malignancy. Solid nodule 9 mm or larger in largest dimension except those 1) presumed benign by CT criteria (use #PUL1), 2) presumed infectious/inflammatory (use #PUL2), 3) presumed metastatic (use #PUL6), or 4) known lung cancer (use #PUL6). Includes growth of a nodule (solid or subsolid) by 2 mm or more; pure ground-glass nodule that develops a solid component on follow-up; part-solid nodule with a solid component 4 mm or larger; other finding suspicious for a primary chest malignancy including suspected primary malignancies in the chest other than lung cancer, such as suspicious mediastinal masses or esophageal tumors. Suspicious. This finding will be reviewed by an expert committee, which will coordinate further follow-up. This finding will be reviewed by an expert committee, which will coordinate further follow-up.
#PUL6. Known lung cancer or suspected metastasis to the lung. Established diagnosis of lung cancer; nodules detected in patients with a known primary malignancy that are considered most likely metastatic. Known lung cancer or suspected metastatic nodule(s) Follow-up by ordering provider
#PULX. Technically limited. Factors prevent exclusion of pulmonary nodules. Technical factors prevent exclusion of pulmonary nodules None

a. The accrual period for our analysis ended in 2017 and used the 2013 Fleischner Guidelines. In 2018, the system was updated to reflect the 2017 Fleischner guidelines.

CT = computed tomography; KP = Kaiser Permanente.

Findings suspicious for lung cancer, tagged #PUL5, are forwarded automatically to a care coordinator. The coordinator forwards new cases to a triage physician, typically a pulmonologist or thoracic surgeon, who performs and initial review. For straightforward cases, the triage physician provides initial management recommendations. More complicated cases are discussed at a weekly multidisciplinary conference that includes a coordinator, pulmonologist, thoracic surgeon, and often a radiologist and oncologist. For some patients, the team takes over management, ordering follow-up imaging studies, arranging biopsies, and scheduling surgery, whereas for others, the team makes recommendations to the primary care physician, with the care coordinator following up to ensure the plan is implemented.

A separate tag, #PUL6 (known lung cancer or suspected metastasis to the lung), is intended for patients whose lung cancer was diagnosed pathologically before their current chest CT or whose cancer is metastatic to the lung. For lung nodules that are benign, subsolid, or indeterminate, the Fleischner Society guidelines4-6 are added to the radiology report along with standard text containing follow-up recommendations for the ordering provider.

As detailed in a separate publication presenting a controlled cohort study that used a step-wedge design, the system resulted in several advantages, with the key advantage being the increased diagnosis of early-stage lung cancer.3 Other advantages include clearer documentation, review of suspicious cases by a multidisciplinary team, reduction in the burden to primary care providers, and the opportunity to measure performance improvement.

The study population included adult health plan members who underwent chest CT examination from August 2015 to July 2017 that was assigned a #PUL tag by the reading radiologist.3 We examined only the first chest CT, which was defined as the index CT. We excluded patients who had a chest CT during the 2-year period before their index CT or who had a history of lung cancer.

Statistical analysis involved computation of percentages and their CIs. Within groups of patients defined by #PUL tag, we calculated the frequency of lung cancer diagnosis7 within 120 days of the initial CT examination, with 120 days selected a priori on the basis of the clinical workflow. For #PUL5 (suspicious for malignancy), we computed the sensitivity, specificity, positive predictive value, negative predictive value, number needed to diagnose (NND), and 95% CIs using exact methods and standard logit confidence intervals.8-10

A separate tag, #PUL6 (known lung cancer or suspected metastasis to the lung), is intended for patients whose lung cancer was diagnosed pathologically before their current chest CT or whose cancer is metastatic to the lung. However, radiologists may not have complete information about the patient and may not review the patient’s chart to confirm a preceding cancer diagnosis. Because radiologists often coded #PUL6 in place of #PUL5, we conducted a post hoc sensitivity analysis combining the 2 tags to gain insight about how changes in the training of radiologists regarding use of the #PUL6 tag and more specific wording of the #PUL6 description could improve the performance of the reporting system.

For the chart reviews of lung cancers assigned tags other than #PUL5, we obtained random samples of up to 10 patients per #PUL tag. Post hoc, we noted several cases of early-stage lung cancer tagged #PUL6 and conducted additional chart reviews for these patients. Chart reviews were performed from 120 days before to 120 days after the index CT by the first author (JRD), with the senior author (LJH) duplicating review of the CT report. We sought to understand the indication for the index CT, the radiologist’s findings, and subsequent clinical actions. This material was analyzed qualitatively, summarized in a table, and shared with the lead radiologist (TU), who performed additional chart review and articulated learnings.

The Research Determination Committee for the Kaiser Permanente Northern California region determined the project does not meet the regulatory definition of research involving human subject per 45 CFR 46.102(d).


The study included 39,409 patients with a tagged chest CT. As reported previously, nearly half were 60 to 80 years old at the time of the index CT, about 40% were nonwhite, 54% were female, and 80% had a preceding chest x-ray.3

Six percent of CTs were tagged #PUL5 (suspicious for malignancy) and 2%, #PUL6 (known lung cancer or suspected metastasis (Table 2). Lung cancer was diagnosed within 120 days of the index CT in 36.4% (CI, 34.4%–38.4%) of #PUL5 patients. It was also diagnosed in 11.7% (CI, 9.6%–13.8%) of #PUL6 patients (with some being metastatic to the lung and some not). In contrast, it was diagnosed in 1% to 2% tagged #PUL2 (infectious/inflammatory), #PUL3 (subsolid), or #PULX (technically limited). It was diagnosed in < 1% tagged #PUL0 (no nodule), #PUL1 (benign), or #PUL4 (indeterminate).

Table 2. Frequency of lung cancer within 120 d of index computed tomography in relation to #PUL tag

#PUL tag n % Disenrolled or died within 120 d, % No. of new primary lung cancers Lung cancer cases, % Patients with lung cancer, % 95% CI
#PUL0: None 21,623 55 5 56 5 0.26 0.19–0.33
#PUL1: Benign 2508 6 4 3 < 1 0.12 0.00–0.26
#PUL2: Infectious/inflammatory 3018 8 8 43 4 1.42 1.01–1.85
#PUL3: Subsolid 690 2 4 12 1 1.74 0.76–2.72
#PUL4: Indeterminate 6365 16 5 44 4 0.69 0.49–0.89
#PUL5: Suspicious 2255 6 17 821 74 36.4 34.4–38.4
#PUL6: Known cancer 898 2 24 105 10 11.7 9.6–13.8
#PULX: Technically limited 2052 5 16 21 2 1.02 0.59–1.45
Total 39,409 100 7 1105 100 2.80 2.71–3.05

Adult members of Kaiser Permanente Northern California who underwent chest computed tomography (CT) without a history of chest CT in the preceding 24 mo, August 2015 to July 2017.

CI = confidence interval.

For the #PUL5 tag, the sensitivity for diagnosing lung cancer was 821 of 1105, or 74% (CI, 72%–77%); the positive predictive value was 821 of 2255, or 36% (CI, 35%–38%), and the NND was 2255 of 821 or 2.7 (CI, 2.6–2.9) (Table 3). The specificity for identifying cases without lung cancer and the negative predictive value exceeded 95%, as expected for rare diseases. As detailed in the following paragraph, #PUL6 was sometimes used incorrectly in place of #PUL5. To assess the opportunity to improve sensitivity through further training of radiologists in use of #PUL6, we combined #PUL5 and #PUL6 in a post hoc analysis, noting the sensitivity increased to 926 of 1105, or 84% (CI, 82%–86%), and the NND increased to 3153 of 926, or 3.4 (CI, 3.3–3.5).

Table 3. Performance characteristics of #PUL5 for detecting lung cancer within 120 d of index computed tomography

#PUL tag Performance characteristic Numerator Denominator Performance, % 95% CI
#PUL5 only Sensitivity 821 1105 74.3 71.6–76.9
Specificity 36,870 38,304 96.3 96.1–96.4
Positive predictive value 821 2255 36.4 35.0–37.8
Negative predictive value 36,870 37,154 99.2 99.2–99.3
No. needed to diagnosis 2255 821 2.7 2.6–2.9
#PUL5 + #PUL6 Sensitivity 926 1105 83.8 81.5–85.9
Specificity 36,077 38,304 94.2 94.0–94.4
Positive predictive value 926 3153 29.4 28.4–30.4
Negative predictive value 36,077 36,256 99.5 99.4–99.6
No. needed to diagnosis 3153 926 3.4 3.3–3.5

Adult members of Kaiser Permanente Northern California who underwent chest computed tomography (CT) without a history of chest CT in the preceding 24 mo, August 2015 to July 2017. For #PUL5 only, the number of true positives (TPs) was 821; true negatives (TNs), 36,870; false negatives (FNs), 284; and false positives (FPs), 1434. For #PUL5 and #PUL6, the number of TPs was 926; TNs, 36,077; FNs, 179; and FPs, 2227. The sensitivity is defined as TP/TP + FN; specificity, TN/(FP + TN); positive predictive value, TP/(TP + FP); negative predictive value, TN/(FN + TN); and number needed.

CI = confidence interval; #PUL5 = suspicious for lung cancer; #PUL6 = known lung cancer or suspected metastasis to the lung.

Two hundred eighty-four cases were newly diagnosed with lung cancer but given a code other than #PUL5 (suspicious for malignancy). To understand this discordance, we performed 66 chart reviews. Reviews of #PUL0 (no nodule) (10 reviews of 56 cancers) showed that radiologists had not updated their templated reports from the default phrase “No Pulmonary Nodule” and overemphasized nodules (≤ 3 cm), leaving masses (> 3 cm) untagged. In addition, in 1 patient, a nodule was detected on the right lung, but the left lung was described as “no pulmonary nodule,” triggering the wrong code. #PUL1 (benign) (3 reviews of 3 cancers) included 1 case that was diagnosed before the CT exam, 1 judged to be granulomatous disease and 1 with a 13.5-mm nodule that was miscoded. #PUL2 (infectious/inflammatory) (10 reviews of 43 cancers) had findings suspicious for malignancy combined with infectious/inflammatory findings for which a repeat CT was ordered to assess resolution. For #PUL3 (subsolid) (10 reviews of 12 cancers), each report described ground-glass findings, but none mentioned a solid component, although the word “nodule” was sometimes used. Multiple ground-glass nodules were sometimes interpreted as infection. The study radiologist concluded that ground-glass nodules represent various clinical scenarios and noted that no guideline recommends upgrading ground-glass nodules to #PUL5 unless they are part solid or growing quickly, regardless of size. #PUL4 (indeterminate) (10 reviews of 44 cancers) had multiple nodules suggesting infection/inflammation, a large cavitated density, and small nodules with other suspicious findings such as lymphadenopathy, with 1 patient having lung cancer diagnosed before their CT scan. In 4 patients, these findings led to the recommendation to repeat CT using contrast or to obtain a pulmonology consultation. We noted that the Fleischner Society guidelines for managing indeterminate nodules are complex, and recommendations for follow-up require synthesis of information of nodule type and size assessed by radiologists with risk factors assessed by the ordering physician such as family history, smoking history, and differential diagnosis.

For patients tagged #PUL6 (known lung cancer or suspected metastasis to the lung), we randomly selected 10 patients of 105 cancers, of which 8 had clear evidence in the CT report of metastatic disease and were appropriately tagged; 2 did not, of whom 1 had early-stage disease. We then reviewed all 13 early-stage cases tagged #PUL6, of which 1 had a recent lung cancer diagnosis; 2, recent diagnoses of cancer other than lung; 4, distant histories of cancer other than lung; and 1, what appeared to the radiologist as a liver cancer with lung metastasis. The other 6 did not have mention of a recent or distant cancer diagnosis as part of their indication. We noted that patient histories were heterogeneous. Chest CT alone did not provide adequate information to the radiologist for making an accurate diagnosis of metastatic disease, and the definition of #PUL6 could be clarified.

Chart review of #PULX (technically limited) (10 reviews of 21 cancers) revealed heterogeneous circumstances, although lung cancer was part of the differential diagnosis or was highly concerning in most. Despite not being coded #PUL5 (suspicious for malignancy), 6 of the radiology reports described masses, and findings were communicated directly to the ordering physician, leading to clinical follow-up.


We assessed the performance characteristics of a standardized reporting system that adds tags to chest CT reports and automatically forwards findings suspicious for lung cancer to an expert review committee. The goals of the reporting system were to expedite workup for new lung cancer cases through the use of a multidisciplinary review group and to assist primary care providers in scheduling follow-ups for nodule management in accordance with clinical guidelines. We previously reported the system’s effectiveness for diagnosing early-stage lung cancer, noting the intervention was associated with a 24% greater odds of diagnosing early-stage lung cancer within 120 days (odds ratio, 1.24; 95% CI, 1.09–1.41), but not with an increased rate of surgical treatment.3 Here, we report #PUL5 (suspicious for malignancy) to have a sensitivity of 74% (CI, 72%–77%), with an NND of 2.7 (CI, 2.6–2.9) for tagging new lung cancer. Although this level of sensitivity is acceptable, since completing this work, we have sought to increase the sensitivity through several workflow improvements, detailed in the following paragraph. The NND of 2.7 means that for every case of lung cancer, the multidisciplinary review group reviews 2.7 imaging reports and associated notes, with some cases being expedited easily and other requiring more time. The multidisciplinary group reports this level of NND to be acceptable.

This study provided important information to improve the system, including the need for radiologists to update reports templated #PUL0, the importance of coding all findings suspicious for lung cancer and not just nodules, challenges with measuring the solid component, challenges coding the combination of inflammation with findings suspicious for cancer, separating cases with new and established lung cancer or metastatic disease, and coding suspicious findings together with technical limitations. In response to these findings, we sought to improve the system through design changes and additional radiologist training to clarify that the definition of #PUL5 includes any finding suggestive of malignancy, not just pulmonary nodules, and #PUL6 provides a tag for patients who have received a confirmed cancer diagnosis, not those in whom cancer appears metastatic on CT.

A 2016 survey of American Thoracic Society members reported that the 3 most common systems-level resources to improve the diagnosis of lung cancer included 1) adding Fleischner Society guidelines to radiology reports, 2) adding prompts to the ordering provider on radiology reports describing new nodules, and 3) employing clinical staff to coordinate recommended follow-up.11,12 The system we describe included all 3 of these components. We are not aware of previous evaluations of these components or other efforts such as described here.

A strength of our study was the inclusion of nearly 40,000 community-based patients undergoing diagnostic chest CT for a variety of reasons, many of whom were elderly and therefore underrepresented in past studies. The key limitation was the study’s scope. We did not assess the time required by the radiologist to tag each report or the user-centeredness of the system. Our study allowed for lung cancer diagnoses through 120 days after imaging, thereby allowing time for reimaging growth of features such as ground-glass nodules. However, we did not have adequate follow-up time to assess lung cancer diagnosis more than 120 days after initial CT examination, as needed to evaluate surveillance of subsolid and indeterminate pulmonary nodules. Additional research is needed to examine lung cancer diagnoses beyond 120 days, the impact of greater diagnosis of early-stage lung cancer on lung cancer mortality, and the effectiveness of tagging subsolid and indeterminate pulmonary nodules for improving nodule management. “Infectious/inflammatory” is a broad concept, and for patients with both infectious/inflammatory findings and a pulmonary nodule, it may be possible to define subgroups for whom the nodule evaluation is especially urgent. For ground-glass nodules that have a well-defined solid component, the Fleischner Society guidelines recommend surveillance of the solid component for growth. However, measuring the solid component is challenging because vessels running through these nodules can be mistaken for solid components, and the solid component can appear to change from one scan to the next, with the difference being technical. This results in high interobserver variability. In addition, subsolid nodules grow slowly,13 and the concept of a lung cancer that grows slowly may be underappreciated by radiologists.14,15 A critical benefit of standardized reporting systems is the opportunity the systems afford, through the creation of data sets, to conduct important research to address unanswered questions regarding lung nodule management and to identify performance improvement opportunities.

Artificial intelligence is increasingly being applied to health-care delivery and, theoretically, could be implemented as an extra precaution against underdiagnosing lung cancer from CT scans.16,17 However, artificial intelligence faces several barriers to full implementation, including performance, computing power needs, the need to train systems, and differences between high-quality, standardized imaging used for algorithm training and nonstandardized clinical imaging in diverse and complex patients.

In conclusion, radiology tagging systems can categorize patients effectively with abnormal lung findings with good sensitivity and low NND. Systems can be improved by ensuring clear categories, prioritizing the purpose of the system over strict definitions during radiologist training, and utilizing user-centered coding schemes that are easy for wide adoption. Tagging systems provide a simple and useful informatics solution for ensuring appropriate patient care, benchmarking, performance improvement, and research into the effectiveness of guidelines in real-world settings.

Disclosure Statement

The investigators are partners and staff of The Permanente Medical Group and report no other conflicts of interest.

Financial Support

This project was supported by The Permanente Medical Group Delivery Science and Applied Research initiative, and in part by K07 CA188142 (LCK). The funders had no role after making a funding decision.

Author Affiliations

1Division of Research, Kaiser Permanente Northern California, Oakland, CA

2Department of Radiology, Kaiser Permanente Northern California, Oakland, CA

3Department of Chest Surgery, Kaiser Permanente Northern California, Oakland, CA

Corresponding Author

Lisa J Herrinton, PhD (

Author Contributions

Lori C Sakoda, PhD, Thomas H Urbania, MD, Sora Ely, MD, Todd Osinski, MD, Ashish Patel, MD, and Lisa J Herrinton, PhD, participated in study design. Jennifer R Dusendang, MPH, and Lisa J Herrinton, PhD, participated in data collection. Jennifer R Dusendang, MPH, Lori C Sakoda, PhD, Thomas H Urbania, MD, Sora Ely, MD, Todd Osinski, MD, Ashish Patel, MD, and Lisa J Herrinton, PhD, participated in data analysis and manuscript preparation. Lisa J Herrinton, PhD, is the guarantor of this manuscript and its content. Jennifer R Dusendang, MPH, Lori C Sakoda, PhD, Thomas H Urbania, MD, Todd Osinski, MD, Ashish Patel, MD, and Lisa J Herrinton, PhD, contributed substantially to the conception or design of the work; Jennifer R Dusendang, MPH, Lori C Sakoda, PhD, Thomas H Urbania, MD, Sora Ely, MD, Todd Osinski, MD, and Lisa J Herrinton, PhD, contributed substantially to the writing and/or revision of the manuscript; and all authors approved the final version of the manuscript. All authors are accountable for the manuscript’s contents.


1. National Institutes of Health. Surveillance, Epidemiology, and End Results Program. Cancer stat facts: Lung and bronchus cancer. Accessed March 11, 2020.

2. Wiener RS, Gould MK, Slatore CG, Fincke BG, Schwartz LM, Woloshin S. Resource use and guideline concordance in evaluation of pulmonary nodules for cancer: Too much and too little care. JAMA Intern Med 2014 Jun;174(6):871–80. DOI:, PMID:24710850.

3. Urbania TH, Dusendang JR, Herrinton LJ, et al. Standardized reporting and management of suspicious findings on chest CT imaging is associated with improved lung cancer diagnosis in an observational study. Chest 2020 Nov;158(5):2211–20. DOI: Online ahead of print.

4. MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: A statement from the Fleischner Society. Radiology 2005 Nov;237(2):395–400. DOI:

5. Naidich DP, Bankier AA, MacMahon H, et al. Recommendations for the management of subsolid pulmonary nodules detected at CT: A statement from the Fleischner Society. Radiology 2013 Jan;266(1):304–17. DOI:, PMID:23070270.

6. MacMahon H, Naidich DP, Goo JM, et al. Guidelines for management of incidental pulmonary nodules detected on CT images: From the Fleischner Society 2017. Radiology 2017 Jul;284(1):228–43. DOI:, PMID:28240562.

7. National Cancer Institute. SEER registrar staging assistant. NAACCR item #390: Date of diagnosis. Accessed April 19, 2019.

8. Garwood F. Fiducial limits for the Poisson distribution. Biometrika 1936 Dec;28(3/4):437–42. DOI:

9. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982 Apr;143(1):29–36. DOI:, PMID:7063747.

10. Mercaldo ND, Lau KF, Zhou XH. Confidence intervals for predictive values with an emphasis to case-control studies. Stat Med 2007 May;26(10):2170–83. DOI:, PMID:16927452.

11. Simmons J, Gould MK, Iaccarino J, Slatore CG, Wiener RS. Systems-level resources for pulmonary nodule evaluation in the United States: A national survey. Am J Respir Crit Care Med 2016 May;193(9):1063–5. DOI:, PMID:27128706.

12. McDonald JS, Koo CW, White D, Hartman TE, Bender CE, Sykes AG. Addition of the Fleischner Society guidelines to chest CT examination interpretive reports improves adherence to recommended follow-up care for incidental pulmonary nodules. Acad Radiol 2017 Mar;24(3):337–44. DOI:, PMID:27793580.

13. Yang J, Wang H, Geng C, Dai Y, Ji J. Advances in intelligent diagnosis methods for pulmonary ground-glass opacity nodules. Biomed Eng Online 2018 Feb;17(1):20. DOI:, PMID:29415726.

14. Nie X, Li L, Huang J, et al. From focal pulmonary pure ground-glass opacity nodule detected by low-dose computed tomography into invasive lung adenocarcinoma: A growth pattern analysis in the elderly. Thorac Cancer 2018 Nov;9(11):1361–5. DOI:, PMID:30144287.

15. Yip R, Henschke CI, Xu DM, Li K, Jirapatnakul A, Yankelevitz DF. Lung cancers manifesting as part-solid nodules in the national lung screening trial. AJR Am J Roentgenol 2017 May;208(5):1011–21. DOI:, PMID:28245151.

16. Li X, Hu B, Li H, You B. Application of artificial intelligence in the diagnosis of multiple primary lung cancer. Thorac Cancer 2019 Nov;10(11):2168–74. DOI:, PMID:31529684, PMCID: PMC6825907.

17. Zhao L, Bai CX, Zhu Y. Diagnostic value of artificial intelligence in early-stage lung cancer. Chin Med J (Engl) 2020 Feb;133(4):503–4. DOI:, PMID:31934937, PMCID: PMC7046256.

Keywords: chest computed tomography, electronic medical record, lung cancer, multidisciplinary care review, pulmonary


Click here to join the eTOC list or text ETOC to 22828. You will receive an email notice with the Table of Contents of The Permanente Journal.


2 million page views of TPJ articles in PubMed from a broad international readership.


Indexed in MEDLINE, PubMed Central, EMBASE, EBSCO Academic Search Complete, CrossRef, and SciVerse/Scopus.




ISSN 1552-5775 Copyright © 2021

All Rights Reserved