Clinical Use Cases for a Tool to Assess Risk in Superficial Bladder Cancer


Carmit K McMullen, PhD1; Maureen O’Keeffe Rosetti, MS1;
Sheila Weinmann, PhD1; Michael C Leo, PhD1;
Matthew E Nielsen, MD, MS, FACS2

Perm J 2019;23:18.276 [Full Citation]
E-pub: 08/19/2019


Background: Among the approximately 53,000 patients newly diagnosed with early-stage (superficial) bladder cancer each year, there is substantial variability in the progression to muscle-invasive disease. Enhancing risk stratification and risk-stratified surveillance could minimize risks and harms to patients, as well as unnecessary costs to health systems.
Objectives: As a preliminary step in developing and validating a risk assessment tool for superficial bladder cancer in a population-based clinical cohort, we interviewed urologists who might use such a tool to assess need, determine potential use cases, and identify key features to include.
Methods: Using an opportunistic and purposeful sampling design, we invited 13 urologists from a variety of practice settings and with a wide range of clinical experience to take part in qualitative interviews; 9 (5 urologic oncologists and 4 general urologists) participated.
Results: All urologists reported using some form of risk stratification to determine surveillance schedules for patients with bladder cancer. The following use cases were endorsed by 4 or more interviewees: 1) provide evidence to guide clinical management in specific situations, 2) generate patient-facing communication aids, 3) improve documentation about recurrence/progression risk, and 4) create scheduling and callback supports to improve the quality of follow-up care.
Conclusion: Our findings demonstrated several potential clinical-use cases for a risk calculator and clinical decision-support tool for patients with superficial bladder cancer. Clinicians stressed the potential utility of such a tool to improve patient communication, scheduling, and tracking in general urology practice.


Follow-up care for cancer is personally and financially costly.1 In the setting of superficial bladder cancer, cystoscopy is uncomfortable and anxiety-producing for patients2 and is expensive for health systems.3-5 Among the approximately 53,000 patients with bladder cancer per year who present with early-stage disease—superficial (nonmuscle invasive) tumors (stage < T2), there is substantial variability in the progression to muscle-invasive disease (stage ≥ T2).6-8 Five-year risk of progression ranges from less than 1% in the majority—patients with low-grade noninvasive papillary (Ta) carcinoma—to 30% to 70% in patients with stage T1 disease or carcinoma in situ.7 Enhancing risk stratification and risk-stratified surveillance could minimize risks and harms to patients, as well as unnecessary costs to health systems.

Accurately assessing and communicating risk of recurrence and progression is critical to providing appropriate and cost-effective follow-up care. The European Association of Urology (EAU) has recommended risk-stratified surveillance schedules for patients with nonmuscle invasive bladder cancer (NMIBC) for more than a decade,9 with low-risk patients receiving only 3 cystoscopies in the first 2 years (assuming negative results). The National Institute for Care Excellence in the UK has promulgated an even less intensive approach than the EAU for low-risk patients, recommending discharge to primary care if recurrence-free at 1 year.10 The American Urological Association (AUA) recently adopted a more risk-adapted approach11 similar to the EAU guideline. An expert panel representing the AUA has called for population-based data and better predictive tools for bladder cancer surveillance.12 Patients and health systems would benefit from a risk-stratified approach to monitoring that focuses resources on those patients who are most at risk.

One tool for calculating the risk of recurrence and progression in bladder cancer comes from the European Organisation for Research and Treatment of Cancer (EORTC). This tool uses an algorithm developed from a pooled analysis of European  clinical trials of NMIBC7 to predict the risks of recurrence and progression to muscle-invasive bladder cancer. This tool forms the basis of current EAU practice guidelines. However, the calculator has faced limited adoption in the US. This may be in part because it relies on aggregate data from clinical trial populations, and it has until recently13 not been validated in a contemporary American population-based cohort. Emerging data from real-world practice settings in the US suggest limitations in the generalizability of the EORTC trial data to observed outcomes in contemporary US practice.13 Additionally, the risk calculator lacks features that apply directly to clinical context, such as decision support, patient communication aids, or documentation and tracking aids.

As a preliminary step in developing and validating a risk-assessment tool for superficial bladder cancer in a population-based clinical cohort, we interviewed urologists who might use such a tool to assess need, determine potential use cases, and identify key features to include.


Using an opportunistic and purposeful sampling design,14,15 we invited 13 urologists from a variety of practice settings and with a wide range of years of clinical experience after residency and fellowship to take part in qualitative interviews; 9 (5 urologic oncologists and 4 general urologists) agreed to participate (Table 1). To provide health system leadership and health information technology perspectives, we included an academic medical center department chair and a urologic oncologist working in an information technology role with expertise in clinical decision support. Urologists were recruited from the integrated health system involved in our larger study and through purposeful outreach by one of the authors (MN), with the intention of recruiting physicians and colleagues who varied in subspecialty training, role in their practice group, and practice setting. Initial outreach was through an email introducing the study and the purpose of the interview.

We developed the interview guide to address best practices for developing and implementing clinical decision-support tools. The open-ended questions asked about current practice, success factors for implementing a clinical decision-support tool,16 and potential barriers and facilitators to changing practice.17 Interviews took approximately 1 hour.

At the end of the interview, or by follow-up email, participants were asked to complete a brief follow-up survey that included questions asking them to rate (on a 5-point Likert scale) how important each of 6 clinical decision-support characteristics would be for a risk-assessment tool. Previous research has identified 9 key criteria for successful clinical decision-support implementation.16 Our survey questions were based on a subset of 6 of these criteria that we considered most in need of stakeholder input for developing a risk calculator: 1) integration with charting and order entry; 2) minimization of clinician data entry; 3) promotion of action vs inaction; 4) local user involvement in the development process; 5) provision of best evidence for practice change; and 6) demonstration of a care recommendation (not just a risk assessment). Seven of the 9 interviewed urologists completed this survey. Urologists were not compensated for their participation in the interviews or survey.

Interviews were audio recorded and transcribed. Interview transcripts were coded by the co-principal investigator (CM), a medical anthropologist and qualitative researcher, using a template coding approach.18,19 This included reviewing all transcripts to identify and summarize text associated with the following codes: Current practice, use of EORTC risk calculator, clinical decision support and health information technology use, conversations with patients about follow-up care, implementation issues, care improvement opportunities, and use cases. “Use case” codes provided specific examples of ways that participants imagined the tool could be used. Co-Principal Investigator CM then reviewed data coded under “use cases,” as well as data across other codes. She consolidated a list of all risk calculator use cases that were mentioned as having utility, as well as concerns about why the risk calculator may not be useful. The other study co-principal investigator (MN), a urologic oncologist, independently reviewed interviews, confirmed the use cases, and helped to clarify descriptions of use cases and their utility.



Current Clinical Practice

All urologists reported using some kind of formal or informal risk stratification to determine surveillance schedules for patients with bladder cancer. One interviewee said he typically followed the AUA guidelines, although he noted that he sometimes recommends longer intervals for some patients. Six urologists specifically mentioned using the EORTC risk categories and/or guidelines, which recommend risk-stratified follow-up schedules, although only 2 reported using the EORTC risk calculator itself. A few outlined specific schedules for patients at low risk of recurrence and progression that involved increasing intervals between cystoscopies after negative results. Several noted that they used “reduced” or “relaxed” surveillance for low-risk patients without providing specific intervals or while noting that their recommendations would depend on the details of a patient’s case. For cases at high risk of recurrence or progression, urologists more uniformly recommended cystoscopy every 3 months for at least 2 years, with some variation in use of bacille Calmette-Guérin treatment.

Potential Use Cases

The following use cases were endorsed by 4 or more interviewees (Table 2):

Provide Evidence to Guide Clinical Management in Specific Situations: When asked about use cases for a decision-support tool providing personalized risk calculations of recurrence and progression, 4 of the 9 urologists indicated that such a tool would have minimal impact on their clinical management and decision making, because they saw risk stratification in bladder cancer as relatively straightforward. However, there were some specific circumstances in which 4 interviewees believed that decision support in clinical management could be helpful. Interviewees thought that it would be valuable to be able to calculate the risk of recurrence and progression after each negative cystoscopy result, each maintenance bacille Calmette-Guérin dose, or after failure of bacille Calmette-Guérin therapy, as well as the risk of recurrence or progression with or without intravesical chemotherapy. They also felt that risk calculators could help determine when to stop surveillance. Tailoring such a tool to general urology practice was seen as important because most superficial bladder cancers are treated in a general urology practice setting, and these general urologists may be most in need of decision support.

Generate Patient-Facing Communication Aids: Five interviewees envisioned using the risk calculator to facilitate communication with patients and stressed the importance of making the tool patient friendly. They reported that such a tool could be helpful for conveying “reasonable expectations” about cancer risk. For low-risk patients, a patient-friendly risk score could help to explain the rationale for less intense surveillance; for higher-risk patients, especially those for whom high risk is not easily explained (eg, high-grade Ta and carcinoma in situ), a risk score could encourage adherence to surveillance plans or  highlight atypical cases.

Improve Documentation about Recurrence/Progression Risk: Five interviewees saw value in a calculator that would document input data (patient, tumor, and treatment characteristics) and output data (risk of recurrence and progression) in the electronic health record, and in having output data that could be printed for patients to take home. They agreed that an ideal risk calculator would be prepopulated using electronic health record data and would provide a clinical note that urologists could use during a patient’s visit. Acknowledging that this may not be feasible in all settings, urologists noted that they would still use the calculator even if they needed to enter the data themselves in a browser window.

Create Scheduling and Callback Supports to Improve Quality of Follow-up Care, Especially for Patients with High-Risk Disease: Noting the challenge of ensuring adherence with follow-up care and the limitations of their existing scheduling systems, 4 interviewees saw great value in a risk management tool that could trigger risk-stratified management protocols. Uses could include automatically setting appointments for follow-up care, providing information to schedulers about who is due or overdue for a cystoscopy or which patients are at highest risk, and sending automatic appointment reminder letters to patients. This type of “tracking system” was endorsed by 1 urologic oncologist and 3 of the 4 general urologists we interviewed. However, 2 of the 5 urologic oncologists explicitly said that they did not need a tool to enhance follow-up completion.

Key Features of Risk Assessment Tools (Survey Ratings)

Urologists were asked in the follow-up survey (n = 7) whether various characteristics of a risk-stratification clinical decision support tool were important. Although all characteristics were rated as valuable, minimizing data entry was rated as the most important feature of a tool (mean rating of 4.6 on a 5-point scale). This was followed in perceived value by providing the best evidence for practice change (mean = 4.4), integration with charting and computerized physician order entry (4.1), promoting action (4.0), the ability to adapt the tool locally (3.9), and the inclusion of a specific recommendation for care (3.9).



Through interviews about the management of NMIBC with urologists from a variety of practice backgrounds, we identified a range of use cases for a personalized risk calculator of recurrence and progression. Participants highlighted the potential of risk assessment tools to improve adherence to follow-up care and to facilitate conversations between patients and urologists regarding appropriate surveillance schedules.

Participants differed in whether they believed that a risk calculator would inform their clinical management. Several reported that risk stratification and follow-up care protocols were simple enough that they did not need a risk calculator. However, others perceived that a risk calculator could help with specific clinical management scenarios, including deciding when to stop surveillance and whether to provide intravesical chemotherapy. Despite the relatively limited clinical management potential of a risk-stratification tool, clinicians were nonetheless enthusiastic about such a tool, particularly as a means of improving patient communication and ensuring adherence to surveillance.

Both general urologists and urologic oncologists stressed the importance of tailoring risk-stratification tools for the general urology setting. Nearly all patients with superficial bladder cancer receive care in general urology practice, and general urologists expressed greater need for clinical guidance and for management protocols to facilitate scheduling and tracking of follow-up care. Several general urologists reported challenges with scheduling cystoscopy appointments and ensuring that patients receive care on schedule. Tools that automate or facilitate tracking and scheduling could reduce the burden on clinicians and staff while increasing care adherence.

Risk calculators and decision-support tools are available for a wide range of conditions, including prostate,20 colorectal, and breast cancer21; cardiovascular disease22; fractures23; and diabetes.24 Consistent with the data presented here, focus group research has found that primary care clinicians use such tools primarily as patient communication aids, although tools are also sometimes used in clinicians’ decision making.25 Clinicians are more likely to adopt tools that are perceived as well validated, referred to in clinical practice guidelines, easy to use, and integrated into the electronic health record system.25 The urologists who participated in our study echo the findings from other clinical contexts. Although they did not anticipate that a risk calculator would have much impact on their clinical management of NMIBC, they anticipated uses for communication and documentation. Similar to primary care clinicians, they highly valued user-friendly, integrated, and evidence-based tools.

Although the clinicians we interviewed represented a range of clinical experiences and practice environments, it is possible that these views may not generalize to the broader population of urologists. We interviewed a small number of clinical stakeholders, and they were not randomly selected. However, they were intentionally chosen to represent a variety of perspectives and practice contexts, and there was broad consensus across this diverse group of participants about the most promising use cases for a risk-assessment tool and the ideal characteristics for such a tool. It is also worth noting that these interviews occurred before the recent changes in AUA guidelines for managing superficial bladder cancer surveillance. We would anticipate that there would be more support in the urology community for the use cases suggested in this article than there would have been before the guideline change.

Although we acknowledge the limitations inherent to this initial developmental stage of work, our study nonetheless provides, to our knowledge, the first documentation of clinician insights into potential use cases and applications of clinical decision-support tools for bladder cancer surveillance. Future efforts could include enrollment of a larger sample of urologists and use of the Delphi method or other approaches to further assess the applicability of such approaches to clinical care. Additionally, crowdsourcing perspectives from a larger sample of clinicians, for instance, through email surveys of urologic professional societies, could further inform future efforts. Principles of user-centered design hold great promise for the development of clinically consequential applications of health information technology to oncology care.


Our findings demonstrate several potential clinical use cases for a risk calculator and clinical decision-support tool for patients with superficial bladder cancer. In particular, clinicians stressed the potential utility of such a tool to improve patient communication, scheduling, and tracking in general urology practice.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.


This study was supported by National Cancer Institute Grant no. R21CA191610. We would like to thank Tullika Garg, MD, MPH, of the Department of Urology, Geisinger Health, Danville, PA; and Neon Brooks, PhD, of the Center for Health Research, Kaiser Permanente Northwest, Portland, OR, for their editorial support.

Kathleen Louden, ELS, of Louden Health Communications performed a primary copy edit.

How to Cite this Article

McMullen CK, Rosetti MO, Weinman S, Leo MC, Nielsen ME. Clinical use cases for a tool to assess risk in superficial bladder cancer. Perm J 2019;23:
18.276. DOI:

Author Affiliations

1 Center for Health Research, Kaiser Permanente Northwest, Portland, OR

2 Lineberger Comprehensive Cancer Center, University of North Carolina-Chapel Hill School of Medicine

Corresponding Author

Carmit K McMullen, PhD (

1. Greenberg CC. Surveillance approaches following active treatment for cancer: A critical target for comparative effectiveness research. Paper presented at: Agency for Healthcare Research and Quality’s Cancer Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Comparative Effectiveness Research Consortium; 2012.
 2. Koo K, Zubkoff L, Sirovich BE, et al. The burden of cystoscopic bladder cancer surveillance: Anxiety, discomfort, and patient preferences for decision making. Urology 2017 Oct;108:122-8. DOI:
 3. Svatek RS, Hollenbeck BK, Holmäng S, et al. The economics of bladder cancer: Costs and considerations of caring for this disease. Eur Urol 2014 Aug;66(2):253-62. DOI:
 4. Botteman MF, Pashos CL, Redaelli A, Laskin B, Hauser R. The health economics of bladder cancer: A comprehensive review of the published literature. Pharmacoeconomics 2003 Dec;21(18):1315-30. DOI:
 5. Riley GF, Potosky AL, Lubitz JD, Kessler LG. Medicare payments from diagnosis to death for elderly cancer patients by stage at diagnosis. Med Care 1995 Aug;33(8):828-41. DOI:
 6. Donat SM. Evaluation and follow-up strategies for superficial bladder cancer. Urol Clin North Am 2003 Nov;30(4):765-76. DOI:
 7. Sylvester RJ, van der Meijden AP, Oosterlinck W, et al. Predicting recurrence and progression in individual patients with stage Ta T1 bladder cancer using EORTC risk tables: A combined analysis of 2596 patients from seven EORTC trials. Eur Urol 2006 Mar;49(3)466-75. DOI:
 8. Herr HW, Donat SM, Reuter VE. Management of low grade papillary bladder tumors. J Urol 2007 Oct;178(4 Pt 1):1201-5. DOI:
 9. Babjuk M, Böhle A, Burger M, et al. EAU Guidelines on non-muscle-invasive urothelial carcinoma of the bladder: Update 2016. Eur Urol 2017 Mar;71(3):447-61. DOI:
 10. National Collaborating Centre for Cancer. Bladder cancer: Diagnosis and management NICE Guideline 2 [Internet]. London, UK: National Institute for Health and Care Excellence; 2015 Feb [cited 2018 May 31]. Available from:
 11. Chang SS, Boorjian SA1, Chou R, et al. Diagnosis and treatment of non-muscle invasive bladder cancer: AUA/SUO guideline. J Urol 2016 Oct;196(4):1021-9. DOI:
 12. National Urology Research Agenda: A roadmap for priorities in urologic disease research. Linthicum, MD: American Urological Association; 2010:38.
 13. Ravvaz K, Walz ME, Weissert JA, Downs TM. Predicting nonmuscle invasive bladder cancer recurrence and progression in a United States population. J Urol 2017 Oct;198(4):824-31. DOI:
 14. Luborsky MR, Rubinstein RL. Sampling in qualitative research: Rationale, issues, and methods. Res Aging 1995 Mar;17(1):89-113. DOI:
 15. Patton MQ. Qualitative evaluation and research methods. Thousand Oaks, CA: SAGE Publications; 2002.
 16. Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep) 2012 Apr;(203):1-784.
 17. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf 2008 Apr;34(4):228-43. DOI:
 18. King N. Using templates in the thematic coding of text. In: Cassel C, Symon G, eds. Essential guide to qualitative methods in organizational research. London, UK: SAGE Publications; 2004. p 256-70.
 19. Silverman D. Doing qualitative reseach. 2nd ed. Thousand Oaks, CA: SAGE Publications; 1999.
 20. Stephenson AJ, Scardino PT, Eastham JA, et al. Postoperative nomogram predicting the 10-year probability of prostate cancer recurrence after radical prostatectomy. J Clin Oncol 2005 Oct 1;23(28):7005-12. DOI:
 21. The Breast Cancer Risk Assessment Tool [Internet]. Bethesda, MD: National Institutes of Health [cited 2019 Apr 12]. Available from:
 22. Assessing cardiovascular risk: Systematic evidence review from the Risk Assessment Work Group [Internet]. Bethesda, MD: National Institutes of Health; 2013 Nov [cited 2019 Apr 2]. Available from:
 23. Centre for Metabolic Bone Diseases. FRAX® Fracture Risk Assessment Tool [Internet]. Sheffield, UK: University of Sheffield [cited 2019 Apr 1]. Available from:
 24. Abbasi A, Peelen LM, Corpeleijn E, et al. Prediction models for risk of developing type 2 diabetes: Systematic literature search and independent external validation study. BMJ 2012 Sep 18;345:e5900. DOI:
 25. Voruganti TR, O‘Brien MA, Straus SE, McLaughlin JR, Grunfeld E. Primary care physicians’ perspectives on computer-based health risk assessment tools for chronic diseases: A mixed methods study. J Innov Health Inform 2015 Sep 24;22(3):333-9. DOI:

Keywords: care improvement, clinical decision support, risk assessment tool, risk stratification, superficial bladder cancer, surveillance

Reprint Permissions

The Permanente Journal welcomes requests for reprints and reproduction. Use of any and all published materials is copyrighted and protected.


Journal subscriptions for The Permanente Journal are entered for the calendar year. Advance payment in US dollars is required.


27,000 print readers per quarter, 15,350 eTOC readers, and in 2018, 2 million page views of TPJ articles in PubMed from a broad international readership.


Indexed in MEDLINE, PubMed Central, HINARI, EMBASE, EBSCO Academic Search Complete, rdrb, CrossRef, and SciVerse/Scopus.

Click here to join the eTOC subscription list or text TPJ to 22828. You will receive an Email notice with the Table of Contents of The Permanente Journal.




ISSN 1552-5767 Copyright © 2019

All Rights Reserved.