Pilot Study of a Web-based Decision Tool on Post-operative Use of Radioactive Iodine

Background: The Thyroid Cancer Care Collaborative developed a web-based clinical decision-making module (CDMM) to inform risk-adjusted decisions on post-thyroidectomy radioactive iodine (RAI) use in papillary thyroid cancer (PTC). Methods: In a pilot study, we evaluated the CDMM in 19 PTC cases representing low- (five), intermediate- (seven) and high-risk (seven) disease. Two PTC experts and 10 PTC physicians reviewed cases and assigned risk level and RAI recommendation. The experts used a standard approach while the others used the CDMM. We assessed agreement between responses using a weighted Kappa. Results: Between experts, risk-assignment was concordant in 100%, 57% and 86% of low-, intermediate- and high-risk cases, respectively. Between CDMM users, risk-assignment was concordant in 100%, 29% and 14% in low-, intermediate- and high-risk cases, respectively (p=0.01). CDMM-assigned risk agreed with the expert-assigned risk in 100%, 25% and 0% of low-, intermediate- and high-risk cases, respectively (Kappa=0.69). For RAI use, the experts agreed in 15 cases while CDMM users agreed in eight. On further analysis, interpretation of extrathyroidal extension and lymph node staging led to discrepancies with the CDMM. Conclusions: For a web-based CDMM to accurately inform appropriate use of RAI in PTC, standard pathological and surgical reports are necessary.

The incidence of thyroid cancer is increasing at a rate of 7% a year; there were an estimated 62,450 cases diagnosed in the US in 2015 alone. The majority of this rise in incidence is explained by the growing number of incidentally detected well-differentiated, early-stage or 'low-risk' papillary thyroid cancers (PTCs). 3 There is a growing awareness that many thyroid cancers may indeed be relatively benign in their behaviour and can be followed without any intervention. 4 As a result, the management of differentiated thyroid cancer has undergone a major paradigm shift over the last two decades from a 'one size fits all' to a 'risk-adapted' approach. The American Thyroid Association (ATA) has led this effort by developing and publishing evidence-based guidelines on thyroid cancer management. The ATA guidelines incorporate tumour and patient characteristics to estimate the initial risk of recurrence (prognostication) and then use this information to inform recommendations on the use of adjuvant radioactive iodine (RAI) or remnant ablation therapy and the intensity and method of surveillance. 5 One major goal of the ATA guidelines is to minimise potential harm from overtreatment for low-risk patients, while appropriately treating high-risk patients.
In well-differentiated thyroid cancer, adjuvant RAI is an effective method of attempting to address microscopic disease both in the thyroid bed (remnant thyroid) and distant metastatic sites.
The phrases remnant ablation and adjuvant therapy are often used interchangeably, but there are distinct differences. A lower dose of RAI, 30 to 50 mCi (or 1,110 to 1,850 MBq) is used for remnant ablation while a higher dose, 100 to 150 mCi (3,700 to 5,550 MBq) is reserved for adjuvant therapy in patients deemed at high risk of micrometastatic disease. 6 The use of RAI improves survival and decreases recurrence rates for high-risk patients with extensive disease, but does not change the already excellent prognosis of patients with low-risk disease. [7][8][9] The use of post-thyroidectomy RAI has dramatically risen over the last three decades as part of the first course of therapy for thyroid cancer from 6.1% of cases treated in 1973 to 48.7% of cases treated in 2006. 10  risk subgroup of patients. 13  The TCCC developed the CDMMs in partnership with the experts who developed the most-recent version of the ATA guidelines. The overall process in the development of the CDMMs was first to isolate data pertinent to the specific clinical decision, and then to determine the necessary questions to ask to collect that data. Next, the applicable ATA clinical practice guidelines were reviewed to create a flow chart algorithm of decisions for the outcomes. Several iteration tests were run in order to verify the logic and to check for any gaps in the algorithm.
Each outcome was cited and supported with a guideline footnote from the ATA clinical practice guidelines. The CDMM used for this study ('When to Administer Remnant Ablation') asks only five questions, one with a follow up if necessary, designed to isolate data points required to assess risk of recurrence based on the ATA guidelines ( see Figure 1).
Then, based on these responses, the CDMM provides a risk level (low, intermediate or high) and one of three recommendations on use of RAI (yes, no or consider). If the answer is 'consider', there is some additional information on which groups may benefit from RAI, but no definitive conclusion due to a lack of consensus in the field.

The Institutional Review Board (IRB) at Memorial Sloan Kettering Cancer
Center (MSKCC) approved this study. We identified study cases from an institutional database of 407 consecutive patients treated with a total thyroidectomy for PTC and stratified by ATA risk of recurrence from 2000 to 2002. We originally selected 20 cases from this dataset using a random number generator. Due to clerical inconsistency in one case upon later review, we only used 19 cases from this database (five from low-risk and seven each from intermediate-and high-risk) for analysis. Only cases with complete clinical information were included (e.g, surgical report, a pathology report, thyroid laboratories pre and post-surgery and records of all pre-surgical radiology).
For each selected case, a de-identified file containing the relevant clinical information that would have been available at time of a decision regarding adjuvant RAI was created. These files were provided to two experts (RMT, LW) who were asked to render a 'yes' or 'no' decision on the use of RAI for each case. The gold standard was deemed to be an agreement decision between two experts. Ten thyroid cancer fellows from endocrinology and surgery in fellowship at a specialised cancer centre were invited to participate (five endocrinology; five head and neck surgical fellows) and asked to use the CDMM to render a response of 'yes', 'no' or 'consider' regarding adjuvant RAI for the same 19 cases.
The number and percent of cases where all fellows agreed was described overall and by risk of recurrence based on ATA guidelines.
The distribution of fellow's responses was described for various types of cases. In cases of disagreement between the fellows, the question(s) on the CDMM causing disagreement was/were identified. Agreement between the fellows was assessed using weighted Kappa with equal weights, i.e. the distance from 'no' to 'consider' is equal to the distance from 'consider' to 'yes'. (All statistical analysis was performed at R 3.1.1.R Foundation, Vienna, Austria) and the irr package was used.

Case descriptions
The 19 cases randomly selected for this study represented a range of PTCs. There were five low-risk cases (26%), seven intermediate-risk cases (37%) and seven high-risk cases (37%). Of these cases, 10 patients were male (53%); seven patients were under 45 years of age (37%) at diagnosis.

Provider decisions regarding use of adjuvant radioactive iodine therapy
The experts had complete agreement in 15 out of 19 cases (79%) (see Table 1). There were 10 cases that were 'yes' agreement; five cases that were 'no' agreement. The experts agreed on 100%, 57% and 86% of low-  Figure 2).   The experts agreed in four intermediate-risk cases 'yes' give RAI, and the fellows agreed 'consider' in one of these cases. The experts agreed in six high-risk cases 'yes' give RAI, and the fellows did not agree on any of these cases (see Figure 3).  Although we did not explicitly ask the experts and fellows to stage the disease, we suspect that there was some confusion interpretation between N1a and N1b disease.

Level of agreement by risk category
It would appear that based on the 19 cases selected for this pilot study, the CDMM for post-treatment RAI may work best for cases that physicians   While this study evaluated a single CDMM, the TCCC has a larger goal: To improve communication between physicians by providing a single repository of clinical, patient and educational material that is tied to the individual patient and can be accessed by any and all providers.
One goal of the TCCC database system is to standardise all narrative reports. Once physicians enter patient data into the TCCC's discrete data points instead of open text boxes, reports are automatically generated.
The TCCC includes narrative output reports on surgical management, pathology, ultrasound and imaging, initial presentation, follow-up exams and nuclear medicine. These standardised narrative outputs enable other members of the care team, specifically those of distinct fellows, to easily locate and understand the reported information, such as the extent of the extrathyroidal extension.
Feedback from this study has been taken back to the TCCC and a morerefined version of the CDMM on RAI therapy will be tested again using cases from after 2005 when our institutional pathology and operative reports for thyroid cancer were standardised to avoid confusion or misinterpretation of these key clinical features.

Conclusions
The TCCC has developed a web-based clinical decision aid that is able to accurately direct clinicians to appropriate use of RAI therapy in some clinical scenarios. The ATA guidelines themselves leave a recommendation to 'consider RAI ablation in intermediate risk' scenario and do not provide a definitive recommendation that reinforces that decision aids are not meant to replace clinical experience and judgement but to help inform more evidence-based care. The goal for clinical guidelines and decision-making modules is to provide the clinician with the evidence available to guide a decision. With this goal in mind, another potential use of a tool like the TCCC CDMM for RAI ablation could also be to help educate trainees and future physicians on guideline-based care. This pilot study identified two problems limiting its broader use in its current format. First, a standardised pathological and surgical report would increase the reproducibility of outcomes between users and may need to be required for those that use the CDMM. Second, while highly effective in low-risk cases, the benefit of the decision aid is less clear on recommendations for intermediate-and high-risk cases and such a disclaimer might be helpful. Before implementation into clinical care, a much-larger validation study on the accuracy and effectiveness of such a tool are required.