1

Medical Dosimetrist perceptions about the efficiency and consistency of auto-contouring

during treatment planning: A Survey

Amber Coffey BS, RT(R)(T)(CT); Jose Moreno, BS, RT(T); Muath Ayyad, BS,BA; Nishele
Lenards, PhD, CMD, RT(R)(T) FAAMD; Ashley Hunzeker, MS, CMD; Matt Tobler, CMD,
RT(T), FAAMD

Medical Dosimetry Program at University of Wisconsin, La Crosse, WI

Key Words: Auto-contouring, Manual contouring, Deep learning, Atlas-based algorithm

Introduction

Advancements in computerized treatment planning systems (TPS) have been substantial,

as evidenced by the multiple tools refined to create a more sophisticated and improved plan.
Computer-aided delineation algorithms are widely available and can save time by reducing the
need for manual-delineation.1 In the last 20 years, auto-contouring capabilities have been
developed using atlas-based methods and deep learning algorithms. It is imperative that
contouring organs at risk (OAR) and target volumes be as accurate as possible to ensure the
radiation dose is targeting the tumor while minimizing dose to healthy tissue and organs.2
However, the quantity of time spent on contouring strongly depends on the experience of the
radiation oncologist and medical dosimetrist.3 As part of the medical dosimetrist’s
responsibilities, contouring is essential to the time saving and efficient workflow necessary for
treatment planning.

Although the efficiency of computer-generated structures is a key benefit in auto-

contouring, several auto-contouring algorithms vary in their individual contouring abilities. The
atlas-based approach uniquely relies on the availability of 1 or more CT series of a specific
anatomical dataset, which has been contoured by a physician following clinical guidelines.4 The
goal of this software is to perform a deformable registration of the atlas subject to transfer to the
CT series that is being contoured. Casati M, et al.4 shows that contours obtained through atlas-
based algorithms can require inconsequential to substantial editing. While some findings indicate
that atlas-based algorithms have yet to replace the human observer in contouring tasks such as
soft tissue in the pelvis region, it does allow for assessing and deploying progressively improved
 2

tools.5 There are diversified results in terms of overall efficiency with atlas contouring methods,
which depends on anatomical regions.5

The deep learning algorithm approach provides an advantage in that the software
automatically generates the most suitable model. Deep learning methods can be trained by
passing a large set of contoured images through convolution neural networks (CNN). This
process allows the algorithm for auto-contouring methods to vary in its consistency of post-
processing contouring technique. Using deep learning contouring significantly reduces the time
in which OAR are contoured.6 Deep learning-based auto segmentation can be clinically useful
when applied to contoured structures with tendencies of increased movement.

Researchers have measured differences between auto-contouring algorithms and manual

contours specific to the delineation of OAR. These studies, with quantitative and qualitative
measures, rationalize the use of auto-contouring algorithms, however, the factor of clinical
workflow was not assessed.7,8,9 While auto-contouring methods can reduce the workload, there
are concerns on whether using auto-contouring will save time if manual contouring edits are
required.7 If a shortfall in the accuracy of auto-contouring arises, necessary edits must be
contrived on every single slice, increasing the potential for time lost. Manual contouring is
known to be time consuming and has considerable variability with contouring of many structures
which may prove daunting.10 While many auto-contoured OAR and target structures still require
manual corrections to be made, researchers have showed efficient time saving ability compared
to full manual contouring.1

Auto-contouring algorithms have shown promising time saving capabilities. Zabel WJ et

al.7 measured mean durations for initial contours generated at 10.9 minutes, 1.4 minutes, and 1.2
minutes for manual, deep-learning, and atlas-based, respectively. Findings showed initial deep
learning contours were more geometrically similar to initial manual contours. Mean duration of
contour editing for manual, deep-learning, and atlas-based contours were 4.1 minutes, 4.7
minutes, and 10.2 minutes, respectively. The extent of required editing was larger for atlas-based
contours when compared with manual contouring and deep learning. Using the deep learning
algorithm for bladder and rectum contours reduced overall time spent on contouring and editing
when compared to atlas-based and manual contouring, without negatively affecting editing
times, contour geometry, or clinically relevant dose-volume metrics.
 3

While comparing manual contouring to auto-contouring, Andrianarison et al11 showed

that it took an average of 7 minutes per structure to contour manually. Looking at average time
required for contouring prostate patients while utilizing auto-contouring algorithm, a study by La
Macchia et al12 estimated an average of 41 minutes time saved from the treatment planning
process for deep learning and an average of 31 minutes for atlas-based. When clinically
evaluating contours for lung cancer between manual contouring and auto-contouring, Lustberg et
al13 found that the total median time saved using auto-contouring was 39% less for atlas-based,
and 50% for deep learning algorithms in contouring, comparing all OAR structures to manual
contouring. Chen14 demonstrated that deep learning algorithm provides accurate, consistent, and
reproducible mastication muscles without the need of any manual correction. Deep learning
algorithms validate the improvement of the consistency and efficiency of auto-contouring in the
treatment planning workflow compared to manual contouring.14

The problem is there is potential for time and efficiency to be diminished during the
treatment planning process through manual contouring methods. In addition, there is a lack of
literature regarding perceptions of auto-contouring benefits. During treatment planning,
automation of contouring can save time by generating organ contours swiftly, which can
otherwise be considered time consuming. Through continued advancements in treatment
planning, we have experienced an increase in the use of auto-contouring tools. The purpose of
this study is to investigate medical dosimetrists' perceptions of how auto-contouring software
impacts efficiency and consistency during the treatment planning process. The research questions
used to guide this study include (Q1) what are medical dosimetrists' perceptions of consistency
on auto-contouring algorithms and (Q2) what are medical dosimetrists' perceptions on treatment
planning efficiency when auto-contouring is used? Measuring both efficiency and consistency of
auto-contouring algorithms, researchers can survey the impact auto-contouring has on the
treatment planning process. This allows for an equitable comparison of auto to manual
contouring. By creating a survey, we were able to measure medical dosimetrist perceptions of
consistency and efficiency during treatment planning.

Methods and Materials

Instrumentation
 4

To gain understanding into the consistency and efficiency of auto-contouring, a 10-

question survey, using Qualtrics, was derived and modified. The structure of this survey was
based on the University of Wisconsin graduate students personal experience of contouring during
our clinical rotation, as well as the experience of medical dosimetrist at our clinical internship.

Two groups of question were used to guide this study. Participant limitations were set to
current certified medical dosimetrists (CMDs) who had previously implemented auto-contouring.
The first group of questions were directed to assess the perceptions of consistency on the auto-
contouring process. The second group of questions focused on the efficiency of auto-contouring
during treatment planning. The questionnaire was estimated to take no longer than 5 minutes to
complete. By creating a survey, researchers were able to better understand perceptions as to what
factors of auto-contouring affected users’ satisfaction with the software, potentially leading to its
implementation and further utilization.

Study Validation

An anonymous qualitative pilot survey was created in Qualtrics in order to validate the
survey instrument. The pilot study was created through several pretest recapitulations through a
variety of pretest and pilot study reviews, to appropriately format the study questions. A 10-
question pilot survey was created with survey questions categorized into 2 types. It was then
distributed to 9 medical dosimetrists within the corresponding clinics. There were 7 study
responses with feedback given for review which allowed for preparation of the final survey and
distribution. Based on the outcome of the pilot study, question 1 was altered to provide clarity.

Participant Selection & Description

The respondent population consisted of American Association of Medical Dosimetrists

(AAMD) members. Participation in the survey was limited to active CMDs, regardless of
experience to collect responses from those most involved during treatment planning. Participant
selection excluded any members that are not CMDs, working outside of the United States and are
not a member of the AAMD network.

Data Collection
 5

Certified medical dosimetrists (n= 2,598) were provided a description of our research and
survey hyperlink, through an email provided by the AAMD network. The entirety of participant
information was anonymous and confidential with implied voluntary statement of consent. The
survey was distributed through email on July 28th, 2021 by the AAMD member service manager
with the request to complete the survey by August 4th, 2021. Survey information was posted to
the AAMD Facebook page on August 2nd, 2021 requesting that CMDs check their email for the
corresponding survey, increasing the number of responses. Qualtrics has the capability to store
responses, allowing researchers to extract information for data analysis.

The survey was comprised of closed and open-ended question. Closed-ended question
combined multiple choice, likert scale, constant sum, ranking order, matrix table and 1 polar
question. The open-ended questions allowed respondents to specify on the type of auto-
contouring software used, most efficient auto-contoured region, most time saving component of
auto-contouring software and optional additional comments. A no response to the polar question
entered the respondent into the exclusion criteria of the study population.

Statistical Analysis

Data analysis was discussed with the Statistical Consulting Center (SCC) through the
University of Wisconsin La-Crosse (UWL), determining that descriptive data analysis was ideal
for this research. Descriptive data analysis was done to determine the medical dosimetrists’
perceptions of efficiency and consistency of auto-contouring. Bar graphs and a pie chart were
designed to visually display the data collected through the survey.

Results

There was a total of 2,598 emails distributed through the AAMD contact list directed
only to CMDs. Through that distribution, 1024 were opened at a rate of 39% (1024/2598). The
1024 opened emails transpired into 86 responses giving a response rate of 8.4% (86/1024). In the
total number of 86 responses, 27 stated they do not use auto-contouring software, excluding them
from the survey criteria. A total of 59 participants stated they do use auto-contouring which led
to the survey data collected.
 6

To determine the usage of auto-contouring software by brand, respondents were given the
ability to comment on which auto-contouring software they use. Determining the usage of the
auto-contouring software by brand allowed for additional data collected, especially if the auto-
contouring software brand was unavailable as an answer choice. Results conveyed the most used
auto-contouring software was MIM, which concluded 45% (33/59) of respondents represented in
(Figure 1).

The first category of questions answered research question (Q1), what are medical
dosimetrists' perceptions of consistency on auto-contouring algorithms. A verity of questions
were asked to analyze medical dosimetrists’ perceptions of auto-contouring algorithms,
including the use of auto-contouring software for generated OAR. From the combined responses,
45.28% (24/53) stated they sometimes accept contours, while 43.40% (23/53) stated sometimes
to deleting auto contours to manual contours. Evaluating the consistency of the auto-contouring
software, respondents stated 37.74% (20/53), 28.30% (15/53), 9.43% (5/53) somewhat agree,
neither agree nor disagree and strongly disagree, respectively (Figure 2). When measuring the
level of agreement regarding inconsistencies in auto-contouring software, 56.60% (30/53) stated
edits are required for auto-contours, 41.51% (22/53) somewhat agree on the inability to
differentiate Hounsfield units among various anatomic structures, while 47.17% (25/53) strongly
agree on the inability to contour streaking artifact (figure 3).

The second category of questions answered research question (Q2), what are medical
dosimetrists' perceptions on treatment planning efficiency when auto-contouring is used. CMDs
were asked to rank auto-contoured anatomical regions from least to most efficient. The results of
most efficient rankings were as follows: pelvis 34.78% (16/46), head and neck 26.09% (12/46),
thorax 28.26% (13/46) (figure 4). When asked how auto-contouring affects the time spent per
plan, 75.52% (37/49) of respondents stated auto-contouring saved time, while 24.48% (12/49)
believed it decreased time spent (figure 5). A total of 38.78% (19/49) of respondents stated auto-
contouring decreased time spent per plan by 7-15 minutes (figure 6). The component of auto-
contouring revealed to have saved the most time was consistency with a selection percent of
35.59% (21/59), followed by the accuracy of contouring OAR with 30.51% (18/59). Mean CMD
ratings of efficiency on manual-contours versus auto-contours were 60.49% and 39.51%
respectively.
 7

References

1. Aliotta E, Nourzadeh H, Choi W, Leandro Alves VG, Siebers JV. An automated

workflow to improve efficiency in radiation therapy treatment planning by prioritizing
organs at risk. Adv Radiat Oncol. 2020;5(6):1324-1333.
https://doi.org/10.1016/l.adro.2020.06.012
2. Vaassen F, Hazelaar C, Vaniqui A, et al. Evaluation of measures for assessing timesaving
of automatic organ-at-risk segmentation in radiotherapy.  Phys Imaging Radiat Oncol.
2020;13(P1-6):1-6. https://doi.org/10.1016/j.phro.2019.12.001
3. Wu X, Udupa JK, Tong Y, et al. AAR-RT - A system for auto-contouring organs at risk
on CT images for radiation therapy planning: Principles, design, and large-scale
evaluation on head-and-neck and thoracic cancer cases. Med Image Anal. 2019;54:45-62.
https://doi.org/10.1016/j.media.2019.01.008
4. Casati M, Piffer S, Calusi S, et al. Methodological approach to create an atlas using a
commercial auto-contouring software. J Appl Clin Med Phys. 2020;21(12):219-230.
https://doi.org/10.1002/acm2.13093
5. Jackson P, Kron T, Hardcastle N. A future of automated image contouring with machine
learning in radiation therapy. J Med Radiat Sci. 2019;66(4):223-225.
https://doi.org/10.1002/jmrs.365
6. Ahn SH, Yeo AU, Kim KH, et al. Comparative clinical evaluation of atlas and deep-
learning-based auto-segmentation of organ structures in liver cancer. Radiat Oncol.
2019;14(1):213. https://doi.org/10.1186/s13014-019-1392-z
7. Zabel WJ, Conway JL, Gladwish A, et al. Clinical evaluation of deep learning and atlas-
based auto-contouring of bladder and rectum for prostate radiation therapy. Pract Radiat
Oncol. 2021;11(1):e80-e89. https://doi.org/10.1016/j.prro.2020.05.013
8. Kazemifar S, Balagopal A, Nguyen D, et al. Segmentation of the prostate and organs at
risk in male pelvic CT images using deep learning. Biomed Phys Eng Express.
2018;4:055003. https://doi.org/10.1088/2057-1976/aad100
9. Delpon G, Escande A, Ruef T, et al. Comparison of automated atlas-based segmentation
software for postoperative prostate cancer radiotherapy. Front Oncol. 2016;6:178.
https://doi.org/10.3389/fonc.2016.00178
 8

10. McCarroll RE, Beadle BM, Balter PA, et al. Retrospective validation and clinical
implementation of automated contouring of organs at risk in the head and neck: A step
toward automated radiation treatment planning for low- and middle-income countries. J
of Glob Oncol. 2018;(4):1-11. https://doi.org/10.1200/jgo.18.00055
11. Andrianarison VA, Laouiti M, Fargier-Bochaton O, et al. Contouring workload in
adjuvant breast cancer radiotherapy. Cancer Radiother. 2018;22(8):747-753.
https://doi.org/10.1016/j.canrad.2018.01.008
12. La Macchia M, Fellin F, Amichetti M, et al. Systematic evaluation of three different
commercial software solutions for automatic segmentation for adaptive therapy in head-
and-neck, prostate, and pleural cancer. Radiat Oncol. 2012;7:160.
https://doi.org/10.1186/1748-717X-7-160
13. Lustberg T, van Soest J, Gooding M, et al. Clinical evaluation of atlas and deep learning
based automatic contouring for lung cancer. Radiother Oncol. 2018;126(2):312-317.
https://doi.org/10.1016/j.radonc.2017.11.012
14. Chen, W., Li, Y., Dyer, B.A. et al. Deep learning vs. atlas-based models for fast auto-
segmentation of the masticatory muscles on head and neck CT images. Radiat Oncol.
2020;15:176. https://doi.org/10.1186/s13014-020-01617-0
 9

Figures

Auto-contouring Software
3; 4%
3; 4%
5; 7%

4; 5%
33; 45%

11; 15%

5; 7%
9; 12%

MiM Smart Segmentaiton/Eclipse

Mirada Radformation/Clear Check
Pinnacle Raystation
Limbus In house

Figure 1. Brand of auto-contouring software used.

Demonstrates Consistency

Strongly agree 5
Level of Agreement

Somewhat agree 10

Neither agree or disagree 15

Somewhat disagree 10

Stongly disagree 5

0 2 4 6 8 10 12 14 16

Number of Responses

Figure 2. Level of agreement on the consistency of auto-contouring.

 10

Inconsistencies in auto-contouring
30
30 25
22
Number of Responses

25 20
20 16
14
15 12
10
10 5
3
5 0 0 1 1 0
0
Strongly Somewhat Neither agree Somewhat Strongly agree
disagree disagree nor disagree agree
Level of Agreement

Edits are required Inability to differentiate Hounsfield Units

Inability to contour streaking artifacts

Figure 3. Level of agreement with inconsistencies in auto-contouring software.

Anatomical Regions Rank Most (1) to Least (5)

42
45
40
Number of Responses

35
30
22
25
18 18 17
20 16
12 13 12
15 10
8 8 8 8
10 5
2 3 1
3 3
5 0 0 0 0 1
0
1 2 3 4 5

Level of agreement
Head+Neck Pelvis Thorax
Abdomen Other- Specifiy

Figure 4. Auto-contouring rankings of anatomical regions.

 11

Time spent per plan

19
20
18
16
Number of Responses

14
12 10
10 8
8 6
5
6
4
1
2
0
Decreases Decreases Decreases Increases time Increases time Increases time
time < 6 time 7-15 time 16-24 < 6 minutes 7-15 minutes 16-24 minutes
minutes minutes minutes

Level of Agreement

Figure 5. Auto-contouring's effect on time per plan.

Which component saves you the most time?

Accuracy of contouring organs 18

Time saving tool

Consistency of auto-generated contours 21

No specific components save time 14

Other- Specify 6

0 5 10 15 20 25

Number of Responses

Figure 6. Time saving components of auto-contouring.