1

A survey of medical dosimetrists’ perceptions of efficiency and consistency of auto-

contouring software

Amber Coffey BS, RT(R)(T)(CT); Jose Moreno, BS, RT(T); Nishele Lenards, PhD, CMD,
RT(R)(T) FAAMD; Ashley Hunzeker, MS, CMD; Matt Tobler, CMD, RT(T), FAAMD

Medical Dosimetry Program at University of Wisconsin, La Crosse, WI

Abstract
Although auto-contouring methods were created to reduce the workload for the radiation
oncology team, concern lies in whether auto-contouring can improve efficiency regarding
generated contours of a treatment plan. Researchers have measured differences between auto-
contouring algorithms and manual contour methods specific to the contouring of organs at risk
(OAR). The problem lies in the paucity of literature specific to perceptions of auto-contouring
and the impact on workflow efficiency. The purpose of this study was to measure medical
dosimetrists’ perceptions of how auto-contouring software impacts the treatment planning
process. To measure perceptions, researchers surveyed medical dosimetrists about their
perspectives on consistency and efficiency of auto-contouring during treatment planning. A
Qualtrics survey was created based on the 2 research questions in this study. The survey was
distributed through email to 2,598 full members of the American Association of Medical
Dosimetrist (AAMD) who were certified by the MDCB; mostly medical dosimetrists but also
included a small group of medical physicists. The email open rate was 39% (1024/2598) but the
response rate for those who read the email was only 8.4% (86/1024). Of the survey respondents,
67% (59/86) used auto-contouring software; thus, eligible to complete the remainder of the
survey. A majority of participants agreed that auto-contouring software decreases time spent
contouring per patient; however, most agreed that manual contouring is more efficient.
Therefore, it was inferred that a combination of both auto and manual contouring have an impact
on workload efficiency.
Key Words: Anatomical contouring, workload efficiency, algorithm, deep learning

Introduction
Advancements in computerized treatment planning systems (TPSs) have been substantial,
as evidenced by the multiple tools refined to create a more sophisticated and optimal plan.
Aliotta et al1 indicated that computer-aided contouring algorithms are widely available and can
 2

save time by reducing the need for manual-contouring.1 In the last 20 years, auto-contouring
capabilities have developed using atlas-based methods and deep learning algorithms. It is
imperative that contouring organs at risk (OAR) and target volumes be as accurate as possible to
ensure the radiation dose is delivered to the tumor while minimizing dose to healthy tissue and
organs.2 However, the quantity of time spent on contouring strongly depends on the experience
of the radiation oncologist and medical dosimetrist.3 As part of the medical dosimetrist’s
responsibilities, contouring is an essential task necessary for treatment planning.
Although the efficiency of computer-generated auto-contours may appear beneficial,
several algorithms vary in capability depending on anatomical areas. The atlas-based approach
uniquely relies on the availability of 1 or more CT series of a specific anatomical dataset, which
has been contoured by a physician following clinical guidelines.4 The goal of this software is to
perform a deformable registration of the atlas subject to transfer the contoured CT series. Casati
et al4 showed that contours obtained through atlas-based algorithms can require inconsequential
to substantial editing. While some findings are clear in stating that atlas-based algorithms have
yet to replace the human observer in contouring tasks, these findings also suggest improved
algorithms have become available to work in conjuncture with medical dosimetrists.5
The deep learning algorithms provide an advantage in which the software automatically
generates the most suitable model. Deep learning methods can be trained by passing a large set
of contoured images through convolution neural networks (CNN). This process allows the auto-
contouring algorithm to vary in consistency of the post-processing contouring technique. Ahn et
al6 suggests using deep learning contouring significantly reduces the time in which OAR are
contoured. Deep learning-based auto segmentation can be clinically useful when applied to
contoured structures with tendencies of increased movement.
Researchers measured differences between auto-contouring algorithms and manual
contours specific to contouring of OAR. These studies with quantitative, qualitative, and clinical
workflow measures rationalize the use of auto-contouring algorithms.7,8,9 Studies were conducted
between manual and auto-contours to validate performance of auto-contouring algorithms;
however, these studies do not access the availability of auto-contouring software for medical
dosimetrists. 7,8,9 Kazemifar et al7 suggested that while auto-contouring methods can reduce the
workload, there are concerns of whether using auto-contouring will be efficient if manual
contouring edits are required. Anatomical regions can have an impact when evaluating efficiency
 3

of workflow when utilizing auto-contouring. The consistency can also vary upon the algorithm
utilized for auto-contouring. Aliotta et al1 stated that the accuracy of OAR contouring can vary
from case to case depending on patient-specific factors. If a shortfall in the accuracy of auto-
contouring arises, necessary edits must be made and potentially decreased efficiency. Manual
contouring is known to be time consuming which may prove daunting.10
While many auto-contoured OAR and target structures still require manual corrections to
be made, researchers have shown efficient time saving ability compared to full manual
contouring.1 Zabel et al7 measured mean durations for initial contours generated at 10.9 minutes,
1.4 minutes, and 1.2 minutes for manual, deep-learning, and atlas-based, respectively.
Researchers showed initial deep learning contours were more geometrically similar to initial
manual contours. Mean duration of contour editing for manual, deep-learning, and atlas-based
contours were 4.1 minutes, 4.7 minutes, and 10.2 minutes, respectively. The extent of required
editing was larger for atlas-based contours when compared with manual contouring and deep
learning. Using the deep learning algorithm for bladder and rectum contours reduced overall time
spent on contouring and editing when compared to atlas-based and manual contouring, without
negatively affecting editing times, contour geometry, or clinically relevant dose-volume metrics.
Andrianarison et al11 showed that, on average, 7 minutes of time per structure was needed
to contour manually. Looking at average time required for contouring prostate patients while
utilizing auto-contouring algorithm, La Macchia et al12 estimated an average of 41 minutes time
saved from the treatment planning process for deep learning and an average of 31 minutes for
atlas-based. When clinically evaluating contours for lung cancer between manual contouring and
auto-contouring, Lustberg et al13 found that the total median time saved using auto-contouring
was 39% less for atlas-based, and 50% for deep learning algorithms in contouring, comparing all
OAR structures to manual contouring. Chen14 demonstrated that deep learning algorithm
provides accurate, consistent, and reproducible mastication muscles without the need of any
manual correction. Deep learning algorithms validate the improvement of the consistency and
efficiency of auto-contouring in the treatment planning workflow compared to manual
contouring.14
While research has shown that efficiency often decreases through manual contouring
methods, the problem is the paucity of literature specific to perceptions of auto-contouring and
the impact on workflow efficiency. Therefore, the purpose of this study was to investigate
 4

medical dosimetrists' perceptions of how auto-contouring software impacts efficiency and

consistency during the treatment planning process. The research questions used to guide this
study included: (Q1) what medical dosimetrists' perceptions about the consistency of auto-
contouring algorithms are and (Q2) what are medical dosimetrists' perceptions about the
efficiency of treatment planning workflow when auto-contouring is used.

Methods and Materials

Instrumentation
To gain understanding of the perceived consistency and efficiency of auto-contouring, a
10-question Qualtrics survey was derived and modified. The structure of this survey was based
on the University of Wisconsin graduate students' personal experience of contouring during
clinical rotation, as well as the experience of medical dosimetrists at the clinical facilities. Two
categories of questions were used to guide the survey based on the 2 research study questions
and were comprised of closed and open-ended questions. The first category of questions was
specific to perceptions of consistency during the auto-contouring process. The second category
of questions was specific to efficiency of auto-contouring. The questionnaire was estimated to
take < 5 minutes to complete.
Survey Validation
A pilot survey was distributed to assist in validation of the survey tool. The pilot survey
was distributed to 9 medical dosimetrists within 3 clinics. Based on the outcome of the pilot
survey, question 1 was altered to provide clarity prior to the final survey distribution.
Population & Survey Distribution
The participant population consisted of active, full members who were certified by the
MDCB (n=2,598); mostly medical dosimetrists but also included a small group of medical
physicists within the American Association of Medical Dosimetrists (AAMD). Participant
selection excluded any members that were working outside of the United States, not certified,
and not a member of the AAMD. The survey was distributed via email on July 28th, 2021 by the
AAMD member service manager with the request to complete the survey by August 4th, 2021.
The participants were provided a description of the research with the survey hyperlink for
completion and the researchers contact information. Participants were informed of anonymity
and confidentiality with an implied voluntary statement of consent if they clicked on the survey
link.
 5

Data Collection & Analyses

Survey responses were collected and stored in the Qualtrics survey tool and extracted for
analysis. There was an email open rate of 39% (1024/2598). Of the 1024 opened emails, only
8.4% (86/1024) completed the survey. Furthermore, of the 86 participants, 27 stated they do not
use auto-contouring software which excluded them from the remainder of the survey. A total of
59 participants stated they used auto-contouring and therefore completed the survey.
Descriptive statistics were used to determine participant perceptions of efficiency and
consistency of auto-contouring. Qualtrics reporting algorithm was used for frequency
distribution reflecting the count of participants responses. The algorithm was used to check for
overall response of the questions. Inferential statistics provided the population-based data with
95% confidence level.
Results
The first demographic question was targeted at identifying vendor-specific auto-
contouring software being used by participants. Forty-five percent of respondents (33/59) used
MIM auto-contouring software (Figure 1). The first category of survey questions was used to
answer research (Q1): what are medical dosimetrists' perceptions about the consistency of auto-
contouring algorithms? Participants were asked about the use of auto-contouring software for
generated OAR. From the combined responses, 45.28% (24/53) stated they sometimes accept
contours, while 43.40% (23/53) stated they sometimes delete auto-contours in order to manually
contour the structures. Participants were asked whether contours generated using auto-contouring
software were consistent and results showed 37.74% (20/53) somewhat agree, 28.30% (15/53)
were neutral, and 9.43% (5/53) strongly disagreed (Figure 2). When measuring the level of
agreement regarding inconsistencies in auto-contouring software, 56.60% (30/53) stated edits
were required for auto-contours, 41.51% (22/53) stated they somewhat agreed on the inability to
differentiate Hounsfield units among various anatomic structures, while 47.17% (25/53) strongly
agreed on the inability to contour streaking artifact (Figure 3).
The second category of survey questions were used to answer research (Q2): What are
medical dosimetrists' perceptions about the efficiency of treatment planning workflow when
auto-contouring is used? Participants were asked to rank auto-contoured anatomical regions from
most efficient to least efficient. The most efficient auto-contoured anatomical regions were
pelvis 34.78% (16/46), head and neck 26.09% (12/46), thorax 28.26% (13/46), and abdomen
 6

21.74% (10/46) (Figure 4). Participants were asked how auto-contouring affected time spent per
plan and 75.52% (37/49) agreed that auto-contouring saved time (Figure 5). More specifically,
38.78% (19/49) thought auto-contouring decreased time spent per plan by 7-15 minutes (Figure
6). Participants were asked about the auto-contouring components that equated to the most time
savings, and 35.59% (21/59) chose consistency of auto-contours while 30.51% (18/59) chose
accuracy of contouring OAR. The mean responses about efficiency of manual-contours versus
auto-contours were 60.49% and 39.51%, respectively.
Discussion
In response to question (Q1) what are medical dosimetrists' perceptions about the
consistency of auto-contouring algorithms, the proportion who frequently edited contours was
statistically greater than those who accepted or deleted contours. The results show a threshold for
the amount of editing required before a participant will completely delete the auto-contour and
manually contour. A respondent stated, “auto contouring is a wonderful tool, but not perfect in
which most edits are needed, but depends on area of the body.” Most medical dosimetrists
agreed that auto-contouring software demonstrated consistency. However, factors deemed not
ideal included the need to edit contours, CT scans with artifacts, and grouping of close structures.
The overall accuracy of the auto-contouring seems to be site specific with anatomical regions
which requires minor edits. These responses coincided with results by Jackson et al5 about how
auto-contouring software is yet to replace the human observer.
In response to question (Q2) what are medical dosimetrists' perceptions about the
efficiency of treatment planning workflow when auto-contouring is used, the most notable result
was that auto-contouring decreased time spent per plan through efficient contours. Even though
auto-contouring can save time, the accuracy of the contour is imperative. Data collection implied
that the accuracy of OAR contours and the consistency of generated auto-contours create a more
efficient workflow. A respondent's perception about auto-contouring software was that it “gives
a good starting point but [medical dosimetrists] always have to edit contours, but actually saves
hours of time.” While the study by Lustberg et al13 confirms auto-contouring software is a good
starting point for contours of OAR, this allows for a considerable time gain with certain
anatomy. While participants rated auto-contouring as an efficient method, the mean population
of respondents rated manual contouring as most efficient which indicated that medical
dosimetrists trust their ability to contour but still utilize the auto-contouring software to help
 7

generate the bulk of contours. La Macchia et al12 and Lustberg et al13 showed that the automatic
contouring time with manual corrections was less than manual contouring time alone. The
findings of the current research survey showed that use of auto-contouring software varied in
efficiency depending on the anatomical location. The top 2 anatomical locations rated for
efficiency were the pelvis and thorax.
Conclusion
Advances in technology can help aid medical dosimetrists with workflow efficiency. The
purpose of this study was to measure medical dosimetrists’ perceptions of how auto-contouring
software impacts efficiency and consistency during the treatment planning process. Medical
dosimetrists’ perceptions of auto-contouring software can vary depending on the anatomical
location being contoured; however, a majority expressed that auto-contouring is a tool that aids
in generating contours, but contours would still need reviewed and edited before planning.
Furthermore, while most medical dosimetrists agreed that auto-contouring is consistent and
efficient, their perception is that manual contouring is most efficient. The results infer that the
combination of both auto and manual contouring is the preferred method among medical
dosimetrists using auto-contouring.
There were limitations in this study which should be noted. Based on the population used,
the response rate of the survey was low and not ideal. Further research with an increase in
participants is necessary. Additionally, demographic factors such as age and experience were not
assessed and could have provided more insight regarding medical dosimetrists favorability of
auto-contouring.
 8

Acknowledgements
The authors would like to thank Dr. Sherwin Toribio and the University of Wisconsin-La Crosse
Statistical Consulting Center for their guidance with data analysis and display of statistical
results for the study. However, any errors in statistics or interpretation of data are the sole
responsibility of the authors.
 9

References
1. Aliotta E, Nourzadeh H, Choi W, Leandro Alves VG, Siebers JV. An automated workflow to
improve efficiency in radiation therapy treatment planning by prioritizing organs at risk. Adv
Radiat Oncol. 2020;5(6):1324-1333. https://doi.org/10.1016/l.adro.2020.06.012
2. Vaassen F, Hazelaar C, Vaniqui A, et al. Evaluation of measures for assessing timesaving of
automatic organ-at-risk segmentation in radiotherapy.  Phys Imaging Radiat Oncol.
2020;13(P1-6):1-6. https://doi.org/10.1016/j.phro.2019.12.001
3. Wu X, Udupa JK, Tong Y, et al. AAR-RT - A system for auto-contouring organs at risk on
CT images for radiation therapy planning: Principles, design, and large-scale evaluation on
head-and-neck and thoracic cancer cases. Med Image Anal. 2019; 54:45-62.
https://doi.org/10.1016/j.media.2019.01.008
4. Casati M, Piffer S, Calusi S, et al. Methodological approach to create an atlas using a
commercial auto-contouring software. J Appl Clin Med Phys. 2020;21(12):219-230.
https://doi.org/10.1002/acm2.13093
5. Jackson P, Kron T, Hardcastle N. A future of automated image contouring with machine
learning in radiation therapy. J Med Radiat Sci. 2019;66(4):223-225.
https://doi.org/10.1002/jmrs.365
6. Ahn SH, Yeo AU, Kim KH, et al. Comparative clinical evaluation of atlas and deep-learning-
based auto-segmentation of organ structures in liver cancer. Radiat Oncol. 2019;14(1):213.
https://doi.org/10.1186/s13014-019-1392-z
7. Zabel WJ, Conway JL, Gladwish A, et al. Clinical evaluation of deep learning and atlas-
based auto-contouring of bladder and rectum for prostate radiation therapy. Pract Radiat
Oncol. 2021;11(1): e80-e89. https://doi.org/10.1016/j.prro.2020.05.013
8. Kazemifar S, Balagopal A, Nguyen D, et al. Segmentation of the prostate and organs at risk
in male pelvic CT images using deep learning. Biomed Phys Eng Express. 2018;4:055003.
https://doi.org/10.1088/2057-1976/aad100
9. Delpon G, Escande A, Ruef T, et al. Comparison of automated atlas-based segmentation
software for postoperative prostate cancer radiotherapy. Front Oncol. 2016;6:178.
https://doi.org/10.3389/fonc.2016.00178
10. McCarroll RE, Beadle BM, Balter PA, et al. Retrospective validation and clinical
implementation of automated contouring of organs at risk in the head and neck: A step
 10

toward automated radiation treatment planning for low- and middle-income countries. J of
Glob Oncol. 2018;(4):1-11. https://doi.org/10.1200/jgo.18.00055
11. Andrianarison VA, Laouiti M, Fargier-Bochaton O, et al. Contouring workload in adjuvant
breast cancer radiotherapy. Cancer Radiother. 2018;22(8):747-753.
https://doi.org/10.1016/j.canrad.2018.01.008
12. La Macchia M, Fellin F, Amichetti M, et al. Systematic evaluation of three different
commercial software solutions for automatic segmentation for adaptive therapy in head-and-
neck, prostate, and pleural cancer. Radiat Oncol. 2012;7:160. https://doi.org/10.1186/1748-
717X-7-160
13. Lustberg T, van Soest J, Gooding M, et al. Clinical evaluation of atlas and deep learning
based automatic contouring for lung cancer. Radiother Oncol. 2018;126(2):312-317.
https://doi.org/10.1016/j.radonc.2017.11.012
14. Chen, W., Li, Y., Dyer, B.A. et al. Deep learning vs. atlas-based models for fast auto-
segmentation of the masticatory muscles on head and neck CT images. Radiat Oncol.
2020;15:176. https://doi.org/10.1186/s13014-020-01617-0
 11

Figures

Auto-contouring software
3; 4% 3; 4%
5; 7%
4; 5%
33; 45%

11; 15%

5; 7%
9; 12%

MiM Smart Segmentaiton/Eclipse Mirada

Radformation/Clear Check Pinnacle Raystation
Limbus In house

Figure 1. Participants selected the vendor of auto-contouring software used.

Demonstrates Consistency

Strongly agree 5
Level of Agreement

Somewhat agree 10

Neither agree or disagree 15

Somewhat disagree 10

Stongly disagree 5

0 2 4 6 8 10 12 14 16

Number of Responses

Figure 2. Participant's level of agreement on consistency of auto-contouring.

 12

Number of Responses Inconsistencies in auto-contouring 30

25
30 2022
16 14
20 12 10
10 0 0 1 3 5 1 0
0
ee ee ee ee ee
agr a gr a gr agr agr
s s s at
di t di r di h gl
y
ly a o w n
ng h e
e n of oAgreement ro
tro ew Level
re m St
S m g S
So era
h
eit required
Edits Nare
Inability to differentiate Hounsfield Units
Inability to contour streaking artifacts

Figure 3. Participant's level of agreement with inconsistencies of auto-contouring software.

Anatomical Regions Rank Most (1) to Least (5)

42
45
40
Number of Responses

35
30
22
25
18 18 17
20 16
12 13 12
15 10
8 8 8 8
10 5
2 3 1
3 3
5 0 0 0 0 1
0
1 2 3 4 5

Level of agreement
Head+Neck Pelvis Thorax
Abdomen Other- Specifiy

Figure 4. Participant ranking of most to least efficient auto-contouring per anatomical region.
 13

Auto-contouring saved time or spent

Decrea extra time per plan
Increas Increas ses
es time es time time <
7-15 16-24 6
minutesminute minutes
Increas
;5
es time s; 1 ;8
<6
minutes
;6
Decrea Decrea
ses ses
time time 7-
16-24 15
minutes minutes
; 10 ; 19
Decreases time < 6 minutes Decreases time 7-15 minutes
Decreases time 16-24 minutes Increases time < 6 minutes
Increases time 7-15 minutes Increases time 16-24 minutes

Figure 5. Participant responses about the effect of auto-contouring on time spent per plan.

Which component saves you the most time?

Accuracy of contouring organs 18

Time saving tool

Consistency of auto-generated contours 21

No specific components save time 14

Other- Specify 6

0 5 10 15 20 25

Number of Responses

Figure 6. Participant perception about time saving components of auto-contouring software.

 14

Appendix A

Survey Introduction

You are invited to participate in a research study titled Medical Dosimetrists Perceptions about
the Efficiency and Consistency of Auto-contouring during Treatment Planning.
The purpose of this study is to investigate medical dosimetrists’ perceptions of how auto-
contouring software impacts efficiency and consistency during the treatment planning process.
The survey should take < 5 minutes to complete. Participation in this study is voluntary. If you
choose to participate, you can withdraw at any time. All collected responses are anonymous and
confidential.
By clicking on the survey link below, you are consenting to participate in this study. Please
complete the survey by July 30, 2021.
[https://uwlax.ca1.qualtrics.com/jfe/form/SV_50CpqzxYXby8qma]
If you have any questions or concerns regarding the research study, please contact research
members of the study.

Amber Coffey coffey6576@uwlax.edu

Jose Moreno moreno7524@uwlax.edu
Muath Ayyad ayyad1870@uwlax.edu
Advisor Nishele Lenards nlenards@uwlax.edu
 15

Appendix B
Questions and Answers to medical dosimetrist perceptions about the efficiency and consistency
of auto-contouring during treatment planning survey

The title of our study is: Medical Dosimetrists Perceptions about Efficiency & Consistency of
Auto-Contouring during Treatment Planning.

The survey consists of up to 10 questions and should take < 5 minutes to complete.
Your participation is appreciated!

1. Do you use auto-contouring software? Yes

No

2. Which auto-contouring software do MIM

you use? Smart Segmentation
Mirada
Raformation (Clear Check)
Other- Please Specify

3. Which of the following do you Accepting contours

Editing contours
perform while using auto-contouring
Deleting auto-contours to manual contour
software for Organs at Risk (OAR)?

4. State your level of agreement.The Strongly agree

auto-contouring software being used d Somewhat agree
emonstrates consistency Neither agree nor disagree
(e.g., reliability, regularity, dependabil Somewhat disagree
ity). Strongly disagree
Edits are required for auto-contours
 16

5. State your level of agreement Edits are required for auto-contours

Inability to differentiate Hounsfield units
regarding inconsistencies in auto-
among various anatomic structures
contouring software. Inability to contour streaking artifacts

6. Order the anatomical regions based on Head and Neck

Pelvis
efficiency when using auto-contouring
Thorax
software. Abdomen
Other- Please Specify

7. How does auto-contouring software Decreases time < 6 minutes

Decreases time7-15 minutes
affect the time spent per plan?
Decreases time 16-24 minutes
Increase time < 6 minutes
Increases time 7-15 minutes
Increase time 16-24 minutes

8. Which component of auto-contouring The accuracy of contouring organs at risk

The consistency of auto-generated contours
saves you the most time?
No specific components save time while auto-
contouring
Other, please explain

9. Please rate the efficiency of both Manual contouring

Auto Contouring
contouring methods.

10. Optional: provide additional

comments about auto-contouring