Neal Kelley's Outside Review of Riverside County's Election Office

ROV Operational Assessment Report August 2023
months so that we could use the new system for the November 2019 Election in order to
gain experience before the Presidential Primary in March 2020. In March 2020 the
COVID pandemic hit and drastically changed all of our procedures leading up to the
November 2020 Election. In 2022 we officially implemented the Voters Choice Act which
requires an Election Administration Plan, public consultation meetings and public
hearings. When you add all of the staff vacancies on top of these unforeseen obstacles
and hectic election cycle, there is absolutely no time for cross training and broad projects.
During the period of my review there were very few staff members that were
consistently on-site3. It was reported that many of the staff members rarely see or
interact with the management team. In an environment where staff taking time off to
rest, and recharge is essential the ROV must prioritize documentation and cross-
training without disrupting normal daily tasks.
ROV Response – The Registrar of Voters is currently operating on a 50% hybrid

operating schedule. Staff are required to come into the office for 50% of their schedule
and are allowed to telecommute for the other 50%. Wednesdays are a mandatory staff
day when everyone is expected to be in the office. The telecommute schedule is only
allowed when there is proper coverage for office functions, especially front counter
customer service. This schedule has only been allowed during 2023 when we have
not had the usual scheduled elections. During regular election cycles we are usually
working 12-hour days, 7 days a week, for 3 months at a time. All staff are in the office
during regular election cycles.
I disagree with the following statement “It was reported that many of the staff members
rarely see or interact with the management team.” The management team is also
allowed to telecommute for 50% of their schedule. All of the management team is in
the office on Wednesdays. The management team has even rotated their
telecommuting days to make sure that staff have the opportunity to interact with
management each week. I attend 4 separate staff meetings every week to make
myself available for questions.
Ultimately, however, if there is a commitment to continual and sustained improvement

expectations should be set early on related to overall engagement and a strategic focus
on improvement projects.
ROV Response – Prior to 2023 the election cycle did not allow for a lot of strategic
planning. In February 2023 the management team held a strategic planning session
and most of 2023 was spent implementing improvement projects.
A common refrain was frustration with the lack of employee feedback or appraisals. It
was reported that many employees have not had an appraisal since they were hired,
and in some cases, this was for a number of years4. While the merit system does not
appear to be tied to appraisals, this is an important feedback loop that should be treated
as critical for meaningful employee engagement.
ROV Response – I agree that employee evaluations have not been performed on a
timely basis. The majority of the staff evaluations fall under the responsibility of the
9
Chief Deputy. Unfortunately, we have had multiple vacancies in the Chief Deputy
positions over the last few years. The vacancies in management positions have
resulted in delays in staff evaluations. Management has made staff evaluations a
priority in 2023.
1 Source: Based on personal interviews of ROV personnel and an examination of the process
2 Source: Based on personal interviews of ROV personnel
3 Source: ROV various 2023 staff schedules
4 Source: Based on personal interviews of ROV personnel
10
position and sent an email explain why the position needed to be in the ROV. Juan
finally agreed to fill the position at a higher level and use a recruitment firm. We worked
with HR to exchange the position and obtain a recruitment firm. The position is
currently open for recruitment.
Recommendations
Recommendation 1: Create a culture of continual improvement.

This should be the stimulus for meaningful change in the organization and must be
more than a cluster of improvement projects. When done correctly this will help to
orchestrate effective leadership behavior, improvement methods and create an
environment for lean practices. This will require a lengthy commitment but a plan for a
department wide culture shift should be one of the first priorities.
ROV Response – I agree with his recommendation but it implies that the Registrar of
Voters has not previously engaged in continual improvement. Every year the
management staff meet and discuss major projects that need to be achieved for the
year. Please see Attachment C for a list of those projects.
Recommendation 2: Develop a learning academy that combines SME education

with outside training support for staff development and cross-training.
Educational classes and training curriculum should be created and offered year-round.
These could include classes on critical tasks within the operation as well as personal
development and leadership classes. This will not only support a continual
improvement atmosphere but will aid in staff retention. It is commendable that the ROV
participates in outside election training, such as the nationally known CERA program.
However, there is a clear need for additional training that goes beyond these programs.
ROV Response – I agree with this recommendation but I think it is important to specify all
of the training opportunities that are currently offered to staff. Please see Attachment D for
a complete list of training that is offered to staff.
Recommendation 3: Partner with outside organizations to showcase similar

industries and set benchmarking goals.
During periods between elections staff should be regularly engaged to aid in personal
development and to provide outside perspective from similar organizations. By
benchmarking off similar industries inspiration can take shape in surprising ways.
ROV Response – I agree with this recommendation but as previously mentioned I think
it is important to clarify and highlight all of the training opportunities currently provided.
5 Although it was out of my scope, I did receive negative feedback regarding the pay for line staff. There is broad
concern over the rate of pay as compared to the Temporary Assignment Program (TAP) employees. I acknowledge
that there are several variables related to these issues, but thought it was important to mention the concern.
12
It became clear during my analysis that many aspects of the operation funnel into a
single point where final action takes place: the ROV personally. This includes critical
and mundane documents needing approval, key decisions that need to be made,
interfacing, communicating, and planning with vendors, questions from observation
groups, software tasks and other aspects to the general flow of the operation7.
ROV Response – I agree that many aspects of the operation funnel to me for final
approval. The reason for this is that 50% of our staff are new to their position and are
still in training. Attachment A contains a list of all of the most recent vacancies and
hires.
On the one hand this is understandable to a degree. As an example, errors in elections

often become highly scrutinized due to the importance of the process and the scale that
elections operate under. Naturally there is a desire to inject quality control into the
process to reduce errors. On the other hand, this should not be done using a single
point of failure, or absent an extensive quality control process. In addition, in an
organization the size of the ROV it is not practical and produces a byproduct of
inefficiencies.
6A review of similar positions in San Diego County, Orange County and Santa Barbara County indicated a gap of up to
$40,000 when compared with the Riverside County position; I also address the concern among line staff related to their
pay in footnote 5
14
that staff member resigned in April 2022 to become a stay at home mom. During the
November 2022 Election we had no staff trained on the process so I had to do it myself.
The role of the Registrar of Voter should involve managing the entire office, including:
• A high-level focus on long-term strategic planning

• Assessing and providing feedback on proposed legislation and its potential
impact on Riverside County and its voters
• Providing and implementing a broad vision
• Improving transparency through communication and community outreach
• Identifying and adopting best practices as used by other jurisdictions
• Managing legal compliance and budgeting
• Organizational risk management across all aspects of the operation
Instead of focusing on these executive level tasks the ROV is currently burdened with
the minutiae of daily office operations due to a variety of factors. This has morphed into
a process of handling multiple tasks and creating unnecessary roadblocks. Ultimately
9 Source: Interviews with the ROV and a review of previously reported or identified errors
10 Source: Interviews with the ROV
11 See Appendix A
16
ISO 9001 sets out the criteria for a quality management system and it can be used by any
organization. This standard is based on several quality management principles including:
• Strong leadership
• Engagement of people
• Process approach
• Quality assurance
• Evidence-based decision making
There is a lack of any organized quality assurance programs currently in place at the
ROV. This creates barriers to many processes and introduces higher levels of risk into
the organization.
12 See International Organization for Standardization at iso.org
18
e-minus (countdown) calendars, standard operating procedures (SOPs), policies,

checklists, and subproject planning documents. The ROV does utilize global calendars
that are referenced during operational meetings. However, there does not appear to be
16 Source: Based on personal interviews of ROV personnel and an examination of existing documents
20
There appears to be a desire among leadership within the ROV to incorporate detailed
project planning into the operations. These are critical, in my opinion, to help create
barricades to potential problems long before they emerge.
22
limited even though it is a high-risk operation. The ROV is using a handful of checklists
for use during vote center operations. However, checklists should be created for all
critical tasks both internal and in the field. There is a substantial time commitment to
fully roll out a robust checklist program, but the dividends pay off with reduced risks.
ROV Response – I agree with this recommendation. As previously mentioned and

contained in Attachment E, we use checklists through out our operations and find them
helpful.
18 Source: Harvard Business Review, November 2017
24
20 See Appendix C for list of documentation
26
21See Caltech voter survey data from Orange County election monitoring: https://monitoringtheelection.us
22The ROV lost a recently recruited individual in the position of Senior Public Information Specialist; the position
remains vacant as of this writing
28
election website to remain off of the county’s central website requirements will allow
additional design opportunities that may be unique to the election workspace.
ROV Response – I do not feel strongly one way or the other about moving the website
to a different platform. I think the new website design is a huge improvement over the
old website design but of course we can continue to work on improvements.
Recommendation 7: Capture data at all levels of the operation and display for
public and internal consumption.
We are in an era of election administration where document requests under the
California Public Records Act (PRAs) and federal Freedom of Information Act (FOIA)
have grown exponentially. In addition, there is a thirst for increased transparency,
education, information, and auditing. Pressure can be eased in each of these
categories by identifying the most requested information and building tools to capture
and display the data. A plan should include any publicly available information and build
tools to transmit or display the data in real-time.
ROV Response - I agree with this recommendation.
30
conduit for critical feedback to operational planning.
ROV Response – I agree with this recommendation.
23Source: Election Advisory Committee Workplan Proposal for formation from Executive Office and ROV, March 29,
2022 Riverside County Board of Supervisors’ agenda.
32
There are several recommendations that should be considered in order to improve the
process for observers as well as ROV staff.
34
Recommendation 4: Re-configure observer stations and pool all observable

operations.
The ROV offices are not designed for effective observation. Keeping many of the
observer functions in smaller rooms, rooms with only a single entry, or in various
sections of the building introduce physical, communication and security challenges.
Consideration should be made to eliminate the majority of storage shelving in the
warehouse (see Office and Warehouse Layout recommendations). By opening this
space up to general operations this would allow for most observer functions to be
36
ROV Response – I agree with this recommendation. I think it is important to note

our current security process. We have two security guards present in each building
24/7. One security guard has to stay in the lobby and the second guard makes
rounds. All of the doors have electronic locks that can only be opened by a badge
from a permanent staff member. TAP staff badges cannot unlock doors, TAP staff
half to be escorted into the building. The Riverside Police Department is stationed
at our office for the 5 days prior to Election Day for weekend voting. The Riverside
County Sheriffs office is stationed at our office on Election Day.
Laws and Regulations Versus Proactive Management
Observations
This is a challenging theme that has emerged but is an important issue to explore. The
California Elections Code is an important tool for all California election officials. The
code book includes nearly 20,000 statutes that govern the conduct of all elections
conducted in California. The courts have been consistent in supporting decisions made
by election officials when carefully following the laws passed by the legislature.
38
25 Agencies include the California Secretary of State; the California Attorney General; the California Auditor-
Controller, etc.
40
Recommendation 3: Develop an election security playbook in conjunction with

RCIT and federal, state and local law enforcement partners.
Following much needed assessments and hardening of the security platform the ROV
should create a comprehensive playbook, both internal and external, that outlines a ring
of security on all aspects of the operations.
27 Formally referred to as the DHS Government Coordinating Council for the Election Critical Infrastructure subsector
42
submitted her two weeks notice for retirement. This left us with absolutely no fiscal
staff. I had the Administrative Services Supervisor cross train with my Executive
Assistant for her last two weeks of employment to cover the basics. The Executive
Assistant had to take on fiscal duties including payroll, accounts payable, accounts
receivable, and budget. We hired two TAP staff to assist the Executive Assistant with
these duties while we opened the vacant positions for recruitment. We hired an
Accounting Technician in 2022 who had experience with accounts payable and
Peoplesoft and was able to assist with those functions. We hired an Administrative
Services Supervisor in March 2022 who came to us from DPSS. We started training
the Administrative Services Supervisor on election specific fiscal information.
Unfortunately, the new Administrative Services Supervisor provided her two weeks
notice in April 2022. The Administrative Services Supervisor told me that she had
been applying for a bunch of positions prior to accepting the ROV position and her
dream job had become available after the fact. The Administrative Services Supervisor
position became vacant for the second time in less than 6 months and the Executive
Assistant had to take over fiscal duties again. The Executive Assistant saw that we
were struggling with filling the position and came to me and suggested that she could
take over the functions permanently. I reached out to HR to find out if she would
qualify for the Administrative Services Supervisor position. HR recommended that we
convert the position to a Senior Administrative Analyst position in order for the
Executive Assistant to take over the functions permanently. I traded in the
Administrative Services Supervisor position for the Senior Administrative Analyst
position and the Executive Assistant was promoted to the position in July 2022. The
Executive Assistant had been performing the duties on a temporary basis for at least 6
months and had previously served as a back up to the position prior to the vacancy.
We currently have a Senior Administrative Analyst, an Administrative Analyst I, an
Accounting Technician, and an Office Assistant III working in the fiscal division.
Both current individuals (Senior Admin. Service Analyst and Accounting Tech. I) came
to the positions with little to no experience in the financial sector. Despite these
challenges both seem dedicated to the work and the mission and seem to be trying to
keep up with the workload.
There are some concerns that should be noted, which include training, resources,
technology and the effects of these on the operation.
• There is little to no formal training that has been offered other than “on the job”
training
• The team shares a single computer that is licensed to operate some of the
necessary software
44
Monday to process all of the time cards and meet ACO payroll deadlines. Our TAP
staff works a lot of overtime and different shifts including a night shift. The payroll
process is not simple because overtime and shift differentials have to be calculated.
We have tried different automated timekeeping systems over the years but they were
not a good fit. We have held meetings with RCIT, the ACO, and HR to describe our
process and see if they have any recommendations or suggestions. One suggestion
was to try Kronos which is the system used by the county hospital. We tried Kronos
and it did not work for our purposes. With the Kronos system the TAP staff had to use
an electronic time clock to clock in and out. They then had to turn in a time sheet at the
end of the pay period and the time sheet was compared to the time stored in Kronos.
Most of the time the time sheets and the Kronos time did not match because the TAP
staff would frequently forget to clock in and out. The fiscal staff would then have to
correct all of the time in Kronos in order for it to be imported into Peoplesoft. The fiscal
staff were often spending more time fixing time in Kronos then if they had just entered
the time manually themselves. The Kronos data would then have to be imported into
Peoplesoft and proofed again. We have talked about giving all TAP staff access to
Peoplesoft to enter their own time similar to the process we use for permanent staff but
that has not worked because Peoplesoft is not user friendly and the TAP staff have a
hard time understanding how to enter time codes for overtime and shift differential.
Again, fiscal staff end up spending more time fixing data entry mistakes then if they
enter the time themselves. We continue to have discussions with RCIT, the ACO, and
HR but an optimal resolution has not been found.
Recommendations
Recommendation 1: Use the time between elections for advanced mandatory

training.
It is critical, given the limited backgrounds of the staff in the financial division, that they
receive enhanced training focused on financial education.
ROV Response – I agree with this recommendation. It is important to note that all of the
fiscal staff started in 2022 and had to dive right into their “on the job” training due to the
vacancies and election cycle. It is also important to note that 2023 was the first
introduction of current fiscal staff to the election billing process, State reimbursement
process, and annual budget process.
Recommendation 2: Use Central HR to re-examine position classifications and

requirements.
It would be wise to ensure the team has the best support and consideration should be
given to providing management oversight (outside of the ROV) to ensure compliance
with financial requirements and experience.
ROV Response – I agree with this recommendation. We have worked closely with HR,
the ACO and the Treasurers department on fiscal trainings.
Recommendation 3: Invest in technology upgrades and additional licensing for

all financial applications.
An IT review of current applications and hardware should be done to upgrade systems
46
to limit downtimes.
ROV Response – I agree with this recommendation. An extra complication was the
switch from RivcoPro back to Peoplesoft. Most of our staff were trained in the use of
RivcoPro for accounts payable. In 2023 the County swicthed back to Peoplesoft and
that was a steep learning curve for the fiscal staff. Our current invoice system is very
old and only contained on one computer within the office. We are in the process of
transitioning that system to the new Peoplesoft invoice system.
Recommendation 4: Increase financial SOPs and regularly test and perform

quality control checks of the procedures.
As noted in the previous section (Project Planning and Standard Operating Procedures)
an increase in SOPs will help ensure that work is done accurately, completely, and in
compliance with state and federal law. The SOPs should be regularly tested by having
supervisors perform quality control checks to determine if the procedures are adequate
and being followed.
ROV Response - I agree with this recommendation.
47
ROV Response – I agree that the ROV needs more RCIT support. I do not feel strongly
either way about RCIT centralization because I have observed pros and cons in the
current environment. On the positive side RCIT has actively been trying to cross train
RCIT staff on election specific technology. This is a good thing for long term succession
planning but it has also presented problems. A lot of election technology is heavily
regulated and most RCIT technicians are not familiar with the regulations. We have
been lucky in the past where certain RCIT technicians have been dedicated to the ROV.
Over time we have seen more responsibilities outside of the ROV assigned to these
RCIT technicians and they slowly start to spend less time at the ROV. Then new RCIT
technicians are assigned to the ROV and they have to go through the election process
training. We have had multiple situations where RCIT projects are not completed
satisfactorily. For example, the old ROV website could only be updated by RCIT staff
due to coding issues. During the November 2022 Election we told RCIT that we would
be doing daily election results website updates and that we would need our website
updated daily. On multiple occasions RCIT staff did not show up to the ROV office for
the daily website update. Fortunately we were able to get other RCIT staff to assist with
the daily updates. Also, during the November 2022 Election we found that some of the
vote center laptops had not been formatted correctly and were nonfunctional. There
was no way of knowing how many laptops were impacted so ROV staff had to go
through all 750 laptops and double check that they were formatted correctly. Staff found
approximately 50 laptops that were not programmed correctly and had to be re-done by
RCIT. Recently we asked RCIT staff to re-format the computers and server in the ballot
counting room so that we could start programming the November 2023 Election. RCIT
staff told us that the process was complete and the ballot counting room was ready for
ROV staff. ROV staff found that the ballot counting room was not functional when they
went to start programming and had to wait several days for RCIT to correct the situation.
There are also areas where RCIT has been very helpful. RCIT staff go out to all 150
vote centers and conduct connectivity testing prior to opening day. They also expand the
help desk to assist with technical problems at vote centers.
49
staff members were trained on the ballot layout and database set up process and ROV
staff will be handling this process internally for future elections. Bringing the ballot layout
functions in house should speed the process up by at least 7 days.
Beyond the internal timing, however, are broader worries about the delays that cause
late mailings of the Voter Information Guide (VIG). The California Elections Code
allows VIGs to begin mailing 40 days prior to any election. In Riverside County the VIG
typically is not mailed for 10 additional days beyond the statutory allowance. This
causes strain on the outbound mailing of vote-by-mail ballots, which must be mailed
beginning 29 days before the election and conclude no later than 24 days before the
election31.
Candidates who purchase statements in the VIG naturally want their statements to
arrive in voters’ hands prior to a vote-by-mail ballot. Delays such as this can cause
animosity and negative public perception. A critical metric that cannot be ignored is the
fact that the VIG and ballot production and mailing in Riverside County is consistently
the last in the state among medium and large sized counties32.
Often the ROV must make adjustments to their mailing operations in order to account
for delays in the production of the VIG. This jeopardizes an organization that already
operates in a high-risk environment as discussed in Appendix A33. Despite the
challenges, I applaud all the staff involved in the production of the VIG and ballot
materials. Assembling all the data in such a short time period is a challenge for any
31 Source: Cal. Elec. Code §3000.5
32 Source: California SOS; USPS and multiple interviews with election vendors serving California counties
33 See: Appendix A, Facts Surrounding Issues Between Runbeck Election Services, Inc. and the ROV
51
provide side-by-side comparisons of documents.

As a part of the tool recommended in the previous recommendation creating digital
versions of the proofing documents will aid in the management process. Seeing what
needs to be proofed, alongside previous changes will aid in the version control process.
Recommendation 6: Assign team leads to manage document process.

There are no “champions” of the proofing process. Team leads should be assigned and
rotated that assist in advancing any documents needing approval on an hourly or daily
basis.

53
Observations
The overall layout of the ROV offices is not conducive to a shared work environment
under optimal conditions. It is clear that the building interiors have been adjusted and
added to over time. This is understandable given the challenges with most county
buildings. Space is at a premium and often times ideal location considerations give way
to short-term solutions.
The addition of the Assessor’s former office space has been a plus for the department.
Despite the shared arrangements it appears to be a positive solution given the addition
of the office space provided.
55
The supplies and items waiting for destruction should be moved to offsite warehouse
space in order to free up the general space for open workspace.
35Source: Previous reviews of existing warehouse space for San Diego County, Orange County, San Bernardino
County and Los Angeles County
57
reference numbers have to be unique and cannot repeat. When we received our
provisional envelopes the reference numbers were not unique and reference numbers
were duplicated. ROV staff had to print and re-label 150,000 envelopes so that they
could be used in the March 2020 Election. Runbeck had to reduce the March 2020
election invoice in order to compensate for the mistake. Runbeck had printed our
provisional envelopes with unique reference numbers in 2018 so it was not a new
process for them. In addition, Runbeck mailed ballots from Phoenix the day before the
election during the March 2021 Election. In February 2021 the Governor signed urgency
legislation that required us to mail a ballot to every registered voter. This went into effect
while we were in the middle of a Special Election. Ballots had already been mailed to
permanent vbm voters but not to voters that had not specifically requested. When the
urgency legislation came into effect I discussed the options with county counsel and the
Executive Office. We all agreed that we should move forward with mailing a ballot to the
remaining 10,000 voters that had not specifically requested a vbm ballot. This all came
about 1 week before the election. I emailed Runbeck to give them a heads up that we
may need to rush 10,000 ballots. Runbeck stated that they could do it. We ran the data
and sent it to them the week before the election. Runbeck did not mail the ballots until
the Monday before Election Day from Phoenix. Unfortunately, these ballots were
received by voters after Election Day. It is not accurate for Runbeck to deny any prior
mistakes.
I focused specifically on issues that occurred during the 2022 election cycles. The
cycles included the June 2022 Primary Election, the November 2022 General Election,
and a municipal election for the City of Banning in December 2022. The issues
identified consisted of the following:
• USPS penalties assessed for postal delivery issues prior to the June 2022
Primary Election
• Ballots sent unsorted for the November 2022 General Election causing significant
delays and unbudgeted cost overruns
36 Source: Vendor observations, personal experience and general observations

59
Runbeck delivered the ballots to the ROV on October 10, 2022. It was soon discovered
that Runbeck had not presorted the mail for postal efficiencies under bulk mailing rules.
Bulk mail is a term used by the USPS to define quantities of mail sufficient to earn a
bulk discount. It costs the USPS less money to process and deliver bulk mail when it’s
41 Source: Interviews with the ROV, Runbeck personnel and review of email exchanges
42 See USPS postal regulations DMM 705.23 et. al.
43 Emails were obtained under consent granted by the Executive Office through a letter dated May 24, 2023
63
from Runbeck about ballot type sorting at that point. On 10/7 I emailed Runbeck and
told them that there was a delay in getting the completed list of ballot types and that
they did not need to sort and could deliver the ballots directly to the ROV. By “sort” I
was referring to “ballot type sort” since our entire conversation was in regards to their
ability to ballot type sort. Again, we never had any discussions about zip code sorting.
On 10/7/2022 Runbeck emailed to confirm that we wanted everything delivered to
ROV. In that same email Runbeck mentioned that they were not “sorting” the ballots. I
thought they were talking about “ballot type sorting” because that was the topic of
conversation. In the 10/6 email I told them to deliver the ballots to our office like they
have for past elections and I reiterated that in the conference call. In past elections the
ballots were always delivered to us with the zip code sort. Later on 10/7/2022 the mail
house vendor started sending us the list of ballot types that were completed. I again
emailed Runbeck to let them know that we now had some of the ballot types. Runbeck
confirmed that they could not sort by ballot type. At that point I told Runbeck to move
forward with delivering the ballots to our office and we would manually sort them by
ballot type.
On 10/10/2022 staff started sorting trays of ballots into ballot type order. We took the
first shipment of vbm ballots to the USPS on 10/11/2022. The USPS inspected some
of the trays and told us they were not in zip code order. On 10/11/2022 I emailed
Runbeck and asked them why the ballots were not in zip code order. Runbeck emailed
back and stated that they decided not to sort in order to get us the ballots by
10/10/2022. Attachment G contains the email from Runbeck stating that they decided
not to do the zip code sort even though they have done it in past elections.
On 10/12/2022 we had a conference call to discuss the sorting issue. During that
conference call there was tension because Runbeck tried to say that I told them not to
sort. This is false. I told them not to sort by ballot type, we never discussed zip code
sorting and I told them numerous times to deliver the ballots like they have in past
elections which was always zip code sorted. I am also including an email from
Runbeck to the Executive Office on 10/11/2022 where Runbeck spells out all of the
steps that they skipped. This information was never provided to me or discussed with
me. These were decisions that Runbeck made on their own without my knowledge.
I find it interesting that the attached 10/7 and 10/11 emails were left out of the original
report as well as the specifics of the 10/6 conference call. Only certain emails were
shared and the context of the 10/6 conference call were completely left out.
I am perfectly aware that with a zip code sort we receive large postage discounts and
that without the zip code sort we pretty much have to pay full first class postage. That
is why I would never authorize the vendor to skip the zip code sort. I know that
skipping the zip code sort would have large cost consequences and that is why I am
adamant that I never authorized Runbeck to skip the zip code sort and we in fact never
had any discussions about zip code sort. All of our conversations were based on
whether or not they had the ability to sort by ballot type. The specific instructions were
that if they could not sort by ballot type they were deliver the ballots to our office in the
same manner as they have in previous elections. Of course, in previous elections the
ballots were zip code sorted.
70
Duplicate Ballots Sent to Voters Prior to the November 2022 General Election
Prior to the November 2022 General Election there were duplicate ballots sent to voters
in Canyon Lake, Menifee, Murrieta, Wildomar and Winchester. The process of
preparing files in order to begin printing ballots is complex and laborious. It is a critical
phase of any election operation and must be handled with quality controls in place.
This particular issue started with the election management system used at the ROV
(EIMS). File exports must be created from the voter files, which tie each voter record to
a specific precinct-based ballot. This takes, on average, three hours of processing time
for each group of 50,000 voters. There is also a strain on the overall management
system due to the large files (which include voter signatures) that are prepared.
Because of this the file extracts are often run over night.
It appears that the actual process of preparing and running these file extracts were
performed by Ms. Spencer herself. The file in question was created by her and was set
to run over night. When she arrived the next morning, the file showed as failed to
complete so she ran it again. None of these lengthy communications, or actual file
preparation, or file proofing, was conducted by any other member of the department47.
After all the files were complete, and the mailing was underway, there was back-and-
forth on multiple occasions with Runbeck on the total number of ballots created.
Runbeck reported 1,352,764, while the ROV expected to see 1,302,853 (which was the
total registered voter count at the time).
There were no duplicate checks run on the files and the reconciliation was taking place
after the ballots were prepared for delivery directly to the ROV for sorting (as previously
mentioned). Runbeck reported that their duplicate checking process was not in place
since they were instructed to bypass sorting by the ROV. The ROV confirmed that
there was no follow up with Runbeck on the files, or inquiries about duplicate checking.
47 Source: Interviews with the ROV, Runbeck personnel and review of email exchanges
71
The ROV was able to sort out the duplicate ballots ensuring that no duplicates were
scanned or included in the tally. Ultimately, however, it is the responsibility of the
election official, not a vendor, to ensure that the files and mailing process is sound.
ROV Response – The above information is inaccurate. As previously mentioned, all of

the mailing files were ran by me because of staff vacancies. All of the staff that had
been trained on this process left prior to the 2022 election cycle and one staff member
was out on unexpected FMLA. The only two staff members that knew how to run the
files were myself and Art Tinoco. Art was busy managing the Operations unit because
that Chief Deputy position was vacant. Matt Ceballos was the brand new Chief Deputy
in voter services and had not been trained on mailing files since staff were out on
FMLA. That left me as the only staff member trained to run these files. It is a very time
consuming and complicated process. We cannot run all 1.3 Million mail files at once
because it will completely crash the server. Instead we have to break the files up into
groups of approximately 50,000 so that the server does not crash. We also have to do
this overnight when staff are not performing data entry on the system. I set the job to
run over night. The next morning I did note receive an error message. The system
showed that the file was complete with no error messages. I ran the job for the next
batch of 50,000 and did not receive any error messages. When all of the batches were
complete I ran reports for how many ballots had been issued and it showed 1.3 Million.
I also looked at the first and last voter in each batch and confirmed that there were no
duplicates. All of the files were then sent to Runbeck. Through out the process I did
receive several error messages because the server would disconnect in the middle of
the night and we would have to restart the job in the morning. I did not receive any
error messages for the batch that was duplicated. Based on these strange error
messages I asked Runbeck to confirm the total quantity received many times.
Runbeck did not confirm the total quantity received until 10/14. At that point we knew
there was a problem because Runbeck’s quantities did not match our quantities. We
were able to determine which files were duplicated. We were able to stop
approximately 46,000 duplicates from being mailed but unfortunately approximately
3,500 duplicates had already been mailed. Runbeck never disclosed to us that they
were not running the duplicate check on the ballots. After reviewing the two duplicate
files and the computer logs it appears that the server disconnected in the middle of the
night after the file had completed. I did not get an error message because it did not
disconnect in the middle of the process, it disconnected at the end of the process. So
the mailing file was complete but the batches in the system were not marked as
complete once the disconnect happened. When I ran the next group of 50,000 it
repeated part of the previous group because they did not get marked as complete when
the server disconnected. The system will only give an error message when it
disconnects in the middle of the batch. If the batch is complete and then it disconnects
an error message is not created. The two batches were not exact copies of each other
which is why it was not caught when I looked at the first and last voter in each file.
When I ran ballot issue reports the duplicate file did not show because the server
disconnect was on the back end and not the front end. The back end saw those groups
as complete but the front end did not.
72
Runbeck sets expectations for performance objectives well in advance of any election
cycle. These objectives do not always align with county expectations based on a
variety of internal and external factors. If deadline targets move, or adjust, it is difficult
for some counties to adapt on short notice. The ROV has not been consistently
responsive in their communications and engagement, which has caused tensions
between Runbeck and the ROV. The ROV did not make sound decisions by asking
Runbeck to bypass the previously agreed upon ballot deliveries to the USPS. This
caused significant increases in labor, postage, and delivery expenses, which could have
been avoided. The ROV requested that ballots be delivered unsorted, which triggered
penalties that were imposed by the USPS. The ROV was not delegating
communication and file preparation and operated in an environment absent a sound
quality assurance program.
ROV Response – As previously mentioned, the above finding is based on inaccurate

information.
74
These deaths are not reported to the California Health Department. Use of national
death data is an excellent method to check California death data and pick up out-of-
state death data.
ROV Response – I agree with this recommendation. This recommendation will require
additional NCC funding.
78
voters to inactive. Bill Essayli then stated that he disagreed with our interpretation of
the law and stated that he might have to take it to court.
Recommendation 6: Create user friendly online tools to allow for third-party

notifications of deaths or moves.
If a residence receives ballot materials that is not in their name, or there has been a
death, they should have an easy mechanism to report this information to the election
office. This should not be a PDF form or cumbersome process. Allowing residents to
report this data will empower citizens to assist in the list maintenance process. This
would simply trigger an investigation by the ROV and would not activate any automatic
list maintenance process.
ROV Response – I agree with this recommendation. It is important to note that we

currently have pdf forms on our website for the above purpose. We would need RCIT
assistance to change the method on the website.
Recommendation 7: Train and cross-train permanent and temporary ROV staff

on voter list maintenance.
Some current permanent ROV staff have learned on the job how to do most of their
tasks but do not have a full understanding of California and federal law regarding voter
registration and why good data entry and list maintenance habits are so important. An
on-site training with all voter registration staff would help lay that foundation while cross-
training would ensure redundancy for these vital job duties.
ROV Response – I agree with this recommendation. It is important to note that all voter
registration staff already attend State training on voter registration and file maintenance
annually. Attachment D shows the State training received annually.
80

Neal Kelley&#39;s Outside Review of Riverside County&#39;s Election Office

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Neal Kelley&#39;s Outside Review of Riverside County&#39;s Election Office

Uploaded by

Copyright:

Available Formats

ROV Operational Assessment Report August 2023

ROV Response – The Registrar of Voters is currently operating on a 50% hybrid

Ultimately, however, if there is a commitment to continual and sustained improvement

Recommendation 1: Create a culture of continual improvement.

Recommendation 2: Develop a learning academy that combines SME education

Recommendation 3: Partner with outside organizations to showcase similar

On the one hand this is understandable to a degree. As an example, errors in elections

• A high-level focus on long-term strategic planning

12 See International Organization for Standardization at iso.org

e-minus (countdown) calendars, standard operating procedures (SOPs), policies,

ROV Response – I agree with this recommendation. As previously mentioned and

20 See Appendix C for list of documentation

ROV Response - I agree with this recommendation.

conduit for critical feedback to operational planning.

ROV Response – I agree with this recommendation.

Recommendation 4: Re-configure observer stations and pool all observable

ROV Response – I agree with this recommendation. I think it is important to note

Laws and Regulations Versus Proactive Management

Recommendation 3: Develop an election security playbook in conjunction with

ROV Response – I agree with this recommendation.

Recommendation 1: Use the time between elections for advanced mandatory

Recommendation 2: Use Central HR to re-examine position classifications and

Recommendation 3: Invest in technology upgrades and additional licensing for

Recommendation 4: Increase financial SOPs and regularly test and perform

ROV Response - I agree with this recommendation.

provide side-by-side comparisons of documents.

ROV Response – I agree with this recommendation.

Recommendation 6: Assign team leads to manage document process.

ROV Response – I agree with this recommendation.

ROV Response – I agree with this recommendation.

36 Source: Vendor observations, personal experience and general observations

ROV Response – The above information is inaccurate. As previously mentioned, all of

ROV Response – As previously mentioned, the above finding is based on inaccurate

Recommendation 6: Create user friendly online tools to allow for third-party

ROV Response – I agree with this recommendation. It is important to note that we

Recommendation 7: Train and cross-train permanent and temporary ROV staff

You might also like

Neal Kelley's Outside Review of Riverside County's Election Office

Neal Kelley's Outside Review of Riverside County's Election Office