Businessweek Feature

Inside the IRS’s Shrinking Band of Wealth

Hunters
Former auditors dish on truant taxpayers, decrepit tech and why $80 billion might
not be enough to fix everything that’s wrong.

By Ben Steverman
April 12, 2023 at 5:00 AM CDT

Illustration: Nick Little for Bloomberg Businessweek

If you ever meet an off-duty Internal Revenue Service auditor, they probably won’t admit where they
work. “What do you do?” is “always a party killer,” says former agent Elyse Katz. Best to keep it vague,
she says, and hope that “accountant” is boring enough to move the conversation along to another
topic.
Few things make Americans more anxious than an IRS agent in their midst, even when the occasion is
social rather than, say, an audit. It’s the audit that inspires many people’s fear of the agency, helping it
poll as the least trusted of major federal departments. Last year, when President Joe Biden and
congressional Democrats secured almost $80 billion in extra IRS funding across the next decade under
the Inflation Reduction Act, it didn’t take long for the agency’s critics to raise the prospect of rampant
nosiness, with Republican politicians claiming that an army of 87,000 armed agents would soon be
poking through Americans’ finances.

“We don’t have guns. It’s just me and a laptop”

The notion was absurd, to say the least, and not only because almost no IRS agents carry guns. Across
the past few decades, the agency has been decimated by cuts, to the point that by most measures it’s
the worst-funded major federal department. It’s down to 84,000 (almost all unarmed) workers, a loss
of 10,000 employees since 2010 and about the same as in 1974, when the US had 120 million fewer
people and an economy a quarter of its current size. The IRS’s most experienced revenue agents have
departed at an even higher rate than other employees, leaving 99.9% of the most complex and opaque
type of business returns unexamined and millionaires’ and billionaires’ audit rates tumbling by 80% to
90%.

The agency’s diminished profile is the result of a conservative campaign that’s been going on at least
since Ronald Reagan became president, thriving through the Trump era even as other core Republican
planks fell aside. When the party took over the House of Representatives in January, the first bill
passed, on a party-line vote, was an attempt to stop most of the $80 billion from being released.

The IRS has already started to spend that money, with the aim of modernizing and helping it catch tax
cheats. “This is our moment in history to transform the IRS,” the agency’s new commissioner, Danny
Werfel, declared at his swearing-in ceremony on April 4. Among his goals, he said, are hiring skilled
staff to unpack complex returns, upgrading customer service and investing in technology that finally
lets the agency interact with taxpayers online rather than via tons and tons of paper.

But employees who’ve been on the job in recent years say more than extra money will be required to
restore the agency to some semblance of its former self. Bloomberg Businessweek persuaded more than
two dozen recently departed employees to open up about their time at the agency. Many asked to
remain anonymous, saying they feared online attacks or professional blowback. Taken together, the
ex-employees describe a dysfunctional agency gutted by budget cuts, staff departures and lousy tech,
leaving it incapable of enforcing the nation’s tax laws fully and fairly. (The IRS declined to respond to
former employees’ criticisms, pointing Businessweek to the agency’s detailed plans to revamp
operations over the next several years.)
Katz at her office in Phoenix. Photographer: Cassidy Araiza for Bloomberg Businessweek

Even some of the fiercest IRS critics concede that the US needs a functioning tax collector. Although
it’s almost impossible to know how much taxpayers are cheating, the agency’s latest estimate pegs the
annual gap between taxes owed and paid at nearly $500 billion—almost an eighth of the $4 trillion a
year it brings in. And academic studies suggest that the rich have hidden away far more than that
offshore and in business activity the IRS can’t easily track. Nor does the estimate account for the
impact of new vehicles for tax evasion such as cryptocurrencies.

Meanwhile, Americans with simple finances find it virtually impossible to cheat. Employers
automatically report worker income to the IRS, which uses basic matching software to spot any
discrepancies on returns. And their simpler returns are, perversely, still relatively straightforward for
the agency to audit. In case that ever happens to you, the former employees also offered Businessweek
readers a few tips.

A Ridiculously Brief History of the IRS (and the Attacks Against It)
The IRS once played a vigorous, if not beloved, role in American society. The modern income tax was
introduced in 1913 by targeting only the wealthiest 0.4% of the population. But the US soon desperately
needed money to fund world wars and build social and physical infrastructure befitting an economic
superpower, so more and more people were required to file. By World War II, most Americans were
paying something, and the Bureau of Internal Revenue started withholding money from paychecks for
the first time.

A decade later, the renamed IRS found it could boost revenue by going door to door looking for tax
cheats, with the approval of 53% of respondents in one poll. In the early 1960s, the IRS became a
computing pioneer, storing Americans’ tax information on giant state-of-the-art IBM mainframes. Its
agents were known for having put Al Capone in prison, for raiding Studio 54 in the 1970s and for
playing the villain or foil in countless sitcom plots, such as the episode of The Mary Tyler Moore Show
in which the title character flirts with her auditor. Wealthy celebrities would joke about the IRS’s
interest in their affairs. “They just took all my money away,” Truman Capote blithely told Dick Cavett
on late-night TV in 1971, explaining he’d just lost a five-year battle with the IRS. The audience laughed.

The agency’s prominence and success made it a rich target for politicians looking to score points with
tax-hating Americans. Then in the 1970s came a tax revolt that would help elect Reagan president.
Once in office, Reagan slashed tax rates, but the IRS didn’t actually start shrinking until the 1990s,
when Republicans won the House for the first time in almost 50 years. As the agency came under
sustained criticism in hearings spotlighting examples of overzealous enforcement, its budgets declined
and its workforce began an epic 38% slide, losing 45,700 full-time staff from a 1991 peak to a 2018
nadir.
Illustration: Nick Little for Bloomberg Businessweek

As the IRS was squeezed, the agency resisted laying off staff, but almost everything else was fair game.
Training budgets were whittled away. Support staff disappeared. Major tech upgrades were out of the
question. The IRS also cut back on the budget for outside expert witnesses such as appraisers, whose
testimony is crucial for competing against well-heeled taxpayers in court.

In the past decade alone, even as the number and complexity of tax filings have steadily risen, the
agency’s inflation-adjusted budget has plunged 15%, from $12.2 billion in 2010 to $10.3 billion last year.
The number of revenue agents—auditors who handle more complex returns—fell almost 40%, from
about 14,500 to 8,500 over the same dozen years. And the chance that an American reporting income
of $5 million or more gets audited fell from 16% in 2010 to 2.35% nine years later.

An Auditor’s Life
On rare occasions, the mere appearance of an auditor can stop tax evasion. One taxpayer started his
audit by asking Kenneth White Jr., a former agent who’s now a financial planning professor at the
University of Arizona, how the process worked. “I’m going to ask you a lot of questions,” White
replied. “Some I know the answers to and some that I don’t.” The business owner leaned back in his
chair to think, then said, “Let me just say that that tax return is totally made up. I will get you the real
tax return if you give me a week or two.”

It’s rarely that easy to catch a cheater. “Revenue agent is really the hardest job in the world,” says Tom
Kane, an IRS attorney who retired as a high-level administrator in 2019. That’s especially so where
wealthy taxpayers and large corporations are concerned. An auditor is assigned a return that could
run to the thousands of pages. Deciphering it can take months and may require specialized knowledge
of a given industry, a profession or an asset class. The auditor must also cross-reference that pile of
paper with the even more voluminous US tax code, hunting for violations and laying out a case that
can survive the scrutiny of judges and creative arguments from teams of high-priced advisers.

Kane at his home in Washington, DC. Photographer: Lexey Swall for Bloomberg Businessweek

While doing this work, agents can request more information from taxpayers, at the risk that
accounting firms will bury them in irrelevant documents. “They have their ways of messing with the
agency,” says Jasmine Jackson, a former revenue agent in the Washington, DC, area. The wealthy also
typically route income and deductions through long chains of holding companies, partnerships and
LLCs, each with their own return. Sometimes the taxpayers themselves don’t understand the
elaborate structures their advisers have set up. A final issue is that the IRS typically has three years to
catch a problem before the statute of limitation expires, and auditors often aren’t even assigned cases
until half that time has elapsed.

Auditing small businesses and poorer Americans doesn’t require as much time or expertise, but the
stress can still be intense. One former agent says he thought he’d found a dream job after graduating
from college, drawn to join in late 2019 by the government pay scale and promised work-life balance
after previous jobs in retail. The months of training he received at the outset were impressively
intense, he says. “And the professors were great, too.” Then, like countless novice auditors before, he
was sent out into the field to work on real cases. “It was sink or swim.” He sank.

Taxpayers would cry and curse, overwhelming him with questions he didn’t know how to answer.
Some seemed to be honest people victimized by unscrupulous tax preparers who’d advised them to
do blatantly illegal things, such as deducting handouts to random homeless people as charitable
expenses. The clunky and slow software the IRS uses to access taxpayer information, a relic from
before he was born, would crash unexpectedly. “I felt like an auditor in the ’90s instead of the 2020s,”
he says. He soon told his manager he couldn’t take it anymore. “I’m going to have a heart attack,” he
said. He calls the job “probably the most stressful six months of my life.” He’s now hoping to change to
a career in information technology.

Another auditor signed up in 2021 because she had a parent who’d loved working at the IRS. With all
instruction being conducted virtually because of the pandemic, “there really wasn’t any training on
how to do your job” beyond the basics, she says. Once in the field, she found herself unsure how to
use the software and even how to get in touch with taxpayers she was supposed to audit. The IRS
doesn’t have a reliable way of communicating electronically with taxpayers, so emailing wasn’t an
option, and taxpayers ignored her letters, perhaps because they didn’t look very professional—with no
support staff, she had to print them out herself, handwriting the address on each envelope. She’d try
calling but found that people rarely pick up the phone. “This whole process is ridiculously inefficient,”
she recalls. “I felt like I couldn’t do my job.” When she quit in 2022, after less than a year, she’d only
made contact with three taxpayers.

Other auditors sometimes had to take the initiative to train themselves. Anthony Kim, a former
attorney in the IRS’s San Francisco office who also taught classes at a local university, would be
surprised to find revenue agents in his tax classes, their fees paid out of pocket. “I don’t feel
comfortable doing my job,” they’d tell him. Peter Orth, an auditor for 33 years, says he taught himself
enough about investing and insurance to get his securities license, then helped the IRS recover money
hidden by complicated tax shelters that used derivatives to create phony loans.
Then there was the often difficult human dimension of the job. Agents unable to reach taxpayers by
phone or email would need to show up at homes or businesses of taxpayers, a situation that could be
volatile. “Sometimes you don’t know what you’re walking into,” says a former agent with eight years of
experience. She remembers rushing out of one audit when a mentally ill taxpayer started making
veiled threats. Despite the rumors to the contrary, only investigators from the IRS’s relatively small
criminal division ever carry weapons. “We don’t have guns,” Jackson, the former revenue agent from
DC, recalls. “It’s just me and a laptop.”

Jackson at home in Baltimore. Photographer: Lexey Swall for Bloomberg Businessweek

“You come into peoples’ lives, and it’s never a good time,” says Jackson. Taxpayers would often seem
terrified, admitting to her they “couldn’t sleep for a week.” Often the audit was connected to other
issues, such as health or financial problems, that led them to tax trouble in the first place. “It could be
depressing.”

Another former agent, ShantaQuilette Carter-Williams, would sometimes have to make site visits to
businesses that had stopped submitting payroll taxes on behalf of their employees. Often the business
was failing. “You have the IRS showing up on top of that to make your life harder,” she says. “I’ve had a
few [owners] get in my face. I’ve gotten kicked off several properties. I’ve had others be extremely
emotional.”
Still, most taxpayers are understanding. Carter-Williams pegs the share at 85%. “There are tax laws
that must be upheld, and when you don’t hold up your end of the agreement, there are
repercussions,” she says. On better days, Carter-Williams could play the role of educator rather than
enforcer. “A lot of times the failure to file or pay was because of a lack of understanding,” she says.

Carter-Williams, a former IRS agent. Photographer: Cormeshia Batty for Bloomberg Businessweek

Cuts, Cuts, Cuts

As budgets dropped, indignities piled up. The most aggravating, to many, involved technology. To get
basic taxpayer information, auditors must still log in to a pre-Windows system with a black-and-green
screen. Computers could take a half-hour to boot up in the morning or would crash if trying to run a
web browser and word processing program at the same time. “You would spend a lot of time just
trying to navigate the software,” says Katz, who’s now a CPA at Eide Bailly LLP. The IRS’s legacy
systems also have trouble merging datasets, making basic financial analyses impossible. When agents
couldn’t access taxpayer histories digitally, they would find themselves in the awkward position of
asking people for copies of old returns.

One former attorney says that “the organization would treat a working computer as a perk of the job.”
An ex-auditor in Florida, Herbert Cantor, who spent 40 years at the agency, remembers the time his
laptop crashed and he was told no replacements were available. He’d have to mail it to Texas to be
repaired. In the meantime, he was forced to cancel all his meetings with taxpayers. “I did not work for
three weeks,” Cantor says.

The parsimony extended to expenses and human resources, too. Kim, the attorney in the San
Francisco IRS office, recalls being asked once by an auditor to come to San Jose as a legal adviser in her
meeting with a taxpayer’s team of attorneys. Driving 50 miles and parking there would have cost at
most $50, but his request arrived at the end of a fiscal year, when the IRS was pinching pennies. His
request was denied, and he couldn’t just pay for the trip himself, because it would have violated
policy.

With the IRS not hiring and attrition rates significantly higher than in other federal departments,
teams were collapsed and combined. Several employees said their departments went more than a
decade without adding anyone to replace people who left. The IRS has said it now has fewer
experienced examiners, the sort who can spot a sophisticated tax dodge in the field, than at any time
since World War II.

Kim cites the brain drain as a key reason why he left the IRS last year. “When I started back in 1995,
the agents just rocked,” he says. By the time he moved on, he was working with new, poorly trained
auditors who were nevertheless being “elevated to take on a lot of cases that are outside their comfort
level and knowledge base.”

The audits that are being done are taking much longer to complete, a function of the increasing
complexity of returns filed by the wealthy and the attrition of the experienced staff capable of parsing
them. The average audit for those earning more than $10 million took 982 days last year, 40% more
than in fiscal 2019, according to the National Taxpayer Advocate, an independent watchdog at the IRS.
When the internet opened up vast new territory for tax evasion and other wrongdoing on the dark
web, the IRS could spare only small teams of investigators to take them on. “There was not a shortage
of cases to pick from,” says Chris Janczewski, who as special criminal agent led a group investigating
cryptocurrency. His team was only 10 people or so, but they managed several high-profile
investigations of illicit financing networks since 2015, including the seizure of stolen Bitcoin worth $3.6
billion in February 2022. “Sometimes you need to do more with less,” says Janczewski, who left the IRS
last year to join a blockchain intelligence company.

Some ex-employees now earn their living guiding taxpayers through the IRS’s broken system. Jackson,
the former auditor in Washington, now has her own accounting company. She says one client has been
waiting for a refund since 2020 and can’t get an answer. Even when Jackson can get through on the
phone to the IRS, she finds herself talking to people who can’t access basic information. “They don’t
know what they’re doing,” she says.

Kim, who went on to co-found a law firm, says he has clients being audited by agents who “are not
correctly applying the law.” He’s confident he’ll win any disputes when he appeals to higher levels of
the agency, if at added cost to his clients. “The lack of agent training has a direct consequence on
taxpayers,” he says.

“How do I like dealing with the IRS? It’s been horrible, absolutely horrible,” says Cantor, the former
Florida auditor who’s now also in private practice. One of his clients has spent two years trying to wipe
out an unjustified tax bill. On the plus side, Cantor says, because so many auditors are inexperienced,
“I can steer them in the wrong direction. If these people knew what they were doing, my client would
probably owe more money.”

Cantor at home in Parkland, Florida. Photographer: Melody Timothee for Bloomberg Businessweek

Nothing Is Certain Except Political Fights About Taxes

Today’s employees have spent most of their entire careers with the agency embroiled in partisan
controversy. House Republicans are promising another round of investigations and oversight hearings,
alleging that the Inflation Reduction Act’s IRS funding will go to audits targeting Americans earning
$75,000 or less, rather than $400,000 or more, as the Biden administration has promised.

The false claim about the agency hiring “87,000 armed agents” won’t seem to die. Even if the IRS was
allowed to hire that many people, most wouldn’t even be “agents” by any definition of the word, let
alone armed ones. The agency needs a variety of employees, including in areas such as IT and
customer service, to replace the 50,000 retirements and resignations expected in the next several
years. Nevertheless, the talking point has had very real effects on IRS staff. Threats against the agency
have surged, and employees have said they fear for their safety, according to the previous
commissioner, Charles Rettig, a tax lawyer appointed to the job by former President Donald Trump.
Rettig ordered a security review of IRS facilities last year.

Despite Republicans’ overwhelming success in diminishing the nation’s tax-collecting capacity, the
party’s politicians and rank and file can still sound certain that the IRS is out to get them. “There is
absolutely no doubt that some of those new agents are going to target Floridians based on their
political beliefs or the simple fact that Florida’s economy has fared far better than most states,” the
state’s chief financial officer, Jimmy Patronis, said in February. He advocated that individual IRS agents
who “target” taxpayers for “political discrimination” be penalized financially.

Employees often profess to be mystified by the anger, suspicion and fear directed at them. Off duty,
few people are less intimidating than a math- and tax-code-loving nerd. Politics never enters into the
job, ex-employees insist, a rule emphasized in annual training. Only two IRS employees are political
appointees, the commissioner and chief counsel. Both must be confirmed by the Senate, with the
commissioner serving a fixed five-year term. “In 27 years, I never had anything that came on my desk
that had a political tint,” Kim says.

Former agents say they were nothing like the pro-tax zealots depicted in conservative media as trying
to soak the rich and squeeze everyone else. If pressure came from managers, they say, it was usually to
close cases more quickly, not to rack up huge settlements. “There was never a vibe with anyone I
worked with that ‘We really need to nail these guys,’ that we have to put the screws to the taxpayer,”
one lawyer says. An auditor says he used to tell taxpayers, “I’m here to make sure that everyone pays
their fair share of taxes, including your competitors.”

If you do happen to be audited, the former employees say, the best approach is to treat your revenue
agent like a person, not the arm of a vast bureaucracy. “It’s a very human organization,” says an
attorney with multiple stints at the IRS. “If you walk in there and you act all defensive and be jerky,
they’re not going to like you.”

A “good faith effort” will go a long way, says White, the Arizona professor and former agent, especially
when it comes to getting auditors the paperwork they need to close your case. They don’t necessarily
need to check every single thing.

Agents facing combative taxpayers are more likely to follow procedures closely, resulting in more time-
consuming and careful examinations, says a former agent who focused on businesses with less than
$10 million in assets. “I’m always very easy to talk to,” she says. But “some people just want to fight.
Then you have to put on your hard hat.”
IRS employees know the agency needs to change. Beyond the day-to-day issues of computer glitches
and staff turnover, there are big-picture concerns to be addressed. A Stanford University study
released in January found the agency over-audits Black Americans, perhaps a consequence of a
strategy of pursuing cheaper audits against the poor instead of resource-intensive investigations of the
wealthy. And almost no one is checking on vast segments of the economy, especially the 4.7 million
returns filed each year for partnerships that hold much of the top 0.1%’s wealth. Only 3,155
partnership audits were initiated last year, one-third the number in 2018.

The agency’s administrative ranks have “gone through a million planning exercises” thinking about
how to modernize the agency, says Natasha Sarin, a Yale professor and former adviser at the US
Department of the Treasury. The newest plan, requested by Secretary of the Treasury Janet Yellen,
arrived more than a month after a mid-February deadline. The 150-page document lays out a strategy
for spending the $80 billion. Priorities include: revamping technology; boosting audit rates on
corporations, partnerships and the wealthy; and providing much better digital tools for taxpayers,
including alerts when they make simple filing mistakes and guidance on claiming deductions and
credits.

The IRS will need to hire—a lot, and not only auditors. The plan calls for adding 19,500 employees in
the next couple years, and almost two-thirds won’t be enforcement staff. The agency needs
programmers to fix its computers systems, and data scientists and data ethicists who can use artificial
intelligence and machine learning to improve the process by which returns are selected for audit. It
needs trainers, communication experts and a much bigger human resources staff to find these new
hires and guide them through the excruciatingly slow onboarding process.

Former employees stick up for the agency. Many say their jobs were the best they’ve had in their
careers. It’s a union shop, with excellent retirement benefits, sane hours and an often collegial culture.
Those who faced discrimination at other employers—everyone including ethnic minorities, older
workers and graduates of less-than-prestigious schools—report finding the IRS unusually welcoming
and meritocratic. The latest figures show that almost two-thirds of the IRS workforce are women and
almost 29% are Black—levels far exceeding those of the federal workforce as a whole.

Still, “the IRS is going to need to be creative in tapping into private sector talent,” Sarin says. One
innovative strategy, she suggests, would be to hire experienced tax pros from the big accounting firms,
which tend to push partners to retire in their late 50s and early 60s. Recruiters for the IRS will have
one key advantage, she says: With the new funding, “it’s going to be a really interesting place to work.”
 

Terms of Service Do Not Sell or Share My Personal Information Trademarks Privacy Policy
©2023 Bloomberg L.P. All Rights Reserved
Careers Made in NYC Advertise Ad Choices Help