Nobody is happy with the FDA.

Almost a year ago, Vox’s Dylan Scott reported that the Food and Drug Administration had been “demoralized and tarnished during the Trump era.” Things haven’t gotten much better for the embattled agency in the months since.

In December, epidemiologist Michael Mina detailed the “frustrating secret” behind why it was so difficult to get rapid tests: The FDA’s “onerous but remarkably useless check boxes,” he wrote, were slowing the approval for different at-home options. Mina explained that the FDA was holding fast to processes that didn’t allow it to consider “the ample data around the world” and was forcing companies to compare rapid tests to lab-run PCRs, preventing hundreds of millions of tests from being purchased by Americans.

If this type of problem seems familiar, it’s because it is. Throughout the pandemic, the FDA has faced criticism about its seeming inability to adapt its processes for an emergency.

It’s the sort of tightrope walk you can only watch with your face behind your hands. Act too quickly, miss something, and lives hang in the balance, not to mention Americans’ dwindling faith in institutions. But it’s just as dangerous to act too slowly when approving needed treatments or tests, to be inflexible in the face of new evidence — lives hang in the balance if you do nothing, as well. And time and again, the FDA seems to have chosen to fear the dangers of action over inaction.

In a statement, FDA spokesperson Michael Felberbaum defended the organization’s pandemic response, arguing it ”made the most appropriate and timely decisions regarding the products we regulate using the best available science, with the health and safety of the American public in mind.”

“Our decision-making must strike a careful balance between the potential risks and benefits of a variety of public health, legal and regulatory considerations,” he said in an email. “These considerations are never as clear-cut as some like to suggest.”

But critics argue the agency could have moved faster. As Conor Friedersdorf reported for the Atlantic last summer, when the FDA was considering authorizing Covid-19 vaccines for children ages 5 to 11, it asked Pfizer and Moderna to gather more safety data, rather than rely on existing evidence that the vaccines were safe for adults and teenagers. Even the American Academy of Pediatrics believed the existing evidence was sufficient to grant authorization: “The FDA should strongly consider authorizing these vaccines for children ages 5–11 years based on data from the initial enrolled cohort, which are already available,” they wrote in a letter to the agency.

The delay approving rapid tests followed the same script. As ProPublica’s Lydia DePillis and Eric Umansky found last November, the source of the FDA’s delay “appears to be a confounding combination of overzealous regulation and anemic government support.” While tests were approved in other countries that prioritized “accessibility and affordability over perfect accuracy,” the FDA blocked the use of such tests in the United States.

It was the same story again when it came to human challenge trials. The FDA brushed off the idea, despite thousands of Americans stepping forward early in the pandemic to volunteer to be infected with SARS-CoV-2, the virus that causes Covid-19, to more rapidly study the efficacy of Covid-19 vaccines. Human challenge trials are a controversial approach, but notably, the United Kingdom was willing to approve this research method in February 2021 because of its potential to help “accelerate” vaccine development.

As months pass and these events pile up, an uneasy question has risen to the forefront: What if the FDA’s failures during the pandemic happen all the time, and most people are just now noticing? If the institution itself is broken, the danger could be far greater than just this moment.

What if the FDA has always been broken?

George Mason University economist Alex Tabarrok has been beating this drum since long before Covid-19 was a concern. Tabarrok, a leading libertarian thinker advocating for institutional reform of the FDA, gained traction criticizing America’s pandemic response. Recently, I asked the economist: what if this goes beyond Covid-19?

Tabarrok has coined a haunting phrase to describe his concerns with the agency: “The FDA is conservative because when it approves a bad drug, its error is visible, but when it fails to approve good drugs, the dead are buried in an invisible graveyard.”

In an interview, he pointed me to a 2017 paper by Leah Isakov, Vahid Montazerhodjat, and Andrew Lo titled “Is the FDA Too Conservative or Too Aggressive?: A Bayesian Decision Analysis of Clinical Trial Design.”

The paper explores the trade-offs the agency faces between what it calls Type I and Type II errors, or false positives and false negatives, respectively. A false positive would be approving a drug that is either harmful or ineffective, and a false negative would be failing to approve a drug that would be helpful or even potentially lifesaving.

Type I errors are much more salient to the FDA than Type II. The agency can drive down the rate of false positives to near zero by being stingy with its approvals — after all, you’ll never approve a drug that harms people if you never approve drugs.

Optimizing for the appropriate levels of risk in both directions — ensuring important treatments are quickly approved, while also guarding against the approval of dangerous or worthless drugs — is an incredibly difficult problem to solve. So how does the FDA score?

According to Isakov, Montazerhodjat, and Lo: badly.

They find that the FDA is way too conservative when assessing clinical trials for therapies of “terminal illnesses with no existing therapies such as pancreatic cancer.” These are the areas where you would hope the FDA would be overly willing to approve therapeutics since the risk of death and disability are already high for patients.

This paper confirms an anecdote from Henry Miller, a former FDA physician who has detailed the flawed incentive structure within the agency:

In the early 1980s, when I headed the team at the FDA that was reviewing the NDA for recombinant human insulin … we were ready to recommend approval a mere four months after the application was submitted (at a time when the average time for NDA review was more than two and a half years). With quintessential bureaucratic reasoning, my supervisor refused to sign off on the approval—even though he agreed that the data provided compelling evidence of the drug’s safety and effectiveness. “If anything goes wrong,” he argued, “think how bad it will look that we approved the drug so quickly.”

In Miller’s telling, the agency was overly sensitive to the possibility of making a Type I (false positive) error — and to safeguard the agency’s reputation, they withheld a drug from the public.

On average, starting the timer at when a drug begins pre-clinical testing, it takes the FDA 12 years to approve a new drug.

And even as the organization has worked to speed up its processes during the Covid-19 pandemic, the agency failed repeatedly on balancing risks.

According to Institute for Progress senior biosecurity fellow Nikki Teran, the US “requires antigen tests to be 80 percent as sensitive as the gold standard RT-PCR tests.” That means, in theory, that an antigen test in the US “needs to be over 30,000 times more sensitive” than in the UK, where Teran notes there are more than 150 different rapid antigen tests available (and for complicated reasons, 30,000 times more sensitive in a clinical trial does not actually mean 30,000 times better in real life).

According to the FDA, as of February 9, just 15 emergency use authorizations have been granted for over-the-counter at-home tests. The agency says there are a few common errors preventing it from authorizing antigen tests, and most boil down to poor data: too few patients tested, not enough proof people know how to use the tests, incorrect types of samples, and more.

Disregarding Mina’s argument that data does exist, just not in the format the FDA wants, there’s still another thing to consider as the FDA rejects applications: The alternative to a complicated test that doesn’t work well is often no test at all.

There are dangerous parallels in history to the current Covid-19 crisis. According to Steven Salbu, then an associate professor of business at the University of Texas Austin, “in the late 1980s the FDA adopted a de facto blanket ban on HIV home-testing kits.” Salbu writes that an FDA spokesperson explained this policy by pointing to the potential for “improper drawing of blood samples, the possibility of blood samples being held for long periods of time, and the potential for blood samples to be affected by temperature changes during in-mail transit.”

So the FDA, seeing that there could be problems with the capability for HIV tests to be performed perfectly accurately at home, chose instead to allow no at-home tests to exist. Sounds familiar.

According to the Kaiser Family Foundation, more than 700,000 Americans have died from an HIV-related illness.

“Now, my view is that for people who have AIDS and cancer and heart disease, it’s always an emergency for them,” Tabarrok said. “Right? It’s always been like a pandemic for them. And now, I hope that people will come to appreciate the opportunity cost of more safety of FDA delay and apply this more broadly.”

Maybe the FDA’s problem isn’t about risk tolerance

Scientist and writer Hilda Bastian disagrees that the FDA was too slow on vaccines. She has pointed out that the agency has moved pretty quickly relative to its normal vaccine authorization process. And that’s not the only thing Bastian finds a little unnerving.

“By the end of the year, Pfizer will have produced an estimated 3 billion doses, the most of any company,” she wrote in the Atlantic in August 2021. “That lightning-fast progress is awe-inspiring—and a little nerve-racking. … The FDA has to be thorough, especially with the first of a new type of drug with completely new production processes.”

A recent controversy over the FDA approval of an Alzheimer’s drug, Aduhelm, showcases the double bind the agency is in.

As Vox’s Dylan Scott has explained, the approval of the drug, which “came over the objections of [FDA’s] own scientific advisers, who cited a lack of evidence for the drug’s effectiveness,” has raised flags that innovation on future Alzheimer’s treatments will decline. Ultimately, Medicare decided to significantly limit which patients could receive the drug — according to the New York Times, officials “concluded that there remain significant doubts about whether the potential benefits of Aduhelm for patients outweigh the safety risks.” (A STAT investigation suggested troublingly close contact between Biogen, the drug manufacturer, and FDA officials.)

This episode pushes back against the narrative that the FDA is too conservative. Instead, it indicates that the relevant question may not be “how risk-averse is the agency?” but instead, “in what situations has it been willing to take risks?” In the case of Aduhelm’s approval, some have alleged that the agency has been too close to the pharmaceutical industry. But when it came to a worldwide emergency, suddenly an abundance of caution ruled the day.

And maybe, as physician Benjamin Mazer suggests, the problem is actually “the long-standing and gradual erosion of the agency’s scientific standards.” Mazer points out that 30 years ago, roughly 40 percent of drugs qualified for a regulatory shortcut, but by 2018, 60 percent of them did.

Whether the FDA’s caution is confined to its decision-making during the Covid-19 pandemic or if it suffers from a much deeper-rooted illness is still up for debate. At the very least, the scrutiny on the FDA demands a rethinking of whether the agency’s risk tolerance is in line with the nation’s best interests.