In late February, the CDC made big changes to its recommendations for monitoring and responding to Covid-19 surges. Now, as US cases are once more on the rise, these recommendations face their first test. But how will we know if they are working?

The CDC used to prioritize cases and positive tests to determine the Covid-19 threat level. Starting in February, the agency placed more weight on hospitalizations. The move invited a lot of scrutiny, and it reflected changes in the CDC’s pandemic response goals: The agency is moving away from trying to eliminate transmission of the virus and toward reducing deaths and health care system strain.

The hard truth, several public health experts tell Vox, is that determining whether they are effective will be difficult.

Even in the best-case scenario, where institutions follow the guidelines and the latest wave recedes, it would be hard to prove that the CDC’s framework deserves the credit.

“We’ll certainly know if it fails,” said Jeremy Faust, an emergency doctor and health policy expert in Boston. The guidelines face the same challenges many public health initiatives do: Failures are easier to spot than successes.

As a new wave begins, it’s worth setting some expectations about what these guidelines can reasonably do, and how easy or hard it will be to measure their success. Ultimately, we might never know how well the guidelines work — and even if they do work, the CDC might not get any credit.

What the new CDC guidelines changed, and why experts like them

For the first two years of the pandemic, there were two main metrics for determining the pandemic’s severity: case counts and test positivity.

Case counts were determined by summing up the positive results of PCR tests conducted in a given time period. And test positivity was determined by calculating the percentage of positive PCR tests. Together, these provided a rough, real-time picture of the Covid-19 threat, which public health agencies and institutions used to guide rollouts of testing programs, mask and vaccine requirements, and other public health measures.

For as long as PCR testing remained vastly more accessible than home-based testing, this approach made sense. However, at-home tests became more widely available over the latter half of 2021; because reporting those tests’ results is not mandatory the way reporting PCR test results is, PCR results have become increasingly unrepresentative of the actual state of play.

The proliferation of home tests rendered the CDC’s key metrics “almost functionally meaningless,” said Jennifer Nuzzo, an epidemiologist and pandemic preparedness expert at Brown University’s public health school. And so, Nuzzo explained, the CDC needed to find a new method for taking the temperature of the pandemic in real time.

The February guidelines did just that, introducing a new way of estimating each county’s Covid-19 burden. The calculation is still partially based on the rate of new cases over the past week, but now it is based largely on the number of new hospital admissions due to Covid-19 and the percent of hospital beds occupied by Covid-19 patients. From this, the CDC assigns each county a “low,” “medium,” or “high” level of burden.

CDC

For each level, the guidelines offer a set of recommendations for institutions and public health departments, and a separate one for individuals. The specifics of the recommendations range from ensuring testing and vaccine access on the low end to calling in backup health care staff on the high end.

Some people balked at the change, in part because hospitalizations are a lagging indicator of transmission intensity, rising one to two weeks after cases increase. However, the model used to create the guidelines accounted for that lag and deliberately set hospitalization thresholds at a level to allow institutions a few weeks to prepare for a rise in deaths.

The new framework also reflected a change in the CDC’s pandemic goals. No longer would the agency focus on eliminating transmission; instead, it would aim to prevent severe illness and death, minimize the burden on the health care system, and protect vulnerable people by using vaccines, therapeutics, and prevention strategies. The new estimates would help accomplish this by focusing on metrics that actually quantified the main indicators of health care system strain and setting the alarms to go off early enough to let public health authorities act.

CDC

Many public health experts felt the shifts were necessary, and organizations representing state, local, and county health officials reported broad support for the changes among their membership.

“A focus on hospitalizations makes a lot of sense right now,” said Justin Lessler, an epidemiology professor at the University of North Carolina’s public health school. He expects that with increasing population immunity, each wave’s severity will likely decrease, making case numbers less relevant. As case numbers do an increasingly bad job of predicting hospitalizations and deaths, there’s just less incentive to focus on them.

“We’d love to prevent infections, but that’s the hardest game of whack-a-mole,” said Nuzzo. However, she said, we can prevent severe illness and death, and “we can prevent our hospitals from becoming overwhelmed, and that is absolutely critical.”

How we’ll know if the guidelines are failing

At the moment, the CDC’s US outbreak severity map shows most counties in green, indicating they have a low community burden of infections.

But within the last month, a handful of counties have changed color to yellow or orange, indicating medium or high Covid-19 levels. Those color changes are intended to provoke public health authorities to make changes, like ramping up testing programs for asymptomatic people and restricting visitation in high-risk settings like nursing homes and prisons. Mask requirements are also on the menu, Nuzzo said.

The timing here is key: The color change is intended to happen early enough to provoke policy changes in time to prevent hospital bed shortages.

Here, we could see clear signs if the guidelines were failing.

If a county goes from green to orange, there should be time to flatten the curve before there’s a big strain on resources. “If we see hospitals overflowing and the CDC’s mask thresholds had not been met, that would be straightforward, incontrovertible proof that [the guidelines] failed to achieve the objective,” said Faust.

Mario Tama/Getty Images

Other red flags would include signs that state and local public health authorities and policymakers are not using the metrics to make decisions. That could suggest a number of problems, including a lack of health department resources, burnout among key personnel, a lack of trust in the CDC’s methods, or insufficient political will to follow the metrics and implement the changes the guidelines suggest.

After all, while the CDC’s guidelines are authoritative, they are not requirements; ultimately, state and local governments can do what they want.

“It’s not the metrics, necessarily, that I think are the thing to test, but it’s how we choose to respond to a change in the metrics,” said Nuzzo. “That’s the wild card.”

Why detecting the guidelines’ success is harder

To determine whether the guidelines are doing their job, we first need to define what it would mean for them to be successful — and that’s currently an open question, said Lessler.

For the CDC’s recommendations to be successful, state and local public health authorities need to use them as the basis for their policy recommendations; policymakers need to act on those recommendations; people and institutions need to follow those recommendations; and the recommendations need to have the desired effect of reducing transmission and increasing access to vaccination and treatment.

But just knowing where the guidelines are being implemented and where they are not is a challenge due to the decentralization of our public health system. Although Covid-19 policy trackers exist, differences in the particulars and the enforcement of different policies impede connecting the dots between mitigation efforts and outcomes. There are 3,006 counties in the US, and it’s hard to keep track of the policies in place in all of them.

“One of the arguments for a diverse public health system is it becomes a laboratory,” said Lessler, “but that’s only true if there’s some sort of central tracking and good reporting of what’s actually being trialed.” In a sense, the CDC’s new guidelines are an experiment in which results cannot be compiled in one place.

Spencer Platt/Getty Images

Another complication in evaluating the success of the guidelines is that individuals nationwide do what they think makes the most sense for themselves, regardless of local policy. That’s not necessarily a sign of anarchy. The CDC’s guidelines actually recommend that people use the agency’s suggested metrics to guide individual choices.

However, individual action tends to happen late in a surge, only “when things are obviously really bad,” said Joshua Salomon, a health policy professor at Stanford University. For example, people in a county where hospitals are overflowing might choose to wear masks even if their governor has forbidden mask mandates. Individual actions like this happening at a large scale change the outcomes, making it even more challenging to link those outcomes with policies.

It will be hard to know if guidelines are working if the BA.2 wave is small

There’s another major challenge to evaluating the new guidelines: If the burgeoning BA.2 omicron subvariant wave of Covid-19 is small, the guidelines may not face a big test at all.

Cases have been rising in the US, and hospitalizations are now rising in several northeastern states, albeit far more slowly than during the explosive wintertime omicron BA.1 wave. The sluggishness of BA.2’s spread (so far) may be attributable to the large number of people who have retained some immunity following infection during that earlier wave.

If BA.2 does not end up producing a large surge of infections in the US, “that will be a welcome surprise,” said Salomon, but “it won’t necessarily be validation of the new community guidance.” Our health care system can’t be threatened — and the CDC guidelines can’t be tested — by a surge that doesn’t happen.

Of course, a big test might be just over the horizon if a variant worse than BA.2 comes into play.

Even if the CDC’s guidelines help prevent disastrous outcomes, people may see the absence of catastrophe as evidence that the guidelines were unnecessary, not as evidence that they worked. Those situations are just as confusing as when people credit public health policies for good outcomes that would’ve happened anyway.

“If the CDC throws a mask mandate on and if things appear to get better, even then that will be correlation, not causation,” said Faust. “It’ll be really hard to tease out.