Scientific experiments are designed to determine facts about our world. But in complicated analyses, there’s a risk that researchers will unintentionally skew their results to match what they were expecting to find. To reduce or eliminate this potential bias, scientists apply a method known as “blind analysis.”
Blind studies are probably best known from their use in clinical drug trials, in which patients are kept in the dark about—or blind to—whether they’re receiving an actual drug or a placebo. This approach helps researchers judge whether their results stem from the treatment itself or from the patients’ belief that they are receiving it.
Particle physicists and astrophysicists do blind studies, too. The approach is particularly valuable when scientists search for extremely small effects hidden among background noise that point to the existence of something new, not accounted for in the current model. Examples include the much-publicized discoveries of the Higgs boson by experiments at CERN’s Large Hadron Collider and of gravitational waves by the Advanced LIGO detector.
“Scientific analyses are iterative processes, in which we make a series of small adjustments to theoretical models until the models accurately describe the experimental data,” says Elisabeth Krause, a postdoc at the Kavli Institute for Particle Astrophysics and Cosmology, which is jointly operated by Stanford University and the Department of Energy’s SLAC National Accelerator Laboratory. “At each step of an analysis, there is the danger that prior knowledge guides the way we make adjustments. Blind analyses help us make independent and better decisions.”
Krause was the main organizer of a recent workshop at KIPAC that looked into how blind analyses could be incorporated into next-generation astronomical surveys that aim to determine more precisely than ever what the universe is made of and how its components have driven cosmic evolution.
Black boxes and salt
One outcome of the workshop was a finding that there is no one-size-fits-all approach, says KIPAC postdoc Kyle Story, one of the event organizers. “Blind analyses need to be designed individually for each experiment.”
The way the blinding is done needs to leave researchers with enough information to allow a meaningful analysis, and it depends on the type of data coming out of a specific experiment.
A common approach is to base the analysis on only some of the data, excluding the part in which an anomaly is thought to be hiding. The excluded data is said to be in a “black box” or “hidden signal box.”
Take the search for the Higgs boson. Using data collected with the Large Hadron Collider until the end of 2011, researchers saw hints of a bump as a potential sign of a new particle with a mass of about 125 gigaelectronvolts. So when they looked at new data, they deliberately quarantined the mass range around this bump and focused on the remaining data instead.
They used that data to make sure they were working with a sufficiently accurate model. Then they “opened the box” and applied that same model to the untouched region. The bump turned out to be the long-sought Higgs particle.
That worked well for the Higgs researchers. However, as scientists involved with the Large Underground Xenon experiment reported at the workshop, the “black box” method of blind analysis can cause problems if the data you’re expressly not looking at contains rare events crucial to figuring out your model in the first place.
LUX has recently completed one of the world’s most sensitive searches for WIMPs—hypothetical particles of dark matter, an invisible form of matter that is five times more prevalent than regular matter. LUX scientists have done a lot of work to guard LUX against background particles—building the detector in a cleanroom, filling it with thoroughly purified liquid, surrounding it with shielding and installing it under a mile of rock. But a few stray particles make it through nonetheless, and the scientists need to look at all of their data to find and eliminate them.
For that reason, LUX researchers chose a different blinding approach for their analyses. Instead of using a “black box,” they use a process called “salting.”
LUX scientists not involved in the most recent LUX analysis added fake events to the data—simulated signals that just look like real ones. Just like the patients in a blind drug trial, the LUX scientists didn’t know whether they were analyzing real or placebo data. Once they completed their analysis, the scientists that did the “salting” revealed which events were false.
A similar technique was used by LIGO scientists, who eventually made the first detection of extremely tiny ripples in space-time called gravitational waves.
High-stakes astronomical surveys
The Blind Analysis workshop at KIPAC focused on future sky surveys that will make unprecedented measurements of dark energy and the Cosmic Microwave Background—observations that will help cosmologists better understand the evolution of our universe.
Dark energy is thought to be a force that is causing the universe to expand faster and faster as time goes by. The CMB is a faint microwave glow spread out over the entire sky. It is the oldest light in the universe, left over from the time the cosmos was only 380,000 years old.
To shed light on the mysterious properties of dark energy, the Dark Energy Science Collaboration is preparing to use data from the Large Synoptic Survey Telescope, which is under construction in Chile. With its unique 3.2-gigapixel camera, LSST will image billions of galaxies, the distribution of which is thought to be strongly influenced by dark energy.
“Blinding will help us look at the properties of galaxies picked for this analysis independent of the well-known cosmological implications of preceding studies,” DESC member Krause says. One way the collaboration plans on blinding its members to this prior knowledge is to distort the images of galaxies before they enter the analysis pipeline.
Not everyone in the scientific community is convinced that blinding is necessary. Blind analyses are more complicated to design than non-blind analyses and take more time to complete. Some scientists participating in blind analyses inevitably spend time looking at fake data, which can feel like a waste.
Yet others strongly advocate for going blind. KIPAC researcher Aaron Roodman, a particle-physicist-turned-astrophysicist, has been using blinding methods for the past 20 years.
“Blind analyses have already become pretty standard in the particle physics world,” he says. “They’ll be also crucial for taking bias out of next-generation cosmological surveys, particularly when the stakes are high. We’ll only build one LSST, for example, to provide us with unprecedented views of the sky.”
