In the wee hours of July 17, an old tree in the Swiss countryside let out a groan, then a crack. A moment later, about 35 miles away, CERN engineer Jose Antonio Ferreira Somoza heard his phone ding. Ferreira Somoza had received a coded message, a series of letters and numbers followed by the words: Vacuum is not OK.
The automated text, generated by the safety system in the Large Hadron Collider, indicated that the thermos-like casing on one of the particle accelerator’s magnets was filling with gas. “It’s very uncommon to have a problem in the insulation vacuum,” Ferreira Somoza says. “I thought it was a false alarm.”
The LHC is the world’s most powerful particle accelerator. It consists of a 17-mile-long ring of superconducting magnets. Each magnet is an intricate collection of electronics and cooling systems, all encased in a vacuum shell. If that shell was filling with gas, it would mean the vacuum was compromised, which could result in a damaged magnet.
Despite the fact that he had just begun his summer vacation, Ferreira Somoza connected his computer to CERN’s remote desktop and pulled up the vacuum controls and diagnostics software to investigate. “The vacuum pressure was increasing a lot,” he says. “Clearly something huge was happening.”
Ferreira Somoza started calling colleagues, including the on-shift operators in the CERN Control Center. They had detected a problem with the LHC’s cooling and were trying to restart the system, to no avail.
The old tree in the Swiss countryside that night had fallen on a set of power lines, which created an electrical perturbation that ricocheted through the power grid all the way to the LHC. The disturbance had caused one of the magnets to lose its superconductivity, which in turn caused it to heat up in a process called a quench.
Magnet quenches are relatively common in particle accelerators, brought on by causes as varied as thunderstorms and curious birds. To deal with them, the LHC is equipped with hundreds of safety systems programmed to immediately dump the particle beam and divert power away from the quenched magnet.
Once the machine is secured and the quenched magnet returned to its operating temperature, operators refill the LHC with protons and ramp it back up to full energy. The process takes about 2 hours and happens a few times a year.
But this time, even after the safety system powered down the affected magnet, its temperature and vacuum pressure continued to rise.
“The only thing we could do was wait for the pressure to stabilize,” Ferreira Somoza says.
Around 3 o’clock in the morning, he finally went to bed. But the problem had just begun.
Finding the leak
Early Monday morning, a team of CERN engineers were finally able to go underground to investigate. When they arrived, they found several of the 7-meter-long magnets covered in frost.
The LHC’s magnets operate at 1.9 Kelvin, just above absolute zero. The magnets sit in a bath of liquid helium, which is suspended inside a vacuum shell that encases the entire magnet. There was clearly a leak in the cryogenic system the magnets shared, but where was it? The technicians needed to locate it before they could address the problem.
“We’ve got one volume that is about 50 meters long, and we didn’t want to warm up the wrong section,” says CERN technician Graeme Barlow. “We knew we had a finite timeline to do this.”
The LHC is designed to contract when cooled and expand when warmed. The electrical cables that connect the LHC magnets in the affected section, for example, are protected by accordion-like bellows. But between thaws and refreezes, the magnets need to be checked, as their interconnections, which carry cooling and power between magnets, can shift in the process.
The LHC’s interconnections are like joints between bones. And like joints, they can be fragile.
The longer it took the technicians to fix the problem, the more the neighboring magnets would thaw, and the more chances there would be for the magnets’ joints to slip out of place or even break. They had no time to lose.
Barlow and around 20 colleagues from different departments worked in shifts. They used ultrasounds, X-rays and helium sniffers to examine one magnet at a time.
The investigation led them to the source of the leak: an interconnect between two magnets. But it was still too cold to work on.
On July 24, a week after the tree fell, the interconnect had finally reached room temperature. CERN technician Tavis Bampton and a small team of specialists went to the tunnel to start the repair.
When they opened the interconnect, they were sure they had found the right spot. “We could audibly hear gas escaping,” Bampton says.
When the magnet had quenched, the liquid helium that cooled it had expanded, surging into the bellow that protected the electrical wires. The affected bellow had a small kink, a weak point that could not quite withstand the rapid change.
The resulting tear was just 2 millimeters long, but it was enough to bring down the LHC.
The team confirmed that this was the only leak, then got to work on the repair. According to Bampton, fixing the LHC was like performing surgery while playing Twister. “You have to be very dexterous and somewhat of a contortionist,” he says. “It’s not the easiest environment to work in.”
This was the first time anyone had performed this type of repair on the LHC, and the team had to learn on the job. They’d never cut open a bellow before. “You can cut around a bellow with a pair of scissors, but if you need to cut latitudinally, it’s a nightmare,” Bampton says.
Their typical tools didn’t work. Finally, they decided to use a rotary saw.
They did not make the decision lightly. The LHC accelerates trillions of protons at close to the speed of light. Some of these protons are absorbed by the machine, making the material radioactive.
Much of this radiation is weak enough to be absorbed by a piece of paper. But cutting the material with a rotary saw creates radioactive dust, which can be inhaled. The team had to implement the highest standards of radiation protection, including protective clothing, specialized vacuums and extensive monitoring.
After several days of careful work, they were able to replace the broken bellow and reseal the interconnect.
Bampton sees this speedy repair as a big achievement for CERN, and multiple people from different teams and departments had to come together to make it happen. “The machine must go on,” he says.
Even though physicists lost a few weeks of data taking, there was a silver lining to the situation.
In 2026, CERN engineers and technicians will turn off the LHC and then replace multiple components as part of the High-Luminosity LHC upgrade. Fixing the leak in the interconnect gave them a practice run for that process, Bampton says. “This repair gave us a rare opportunity to work in the same environment as the HL-LHC before we start the upgrade.”
The team managed to repair the LHC before the neighboring magnets warmed, and CERN is preparing to start colliding lead nuclei for its regularly scheduled heavy-ion run this week.
A tree in the Swiss canton of Vaud fell on power lines and disrupted the power system, triggering the safety system in the LHC to dump the beam.
Safety, vacuum, cryogenics, magnets, engineering, powering, magnet protection, survey, beam instrumentation, and operations experts discussed how to tackle the problem and established a procedure on the spot.
The teams found a crack in the M2 bellows.
Experts led by Sandrine Le Naour and Said Atieh discussed possible repair solutions on site.
The vacuum and mechanical teams discussed the action plan while the repairs were in progress.
Graeme Barlow of the vacuum team installs a leak-test machine with Paul Cruikshank.
The team reinstalls the beam position monitoring cables, which had to be removed during the opening of the Q1-Q2 interconnection.
Wim Maan and Marcel Knoch check the tightness of the final weld.
The intervention teams prepare to close the affected magnet.
