The magnet detectives

The reading on the current-meter was gradually climbing. Until it wasn’t. In fact, it suddenly dropped. “Uh-oh,” Giorgio Ambrosio remembers thinking. “Houston, we have a problem.”

Ambrosio is a scientist at the US Department of Energy’s Fermi National Accelerator Laboratory who oversees the assembly of the US-built magnets for the high-luminosity upgrade to the Large Hadron Collider. By doubling the number of protons inside the LHC and improving the beam dynamics, the upgrade will increase experimental datasets by a factor of 10. This will enable physicists to study rare physical phenomena and learn more about the origin of mass and matter in the universe.

The HL-LHC upgrade needs 150 new accelerator magnets. In the US, the LHC Accelerator Upgrade Project is building new focusing magnets, which will squeeze the beam just before collisions and will be twice as powerful as their predecessors.

But the new focusing magnet that Ambrosio and his US colleagues were testing had failed; the magnet was dead.

The 5-meter-long electromagnets need to carry 16,530 amperes of current—roughly as much as a bolt of lightning. Because they become superconductors when cooled to very low temperatures, the magnets can carry this huge current without generating any heat. Scientists like Ambrosio introduce and gradually increase the current to test and “train” the magnets before use.

Ambrosio and colleagues like CERN fellow Alice Moros know that “things not working” is part of the process of science—especially when working with complex, one-of-a-kind, not commercially available technology. "A magnet, like everything else in the world, cannot be perfect," Moros says.

So, the US team put the magnet to the side and started testing another one. Again, the current ramped, and again, it suddenly dropped. They tried to revive it, but the current refused to climb: A second magnet was dead.

“The worst part is that this happened twice—on two magnets—back-to-back,” Ambrosio says. “It felt like being on the Titanic after it encountered the iceberg.”

Was it just two unlucky coincidences? Or was a weakness in the magnets’ design about to sink the entire LHC upgrade project? Finding the culprit was vital: The future of the LHC research program depended on it.

The unusual suspects

The production of HL-LHC focusing magnets involves hundreds of steps over the course of a six-month period. The problem could have arisen from any part of the process.

Ambrosio and his colleagues turned their attention to the first magnet that failed. What secrets could its untimely demise reveal? “We started the investigation right away,” Ambrosio says.

Suspect no. 1: the cable winding.

Each magnetic coil consists of a single cable, which is wrapped 50 times lengthwise around a 4.2-meter-long titanium pole. The cable is delicate because it consists of 40 twisted wires filled with brittle niobium-tin filaments. If one of these wires fell out of place, it could create a pinch point that would compromise the entire coil.

In the broken coil, “we had to fix a wire two times during the winding,” Ambrosio says. “So we had the question, did the wire move again?”

Suspect no. 2: an unexpected delay.

After winding, technicians transform the raw coil into a functional and sturdy superconducting magnet through heat treatments, applying binding agents, and soldering on components. The winding and curing usually take a couple of weeks, but their victim happened to be one they were working on when COVID first struck.

“We started the coil fabrication and then had to stop for 12 weeks,” Ambrosio says. “This is pretty unusual... But did it have an impact?”

Suspect no. 3: a broken symmetry.

The focusing magnets were quadrupoles, meaning they were each made from four separate coils fitted together to form a long and hollow cylinder. When the magnet is switched on at full current, electromagnetic forces pull the ends of the four coils with a 100-ton force.

This is usually fine: If the magnet is perfectly symmetrical, then the forces are equally balanced, and the strong support structure prevents damage to the delicate conductor. But what if they hadn’t been?

The support structures have small alignment gaps to compensate for the expansion and compression of the material when the magnets are cooled and warmed. Ambrosio and his team saw that the alignment gaps had closed in the broken coil’s support structure but had stayed open in the other three. Could that have condemned the coil?

A loose wire. An unexpected delay. A broken symmetry. Ambrosio and his team had gathered circumstantial evidence against each suspect. But that wasn’t enough. They needed an autopsy.

Physics forensics

A few months after the first failed test, a small, heavy package arrived at CERN Goods Reception. Its content, wrapped in foam, was a 20-kilogram chunk of the failed coil that had been extracted from the first magnet.

“The patient is dead,” Ambrosio says. “But we want to learn for the future ones.”

Scientists in the US had analyzed test data and made computer simulations to determine where the fatal flaw may have occurred. Now it was time to see if a small team of forensic physicists could confirm or rule out the three leading theories about what had gone wrong.

The first step was to send the coil to a lab in Germany for a CT scan. Much like doctors use CT scans to look for displaced vertebrae, the team used CT scans to investigate suspect no. 1: the cable winding.

"It looked fine at the level of the cable, so we knew we needed to go deeper,” says Mickael Crouvizier, a materials technician at CERN.

“Going deeper” would allow them to investigate suspects no. 2 and 3. But it would be trickier. Crouvizier and Moros—who were responsible for the forensic analysis—would need to cut the coil and extract several samples. Knicks, scratches and any other unforeseen blemishes introduced during this phase could contaminate the samples and compromise the entire operation.

Crouvizier and Moros used a diamond-studded wire to slowly slice through the coil, and two days of polishing and preparation to get the samples to a mirror finish.

"It is very similar to watch-making," Crouvizier says.

Then came the analysis. According to Moros, one of the most difficult parts of the job was differentiating between the damage that killed the coil and the damage of simply preparing the sample. They tried to use artificial intelligence, but it failed. “If you have scratches, if you have oxidation, the AI cannot recognize it,” Crouvizier says. “It’s a nightmare.”

They quickly realized that the only way to get accurate results was to do the analysis the old-fashioned way: inspecting every micron of the samples visually. Crouvizier and Moros spent several days hunched over a microscope. “It’s not so much the neck that hurts—it’s your back,” Moros says.

After weeks without unearthing any conclusive evidence, Ambrosio had an idea.

“We had been cutting [the coil] like salami—perpendicular to the wires—and finding nothing,” Ambrosio says. “But what if we cut it longitudinally—along the wires?”

“Salami” cuts are optimized to look for cracks that travel along the length of the wires. But more sinister damage—a tear that rips a delicate filament in half—is usually invisible to a salami cut.

Because longitudinal cuts traverse the length of the wire, they can reveal very small tears. But longitudinal cuts are very tricky to do; Crouvizier and Moros would have to cut very precisely along sub-millimeter wires without destroying the evidence they were looking for.

Crouvizier and Moros decided to try.

After a few weeks, Crouvizier saw something weird: a segment of wire in which each of several filaments appeared to be missing a chunk.

"When I saw the first events, I was not sure if it was an artifact from the polishing or something else,” Crouvizier says. “It was only after we repeated on several samples and saw the same thing that I knew we had found something.”

To be sure, they would need to repeat the analysis, going back to the “salami” cuts. “The salami cuts had to be close to the point of investigation, but not so close that they would destroy it,” Ambrosio says.

When they looked under a microscope, they could see small holes where the wires had ripped and delicate niobium-tin filaments had fallen out. Each point of damage was five times smaller than the width of a human hair. In total, they found hundreds of tears in a cable segment less than 2 millimeters long.

Were the tears a result of suspect no. 2, the delay in the coil fabrication process? Or of suspect no. 3, a broken symmetry during the final magnet assembly? The location of the damage was all they needed to identify the culprit. The micro-tears were all close to the joint where the cables turned around the support structure. According to Ambrosio, this is what the computer simulations predicted would happen if the coil had been over-stretched due to an asymmetry.

“It was the smoking gun,” Ambrosio says. “It was suspect no. 3.”

But suspect no. 2 wasn’t off the hook. Ambrosio says he thinks that modified timelines and procedures might have acted as an accomplice. “Because of COVID, the way this magnet was assembled changed,” he says. “We didn’t want people to be close, so we had them stand farther apart. But it’s much harder to make something symmetrical when you’re standing far apart.”

Luckily, these were the only two magnets that failed the testing process, and the scientists were able to learn from the incident. “Subsequent magnets have a design change and a procedure change to prevent this issue” Ambrosio says.

The first magnet was also given a second chance. Because a quadrupole magnet consists of four coils—and because only one out of those coils was incapacitated—the scientists were able to perform a transplant and bring the magnet back to life.

“We put in a new coil, re-tested the magnet, and it performed very well,” Ambrosio says. “That’s an important step, because it shows that you don’t have to throw away everything. If it happens again, we can replace just one coil—not all of them.”

The US team is planning to repeat the transplant on the other failed magnet.

Moros and Crouvizier say they are proud of the results of the investigation. "For me, I really feel that my work is useful,” Moros says. “It was super challenging, but really motivating.”

According to Ambrosio, solving this case was possible only because of the unwavering dedication of Moros, Crouvizier and many other colleagues at collaborating laboratories around the world.

“Teamwork is the key to success, even in a detective story,” he says.