Essay: Robin Hanson

Photo courtesy of Robin Hanson

LHC forecasts: better than horoscopes?

My horoscope today says, “Focus on the small stuff.” Now, such advice does have content. It predicts that when readers interpret its words in the usual way as a guide to action, those who do what they think it recommends will, on average, feel they got more of what they wanted than those who ignored it.

Even so, astrologers sure don’t make it easy for us to test their claims. If they wanted to make it easier, they would do what forecasters in weather, sports, and business do: issue forecasts in a clear, standardized form. A horoscope is like a weather forecast that says, “Some rain likely nearby, soon.” It’s much easier to evaluate a forecast that a particular rain gauge has a 15% percent chance of collecting more than an inch of rain during the next calendar day.

When multiple forecasters each make many such forecasts on overlapping topics, we can use standard scoring rules to evaluate their relative accuracy. Such scores are a standard way to evaluate forecasts in weather, sports, and business, and to evaluate students on tests. Which bring us to the issue of forecasts related to the Large Hadron Collider.

Physicists are rightly proud that their theories usually support scoreable forecasts. While in private a physicist might suggest, “Something like this should work,” for a critical audience he or she will try to calculate probability distributions over yet-to-be-observed parameters. In contrast to horoscopes, physics forecasts are commonly clear enough to be scored against other forecasts.

Physicists are also proud of the LHC and its public image. The odds are good that within the next few years the public will see news articles in which big-shot theorists crow that some new LHC result has vindicated their approach to particle physics, and implicitly vindicated physicists in general for having supported that approach against skeptics who questioned their judgments on funding, etc.

In this context it is worth noticing: There are today no public predictions about LHC results by high-profile physicists that are stated precisely enough to be clearly scored for accuracy.

Yes, some have made qualitative LHC forecasts, such as these:

Brian Greene: The LHC could provide evidence for more than three dimensions of space.

Nima Arkani-Hamed: “They will find the Higgs particle. There’s also a pretty fair chance that they might observe some of the particles that make up the mysterious dark matter.”

Sir Chris Llewellyn-Smith: “A Higgs boson will be found (95% probability), and (with 60% probability) supersymmetry.”

These LHC forecasts are not without content, but as in the case of most horoscopes, scoring them may require lots of interpretation; in many plausible scenarios the accuracy of such forecasts could be unclear.

Clearer forecasts would be phrased in terms of new particle mass, spin, charge, decay rates, etc. And the academic literature does contain papers with more precise indications of what the LHC might see. But different papers predict different things, and prominent physicists have not picked particular papers to endorse. Yes, the space of possible new particle properties is very large, and each paper may only (with some probability) exclude some regions of this large space; but we have ways to score such forecasts.

The bottom line is that to score physicists for LHC forecast accuracy, we must either attempt the enormous task of averaging over forecasts from thousands of diverse academic papers, or we must apply great judgment to interpret a few high-profile quotes. Today’s LHC forecasts are no easier to score than the typical horoscope.

Yes, physicists eventually will form a consensus on what if any supersymmetry or Higgs-like process best explains the LHC data, and yes, that consensus will allow precise, scoreable forecasts of particle phenomena. But geez—the LHC costs more than $10 billion of public money. Shouldn’t we expect big-shot physicists who hope to crow to the public about LHC vindication to express their predictions in a more scoreable form? We don’t accept less from weather, business, or sport forecasters; why accept less from physicists?

Robin Hanson, an associate professor of economics at George Mason University, is a pioneer in the field of prediction markets, also known as information markets or idea futures.
 

Click here to download the pdf version of this article.