One of my relatives recently picked up a copy of the Old Farmer’s Almanac. So, I asked what was inside. He said that one of the things inside the book was predictions about how warm or cold the next season would be, which is useful for farmers. I was skeptical, and thought “that sounds like something that can be tested mathematically”. Were their predictions any better than chance?
Some predicted values were very close to their actual values, but others were off in a major way. Adding up the numbers, the gap between their predicted value and the actual value was an average of 1.92 degrees.
For comparison, if you predicted nothing at all (i.e. zero temperature variation from normal), your prediction would be off by an average of 1.99 degrees. If you made predictions using random numbers between -2.0 and +2.0, you’d be off by an average of 2.12 degrees. Since their predicted values were between -2.0 and +2.0 in 15 out of 16 cases, this is probably a good comparison to random. So, the Farmer’s Almanac did slightly better than random, but not by much. When thousands of random predictions were generated between -2.0 and +2.0, the Farmer’s Almanac scored in the 78th percentile. This is the equivalent of correctly guessing 6 out of 10 coin flips, or having 2 or 3 of the 16 predictions being exactly correct, while the other 13 or 14 predictions being completely random.
Based on this, I’d say that either the Farmer’s Almanac predictions were no better than random but they had a good year (something that could be double-checked by looking at predictions in other years), or maybe they have some slight indication of the upcoming season (for example, if there’s a slight tendency for a warm summer to follow a cold winter).