It is an awkward fact that, except for extremely simple or routine tasks, it is impossible to make accurate predictions of the difficulty of achieving a goal. It usually takes more time and money than expected, and often we give up having achieved nothing. Similarly, when predicting things that result from human social behavior such as stock market moves or movie sales we find that we can do no better than predicting the stock price will be the same as yesterday or the movie sales will be the same as a similar movie. In these areas there are people who claim to be experts, but as a whole they do no better than the trivial prediction of sameness, and often somewhat worse.

The practical intractability of prediction is a straightforward consequence of Physical Chaos and the dynamics of complex systems. The Black Swan has much to say on the practical difficulties of prediction and the failure of experts to do any better than cabdrivers in predicting economic or political events. Gut Feelings has some wonderful examples of how financial experts do worse than chance and how we can do better than experts in sports or finance by averaging man-in-the-street opinions.

A subjectively similar phenomenon is that when we do achieve our goal we usually find that our resulting happiness is much less than we imagined, and our personal interpretation of the meaning is much different than we expected. We consider these paradoxes of Affective Forecasting elsewhere because they result in part from our failure to understand how our motivational systems work.

We have a difficult relationship with the fact of predictive intractability. One one hand, it something that “everyone knows”, but on the other hand, we persist in acting as though it isn't so. This is closely tied to our Positive Illusions. The beliefs that we are unusually competent, can beat the odds, and have more control than we do encourages us to believe that we can make accurate predictions even if nobody else can. It seems that we cling to our positive illusions because they foster an unjustified sense of optimism which helps us to persist in the face of uncertainty. This clueless persistence is the best we can do because we really don't know whether we will succeed or not (see Sunk Costs.) In other words, positive illusions are adaptive because they aid in risk-taking.

We are incorrigibly persistent in making and believing predictions in spite of the repeated failure of past predictions. Nassim Nicholas Taleb gives a wonderful analysis of this habit in The Black Swan. In particular, we like the idea of the narrative fallacy. When an unexpected event happens, we come up with a Story (theory) explaining why this event happened and why it was not expected, and then we somehow feel that we truly understand why the event happened. Also, because we understand where we went wrong and have learned our lesson, that error will never happen again, so our predictions will never be wrong again.

Of course, nobody says that they follow this silly reasoning, but most people act this way over and over again. In our retrospective analysis we need to find an explanation and determine blame, then we are satisfied and go back to business as usual. Taleb is clearly exasperated that his repeated cries that “the King has no clothes” seem to fall on deaf ears.

So perhaps all this is true, but what should we do about it? In many areas our predictions are better than chance, so it is worth making some effort at predicting.

In unpredictable areas with potential for dire consequences such as economic policy, politics and the environment, it would be good to cultivate some humility and a sense of caution. In pathologically unpredictable areas such as investment decisions we should reduce our expectations and not pay for lots of expensive advice.