====== Prediction is Intractable ======

It is an awkward fact that, except for extremely simple or routine tasks, it is impossible to make accurate predictions of the difficulty of achieving a goal.  It usually takes more time and money than expected, and often we give up having achieved nothing.  Similarly, when predicting things that result from human social behavior such as stock market moves or movie sales we find that we can do no better than predicting the stock price will be the same as yesterday or the movie sales will be the same as a similar movie.  In these areas there are people who claim to be experts, but as a whole they do no better than the trivial prediction of sameness, and often somewhat worse.  

The practical intractability of prediction is a straightforward consequence of [[chaos]] and the dynamics of complex systems.  [[books:black_swan]] has much to say on the practical difficulties of prediction and the failure of experts to do any better than cabdrivers in predicting economic or political events.  [[books:gut_feelings]] has some wonderful examples of how financial experts do worse than chance and how we can do better than experts in sports or finance by averaging man-in-the-street opinions. 

A subjectively similar phenomenon is that when we do achieve our goal we usually find that our resulting happiness is much less than we imagined, and our personal interpretation of the meaning is much different than we expected.  We consider these paradoxes of [[analysis:emotion:affective_forecasting]] elsewhere because they result in part from our failure to understand how our motivational systems work.

===== Positive Illusions =====

We have a difficult relationship with the fact of predictive intractability.  One one hand, it something that "everyone knows", but on the other hand, we persist in acting as though it isn't so.
This is closely tied to our [[analysis:bias:positive_illusions]].  The beliefs that we are unusually competent, can beat the odds, and have more control than we do encourages us to believe that //we// can make accurate predictions even if nobody else can.  It seems that we cling to our positive illusions because they foster an unjustified sense of optimism which helps us to persist in the face of uncertainty.  This clueless persistence is the best we can do because we really don't know whether we will succeed or not (see [[analysis:bias:sunk_costs]].)  In other words, positive illusions are  [[analysis:evolution:adaptive | adaptive]] because they aid in risk-taking.

===== Incorrigibly Confident =====

We are incorrigibly persistent in making and believing predictions in spite of the repeated failure of past predictions.  [[wp>Nassim Nicholas Taleb]] gives a wonderful analysis of this habit in [[books:black_swan]].  In particular, we like the idea of the //narrative fallacy//.  When an unexpected event happens, we come up with a [[analysis:mind:story]] (theory) explaining why this event happened and why it was not expected, and then we somehow feel that we truly //understand// why the event happened.  Also, because we understand where we went wrong and have learned our lesson, that error will never happen again, so our predictions will never be wrong again.  

Of course, nobody says that they follow this silly reasoning, but most people act this way over and over again.  In our retrospective analysis we need to find an explanation and determine blame, then we are satisfied and go back to business as usual.  Taleb is clearly exasperated that his repeated cries that "the King has no clothes" seem to fall on deaf ears.

===== The Best We Can Do? =====

So perhaps all this is true, but what should we do about it?  In many areas our predictions are better than chance, so it is worth making some effort at predicting.

In unpredictable areas with potential for dire consequences such as economic policy, politics and the environment, it would be good to cultivate some humility and a sense of caution.  In pathologically unpredictable areas such as investment decisions we should reduce our expectations and not pay for lots of expensive advice.

{{tag>unpredictability bias story chaos emergence}}