What can colonoscopies teach us about cricket?

My theme is happiness and memory, but it begins with an unpleasant question: what can colonoscopies teach us about cricket? A colonoscopy is when… No, on reflection, I'm not going to go there. You can google.

Anyway, Daniel Kahneman, the psychologist who won the Nobel Prize for economics, used colonoscopies to devise an experiment that addressed a central question about human experience. Do we remember the amount of pain (or pleasure), do we remember the sum total? Or do we instead remember the peaks - what the pain or pleasure felt like at its most intense? And how important is our final recollection of the experience?

Two groups of patients experienced painful colonoscopies. Patients in Group A experienced the usual procedure. So did the patients in Group B, except a few extra minutes of mild discomfort were added at the end. Which group suffered more? Group B experienced all the pain of Group A, and then some more. And yet, because the procedure ended less painfully, patients in Group B said they minded it less.

The two crucial factors were the intensity of pain at its most intense and then the level of pain at the end of the experience. Kahneman called this the "peak-end" rule. In contrast, the overall duration of the experience had no effect on the way people rated total pain. Kahneman termed this "duration neglect".

In other words, our memories are very ineffective at quantifying the sum total of the pain (or pleasure) we have experienced.

I strongly suspect that the same biases affect our judgement of "great" and "less than great" sportsmen. First, we remember the highest achievement and the lowest dip rather than the sum of the competence. Secondly, our assessment of the whole career is distorted by the way the story ends.

The first bias leads us to underestimate excellent players who never achieved a concentration of brilliance. Compare the careers of Tony Greig and Andrew Flintoff. Greig has better overall statistics, but his achievements were spread out more evenly, which is why Flintoff is probably regarded as the greater player. During the 2005 Ashes, Flintoff was such a talismanic force that he became a national hero. We need not take anything away from Flintoff to see that we have missed something in our judgement of Greig.

And yet the first bias also leads us to misjudge essentially good players who suffered one mistake that lingers in the memory. The England footballer Gareth Southgate was an intelligent, underestimated player, who is unfairly remembered for missing a crucial penalty in the Euro 1996 shootout.

The second bias - the distorting effect caused by the way the story ends - explains why great sportsmen are so obsessed with their swansong. They intuitively grasp that the final chapter carries undue significance. They see their swansong as linked to their legacy. The problem, of course, is that holding on too long for the perfect ending has the unpleasant consequence of making the desired finale even harder to achieve. As you perform well less often, the probability of writing a happy final page becomes progressively less likely. If Sachin Tendulkar's career peters out, for example, it will disproportionately affect the way he is remembered (though his 81 last week shows he can still influence vital matches). Arsene Wenger, now eight seasons without a trophy at Arsenal, is already suffering that fate. He is an unquestionably great manager. And yet his relative lack of recent success is souring his whole reputation.

That is unfair, which is why I am now going to argue slightly against my own column here a fortnight ago. I think there are times when we should try to fight against the tyranny of our memory. We should try to remember the sum of the experience rather than only the "peak" and "end". We should try to be grateful for the total contribution of the player, not just the shortfall between what we see now and the memory of his greatest achievements.

A friend of mine has loved and studied Bob Dylan for 45 years. He is a world expert. And yet he invariably rushes to criticise Dylan's latest album. In fact, I sense that my friend's frustrations with Dylan's work today poison his attitude towards the singer's entire corpus. I try to take a different view. Given all the happiness Dylan has given me, I feel only gratitude for still further, new pleasures I get from his work today - even if it is a lesser type of pleasure that I now experience. Some fans believe that artists "owe" them an output of a requisite quality. I think it is me, the fan, who owes a debt of thanks.

The same problem afflicts great film directors. A colleague of mine recently complained that Woody Allen's To Rome with Love wasn't as good as Annie Hall or Manhattan. Well, no, it isn't as good. But can't we enjoy To Rome with Love without the experience being contaminated by comparisons with Annie Hall ? I think we can, and, indeed, that we should.

The same point may even apply to friendship. Must we remember a long, sustaining friendship that ends in brisk estrangement as a "failure" overall? Surely the happiness was real at the time?

All of which is another way of saying that fans ask too much of players. Watching sport when you care passionately about the outcome is often a frustrating experience. Why can't players perform at their best more often, especially when it matters most? Why can't they produce careers with the shape and narrative that gives fans the most satisfying experience? The short answer is because sportsmen are human beings. They are not entirely in control of form. The muse comes and goes.

I can't improve upon the story Dylan tells in his autobiography Chronicles. He had, metaphorically, lost his voice as a songwriter. He reached out for advice. A producer replied: why didn't he just write some new songs that were as good as "The Times They Are A' Changin" and "Masters of War"? Gee, thanks for the advice.

Sport, ultimately, is not about winning and losing, or even about achievement or excellence. It is about pleasure and entertainment. But our assessment of that pleasure is determined by our memory, and our memory is skewed by the "peak-end" rule and "duration neglect". Tastes and judgements, as Kahneman concluded, "are shaped by memories, and the memories can be wrong".

So we should, I believe, try a little harder to remember the sum of the happiness that great players have given us, not just bemoan the shortfall between their finest hour and their present ordinariness. For our own sakes, as well as theirs.

My thanks to Kieran McMaster, an ESPNcricinfo reader and cricket statistician. McMaster wrote a brilliant response to my column about measuring greatness two weeks ago, reminding me that Kahneman had addressed similar questions from a different perspective. I interviewed Kahneman here but I'd failed to make the connection until McMaster prompted me.

Another ESPNcricinfo reader, Xolile, made an observation that is now even more pertinent, given that this column at least partially opposes my previous one: "When Ed first started writing I optimistically judged him to be a man who values reason. But time has revealed that Ed has a preference for the heroic, the sublime and the mysterious."

I can't imagine a more central tension. And, with luck, I look forward to examining that tension for the rest of my life - without, I hope, ever entirely resolving it.