March 26, 2015

Why ball-tracking can't be trusted

In the absence of information on how the technology works, it's hard for some of us to shake doubts about why what we're seeing with our eyes differs significantly from the reading of a computer

Imran Tahir's appeal against Martin Guptill looked straightforward but Hawk-Eye differed © AFP

I don't trust the data. Don't worry, this isn't another Peter Moores thinkpiece. It's Hawk-Eye or ball-tracker or whatever you want to call it. I don't trust it. I don't trust the readings it gives.

This isn't a flat-earth theory, though flat earth does come into it, I suppose. How can six cameras really predict the movement of a ball (a non-perfect sphere prone to going out of shape at that) off a surface that is neither flat nor stable? A ball that is influenced by constantly changing amounts of torque, grip, flight, speed and spin, not to mention moisture.

When Hawk-Eye, a prediction system with a known and well-publicised propensity for minor error (2.2 millimetres is the most recent publicly available figure) shows a fraction less than a half of the ball clipping leg stump after an lbw appeal, can we take that information at face value and make a decision based upon it?

During Tuesday's World Cup semi-final, my long-held conspiracy theory bubbled over. How, I wondered, could the ball that Imran Tahir bowled to Martin Guptill in the sixth over - the turned-down lbw shout from which the bowler called for a review - have passed as far above the stumps as the TV ball-tracker indicated? To the naked eye it looked wrong. The predicted bounce on the Hawk-Eye reading looked far too extravagant.

Worse, why upon seeing that projection did every single person in the pub I was in "ooh" and "aah" as though what they were seeing was as definitive and irrefutable as a ball sticking in a fielder's hands, or the literal rattle of ball on stumps? Have we just completely stopped questioning the authority of the technology and the data?

Later I checked the ball-by-ball commentary on ESPNcricinfo. Here's what it said:

"This is a flighted legbreak, he looks to sweep it, and is beaten. Umpire Rod Tucker thinks it might be turning past the off stump. This has pitched leg, turned past the bat, hit him in front of middle, but is bouncing over, according to the Hawkeye. That has surprised everybody. That height has come into play here. It stays not-out."

Why don't we question the authority of a technology that has a well-publicised margin of error? © Getty Images

It surprised me, but did it surprise everybody? Probably not. More TV viewers seemed to accept the call than question it. When you've watched enough cricket, though, some things just look a little off. To me this one didn't add up. Guptill made another 28 runs, not a trifling matter in the context of the game.

A disclaimer: though I distrust it for lbw decisions, I'm not saying that Hawk-Eye is all bad. It's great for "grouping" maps to show you where certain bowlers are pitching the ball, because tracking where a ball lands is simple. What happens next I'm not so sure on, particularly when the spinners are bowling.

To be fair, Hawk-Eye's inventor Paul Hawkins was a true pioneer and has arguably made a greater contribution to the entertainment of watching cricket on TV than many actual players manage. That's the thing, though: it's entertainment. In 2001, barely two years after Hawkins had developed the idea, it had won a BAFTA for its use in Channel 4's Ashes coverage that year. It wasn't until 2008 - seven years later - that it was added as a component of the Decision Review System. Quite a lag, that.

On its website, admittedly not the place to look for frank and fearless appraisal of the technology, Hawk-Eye (now owned by Sony) claims that the fact TV viewers now expect a reading on every lbw shout is "a testimony to Hawk-Eye's reputation for accuracy and reliability". But it's not, is it? All that really tells us is that we are lemmings who have been conditioned to accept the reading as irrefutable fact upon which an umpiring decision can be made. But it's a prediction.

Not even Hawk-Eye itself would call it a faultless system. Last December the company admitted it had got a reading completely wrong when Pakistan's Shan Masood was dismissed by Trent Boult during the Dubai Test. In this instance, the use of only four cameras at the ground (Hawk-Eye requires six) resulted in the operator making an input error. Why it was even being used under those conditions is more a question for the ICC, I suppose.

It's not all bad: Hawke-Eye gives great insight into where bowlers are pitching their deliveries © Hawk-Eye

The Masood debacle highlights an interesting issue with regards to the cameras though. Understandably, given the pay cheques at stake and that Hawk-Eye is a valuable component of their coverage, TV commentators rarely question the readings even in cases as puzzling as the Masood verdict. Mike Haysman is one who stuck his neck out in a 2011 Supersport article. Firstly, Haysman echoed my earlier thought: "The entertainment factor was the exact reason they were originally introduced. Precise decision-making was not part of the initial creative masterplan." The technology has doubtless improved since, but the point remains.

More worryingly, though, Haysman shone a light on the issue with the cameras upon which Hawk-Eye depends. At that point an Ashes Test, for instance, might have had bestowed upon it a battalion of deluxe 250 frame-per-second cameras, whereas a so-called lesser fixture might use ones that captured as few as 25 frames-per-second. Remember: the higher the frame rate the more accurate the reading. Put plainly, for the past five years the production budget of the rights holder for any given game, as well as that game's level of perceived importance, has had an impact on the reliability of Hawk-Eye readings. Absurd.

As a general rule, the more you research the technology used in DRS calls, the more you worry. In one 2013 interview about his new goal-line technology for football, Paul Hawkins decried the lack of testing the ICC had done to verify the accuracy of DRS technologies. "What cricket hasn't done as much as other sports is test anything," he started. "This [football's Goal Decision System] has been very, very heavily tested whereas cricket's hasn't really undergone any testing." Any? Then this: "It's almost like it has tested it in live conditions so they are inheriting broadcast technology rather than developing officiating technology." Does that fill you with confidence?

Hawkins and science-minded cricket fans might bray at the suggestion that Hawk-Eye can't be taken as law, but in lieu of any explanation of its formulas, machinations and the way it's operated (also known as proprietary information) it's hard for some of us to shake the doubt that what we're seeing with our eyes differs significantly from the reading of a computer.

Russell Jackson is a cricket lover who blogs about sport in the present and nostalgic tense for the Guardian Australia and Wasted Afternoons. @rustyjacko

Comments have now been closed for this article

  • Naser on April 1, 2015, 20:12 GMT

    I think overall, DRS is a good system that can help improve some decisions. The technology used is capable of judging any faint snicks better than naked eye and the bat pad decisions for lbw and close catching, generally has been significantly improved. However the ball tracking leaves alot of unanswered questions. Every ball turns and bounces uniquely and the DRS somewhat generalizes this fact. This is where major improvement is needed.

  • Francis on April 1, 2015, 9:34 GMT

    Many years ago (wish I could recall the game) the Channel 9 commentators were railing against the umpire for not giving an LBW. Hawk-Eye backed the commentators up, However a slow motion replay showed a quite thick edge that the commentators missed (at normal speed) as did Hawk-Eye. Some how the Umpire saw it. I've never paid Hawk-Eye much heed since then. Certainly laughed when I read it uses military technology (scud missiles perhaps). It's just another gimmick that we apparently need because the game is "boring".

  • Indian on April 1, 2015, 9:07 GMT

    It is indeed farcical that the accuracy of the system depends upon the number and quality of cameras being used. Why do we need this complexity and cost?!

  • amakan on April 1, 2015, 8:39 GMT

    Till we get something better I am very very happy with use of this technology. Umpires decisions are respected through this as we have Umpires call on all sides/top of wickets. So everyone (players/audience) is a winner.

  • randolf on April 1, 2015, 6:09 GMT

    Russell, I'm almost certain, that not only "science-minded cricket fans" who are apprehensive about the accuracy of Hawkeye's ball-tracking technology. I think that every cricket fan with common sense is so inclined. But what is good about it is the fact that none of us knows whether or not it is accurate on the spur of the moment - hence, we all accept its verdicts with significant unanimity, which is what the sport mainly requires. It's left to the operators of the technology only, to subsequently make some sort of honest confession - but when it doesn't really matter any more - the sting in the excitement already over! On the other hand, when an umpire via his natural instincts makes a mistake, every normally functioning spectator sees and knows! I'm also not too concerned about the the actual ball-tracking accuracy (which would improve with time); my main concern has to do with the possibility of the so called operator-on-call being able to instantaneously influence the decision!

  • Dummy4 on March 31, 2015, 5:35 GMT

    I disagree with Russell. The margin of error and the inherent problems in the predictive technology are well researched and published. Also, Hawk-Eye would give the same decision over and over for the same ball. Can the same be said for the umpires? Everyone knows that many times there are close calls and the decisions could go either way but some decision has to be given. What is important is that it is consistent and Hawk-Eye is currently the best.

    Issues with the DRS currently have more to do with the way in which the rules have been made and not with the technology itself. (a)third umpire should intervene when there are clear bloopers (b)increase no. of reviews and marginal calls shouldn't be counted (d)umpires can get two reviews per game (e)third umpires should be enforced to zoom in of the foot near the rope to avoid controversy (f)The ball should be in play till both teams complete their "action". Umpires could wait for the action to complete, e.g. delay LBW till run is over.

  • Philip on March 30, 2015, 13:26 GMT

    We have seen, heard and read so much about the DRS and another wonderful piece written here. Thank you

    Tracking or prediction via Technology should not be too far away from what the naked eye can see. A replay to show the slow motion is more than sufficient for the third umpire to have a rethink. It is not rocket science.

    The slow motion does help us in many ways. It human error with DRS too and why take that to be totally factual and not use common sense?

    We have see far too many dubious decisions by umpires and this is the very reason that the review system was brought in in the first place.

    The viewer is able to see what the umpire is seeing and not a ridiculous robot. let the decision be made by the umpire where ever prediction is required.

    The snicko and the hot spots are not 100% either. At least we are able to see point of contact on the snicko to establish if it relates to a particular contact.

    Hot spot in my view is hight subjective too.

    Philip Gnana, Surrey.

  • Manikantha on March 30, 2015, 13:00 GMT

    It is surprising to see people comment that no sport uses predictive technology to reverse umpires' decisions because I can't think of any sport that actually lets the umpire make a decision on his prediction either. The LBW law is such. The umpire HAS to predict based on what he has seen. It is not objective, it is based on assumptions whether or not the ball will go on to hit the stumps. And all those comments about various parameters to be considered such as wind, shape of the ball, moisture........ the list goes on, does the umpire compute all these before making his prediction? I'm guessing no. That being said I don't believe Hawkeye. And I don't believe it should be used to make decisions based on ball tracking. And I wish that the so called extensive testing be made public for the viewers to better understand the process.

  • Dummy4 on March 30, 2015, 9:44 GMT

    Brilliant article, totally agree, we have been having this predictive element debate since the CWC 2011 and the LBW OUT decision of Tendulkar from Saeed Ajmal. You can watch it a million times and it is plum Lbw and the umpire gave out immediately. Subsequent events showed that the only reason Tendulakr reviewed was because they knew Hawk Eye would show it missing. That decision was an engineered result and Pak were never going to be allowed to play the final in Mumbai. Anyone who has ever played or watched cricket with any kind of understanding, the predictive turn to show the ball missing leg stump was complete fabrication. HE should be used for the pitching of the ball, the height and should be frozen on-screen at point of impact with pad and then the 3rd umpire makes a decision based on distance and height. if he has any doubt then stay with umpires call. if Howler, then reverse decision.

  • Dummy4 on March 30, 2015, 5:36 GMT

    While I have my doubts about HawkEye, the bigger problem is around the protocol design of DRS in itself:

    1. What is the objective of DRS? if it is to prevent howlers how come only restricted number of referrals and that to with the players 2. What is the definition of benefit of doubt? and how do you implement this in the context of a so called precise system called HawkEye

    Around Hawk Eye itself, I would like to know 1. The test cases used to test the system? Are they available in public domain 2. How were the test cases simulated? 3. Where are the test results? again are they available in the public domain? 4. Has this testing been done is laboratory conditions or real conditions or both? 5. What are the variables that impact the accuracy of HawkEye? Is this available?

    The issue is more around the confused approach of the ICC. The lack of transparency around the whole thing?