Facebook Admits Metrics Errors... Again

28 December 2016

Facebook Admits Metrics Errors... Again

The old joke about “lies, damn lies and statistics” may need to be updated to reflect Facebook’s apparent inability to get its metrics straight although, to be fair, this does appear to be a case of (a series of) genuine mistakes, rather than a deliberate attempt to mislead.

Previously this year…

Back in November, Facebook announced that it was committing to a major overhaul of its metrics-generation process, largely in response to the fact that it had been forced to issue a series of corrections to its initial metrics calculations.

The most embarrassing of these was the fact that it had made a very basic error in its method for calculating the “average duration of video viewed”. Essentially, it had based its calculation on the number of “views” (when a video is played for at least 3 seconds) rather than on the number of viewers (i.e. real people) choosing to watch the video. This blunder was serious enough to leave some of Facebook’s corporate customers seething.

Other errors, announced around the same time, were errors in the video completion rate (understated), errors in the organic reach of Facebook posts (overstated), errors in the average time spent on Facebook Instant Articles (overstated) and errors in the measurement of referral traffic from Facebook to external apps and websites (overstated). This last problem has yet to be corrected (as of December).

The latest update

Fast forward to the vital pre-Christmas period of mid-December and Facebook has released a new flurry of corrections. Presumably to vast embarrassment, these included new issues with the organic reach of Facebook posts (overstated) and Instant Articles (only for iPhone users and understated). There were further problems with videos, albeit only for live streaming (both overstated and understated) and tracking engagement with off-Facebook links (unclear).

Facebook is currently (mid-December) still working to fix both the second problem with the organic reach of Facebook posts and the problem with tracking off-Facebook links.

The impact in real terms

In any data-driven activity, it’s pretty much impossible to overstate the importance of having reliable data. Advertisers and marketers are therefore understandably furious that they have been making decisions based on inaccurate data.

The situation has been made even worse because Facebook has been perceived as reluctant to allow third-party monitoring agencies (i.e. agencies answerable to Facebook’s customers rather than to Facebook itself) access to Facebook’s raw data. Had external agencies been allowed to do their own measuring then it is entirely possible that the problem with the “average duration of video viewed” metric might have been picked up long before the two years it took for Facebook to inform its customers of the issue.

Facebook has since announced that it will open up access to third-party agencies to allow for more independent monitoring. It remains to be seen whether the moves they have made so far will be enough to placate its disgruntled customers.