Did that Just Happen?

Much of the movement we see in brand trackers is simply noise. In this post we discuss how statistical significance can help to cut through that noise and confidently assert brand performance.

Chelsea manager Jose Mourinho¹ once said “I know all about the ups and downs of football”.  Sure.  Some weeks you win.  Some weeks you get thrashed by Dortmund 4-1.  To be fair, in the long run, it is about skill and money.  But game to game, anything can happen.  So, in this life, how much should you read into this week’s result?

Similarly, as a brand manager, your tracker goes up and down from month-to-month, as the consideration time-series below illustrates.  Previously we’ve noted that much of this movement is simply noise and, like football, there are ingenious methods available for smoothing out variations and extracting underlying performance.  But inevitably people, notably the rest of the business and their management, will also want to know what to read into this week’s result.

Retailer Consideration

The figure relates to a medium-sized retailer which ran three ad campaigns during that year and, at first glance, they each seem to have been effective, with consideration rising the following month for all three.  But the picture changes when we factor in the measurement uncertainty, which is shown by the error bars on each data point.  For the last two campaigns, the error bars overlap considerably with the previous consideration rates.  You cannot say that these campaigns had an impact.

But the first campaign is different.  Considerations rates were 31% and then rose to 36%, give or take 4%.  So consideration went up to at least 32% and may even have climbed as high as 40%.  This was a great campaign and applying the truth serum of statistical significance allows the brand manager to confidently adjust their commentary and future campaign strategy accordingly.

It also means that they can talk knowledgeably about what worked and what failed, thereby avoiding the contentless truths oft deployed by their footballing peers.  “Sometimes in football you have to score goals”, “we lost because we didn’t win,” and, “everything’s been really positive and smooth, apart from, obviously, the season” being wonderful examples from the beautiful game.

1. Actually it’s José Mário dos Santos Mourinho Félix if you’re short on banter for the office party.