How Much “Signal” is there in Your Brand Tracker? Part II

In a recent post we asked how much replicable signal there is in a typical brand image tracker. We went on to report that such survey questions probably contain around 35% real opinion and 65% noise¹. As such, the margin of error normally cited for a tracker rating, based on sampling error alone, is generally 2-3 times too small. So why are trackers so noisy and what can be done about it?

The fundamental problem is that brand trackers are trying to measure something that isn’t really there. People don’t have pre-formed brand beliefs and when you survey them about these non-existent beliefs they understandably make something up on the spot. So their survey responses are based on, in order of priority, what mood they are in, what you asked them earlier in the survey, whether they like the brand and, finally, what they actually think about, for example, the customer service.

The solution to this problem is to find questions that people can truthfully answer. And in the case of customer service levels, even if people don’t have a pre-formed abstract opinion about service image, they will have concrete memories about service events. Hence, it makes intuitive sense to try and track experiences rather than attitudes. The analysis of a client’s data from two overlapping parallel trackers supports this hunch.

Reliability of In-Service Events

The main finding cited earlier is that there’s a 35% R-Squared between the brand images obtained from the questions these two trackers have in common. Since R-Squared represents the amount of the variation in one survey explained by the other survey, this is a measure of how much content is shared between the two surveys. The remaining 65% of the observed rating variation comes from different sources of circumstantial noise, such as what’s been primed by the rest of the survey and so forth.

But the other intriguing result is that the amount of real opinion, compared to noise, varies between questions. As the graphic shows, the R-Squared ranges from 52% to 23%. The lower content ratings are associated with more traditional, abstract image questions about satisfaction with stock levels and the cleanliness of the store. The higher content responses are associated with more tangible, experiential questions about staff offering to help pack and the length of the checkout wait.

In summary, experience and behaviourally oriented type questions contain more signal and have less noise. So, don’t ask, “What do you think of our tracker?”. Instead ask, “Have you recently used our tracker to make a commercial decision?”

1. This view is also supported by the literature. For example see Dolnicar S. & Rossiter, J. R. (2008) The Low Stability of Brand-Attribute Associations is Partly Due to Market Research Methodology. International Journal of Research in Marketing, 25(2), pp:104-108 for further evidence regarding the depressingly low levels of content in brand image measures.