when normative isn’t normal

Man circling a group on people on a window

One of the most common questions asked of market research is “how do my results compare to others?”  In-survey benchmarking and normative data held by research suppliers are the most common ways to answer the question about your results relative to others.

In this article we want to highlight some of the challenges that we see with normative data – from any and all research suppliers who make a claim to being able to provide normative data.  Normative data is attractive in principle but based on the evidence the use of norms is dangerous in general.  The following questions should be asked in every case where norms are being proposed:

    1. For advertising research.  What was the exact purpose for each ad represented in the normative database?  Were the ads intended to drive retention, acquisition, public image / reputation?  The purpose of the ad will affect who sees it and what messages they take away from it, so a norm used to compare your ad should reflect exactly the same “purpose” as the client ad being evaluated.  We are not aware of any normative database that can separate advertising norms based on the intent of the ads.
    2. How do the norms reflect usage within the category?  What is the brand usage profile of the respondents in every survey that generated the normative data?  For the most part large brands will be over-represented in normative databases and that is to be expected as those are the brands most likely to buy research.  Therefore, most norms will represent only the top few brands in a category.  With greater distribution and a larger base of users it is these leading brands that will usually generate higher scores (at least for ads).  If your brand is not in the top few, it can easily look the worse simply because you don’t reach as many people.
    3. What was the pre-existing strength of relationship between each respondent to the brands in the category and even the category itself?  If a sample for a survey is comprised of strongly loyal customers then many norms will be positively biased compared to a general market survey.  But, most normative databases contain little information about the sample profile.
    4. What geographic regions are represented?  If the norms are not based on the same geographic markets as your data then it is unlikely the norms are truly comparable to your results.
    5. What was the study methodology?  There is overwhelming evidence that data collected through the internet or some other form of self-complete survey will be different from that collected via telephone interview.  Studies using these different methods cannot be combined to create norms.  The norms should reflect only studies conducted with the same method that you intend to use in your study.
    6. How is the category defined?  Would you expect, for example, that advertising for life insurance is a valid benchmark for property and casualty insurance?  We have certainly seen research companies provide norms that are not for the same products even when in the same industry.
    7. How many individual studies are represented in the norm – do not accept multiple counts for waves of the same survey for the same client. We have seen normative data counts that are based on 1-2 brands (accounting for dozens of surveys and thousands of interviews) but they are simply represented as the number of respondents or surveys in the database without mention of the number of brands.
    8. How many individual companies are represented in the norm?  Most research suppliers work for only a few companies in any category so their normative data will tend to reflect only those clients.  And in many cases if they conduct research for Brand A they do not also conduct research for Brand B or C meaning their “normative” data is highly biased towards Brand A – their client.  It is doubtful that many research providers (perhaps none) can truly provide category norms for most categories.
    9. How old is the oldest study being reported as part of the norms?  Given the pace of change in consumer preferences and our marketplaces it is doubtful that any norm older than 2-3 years can be relevant today (an argument could be made that nothing older than a year should be considered.  If that is accepted then probably no normative database would contain enough recent data to provide a “norm” for any product or service.

We advocate in-survey benchmarking when it is done properly.  In a future article we will provide an overview of how to benchmark properly and what type of benchmarking you should use against your brand strategy.