Our recently retired MD Steve Wilcox considers the benefits and challenges of a larger panel size and finds that bigger isn’t automatically better – and when it comes to TAM panels can even lead to smaller effective sample sizes!
In the UK, BARB has a long and proud history of maintaining the highest quality audience measurement in a TV delivery ecosystem which has steadily become increasingly fragmented and challenging. The essence of this challenge is to strike the right balance between the accuracy of the measurement technique, the highest standards of survey design and execution, rigorous quality control and of course sample sizes which are big enough to measure a wide range of audiences with acceptable precision. The scrutiny with which BARB and its stakeholders review any change to any component of the system, bears testimony to their desire to understand the potential downsides of any proposed improvement in design or efficiency. There are always trade-offs, some can be painful.
One such trade-off is the investment needed to grow panel size. In a straitened economic climate it may be tempting to save money in one part of the system to invest in another. In this blog I’ll consider the issues.
TAM panels are usually under pressure to increase sample size and this is not a new phenomenon.
At the 2018 asi conference, RSMB calculated that 86% of the programmes reported by BARB each have less than 10 viewing panel members, and that didn’t happen overnight. So, how far would simply increasing reporting panel sizes help, for example by an additional 2,000 on a UK-like panel of 5,100 reporting homes? In purely mathematical terms, there would be a statistical improvement in both accuracy and precision for the bulk of the BARB audience measurement. It would also bring the next tier of channels into play by significantly reducing their numbers of zero reported programme audiences. In the real world context of hundreds of channels however, it must be recognised that it wouldn’t affect the long tail of very small programmes and ad breaks on the lower tiers of channels.
Overall, RSMB would never argue against an increase in sample size. Well, I don’t think it would. Unless there is some hidden downside?
In principle this is an easy one. All things being equal, if you increase the sample size, then you improve both accuracy and precision. Yes, there is a statistical law of diminishing returns and it is tempting to plot sampling error against increasing sample size, then look for the point of inflection and pronounce this to be the optimum sample size. In practice this doesn’t work, mainly because there isn’t a point of inflection! Further, this law doesn’t apply to all aspects of the reporting system. For example, the probability of reporting a small audience as zero decreases in direct proportion to increases in sample size. So, if we can afford to double the TAM panel size, then we reduce the sampling error by one third and halve the incidence of true audiences being erroneously reported as zero. What’s not to like?
Panel size is a key consideration but by no means the only consideration. However much an observer might understand the intricacies of TAM operations, it is easy to forget the extent of the interdependencies in the end-to-end processes which are the bedrock of users’ faith in the ratings they produce. The bottom line is that there is no point in building the panel size if the other elements of the system such as quality control of panel member compliance and data calculation are not maintained. In the UK for example, our experience is that panel size is not the first port of call when BARB need to investigate and defend the audience measurement. So what is?
There is no gain in building a big sample which is not representative of the viewing population. If we recruit too many homes in a particular demographic, just because it’s easy, then we have to down-weight their contribution to avoid a bias in the audience estimates. If each panel home costs several thousand pounds a year, any down-weighting of additional homes means that we are wasting a lot of money. And of course, once we’ve recruited an unrepresentative panel, it costs more to put it right and audience estimates are prone to panel turnover effects.
In the UK, BARB have never succumbed to the temptation of such convenience sampling, even under the pressure of achieving targets for a new panel build. In our view BARB has always made the correct decision to start reporting with an under-strength but well- balanced sample. This is always informed by a comprehensive and independent understanding of alternative recruitment strategies and projected outcomes.
Each panel home costs several thousand pounds a year... any down-weighting of additional homes means wasting a lot of money.
It’s amazing that rim weighting is now such common parlance in the audience research world. It’s variously seen as either the global saviour or the root of all evil. The truth is that it can be both. Its objective is to fine tune the representativeness of the panel in terms of a large number of demographic profiles and important behaviour related characteristics. But it’s sensitive and can produce high weights (destabilising) and low weights (rendering the panel home a waste of money). It needs nurturing, in particular to anticipate the effects of changes in panel balance and universe profiles. In particular, a poorly representative sample will put rim weighting under pressure. This in turn means that that any gains in increasing the sample size can be quickly lost if the panel is not well balanced, and you could end up with an effective sample size that is lower than that of your smaller sample.
Allowing poor quality homes into the reporting panel has the most serious impact on the reliability of the audience estimates; sampling error is a fact of life but this sort of bias is avoidable. Although the electronic meters provide a reliable measurement of viewing device activity, we are still reliant on the diligence of panel members to register their presence on a continuous basis. We cannot expect perfection, therefore there is a considerable skill in the panel liaison function to identify erroneous household profiles, to root out poor compliance and to continually refine the rules by which homes are excluded from reporting or dropped from the panel. We believe that BARB are world leaders in the rigour of their independent quality control procedures.
So what happens when things go wrong? When all of this comes together in a major reconciliation exercise, it is essential to be able to track the end-to-end components from design, through quality control to audience calculation. This requires a significant panel management information system and expertise that has to be available “on tap”. There is no doubt that this has come to the fore on the occasions when the direction of travel needs serious review. At this point, BARB and its stakeholders require a transparent and searching account of the issues, their impact on the data, proposed solutions and their projected outcomes. Otherwise, it is impossible to agree changes to a “black box”. Thankfully this doesn’t happen often because these same, comprehensive systems are usually preventative.
Of course, we’re not suggesting that anyone would wilfully sacrifice one for the other. We would always argue that the industry should invest in the largest panel sizes that it can afford. It’s important to be mindful of the trade-offs that might be made to free up the budgets. Just because the design and quality control systems have been running effectively with few visible problems, don’t assume that the allocated resource can be reduced.
A smooth running TAM system, such as BARB, runs well because of the balance between the different elements such as recruitment mechanisms to ensure a balanced panel, independent and continuous quality control to ensure panellists are properly recording their viewing and are properly classified and in-depth investigation of data anomalies to ensure problems are identified and resolved as quickly as possible. Disrupt that balance, for example by increasing panel size at the expense of quality control and there is a real possibility that the system will be destabilised and problems will occur. If fundamental changes have been made then gaps will appear and the necessary systems will be so much more difficult to recreate once they’re gone. It’s happened before.
"Bigger doesn’t necessarily mean better when it comes to TAM panels"
So in a nutshell: bigger doesn’t necessarily mean better when it comes to TAM panels, and should not be an objective in itself. The objective has to be the accuracy, precision and stability of television ratings, which after all is the currency on which the industry relies. To deliver that, every element of the system has to work well and in practice that means properly investing in each. If they don’t, it’s actually possible for a larger panel to result in a smaller effective sample size.
