“Balance is not something you find, it’s something you create” – Jana Kingsford
Getting a perfectly balanced sample is the dream for any research panel. Although in practice this is unachievable, the aim is to get as close as possible. However, getting the sample to the best possible state can feel like juggling multiple balls or balancing a number of spinning plates.
Panel balance is a measure of how closely a sample is aligned to given targets and these are typically based on population measures (unless there is deliberate disproportionality in the sample design). When targets are representative of a population then poor panel balance implies that the sample is misaligned with the population and conversely good panel balance suggests the sample is fairly representative.
It is important that a sample exhibits good panel balance for key demographics so that any resulting measurement accurately represents population behaviours. In the context of audience measurement, good panel balance ensures that differential viewing behaviours across demographic groups are accounted for in any data collected.
The panel will be weighted to ensure representation. If a sample is not well balanced then more weighting correction is necessary to correct for any imbalances. However, if a sample is sufficiently imbalanced then this increases variability in the measurement and may also restrict what demographics can be corrected for in this process and therefore the credibility of the audience estimates.
In an ideal world, at the recruitment stage all homes would have similar co-operation rates making the task of maintaining the desired demographic easier. However, in reality there will always be groups that are less co-operative when it comes to the recruitment process. These differences in co-operation rates are one key source of panel imbalances when recruiting homes onto a research panel.
The issue of differential co-operation rates is not limited to recruitment but also maintenance of a panel. Certain demographics tend to have a higher propensity to drop out of the panel and so need replacing at a greater rate. Additionally, if targeting fluid demographics like age groups, you will also need to account for migration of homes when considering how your panel will be replenished and maintained. Migration is when homes or individuals move from one demographic group to another; different demographic characteristics have different migration rates. For example, the ethnic group an individual belongs to does not change and therefore individuals remain in the same classification cell, whilst individuals may move age classifications as they grow older.
Differential recruitment, dropout and migration rates can all contribute to panel imbalances. To ensure that panel recruitment and management is as effective as possible, it is important to identify the key characteristics that influence likelihood of recruitment and co-operation. The panel recruitment and management should then be designed with these factors accounted for.
The flow chart below shows the steps that RSMB take to improve panel balance on the Barb panel.
Whilst having a panel that perfectly represents your population is the ideal situation, it is not usually possible to have a sample that replicates the population for every conceivable demographic. As a result, a key stage to creating a balanced sample is to identify the most important demographic factors that influence what it is you are trying to measure.
In the case of the Barb panel, the measurement is TV and other screen audiences. It is RSMB’s responsibility to identify the key demographic splits that explain as much of the variation in viewing across different homes and bring the measurement panel into line with the UK viewing population.
To identify these key demographics, RSMB carries out numerous variance analyses on actual viewing data. These analyses look at various demographics and types of viewing which enables RSMB to make informed recommendations on which demographic splits to use as panel controls. Panel controls are segmentations of the population/panel that are prioritised when looking to improve panel balance.
When making recommendations on the panel controls that should be used there are practical considerations to be accounted for. When considering how many panel controls to proceed with, the number needs to be indicative of the volume that can feasibly be targeted. The more controls that are targeted the less likely you are to hit the various targets but targeting too few controls will make your sample representative of only the few demographics selected therefore limiting the analyses that can be done on the output data. It should be noted that there are other practical considerations for the use of a potential panel control, and these include the robustness of representative sample sizes and the accuracy of classifications at recruitment. RSMB consider each of these factors when making panel control recommendations.
This exercise is carried out annually so that any changes in viewing behaviour and new potential demographics can be accounted for as soon as possible.
Demographic targets then need to be derived to determine what the demographic profile of the sample ought to be. This is done using up to date population estimates and the panel target sizes. The targets are updated monthly to account for any observed population changes and trends.These targets then feed into panel maintenance procedures that aim to improve panel balance.
The methodology behind recruiting homes onto your research panel will dramatically impact the panel balance of your resulting sample. Utilising a more simplistic recruitment approach, one that doesn’t account for the various factors that contribute to panel imbalances, will result in a panel with limited panel balance. Any recruitment method should account for the known differential co-operation rates and try to bias any selection of homes in a way that will benefit the balance of the panel.
In a perfect scenario, each new home joining the sample should improve panel balance and not add to any imbalances that may exist. However, due to the limits within the recruitment pool, it is not always possible to select only homes that improve panel balance for all controls. Therefore,it is important to target homes that are expected to improve panel balance the most while still fulfilling necessary replenishment of the panel volumes.
To recruit homes that are generally beneficial to panel balance onto the Barb panel, RSMB produces a probability matrix which segments the recruitment sample according to the recommended panel controls and assigns the home a probability which is then used in the recruitment process. These probabilities are derived based on panel requirements for each demographic relative to target and are revised monthly to account for the changing demographics and requirements of the panel. These probabilities account for what sample can be expected based on historical data and differential recruit and co-operation rates for different demographics. It is not possible to target all panel controls each month in every area as this would require an extremely large recruitment sample. RSMB instead takes a pragmatic approach and targets demographic controls where the panel balance is the worst at that point in time. Continuous monitoring of panel balance is required to determine which demographics should be targeted each month. This is where the ‘spinning plates’ analogy referred to at the start, comes in.
RSMB has developed a highly sophisticated algorithm that assigns each household on the Establishment Survey a probability based on the household geography and the demographic requirements of the panel.The probability is indicative of the proportion of homes of that type that should be attempted for recruitment. This algorithm accounts for any differential co-operation rates and sample availability.
Whilst it may be ideal to have a sample that is perfectly balanced for all key controls this is not usually possible in practice due to differential co-operation rates at recruitment and once on the panel. It is often the case that to reach desired volumes panel balance is compromised slightly. This can result in an abundance of those homes that are easier to recruit. To address imbalances of this sort, a level of enforced churn of the sample is beneficial and arguably necessary.
For the Barb panel, 5% of the panel each year are churned in this way. RSMB implements an algorithm that identifies the ideal households to be discarded from the panel to best improve overall panel balance.
A combination of discriminatory recruitment and a sensible level of enforced discards are both vital to ensure that panel imbalances are corrected for, and the panel moves towards being as representative as possible.
On the Barb panel, corrective weighting using population estimates is applied to the reporting sample to ensure that the resulting viewing data is representative of all controlled for demographics.
The more imbalance the weighting has to correct, the lower the effective sample size and therefore increased sampling error around the measurement and less cost effective.
Whilst RSMB are able to use corrective weighting to account for some panel imbalances, a well-balanced panel is still important to ensure that a diverse range of demographics can be corrected for. The more imbalance the weighting has to correct, the lower the effective sample size and therefore increased sampling error around the measurement and less cost effective.
In addition, if a demographic sample is under-represented on the panel, then the corrective weighting will assign individuals higher weights to compensate. The issue with this is that the resulting audiences become reliant on fewer individuals with higher weights which in turn causes volatility in these demographic audiences. On the other end of the weighting spectrum, the imbalance may also mean an over-representation of some demographics leading to very low weights with individuals effectively then barely used so not cost effective.
Under-represented samples can mean that the weighting process becomes strained or fails and so the demographic samples need to be robust to allow for corrective weighting to be effectively applied.
A balanced panel with robust samples is required to ensure that a range of demographics can be corrected for in the rim weighting and to therefore ensure stability of demographic audiences.
You can read more about rim weighting in a previous blog.
To make the decision on which controls should be targeted through recruitment and enforced discards we need to have measures in place to determine the balance of each control.
One measure that is used by RSMB to determine panel balance is the effective sample. The effective sample is an estimate of the sample size of a perfectly balanced sample that when unweighted would achieve the same level of precision as your weighted sample. Therefore, the objective of improving panel balance is to bring the effective sample size as close to the actual sample size as possible.
Control Efficiency is a measure also used by RSMB to determine how close to target each demographic split is within a control. A control efficiency of 100% implies the demographic split of the sample is aligned with the target volumes and an efficiency of less than 100% suggests the panel balance could be improved for that demographic split.
In addition to these two measures RSMB also look to see if there are any demographic groups with a sample under tolerance. This is a measure that accounts for under-representation that is even further under an allowable tolerance limit that is set due to the stochastic nature of recruitment. RSMB use this measure in combination with control efficiency to identify demographics for which it would be useful to carry out discriminatory recruitment.
In summary, at RSMB we believe panel balance is important in creating and maintaining an effective panel and it should be a key consideration for any recruitment or maintenance activity of your panel.
I hope this has been informative and helpful and not left you feeling too off balance!
