This area is a very tricky one. Most pollsters have a theory of how people will tend to answer a particular issue. But the actual wording and location of the question within the questionnaire may also influence how people will answer.

The assumption that underlies most sampling decisions is that you will have a 50% distribution of responses i.e. half of the people may be on either side. But in a case of an opinion poll where the answers may be skewed if driven by other considerations then the pollster has to be careful to consider the size of the sample. Margin of error for a study depends upon and includes the assumption made about the response rate distribution. This issue may yet be an important one for all of our pollsters given the deepening divide we are witnessing in our country that may only be resolved through better sampling or better weighting of the sample. Different groups will have different opinions and so it is critical for us to factor these biases of the respondents into our sampling decisions.

Another consideration is the timing of the survey. Both pollsters reported very tight field times (less than a week) and both indicate that the polls were done around the time of our Independence Day celebrations. There are two difficulties with this. Firstly, the tight timelines reduce the ability of the pollster to perform callbacks if a person is not available. This can compromise the quality of the survey that is the random selection process, depending on the way they handled the substitution. Secondly, the tight time frame would mean that the impact of the immediate circumstances – the Ferry issue etc. – will have a disproportionate impact on the assessment of the participants. Thirdly, the timing of the fieldwork may be negatively influenced by the celebratory mood or lack thereof of the respondents.

It is a well-known fact that the order and wording of the questions may influence the responses. In this both cases, the pollsters did not always disclose the exact question asked. Secondly, responses to early questions can influence the later ones. Even though they may ask the same question about the job approval rating for Dr. Rowley or the Opposition Leader, if the position of the questions comes after other questions, in one but not the other poll, it is likely that there will be different responses. This is why it is important to know not only what was asked but when a particular question was asked.

Finally, the question of weighting needs to be explained. What is weighting? Why does it happen? Weighting is a common task in polling and it attempts to get your final sample back in line with known population data. For example, CSO – our only official statistical organisation – publishes a census every ten years and from that, we know important data about age, gender, race etc. No sample is perfectly aligned to this database and as such we, pollsters, all practice weighting which is the adjustment of the sample to it. This is routine and to pretend otherwise is not wise. If we know that 30% of our population is in a particular age group and our sample picked up 40% then we need to adjust this group to reflect their true proportion. If we fail to do so then their responses will be given more importance than they should be given. The math to do this adjustment is simple and SPSS, the longstanding gold standard of data analysis, does it for pollsters with little more than a command.

This variability and the way it can influence headlines means that we, as pollsters, need to be more careful in our reporting of our findings. A good way to start is to declare who was sampled and how they were sampled so that the reader can judge how much to trust the findings.