CRIC Requirements for Reporting Survey Precision

Background

The American Association for Public Opinion Research (AAPOR) has stated that margin of error (MOE) calculations are only appropriate for probability samples, and that alternative measures should be used for non-probability samples. Other professional organizations, such as the World Association for Public Opinion Research (WAPOR) and the European Society for Opinion and Marketing Research (ESOMAR), have also recommended against using margin of error for non-probability samples.

In Canada, when reporting non-probability samples, it has become common practice for some companies to provide a comparison of MOE for a similar size probability sample. The media, however, is interpreting such a comparison as the non-probability sample being essentially the same as a probability sample. The use of MOE is a common complaint topic to CRIC and one that warrants an official position by our organization.

While research societies in no country is known to explicitly limit or restrict the use of margin of error for non-probability samples, it is widely considered inappropriate and not recommended by professional organizations in the research sector.

CRIC Requirements

Probability Samples: For probability sample surveys, results will provide an estimate of sampling error (margin of error). Additionally, it is important to disclose any limitations related to non-sampling error and the potential for bias in any conclusions drawn from the data.

Non-Probability Samples: Results written based on non-probability samples must clearly state that it is a non-probability sample. Such reports will only provide measures of precision if they are defined and accompanied by a detailed description of how the underlying model was specified, its assumptions validated, and the measure(s) calculated. Additionally, it is important to disclose the limitations of the sample and the potential for bias in any conclusions drawn from the data.

*Note, the Government of Canada Standards for Public Opinion Research state that for non-probability surveys, no alternative measures of precisions may be cited, and no statements may be made to indicate that the sample has a level of sampling error equivalent to a probability sample of similar size.

If a probability sample MOE is provided for comparison purposes, it must clearly state that a Margin of Error cannot be calculated on a non-probability sample.  The statement must appear in the methodology section only and not included with individual results.  If included, the worlding must be presented as follows:

A margin of error cannot be calculated on a non-probability sample.  For comparison purposes, a probability sample of ## respondents would have a margin of error of ##%, 19 times out of 20.

Additionally, it is important to describe the quality procedures in place to ensure the sample is representative of the intended population and to outline any limitations and the potential for bias in any conclusions drawn from the data.  The methodology should also list the panel or panels that the sample was drawn from.

A statement must be included in all releases indicating compliance with CRIC Standards: “Company Name” is a member of the Canadian Research Insights Council (CRIC) and readers can have confidence in the results because it was completed in compliance with CRIC standards.

CRIC Requirements for Open-link Surveys

Background

An open-link survey is when a link to a survey is shared publicly, allowing anyone to participate. This methodology has become increasingly popular as a public engagement tool. While this approach has some advantages, such as convenience, cost, and potentially larger number of responses, there are notable risks associated with open-link surveys, including bias, susceptibility to poor quality response (including fraudulent or spam responses or multiple responses or attempts by interest groups to manipulate results), and lack of representativeness.

CRIC Requirements

Researchers should be aware of the potential risks and limitations of implementing an open-link survey and take steps to minimize bias, where possible, and ensure the quality and integrity of the data collected. Methodology must disclose the limitations and the potential for bias in any conclusions drawn from the data.

Open-link survey data should never be combined with data collected separately from a probability or a non-probability sample survey data set.