The discussion arose again with the presidential elections in the United States in November 2024 as to what extent the results of pollsters based on surveys can be trusted, given that no institute was able to predict Donald Trump’s unchallenged victory. Irrespective of whether the polls were conducted by telephone or online, the surveyors largely consistently undervalued Donald Trump and his election victory. Basically, however, as a researcher, I am left with the conclusion that the choice of survey methodology (telephone vs. online) did not deliver more representative results. In the past, for example, YouGov has often been able to make impressively reliable predictions for the actual election result using its online panel. That is why telephone surveys are not automatically synonymous with more representative results for a study.
The validity of online research has long been debated, involving not only questions of whether the research methods were appropriate and whether the study was well-executed but also who was sampled and what conclusions were drawn. Online surveys are not ideal for descriptive research (e.g., opinion polls) as they do not form a representative sample of the general population. Compared to the general population, online users are younger, better educated, more liberal, and use the media more frequently. Certain groups of people (e.g., older users aged 55 and up, or with lower levels of education and political knowledge) are still rarely reached via a mobile or online survey, as they simply lack the technical affinity and media skills to use it.
Sampling is of fundamental importance for more valid online research
I have been involved in this debate over the last 25 years, ever since I set up one of the very first online panels in Germany in 1999. In hundreds of studies that I have carried out for my clients of the top 100 companies, such as Microsoft, Deutsche Post, and many others, I have performed valide surveys using international online panels.
The ComCult Research GmbH, my German company, applied an elaborate sampling process for each survey, which has already allocated individual subgroups according to age, gender, and education as well as soft factors (such as media use) in the sample. With the purpose to have a representative survey, we then kept inviting participants until all subgroups (e.g., women aged between 18 and 25 with a high school diploma) were filled. Anyway, such a time-consuming survey method using a stratified sample is no longer feasible today due to market demands for cost-effective and quick studies. By this I do not mean a simple quota sampling, as is common practice in the majority of these days’ online panels, where quotas are set for individual demographic characteristics that are only intended to ensure an overall proportion of, for example, 50% women in the study. Which is generally not sufficient to collect more valid data in the future.
Another practice is weighting the data in order to balance out the differences in the individual subgroups of the sample. Personally, I have never been a convinced practitioner of weighting after data collection, whereby participants from underrepresented subgroups with their responses are upweighted to bring the sample in line with the overall population, for example. Sadly, this does not correct existing biases in the study, as systematic errors, for instance in sampling, remain.
Based on my experience regarding reliable study results using their online panel, “sample framing” appears to be of fundamental importance for valid online research. A simple random sample would be desirable, in which each unit has the same probability of being drawn and included in the sample. As this cannot be guaranteed with either telephone or online surveys, the only option is to use a randomized, stratified sample. Only active sampling ensures that the right people are invited in the right proportions. Well, it is a labor-intensive task that can only be achieved with a high level of experience and expertise in sampling.
A well-maintained panel guarantees the accuracy of survey results
In the past 25 years, the use of computer-assisted questionnaires has led to an increase in very specific groups of people being surveyed, e.g., only customers of a certain brand, as complicated filter and selection criteria have become easy to set as part of the questionnaire. Yet, through excessive screening of participants in surveys, a conditioning of its members can be expected in every online panel. That is because respondents generally tend to answer according to what is awaited of them during the survey rather than commenting on what is actually asked. Panel members therefore increasingly try, consciously or unconsciously, to bypass the selection for the survey by giving answers that they believe the researcher wants in order to be given full participation in the survey. The outcome is a steady growing proportion of participants in each survey who only marginally belong to the target group of the study, causing study measurement errors. On account of this, studies in my proprietary online panel are not conducted with screening questions but rather with personal profiling for audience selection. Every survey should therefore have a 100 percent incidence and, if applicable, a pre-selection of respondents by targeting a specific audience. Only strict adherence to this principle will ensure valid results from an online panel in the future, based on truthful answers from the respondents.
Another important criterion is the respectful and trustworthy interaction with the test persons in the online panel. Over the past decade, I have advised a comparability platform of survey providers for recruitment of online panels, whose users have repeatedly commented on the need for improvement in respectful treatment and incentivization practices.
Also, a technically reliable survey process is of immanent importance for the trust of the participants. This is why there is no access to my online panel for third parties, just as, on the other hand, surveys are not allowed to be hosted elsewhere. Beyond that, participants’ incentives should be based on the half of the minimum wage in their respective countries in order to convey a sense of appreciation; a key element of this is managing an online panel in a socially responsible and ethical way. A well-maintained panel guarantees the accuracy of survey results.
Online Panel NET for rapid and cost-effective micro-surveys
Using Online Panel NET, you can carry out micro-surveys quickly and cost-effectively. Support your strategy with a small number of opinions. Conduct micro-research projects (e.g., tests) with your target groups or send your survey to a general population sample. A representative sample is weighted according to socio-demographic characteristics, in line with the country-specific census. Online Panel NET is small but effective – rather than increasing the size of the online panel, the focus is on response quality and country diversity. Reliable responses from well-maintained, real people.
In fact, there is no sample method anymore that has perfect coverage and known probabilities for any respondent in order to conduct representative studies. But surveys do not necessarily have to be always representative either for the population or for a target group. For example, online surveys appear well-suited for correlational research, so for questions that seek to understand whether two or more things are related to each other, or for experimental study designs, with questions that attempt to show one thing causes another thing. Fortunately, these categories capture most of the research, and most correlational and experimental findings found in representative samples can be replicated via online panels. Good for the use of a variety of research methods and test procedures, from A/B testing to UX research.
Requirements for more panel quality and new members
Every panel provider naturally has to deal with the quality of its members’ answers. The dangers of panel fatigue and panel conditioning thus mean that findings can be seriously skewed, especially as any online panel has members who are exposed to topics they are becoming experts on over an extended period of time. With no regular refreshing, under-represented target groups among panelists will become unreachable, such as those with low levels of sophistication. It would therefore be desirable to avoid any repetitive thematic questioning of panel members and to ensure a regular exchange between its members.
Currently, it is more difficult than ever to recruit a fresh replacement sample for an online panel. Regardless of whether the members were recruited by offline recruitment procedures or whether self-selection on the part of its members is permitted on the Internet. Competition between panel providers with regard to the size of their online panels is also of no use if the same people take part in their surveys over and over again. It would be advisable to set a limit to the number of permitted survey participations per member over a certain time period, e.g., a quarterly maximum of 10 surveys. As a subsidiary effect, the so-called “power users” would lose motivation, so the proportion of profit-oriented participation fraudsters would fall as well.
The Survey Scientists, a new brand for market research services
Running an international online survey panel today is more challenging than ever before. With this in mind, in 2024 I launched a new brand, The Survey Scientists, to ensure quality data and survey results for more valid online research. This also calls for the development of new study and questionnaire designs.
As an example, it is advisable not to select only your particular target groups for a study but also to consider an analysis of non-users. It is the comparison, e.g., of one’s own customer base with the general user base for the product group, especially from competitors, that provides far-reaching insights into one’s own brand. It is often only in the context of others that one’s own strengths and weaknesses can be recognized. In a large number of comparative studies, I have repeatedly been on the spot to contribute one of my research-relevant strengths to my clients. Cross-country comparisons are a matter of course for me as an international market researcher, so I always consider a study design with the option of conducting an international survey and develop it if desired.
My years of experience in the semantic translation of questionnaires, the country-specific selection of respondents in sampling, the timely programming of different language versions of questionnaires, the simultaneous management of several surveys in different countries, the processing of each data set (including the translation of open-ended questions), and the presentation of the study results in a country comparison are as natural to me as any other in-depth and custom-related research. Get in touch with me.
