The discussion arose again with the presidential elections in the United States in November 2024 as to what extent the results of pollsters based on surveys can be trusted, given that no institute was able to predict Donald Trump’s unchallenged victory. Irrespective of whether the polls were conducted by telephone or online, the surveyors largely consistently undervalued Donald Trump and his election victory. Basically, however, as a researcher, I am left with the conclusion that the choice of survey methodology (telephone vs. online) did not deliver more representative results. In the past, for example, YouGov has often been able to make impressively reliable predictions for the actual election result using its online panel.
The validity of online research has long been debated, involving not only questions of whether the research methods were appropriate and whether the study was well-executed but also who was sampled and what conclusions were drawn. Online surveys are not ideal for descriptive research (e.g., opinion polls) as they do not form a representative sample of the general population. Compared to the general population, online users are younger, better educated, more liberal, and use the media more frequently. Certain groups of people (e.g., older users aged 55 and up, or with lower levels of education and political knowledge) are still rarely reached via a mobile or online survey, as they simply lack the technical affinity and media skills to use it.
Sampling is of fundamental importance for more valid online research
I have been involved in this debate over the last 25 years, ever since I set up one of the very first online panels in Germany in cooperation with Emnid (later TNS Infratest, Kantar Emnid) in 1999. In hundreds of studies that I have carried out for my international clients of the top 100 companies, such as Microsoft, Deutsche Post, T-Mobile, and others, I have performed representative surveys via the Internet using international online panels.
The ComCult Research GmbH, my German company, applied an elaborate sampling process for each survey, which has already allocated individual subgroups according to age, gender, and education as well as soft factors (such as media use) in the sample. With the purpose to have a representative survey, we then kept inviting participants until all subgroups (e.g., women aged between 18 and 25 with a high school diploma) were filled. Anyway, such a time-consuming survey method using a stratified sample is no longer feasible today due to market demands for cost-effective and quick studies.
By this I do not mean a simple quota sampling, as is common practice in the majority of these days’ online panels, where quotas are set for individual demographic characteristics that are only intended to ensure an overall proportion of, for example, 50% women in the study. Another practice to watch out for carefully is weighting the data in order to balance out the differences in the individual subgroups of the sample. No matter, additional questions on demographics, attitudes, and lifestyle are needed to weight and correct for any specific characteristics of the respondents.
In other words, based on my experience and confirmed by YouGov’s statements regarding reliable study results using their online panel, “sample framing” appears to be of fundamental importance for valid online research. Active sampling ensures that the right people are invited in the right proportions. Well, from personal and professional knowledge, it is a labor-intensive task that can only be achieved with a high level of experience and expertise in sampling.
A well-maintained panel guarantees the accuracy of survey results
In the past 25 years, the use of computer-assisted questionnaires has led to an increase in very specific groups of people being surveyed, e.g., only customers of a certain brand, as complicated filter and selection criteria have become easy to set as part of the questionnaire. Yet, through excessive screening of participants in surveys, a conditioning of its members can be expected in every online panel. That is because respondents generally tend to answer according to what is awaited of them during the survey rather than commenting on what is actually asked. Panel members therefore increasingly try, consciously or unconsciously, to bypass the selection for the survey by giving answers that they believe the researcher wants in order to be given full participation in the survey. The outcome is a steady growing proportion of participants in each survey who only marginally belong to the target group of the study, causing measurement errors. On account of this, studies in my proprietary online panel are not conducted with screening questions but rather with personal profiling for audience targeting. Every survey should therefore have a 100 percent incidence and, if applicable, a pre-selection of respondents by targeting a specific audience. Only strict adherence to this principle will ensure valid results from an online panel in the future, based on truthful answers from the respondents.
Another important criterion is the respectful and trustworthy interaction with the test persons in the online panel. Over the past decade, I have advised a comparability platform of survey providers for recruitment of online panels, whose users have repeatedly commented on the need for improvement in respectful treatment and incentivization practices.
Also, a technically reliable survey process is of immanent importance for the trust of the participants. This is why there is no access to my online panel for third parties, just as, on the other hand, surveys are not allowed to be hosted elsewhere. Beyond that, participants’ incentives should be based on the minimum wage in their respective countries in order to convey a sense of appreciation; a key element of this is managing an online panel in a socially responsible and ethical way. A well-maintained panel guarantees the accuracy of survey results.
Online Panel NET for rapid and cost-effective micro-surveys
Using Online Panel NET, you can carry out micro-surveys quickly and cost-effectively. Support your strategy with a small number of opinions. Conduct micro-research projects (e.g., tests) with your target groups or send your survey to a general population sample. A representative sample is weighted according to socio-demographic characteristics, in line with the country-specific census. Online Panel NET is small but effective – rather than increasing the size of the online panel, the focus is on response quality and country diversity. Reliable responses from well-maintained, real people.
In fact, surveys do not necessarily have to be always representative either for the population or for a target group. For example, online surveys appear well-suited for correlational research, so for questions that seek to understand whether two or more things are related to each other, or for experimental study designs, with questions that attempt to show one thing causes another thing. Fortunately, these categories capture most of the research, and most correlational and experimental findings found in representative samples can be replicated via online panels. Good for the use of a variety of research methods and test procedures, from A/B testing to UX research.
Invest in new technologies and innovative recruitment methods
Currently, it is more difficult than ever to recruit a fresh replacement sample for an online panel. The dangers of panel fatigue and panel conditioning thus mean that findings can be seriously skewed, especially as any online panel has members who are exposed to topics they are becoming experts on over an extended period of time. With no regular refreshing, under-represented target groups among panelists will become unreachable, such as those with low levels of sophistication.
During more than 15 years, I have commissioned most of the well-known online panels for my clients to support the study designs I have developed for various countries. As of today, I favor other sampling survey methods to complement the surveys in my own online panel. There is no sample method anymore that has perfect coverage and known probabilities for any respondent. This is one of the reasons I invest in new technologies and innovative recruitment methods to create pioneering solutions. As there is, the newly developed “ComCult Web App” for an innovative test method with opinion polls.
The Survey Scientists, a new brand for market research services
Running an international online survey panel today is more challenging than ever before. With this in mind, in 2024 I launched a new brand, The Survey Scientists, to ensure quality data and survey results for more valid online research. This also calls for the development of new study and questionnaire designs.
As an example, it is advisable not to select only your particular target groups for a study but also to consider an analysis of non-users. It is the comparison, e.g., of one’s own customer base with the general user base for the product group, especially from competitors, that provides far-reaching insights into one’s own brand. It is often only in the context of others that one’s own strengths and weaknesses can be recognized. In a large number of comparative studies, I have repeatedly been on the spot to contribute one of my research-relevant strengths to my clients. Cross-country comparisons are a matter of course for me as an international market researcher, so I always consider a study design with the option of conducting an international survey and develop it if desired.
My years of experience in the semantic translation of questionnaires, the country-specific selection of respondents in sampling, the timely programming of different language versions of questionnaires, the simultaneous management of several surveys in different countries, the processing of each data set (including the translation of open-ended questions), and the presentation of the study results in a country comparison are as natural to me as any other in-depth and custom-related research. Get in touch with me.
