Remote Data Collection: The Solution to WEIRD Sample Groups

Most psychological and neuroscience research methods rely on WEIRD (White, Educated, Industrialized, Rich, and Democratic) sample populations. This creates contextual and cultural limitations. If these limitations go unacknowledged, then the generalizability of research findings to the human population is limited. The broad-strokes, WEIRD approach to research has led to problematic applications of treatments, therapies, and even laws.

Research programs that employ narrow sample sets consisting solely of WEIRD participants can create harmful policies and practices. This is because they can be based on findings that may exclude people with underrepresented characteristics. A study published in the Proceedings of the National Academy of Sciences found that 96% of studies in six top American Psychological Association journals relied on WEIRD samples that represented only 12% of global population characteristics.

Complicating this problem is the fact that underrepresented populations may not be a minority in many cases. Research suggests that WEIRD individuals are actually outliers in many psychological measures—including spatial reasoning, visual perception, inferential induction, and moral reasoning.

To help alleviate the WEIRD bias, researchers can turn to online data collection tools that offer access to people outside a limited geographic area. By expanding the potential sample pool beyond locally accessible individuals, remote data collection platforms—like EmotivLABS—can increase the demographic diversity and representation in neuroscience research. This increase in diversity does not require a significant increase in the cost of subject recruitment and may be cheaper in the long run.

Expanding the subject pool globally can provide more statistical power to your research. It can also make it more likely to be reproduced — a major issue discussed in previous posts, 3 Approaches to Fix the Neuroscience Reproducibility Crisis and The Replication Crisis in Cognitive Neuroscience.

Remote data collection platforms directly address the WEIRD problem in modern psychological, neuroscience, and social science research. Given the availability and accessibility of online, remote research technology, psychological laboratories have no excuse to rely on WEIRD sample pools solely.

The Problematic Normalization of WEIRD Population Sampling

The first step in fixing a problem is acknowledging it exists. Mostafa Salari Rad et al. examined psychological research reports in the journal Psychological Science. The authors reviewed 428 studies and coded them based on the sample group’s geographic origin, education level, socioeconomic status/income, race/ethnicity, gender, recruitment techniques, and compensation. The vast majority of studies sampled (94.15%) exclusively used western, English-speaking participants.

Overall, it seems that researchers did not modify their research methodologies to address the lack of diversity and representation in the samples. Admittedly, modifying a lab’s experimental design and methodological approach may be prohibitive from a time and financial perspective.

Simply reporting details about sample demographics and accounting for them in conclusions can certainly be impactful while also far less expensive. Unfortunately, this simple practice has not become the standard. For example, a 2018 paper suggested that:

• Over 72% of abstracts reviewed contained no details about the population sampled.
• 83% did not perform a statistical analysis of possible effects related to sample diversity.
• 85% did not even discuss the potential impacts of context/culture.
• Only 16% recommended that future research be performed on the topic in other cultures or societies.

More recent research has reported similar results—a lack of diversity without acknowledgment of its potential limitation. E.Kate Webb and the National Center for Advancing Translational Sciences found that neuroscience tools (e.g., EEG) introduce racial bias into studies through exclusions of subjects based on phenotypic differences in hair type and skin pigmentation. The goal, as Webb concludes, is “to challenge racism in scientific work and propose procedures and changes that may lead to more equitable science”.

Overreliance on American Undergraduates

A big reason for the persistence of the WEIRD problem is an overreliance on American undergraduates as research subjects. In 2007, 67% of American studies published in the Journal of Personality and 20% in Psychological Science utilized American undergraduate students as test subjects. This is common because there is a genuine difficulty in gathering diversified subjects. Often, the process of selecting, recruiting, and compensating participants requires more time and energy than actually designing, administering, and analyzing the results of experiments. This means researchers spend more time focused on organizing participants than they do discovering insights about participants.

Increasing the Diversity of Sample Pools with Remote Online Research Platforms

Global communities

It is rapidly becoming standard practice to approach, recruit, and evaluate research subjects online. Using cloud-based research platforms frees researchers from logistical (advertisement, scheduling, registration) and physical (participant location) obstacles so they can dedicate more resources to their actual research.

These platforms promote cost reduction and increased efficiency while allowing researchers to access global networks of subject pools that better characterize the cultural context of the research.

Moving Beyond WEIRD psychology research

The WEIRD problem raises difficult questions about research reliability, generalizability, validity, and robustness. These studies often attempt to answer research questions about humans but employ individuals who represent less than 12% of the global population.

Several recommendations have been advanced to address the WEIRD problem. Usually, these recommendations involve more specific reporting requirements from the editorial boards of scientific journals. Given the necessity for researchers to publish their research, there should be changes to the submission requirements: reports must contain more detailed demographic information and discuss their results within the context of the sample’s demographic characteristics.

Overcoming logistical and physical challenges with EmotivLABS

The most frequently cited benefits of online neuroscience or psychological experiments include increased study sample size, varied demographic diversity quickly, and low-cost technology.

EmotivLABS integrates many of these best practices, including a certification process for participants that use Emotiv EEG headsets as well as data quality control that uses machine learning to verify signal quality. This allows research to conduct online experiments that generate high-quality data. In addition, Emotiv continually monitors platform usage and engages with researchers to inform regular release updates that improve both software and hardware.

Streamlined Subject Selection

Individuals and researchers are no longer required to meet in person for data collection. EmotivLABS streamlines subject selection, recruitment and compensation by matching each experiment with the most suitable individuals in the global subject pool.

Researchers can specify demographic attributes in their experimental design, similar to the granular targeting abilities of advertisements on social networks. The researcher’s experiment is automatically made available to certified individuals who best match the desired specifications. The certification process ensure data collected from this community are high-quality and complete.

Accessibility to subjects of different ages and ethnicities will increase studies’ power and provide more nuance to interpreting human data. As of 2021, EmotivLABS comprised individuals from more than 80 countries with diverse educational, occupational, and socioeconomic profiles.

EmotivLABS allows researchers to access participants outside the US and other western countries; a crucial factor in alleviating the WEIRD problem.

Budget With Ease

Emotiv’s EEG headsets are considerably less expensive than laboratory research devices and are easier to set up. Personal users, or citizen scientists, have their own EMOTIV headsets, so researchers don’t have to supply their own.

Initially, established researchers expressed skepticism about commercial EEG hardware and remote data collection. These scientists are uniquely aware of the challenges in collecting high-quality, valid signal data within the lab when subjects are directly under their supervision. However, numerous research teams have reported the validity of Emotiv EEG headsets. Most of these validation studies report equivalent research data between Emotiv systems and laboratory systems. This supports Emotiv headsets as a viable alternative for research-grade data collection.

Explore Un-Touched Research

Using portable EEG hardware permits researchers to ask new questions about humans, who are walking, talking beings that spend their lives outside of the lab. For example, studying social cognition in real-world interactions is a more desirable experimental design than using isolated individuals who are stuck indoors and immobilized. In addition, with this flexibility, longitudinal studies become more feasible.
Ultimately, it is not just that experiments are performed on limited, non-representative subject pools.

Most published research reports do not even address or detail the demographic attributes of their subjects beyond age and gender.

It’s also been demonstrated that WEIRD subjects are often outliers on the spectrum of human behavior. In other words, they are the least ideal sample group with which to gain insight into universal human characteristics.

If best practices and inclusivity are incorporated into your experimental design, online research can actually be more impactful and relevant than offline, in-lab research. In either case, the contextual framework in which the research question(s) are being asked must be thoughtfully considered.
There are pros and cons to both online and offline studies, and the specific circumstances of the research question(s) should dictate the researcher’s choice.

Want to Learn More About What EmotivLABS Could Do for Your Research?

Build your experiment with Emotiv’s Experiment Builder, then deploy it to EmotivLABS. Recruit from a global panel of certified participants to collect high-quality EEG data, all from one platform. Click here to learn more or request a demo.