The Replication Crisis in Cognitive Neuroscience

Targeting the replication crisis in cognitive neuroscience to improve statistical significance.

In cognitive neuroscience and social behavior research, EEG research methodology aims to understand the human mind by studying the nature of brain activity associated with different activities or external environments. An important differentiator of EEG research, now that wireless, portable EEG headsets are available, is the ability to examine longitudinal brain activity and social behavior in real-world locations, rather than being confined to a lab.

What is the Replication Crisis?

The replication crisis refers to when researchers cannot replicate or reproduce the results of other researchers’ experiments. As a result, their findings cannot extend from the sample group to the general population.

Unfortunately, small sample sizes are at the core of current challenges in neuroscientific endeavors. Small sample sizes affect statistical significance, increase difficulty in drawing meaningful conclusions, and worsen the burgeoning replication crisis.

Because replication is such a crucial step in the scientific process, solving this replication crisis is imperative. If not, empirical results that cannot be reproduced undermine the credibility of the theories in question and therefore, any therapies, treatments, or laws that come about as a result of the process.

This post provides an overview of the replication crisis and how it affects neuroscientists’ ability to unlock the full potential of data collected to understand the real world. We will then introduce how 21st-century technology, like AI-mediated crowdsourced research, provides relief from the replication crisis.

The importance of reproducibility in neuroscience research

Modern day empirical research involves both obtaining and analyzing data. As such, considerations about its reproducibility fall into two questions:

1. Reproducibility: Does your experiment possess the validation of analyses and necessary certifications for the interpretation of the data?
2. Replication: Does your experiment have the capacity to be repeated to obtain new, independent data

Unfortunately, the answer to either question in neuroscience research is ‘no’.

In 2016, Nature surveyed 1,576 researchers and found that more than 70% of researchers tried and failed to reproduce another scientist’s experiment. And, more than half failed to reproduce their own experiments. Despite their failures, only 52% of researchers agreed there was a crisis, while 31% thought the results are probably wrong.

The inability to reproduce research results is typically due to the unique nature of the experimental conditions that cannot be compensated for or detected statistically. Everything from the weather that day, the individual lab technician performing experiments, and the analysis or statistical tools developed to evaluate experimental results can have unique elements that complicate replication.

Furthermore, because of the limited resources and expertise required for neuroscience research, its experiments tend to be undersampled and have WEIRD (western, educated, industrialized, rich and democratic) limited population samples already.

Why is Replication in Research Important?

Logistical constraints, such as recruitment issues, being restricted to a lab, and small sample sizes, have meant that researchers have relied on legacy research practices and patterns. As a result, researchers studying real-world social behaviors and personal information cannot run experiments diversely or creatively. These constraints have impacted neuroscience researchers’ capacity to transform their findings for real-world applications, such as therapies, treatments, and even laws. These issues, sample group constraints, and limited ability to test in real-world situations are at the core of the replication crisis.

The inability to replicate experimental results in a lab makes it difficult to draw conclusions with high statistical power. When there is doubt in the inferences drawn from experimental results, it undermines the confidence of the whole system. This doubt can also reflect negatively on the grant funding bodies tasked with supporting research that will have broad, real-world impacts.

Improved reproducibility often comes from pinning down research methods. Replicating results is improved when researchers achieve strict standardization of data collection, quality control, and analysis procedures in experiments with larger sample groups.

In the last decade, technological innovations in crowdsourcing software and hardware have empowered researchers to provide these core standardization elements that tackle the replication crisis, the statistical power problem, and reproducibility crisis.

Crowdsourcing Research Practices

Going global for more subjects with crowdsourced research

Crowdsourcing technology has developed rapidly in the last two decades. It is an approach that allows the researcher to collect much more data from individuals through a globally connected network. For example, consider the progress made in computer speed recognition. For decades this research was essentially at a standstill until companies like Google, Amazon, and Apple collected the cumulative input of hundreds of thousands of internet users. With such a massive data set, they could refine voice recognition software into what is commonly used in Google, Alexa, and Siri services today.

However, as with most technological innovations, crowdsourcing has been adapted to several commercial and scientific research practices. Zooniverse, Folding@Home, and Seek by iNaturalists are three of the most commonly known crowdsourcing platforms.

Zooniverse is the largest people-powered research platform, with two million accounts registered. Humans are uniquely adapted to identify patterns, while computers have difficulty. The general public can help researchers identify and classify everything from galaxy shapes to animal types. The Zooniverse community contributed to many discoveries and anyone can upload their data into a project open to the community.

Other popular crowdsourcing platforms include Folding@Home and Seek by iNaturalist. Folding@Home borrows computing power while the screensaver is active, calculating the shape and interaction possibilities of proteins translated from DNA codes. iNaturalists’ Seek is a mobile phone application that identifies the genus and species of any plant or animal via the camera.

In terms of data collection, Seek averages almost 200,000 images uploaded daily. Once uploaded, a community of over two million scientists and citizen scientists interact with the data serving as data quality analysts, suggesting and confirming identification accuracy.

Crowdsourcing and Neuroscience Were Always Meant to Be Partners

Crowdsourcing research is a way to improve the practical significance of experiments. It brings together individuals who have an interest in contributing to research and helping researchers make an impact. With crowdsourcing, the statistical power of these research practices increases. Researchers are more likely to detect a statistically significant difference when one exists because there is a big sample size to guarantee enough statistical power.

Recent developments in crowdsourcing technology and high-quality, portable research-grade EEG headsets and equipment offer hope to solving the replication crisis. Crowdsourcing research-grade data from thousands of diverse and dispersed individuals do appear to provide more substantial statistical power to research practices. To answer the most vexing innovation and research questions, crowds are becoming the partner of choice.

While some discussion continues about the replication crisis, EMOTIV has reframed the problem more as an opportunity—a challenge worth trying to solve—and they have made significant progress doing just that. To address the replication crisis in cognitive neuroscience, EMOTIV has developed a scalable distributed neuroscience research platform called EmotivLABS.

Getting Ahead of the Replication Crisis with EmotivLABS

We can accelerate our research by working together.

EmotivLABS is EMOTIV’s scaleable distributed research platform. Participants from around the globe can participate in neuroscience research with their own EMOTIV EEG headsets and be paid for their contribution.

An integral feature of the platform is its sophisticated quality assurance processes that ensure researchers acquire high-quality, research-grade data from participants. Users must complete a certification process: demonstrating that they know how their headset works and they can obtain high-quality EEG data. Once certified, users can participate in neuroscience research experiments on the platform and, in some cases, even receive compensation.

In addition to raw EEG, researchers also have access to band power data and a suite of affect and cognitive detection algorithms that include attention, frustration, interest, relaxation, engagement, excitement, and cognitive stress.

Research experiments can be built using EMOTIV’s Experiment Builder, then deployed to EmotivLABS. Connect and recruit from a global panel of certified participants, and collect high-quality EEG data all in one platform.

EMOTIV’s EEG headsets, paired with EmotivLABS address the replication crisis’s three main issues: recruitment logistics, statistical significance, and access to a more diverse, inclusive demographic.

Ultimately, as the number of disciplines and commercial markets embracing neuroscience tools and methodologies increases, EMOTIV’s low-cost, research-grade headsets is being used in neuroscience research, health and wellness, automotive, neuromarketing, consumer research, education, and entertainment settings.

Ultimately, innovations in neurotechnology of this magnitude allowed us to gain greater insight into our emotional and intellectual lives. Knowledge once out of reach due to legacy experimental designs and research practices. Applying such insights will give us greater control in consciously adapting our personal and professional lives to maximize performance and enrich our innate capabilities personally and in our relationships with others.

Learn more about how to enhance your research here.

Learn More About EMOTIV

Founded in 2011, EMOTIV is a San Francisco-based bioinformatics company with a mission of advancing our understanding of the human brain using custom electroencephalography (EEG) hardware, analysis, and visualization.

At the center of open science is collaboration. EMOTIV’s research platform and staff aim to promote scientific integrity and experimental rigor. EMOTIV scalable research platform, EmotivLABs, connects cognitive neuroscientists worldwide with a global population of research participants and investigators. Recognizing the additive linearity of neuroscience research, we aid researchers by providing extensive, multi-dimensional, rich data sets, allowing you to draw meaningful conclusions from a wide sample.

EMOTIV Headsets

EMOTIV Insight

EMOTIV serves a wide community of users, from professionals to individuals who simply seek to learn about their brains. EMOTIV Insight is a 5-channel EEG headset designed for brain-computer interface (BCI). Insight combines a sleek easy-to-set-up design with revolutionary sensor technology.

EMOTIV EPOC X and EPOC Flex

EMOTIV EPOC X is a 14-channel EEG headset and EPOC Flex is a 32-channel EEG system. Designed for neuroscience research in every setting, the EPOC headsets are wireless with improved sensor technology. Both headsets have been validated by independent research groups and are proven to provide high-quality research-grade data.

A complete comparison of the technical capabilities of EMOTIV’s EEG headsets is available on the EMOTIV website.

EMOTIV Technology

EMOTIV has designed a suite of tools to support every step of neuroscience research along the way.
EmotivPRO software allows users to process, analyze and visualize trial results. Researchers can also design experiments at the professional level in which any participant with an EMOTIV headset can participate if compliant with the experimental design.

A Software Development Kit (SDK) for EMOTIV is also available so that custom apps, interactions, or experimental designs can be performed on the go using the headset and smartphone alone.

As the number of disciplines and commercial markets embracing neuroscience tools and methodologies increases, EMOTIV’s low-cost, ease-of-use EEG headsets are being used in:

• Neuroscience research
• Health and wellness
• Automotive industries
• Neuromarketing
• Consumer research
• Education
• Entertainment settings