Mode
Text Size
Log in / Sign up

ChatGPT-4o shows modest accuracy compared to human-executed steps in screening systematic review tasksAI Can Help Sort Research, But Humans Must Lead

AI-generated summary of the cited source, checked by automated accuracy review. How we work

Key Takeaway
Note that ChatGPT-4o shows modest accuracy in screening tasks, requiring human oversight for systematic reviews.

This meta-analysis assessed the performance of ChatGPT-4o against human-executed steps for screening and full-text review tasks. The study population involved emotional functioning after spinal cord stimulation, though specific sample size and setting details were not reported. The analysis focused on accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for these tasks.

ChatGPT-4o demonstrated modest to moderate accuracy in title and abstract screening at 70.4%, with a sensitivity of 54.9% and specificity of 80.1%. For full-text screening, accuracy was 68.4%, sensitivity was 75.6%, and specificity was 66.8%. Data pooling accuracy for five forest plots was reported as 100%, and no significant discrepancies were found in forest plot generation.

Safety and tolerability data were not reported in this analysis. Key limitations include the observation that ChatGPT demonstrates only modest to moderate accuracy in screening and study selection tasks, alongside minor discrepancies in tau-squared values ranging from 0.01 to 0.05. The study phase was not reported, and funding or conflicts of interest were not disclosed.

The practice relevance underscores the potential of AI to augment systematic review methodologies, while also emphasizing the need for human oversight to ensure accuracy and integrity in research workflows. Clinicians should interpret these results as indicative of current AI capabilities rather than a replacement for human expertise in systematic reviews.

Imagine sifting through hundreds of medical papers to find the right treatment. It is exhausting work that takes months. Now, imagine a tool that does the heavy lifting for you.

New research shows that artificial intelligence can help doctors and researchers organize medical evidence. However, it cannot replace human judgment just yet.

Finding the best way to treat chronic pain is hard. Patients often try many things before finding relief. Doctors need clear answers from research to help them decide.

But reading every single study is impossible for busy medical teams. They often miss important details or waste time on irrelevant papers. This slows down progress for patients who need help.

The surprising shift

For years, scientists believed only humans could review medical studies safely. We thought computers were too confused by complex medical language.

But here is the twist. A new test shows AI is surprisingly good at some specific tasks. It can read titles and summaries very well. It can also do the math for combining results from many studies.

What scientists didn't expect

The team tested ChatGPT on real research about spinal cord stimulation. This is a therapy that uses electricity to manage pain. They wanted to see if the AI could match human experts.

The results were mixed. The AI was not perfect at picking which studies to keep. It sometimes missed good studies or kept bad ones.

However, when it came to the math, the AI was flawless. It calculated the final numbers correctly every single time. It built the charts that show how well a treatment works without making mistakes.

The biology of the data

Think of a systematic review like a giant puzzle. You have thousands of tiny pieces from different hospitals and doctors. Your job is to sort the pieces and build the picture.

The AI is like a very fast assistant. It can quickly throw away pieces that clearly do not fit. It can also glue the matching pieces together perfectly.

But the AI sometimes looks at a piece and thinks it fits when it actually does not. A human expert must check the work to make sure the puzzle is correct.

The study snapshot

Researchers took an existing study on spinal cord stimulation. They used ChatGPT-4o to do the work. The computer looked at titles and abstracts first. Then it read the full text of the papers.

Finally, the AI combined the data from all the studies. It created the graphs that show the overall results. The team compared the AI's work to a team of humans doing the same job.

The AI was very good at the math part. It got 100% accuracy on the calculations. It found the right numbers for how much pain relief patients felt. The charts it made looked almost exactly like the ones humans made.

The screening part was different. The AI was right about 70% of the time. It was better at saying "no" to bad studies than "yes" to good ones. This means it might miss some helpful studies if not checked by a person.

This doesn't mean this treatment is available yet.

The tool is for researchers, not for patients to use at home. It helps make research faster and cheaper. But it needs a human to double-check the work.

Medical experts say this is a helpful new tool. It does not replace the doctor or the researcher. Instead, it acts like a powerful assistant.

It allows teams to look at more studies in less time. This could lead to better answers for patients with chronic pain sooner. But the human eye is still needed to catch small errors.

If you have chronic pain, this news is not about a new pill or device. It is about how doctors find answers.

Your doctor might use these tools to review new treatments faster. This could mean you get access to better options sooner. Talk to your doctor if you have questions about new therapies.

This study had some limits. It only looked at one type of pain treatment. It also used a specific version of the AI. Different studies might get different results.

The AI made mistakes in picking which studies to include. It missed about half of the good studies it should have kept. This shows why human oversight is necessary.

Researchers will keep testing these tools. They will try to make the AI smarter at picking studies. They also want to make sure the math stays perfect.

It will take time before these tools are used everywhere. Safety and accuracy come first. We must be careful not to trust the computer too much.

The future of medical research is a partnership. Humans provide the wisdom, and AI provides the speed. Together, they can help patients find relief faster.

Study Details

Study typeMeta analysis
EvidenceLevel 1
PublishedApr 2026
View Original Abstract ↓
INTRODUCTION: Artificial intelligence (AI), particularly large-language models like Chat Generative Pre-Trained Transformer (ChatGPT), has demonstrated potential in streamlining research methodologies. Systematic reviews and meta-analyses, often considered the pinnacle of evidence-based medicine, are inherently time-intensive and demand meticulous planning, rigorous data extraction, thorough analysis, and careful synthesis. Despite promising applications of AI, its utility in conducting systematic reviews with meta-analysis remains unclear. This study evaluated ChatGPT's accuracy in conducting key tasks of a systematic review with meta-analysis. METHODS: This validation study used data from a published meta-analysis on emotional functioning after spinal cord stimulation. ChatGPT-4o performed title/abstract screening, full-text study selection, and data pooling for this systematic review with meta-analysis. Comparisons were made against human-executed steps, which were considered the gold standard. Outcomes of interest included accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for screening and full-text review tasks. We also assessed for discrepancies in pooled effect estimates and forest plot generation. RESULTS: For title and abstract screening, ChatGPT achieved an accuracy of 70.4%, sensitivity of 54.9%, and specificity of 80.1%. In the full-text screening phase, accuracy was 68.4%, sensitivity 75.6%, and specificity 66.8%. ChatGPT successfully pooled data for five forest plots, achieving 100% accuracy in calculating pooled mean differences, 95% CIs, and heterogeneity estimates ( score and tau-squared values) for most outcomes, with minor discrepancies in tau-squared values (range 0.01-0.05). Forest plots showed no significant discrepancies. CONCLUSION: ChatGPT demonstrates modest to moderate accuracy in screening and study selection tasks, but performs well in data pooling and meta-analytic calculations. These findings underscore the potential of AI to augment systematic review methodologies, while also emphasizing the need for human oversight to ensure accuracy and integrity in research workflows.
Free Newsletter

Clinical research that matters. Delivered to your inbox.

Join thousands of clinicians and researchers. No spam, unsubscribe anytime.