MRIRA annotation scheme shows high agreement for medication safety incident reports in NHSCan computers reliably read your hospital safety reports about medication errors?

Frontiers in Medicine Published April 18, 2026 Study authors: Huda Mohammad Alshammari, Denham L. Phipps, Elizabeth Le, Haifa Alrdahi, Penny J. Lewis, Riza Batist… DOI ↗ Editorial oversight: Dr. Julia Lee, PhD · Oncology, Genomics & Drug Development

AI-generated summary of the cited source, checked by automated accuracy review. How we work

Key Takeaway

Consider MRIRA for structuring medication safety reports, but note it's a methodological tool.

This methodological study involved two phases: Phase 1 developed the Medication-Related Incident Report Annotation (MRIRA) scheme by manually annotating 55 Controlled Drug incident reports, and Phase 2 evaluated it through independent annotation of 30 incident reports (15 Controlled Drug reports and 15 from the National Reporting and Learning System/Learn from Patient Safety Events) from the English NHS. The primary outcome was inter-annotator agreement across multiple metrics. Under strict evaluation, entity recognition showed high agreement with F1 scores of 0.85 and 0.91 across datasets, entity–relation extraction had high agreement with F1 = 0.75 and 0.83, event extraction had moderate agreement with F1 = 0.62 and 0.72, and event attribute tagging had acceptable agreement with F1 = 0.61 and 0.51. Agreement improved under relaxed matching criteria, though specific effect sizes were not reported.

Safety and tolerability data were not reported, as this study focused on annotation scheme evaluation rather than clinical interventions. Key limitations include that few annotation schemes have been developed specifically for medication safety and validated using real-world healthcare incident data, and the study design involved purposive sampling in Phase 1. Follow-up duration, funding, and conflicts of interest were not reported.

In practice, the MRIRA scheme provides a robust and reliable framework for structuring narrative medication safety reports, enabling systematic extraction of entities, events, and contextual relationships. This could support automated natural language processing tools and enhance organizational learning from medication-related incidents in healthcare systems. However, as a methodological study, it does not assess clinical outcomes or causality, and its direct application requires further validation in broader settings.

Imagine trying to teach a computer to read handwritten notes about medication mistakes. That is exactly what this study did. Researchers worked with reports from the English National Health Service to build a new way of organizing safety data. They wanted to know if a computer could accurately pull out who was involved, what happened, and what medicine caused the issue.

In the first part, they manually labeled 55 reports about controlled drugs. Then, in the second part, they had a different team label 30 reports to see if the computer's rules matched up. The results were mixed but promising. The system agreed with human experts 85% to 91% of the time when finding basic facts like drug names. It did even better at 75% to 83% when figuring out how those facts connected.

However, the computer had a harder time with more complex details. It only matched human experts 62% to 72% of the time for identifying the full event, and just 51% to 61% for tagging specific attributes. The researchers noted that using stricter rules made the agreement better, but the system is not perfect yet. This study was not a test of a new drug or a treatment, but a test of a new tool to help hospitals learn from their own safety reports.

This new framework gives hospitals a reliable way to structure messy safety stories. It allows automated tools to read these reports and help organizations learn from medication incidents. While the tool shows strong promise for finding basic information, it still needs work to handle the full complexity of safety events before it can be trusted for every job.

What this means for you:

A new computer tool reads safety reports well for basics but still struggles with complex details.

Study Details

Study typePhase2

EvidenceLevel 3

PublishedApr 2026

View Original Abstract ↓

Narrative reports of medication-related incidents contain valuable information about the causes and consequences of errors, but their unstructured format limits systematic analysis. Although natural language processing (NLP) can convert narrative reports into structured data, few annotation schemes have been developed specifically for medication safety and validated using real-world healthcare incident data. This study aimed to develop and evaluate the Medication-Related Incident Report Annotation (MRIRA) scheme, a multi-layer framework designed to structure narrative medication safety reports to support both qualitative analysis and automated text processing. Using narrative incident reports from the English National Health Service (NHS), a two-phase study design was implemented. In Phase 1, a purposive sample of 55 Controlled Drug incident reports was manually annotated to iteratively design the MRIRA scheme. The framework incorporated multiple annotation layers, including entities, events, attributes, and relations. The final scheme comprised 16 entity types, 11 event types, 5 attributes, 9 relation types, and 6 event argument roles. In Phase 2, two annotators independently applied the scheme to 30 incident reports, including 15 Controlled Drug reports and 15 reports from the National Reporting and Learning System/Learn from Patient Safety Events (NRLS/LFPSE). Inter-annotator agreement was evaluated using F1 scores under both strict and relaxed matching criteria. Under strict evaluation, agreement was high for entity recognition (F1 = 0.85 and 0.91 across the two datasets) and entity–relation extraction (0.75 and 0.83). Agreement was moderate for event extraction (0.62 and 0.72) and acceptable for event attribute tagging (0.61 and 0.51). All metrics improved under relaxed matching criteria, indicating greater consistency when allowing minor boundary variation between annotations. The MRIRA scheme provides a robust and reliable framework for structuring narrative medication safety reports. By enabling systematic extraction of entities, events, and contextual relationships from incident narratives, the scheme offers a high-quality annotated resource that can support the development of automated NLP tools and enhance organisational learning from medication-related incidents in healthcare systems

MRIRA annotation scheme shows high agreement for medication safety incident reports in NHSCan computers reliably read your hospital safety reports about medication errors?

Study Details

Mazdutide monotherapy reduces HbA1c and promotes weight loss in Chinese adults with type 2 diabetes over 48 weeks

New Drug Lowers Sugar and Shrinks Weight in Diabetes

Clinical research that matters. Delivered to your inbox.

MRIRA annotation scheme shows high agreement for medication safety incident reports in NHSCan computers reliably read your hospital safety reports about medication errors?

Study Details

Mazdutide monotherapy reduces HbA1c and promotes weight loss in Chinese adults with type 2 diabetes over 48 weeks

New Drug Lowers Sugar and Shrinks Weight in Diabetes

Clinical research that matters. Delivered to your inbox.

Related in Drug Pipeline

From Other Specialties