Doctors, nurses outperform AI in emergency triage, with AI no better than chance in many cases, study finds

Doctors and nurses have outperformed artificial intelligence (AI) in triaging emergency patients, according to research presented at the European Emergency Medicine Congress in Vienna. The findings indicate that while AI can assist in prioritising critical cases, it lacks the reliability and clinical intuition required to replace trained medical staff

A study presented at the European Emergency Medicine Congress in Vienna, Austria, has found that doctors and nurses outperformed artificial intelligence (AI) in triaging patients in emergency departments. The research, presented by Dr Renata Jukneviciene of Vilnius University in Lithuania, suggested that AI might serve a supporting role but should not replace clinical staff in triage.

“We conducted this study to address the growing issue of overcrowding in the emergency department and the escalating workload of nurses,” said Dr Jukneviciene.

“Given the rapid development of AI tools like ChatGPT, we sought to explore whether AI could support triage decision-making, improve efficiency and reduce the burden on staff in emergency settings.”

The investigators distributed both paper and digital questionnaires to six emergency physicians and 51 nurses at Vilnius University Hospital Santaros Klinikos. Participants were asked to triage clinical vignettes drawn from 110 case reports available in the PubMed database. They categorised patients’ urgency level using the Manchester Triage System of which there are five levels. The same cases were also processed by ChatGPT version 3.5.

Of the participants, 44 nurses (86.3 per cent) and all six doctors (100 per cent) completed the questionnaires.

“Overall, the AI underperformed when compared to nurses and doctors across most metrics,” Dr Jukneviciene stated.

“AI’s overall accuracy was 50.4 per cent, compared with 65.5 per cent for nurses and 70.6 per cent for doctors. Sensitivity – how well it identified true urgent cases – for AI was 58.3 per cent versus 73.8 per cent for nurses and 83.0 per cent for doctors.”

Doctors achieved the highest scores in all analysed urgency categories.

“However, AI did outperform nurses in the first triage category – the most urgent cases – showing higher accuracy and specificity in identifying truly life-threatening cases. For accuracy, AI scored 27.3 per cent compared with 9.3 per cent for nurses, and for specificity AI scored 27.8 per cent versus 8.3 per cent.”

“These results imply that while AI tends to over-triage, it may err on the side of caution in flagging critical cases which can be both an advantage and a drawback,” Dr Jukneviciene observed.

In surgical cases, doctors scored 68.4 per cent in reliability, nurses scored 63 per cent but AI only managed 39.5 per cent. In therapeutic (non-invasive) cases, doctors scored 65.9 per cent, nurses 44.5 per cent whereas AI here achieved 51.9 per cent – exceeding nurses but still behind doctors.

Dr Jukneviciene said that while she had anticipated that AI would not surpass experienced clinicians, she was surprised that in the most urgent triage category AI achieved higher accuracy than nurses.

“This indicates that AI should not replace clinical judgement but could act as a decision-support tool in specific clinical contexts and in overwhelmed emergency departments. AI may assist in prioritising the most urgent cases more consistently and in supporting less experienced staff.

“However, excessive triaging could lead to inefficiencies, so careful integration and human oversight are crucial. Hospitals should approach AI implementation with caution and focus on training staff to critically interpret AI suggestions,” she concluded.

The research team has planned follow-up studies using more recent AI models and models fine-tuned for medical application. They intend to test them with larger participant groups, include electrocardiogram – ECG – interpretation and explore how AI might be integrated into nurse training for triage and mass-casualty incidents.

The study’s limitations include its small sample size, being conducted at a single centre and the fact that AI analysis occurred outside a real-time hospital environment, which prevented assessment within daily workflows, interaction with patients, measurement of vital signs or follow-up outcomes.

Moreover, ChatGPT 3.5 was not trained specifically for medical purposes. The study’s strengths included use of real clinical cases, participation of multidisciplinary staff, mixed questionnaire formats – digital and paper – relevance to practical challenges such as overcrowding and staff shortages and the identification of AI’s tendency to over-triage, a finding vital for the safe implementation of AI in emergency departments.

“AI has the potential to be a useful tool for many aspects of medical care and is already proving its worth in areas such as interpreting x-rays. However, it has limitations, and this study shows very clearly that it cannot replace trained medical staff for triaging patients entering emergency departments,” commented Dr Barbra Backus, chair of the EUSEM (European Society for Emergency Medicine) abstract selection committee and an emergency physician in Amsterdam, Netherlands. Dr Backus was not involved in the study.

“This does not mean it should not be used, as it could aid in accelerating decision-making. However, it must be applied with caution and under oversight from doctors and nurses. I expect AI will improve in future, but it must be tested at every development stage,” she concluded.