New Research Shows Promise and Limitations of Physicians Working with GPT-4 for Decision Making

Published in JAMA Network Open, a collaborative team of researchers from the University of Minnesota Medical School, Stanford University, Beth Israel Deaconess Medical Center and the University of Virginia studied how well doctors used GPT-4 - an artificial intelligence (AI) large language model system - for diagnosing patients.

The study was conducted with 50 U.S.-licensed physicians in family medicine, internal medicine and emergency medicine. The research team found that the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources. Other key findings include:

  • GPT-4 alone demonstrated significantly better scores in diagnostic performance, surpassing the performance of clinicians using conventional diagnostic online resources and clinicians assisted by GPT-4.
  • There was no significant enhancement in diagnostic performance with the addition of GPT-4 when assessing clinicians using GPT-4 against clinicians using conventional diagnostic resources.

"The field of AI is expanding rapidly and impacting our lives inside and outside of medicine. It is important that we study these tools and understand how we best use them to improve the care we provide as well as the experience of providing it," said Andrew Olson, MD, a professor at the U of M Medical School and hospitalist with M Health Fairview. "This study suggests that there are opportunities for further improvement in physician-AI collaboration in clinical practice."

These results underline the complexity of integrating AI into clinical practice. While GPT-4 alone showed promising results, the integration of GPT-4 as a diagnostic aid alongside clinicians did not significantly outperform the use of conventional diagnostic resources. This suggests a nuanced potential for AI in healthcare, emphasizing the importance of further exploration into how AI can best support clinical practice. Further, more studies are needed to understand how clinicians should be trained to use these tools.

The four collaborating institutions have launched a bi-coastal AI evaluation network - known as ARiSE - to further evaluate GenAI outputs in healthcare.

Funding for this research was provided by the Gordon and Betty Moore Foundation.

Goh E, Gallo R, Hom J, Strong E, Weng Y, Kerman H, Cool JA, Kanjee Z, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Rodman A, Chen JH.
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial.
JAMA Netw Open. 2024 Oct 1;7(10):e2440969. doi: 10.1001/jamanetworkopen.2024.40969

Most Popular Now

When it comes to Emergency Care, ChatGPT…

If ChatGPT were cut loose in the Emergency Department, it might suggest unneeded x-rays and antibiotics for some patients and admit others who didn't require hospital treatment, a new study...

MEDICA 2024 + COMPAMED 2024: Adapted Hal…

11 - 14 November 2024, Düsseldorf, Germany. The final preparations for MEDICA 2024 and COMPAMED 2024 in Düsseldorf have begun. A total of more than 5,500 exhibitors from approximately 70 countries...

AI does Not Necessarily Lead to more Eff…

The use of artificial intelligence (AI) in hospitals and patient care is steadily increasing. Especially in specialist areas with a high proportion of imaging, such as radiology, AI has long...

AI Drives Development of Cancer Fighting…

University of Houston researchers and their students are developing a new software technology, based on artificial intelligence, for advancing cell-based immunotherapy to treat cancer and other diseases. CellChorus...

Revolutionizing Cardiovascular Risk Asse…

A recent position paper in the Asia-Pacific Journal of Ophthalmology explores the transformative potential of artificial intelligence (AI) in ophthalmology. Led by Lama Al-Aswad, Professor of Ophthalmology and Irene Heinz...

HWL 2024 Brings Together a Record Number…

1 - 2 October 2024, Luxembourg. The second edition of Healthcare Week Luxembourg on 1 and 2 October 2024, organised by the Federation of Luxembourg Hospitals (FHL), in partnership with the...

Why the NHS Needs a Transparency Revolut…

Opinion Article by Dr Mark Ratnarajah, NHS paediatrician and UK Managing Director for C2-Ai. Wes Streeting wanted 'no stone left unturned' when he asked Lord Darzi to examine the current state...

Why the NHS is Seeking to Make Media Ser…

Opinion Article by Dean Moody, Healthcare Services Director, Airwave Healthcare. Tim Kelsey and Martha Lane Fox called for WiFi to be made available free of charge throughout the NHS back in...

An AI-Powered Pipeline for Personalized …

Ludwig Cancer Research scientists have developed a full, start-to-finish computational pipeline that integrates multiple molecular and genetic analyses of tumors and the specific molecular targets of T cells and harnesses...

Great Start for Ideas and Innovations: D…

8 - 10 April 2025, Berlin, Germany. From 15 October to 15 November 2024, the DMEA invites experts from business, science, politics and practice to actively participate in shaping the congress...

Commission Joins Forces with Venture Cap…

The Commission has launched a Trusted Investors Network bringing together a group of investors ready to co-invest in innovative deep-tech companies in Europe together with the EU. The Union's investment...

AI could Transform How Hospitals Produce…

A pilot study led by researchers at University of California San Diego School of Medicine found that advanced artificial intelligence (AI) could potentially lead to easier, faster and more efficient...