Unfounded generalizations and false conclusions: RUDN scientists have identified AI “hallucinations” in the diagnosis of mental disorders

“Our research is an important step towards trusted AI in medicine. We don't just point out the shortcomings of AI tools, we offer approaches to overcoming them. The key task today is not blind trust in algorithms, but their integration into the work of doctors as a proven and understandable decision-making support tool. Patient safety and understanding the limitations of technology are our absolute priorities,” said Anton Poddubsky, Dean of the Faculty of Artificial Intelligence.

The main value of the study is a detailed comparison of existing large language models (LLMs), as well as methods for their use and retraining for the tasks of detecting depression and anxiety in text, and an analysis of AI errors and “hallucinations” in these tasks with the involvement of experts in the field of psychology. The work of RUDN scientists has been recognized and presented at the highly rated international conference Empirical Methods in Natural Language Processing (EMNLP). We spoke with the authors of the article and learned how the idea for the work came about, what AI “hallucinations” were identified, and what the prospects are for further research.

How did the idea for research on this topic arise, and why is it relevant and pivotal?

In recent years, there has been growing interest in diagnosing mental states based on text and using AI in this field, as well as applying LLM in medicine in general. However, most studies rely on English-language data and ML models; there have been no comprehensive comparisons for the Russian language to date. This prompted us to investigate LLM and other machine learning models for detecting depression and anxiety from text. We compared different models for both tasks and showed which ones work best in each case. In addition, we conducted additional experiments to evaluate the quality of LLM generation from the perspective of expert psychologists. It turned out that at the current stage, LLMs provide low-quality answers. In particular, in one experiment, we used LLM not only to determine the presence or absence of depression in the author of the text, but also to generate an explanation of why the model came to that conclusion. It was in this experiment that we found that the explanations of modern models contain a large number of errors from an expert point of view.

What is the main danger of such errors?

The danger lies in the fact that LLMs can produce unfounded or false conclusions (“hallucinations”) that appear plausible to the end user. Such errors are difficult to detect without the help of an expert, but they can lead to misinterpretation of signs of depression.

What causes of AI errors have you identified? What is it about conversations about mental health that is so “confusing” even for the most advanced language models?

Clinical psychologists analyzed LLM's responses and noted errors in them from an expert point of view. We identified six main types of errors: tautology, unfounded generalizations, false conclusions, confabulations, distortion of medical concepts of depression, and incomplete listing of its symptoms. It is worth noting that from a machine learning perspective, all these errors can be described as “hallucinations,” but in psychology-related tasks, a more precise categorization is needed. The peculiarity of the texts used to identify depression is related to the complexity of their interpretation. People often describe their condition indirectly, using metaphors, and the text does not always directly reflect the signs of mental disorders. In addition, the task of detecting depression from text is difficult for non-specialized models, since most of them have not been trained on psychological or medical data.

What are the prospects for further development of this research?

The next step could be specialized LLM retraining on large datasets for the purpose of detecting depression and anxiety. The current experiments used a relatively small amount of data, which may have limited the final quality of the models.