Neural language models as content analysis tools in psychology

This study investigates the potential use of current neural language models in psychological practice, particularly in diagnosing patients through the analysis of textual content. The growing interest in the capabilities of neural language models, especially in areas where natural language serves as a primary information source, and specifically by psychology, is the motivation behind this study. As a case study, we tackled the assessment of coherence in autobiographical narratives, a diagnostic methodology widely recognized in psychology for its pivotal role in understanding personal identity. This is a particularly intriguing case study for two reasons. On the one hand, it demonstrates how these tools can be a valid aid in psychological practice, in a task notoriously burdensome and time-consuming for human operators. On the other hand, it tests the ability of neural language models to grasp deep, global conceptual aspects from a text, a task not easily accomplished even by professional psychologists. Coherence is evaluated here quantitatively using the standard procedure known as the Narrative Coherence Coding Scheme. The results obtained are particularly promising, especially when using models subjected to fine-tuning with some examples of stories and their evaluations. However, even without fine-tuning, models prompted with appropriate prompts also perform acceptably.