Reputation at Risk: Sentiment Analysis and Social Media Listening Tools under the Lens of Critical Multimodal Discourse Studies

In the field of online crisis management communication, AI-based social media listening tools (i.e., tools designed to track and monitor online conversations about a topic or brand) play a pivotal role in opinion mining and reputation audits. Com-pared to manual analyses, AI enables a faster large-scale collection and classification of vast amounts of data from several online platforms, thus facilitating the task of detecting and monitoring the sentiment linked to a brand and/or product. Nonetheless, AI-based analyses are far from unbiased. This paper adopts a critical multimodal perspective to explore the challenges linked to sentiment analysis in digital multimodal aggregations. It presents an empirical study in which the results of sentiment analysis performed by the social media listening tool Meltwater and manual tagging are compared to evaluate the efficacy of the tool in assessing online reputation damage following a crisis event. The findings of this study suggest that, in addition to possible incorrect classifications of texts (e.g. lack of understanding of pragmatic features or texts in languages other than English), the AI-based tool’s misinterpretation of emotional cues also includes multimodal ensembles. This is due to social media listening tools’ dependence on unimodal (verbal-only) classifiers that fail to produce reliable outputs. Despite the predictive power of these tools, the findings ultimately indicate that the accuracy of sentiment analysis is still affected by a hard-to-die bias concerning the primacy of language over non-verbal communication – a trend which is in contrast with the multimodal nature of semiosis and the proliferation of complex multimodal artifacts online.