Divergent user expectations of conversational AI: a comparative sentiment and thematic analysis of ChatGPT and Replika app store reviews

Conversational AI apps are increasingly serving two different purposes: instrumental productivity and emotional support. Although both are built on the same underlying language technology, users may evaluate these systems according to different instrumental and relational expectations. This study examines the differences in user evaluation of a task-oriented assistant (ChatGPT) and a companionship-oriented agent (Replika) through computational analysis of the sentiment scoring, NRC emotion analysis, topic modelling, and inferential tests of the 6,000 publicly available app store reviews (3,000 per application).

The two applications differed only marginally in overall review polarity, and the direction of the difference depended on the method used: under the Bing lexicon Replika reviews scored slightly higher (M = 0.312 vs. 0.275), whereas under sentimentr ChatGPT scored higher (M = 0.135 vs. 0.102); aggregate polarity was therefore not a robust point of difference between the two corpora. The rating distributions differed significantly between the applications (χ² (4, N = 6,000) = 242.26, p < .001, Cramér’s V = 0.20), with ChatGPT ratings more polarised at both ends of the scale (36.8% vs. 21.7% five-star; 27.4% vs. 23.8% one-star) and Replika ratings more concentrated in the intermediate categories. Once scores were normalised for review length, Replika reviews contained more joy-, anger-, surprise-, and anticipation-associated language, with all effect sizes small; fear-, sadness-, and trust-associated language did not differ significantly between the applications.

The coupling between review language and star rating was weaker for the companion application, though the difference was small (App × Rating interaction, partial η² = 0.016); because the two samples span different publication periods, this interaction is reported as provisional. Topic modelling yielded a six-topic solution in which conversation and companionship accounted for 53.6% of Replika reviews against 10.9% of ChatGPT reviews, while task assistance and usefulness accounted for 30.9% of ChatGPT reviews against 1.2% of Replika reviews. Together, the findings are consistent with differences in evaluative framing between the two applications examined in this study, illustrating two representative cases rather than establishing distinct categories of conversational AI systems.

Open access funding provided by Vellore Institute of Technology. This study was supported by internal research funding from Vellore Institute of Technology (VIT) under its faculty research support scheme. VIT Business School, Vellore Institute of Technology, Vellore, Tamil Nadu, India The authors declare no competing interests.

This article does not contain any studies with human participants or animals performed by any of the authors. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.