User perceptions of serendipity-control, cognitive-level labelling, and explanation features in an LLM-based academic article recommender: a single-institution pilot study

Serendipity in recommender systems has been studied mainly as an algorithmic property rather than as an interface-design question. This pilot study examined how undergraduates perceived three interface features of ScholarMind, a prototype in which a large language model (LLM) generates article recommendations directly: accuracy and serendipity sliders, LLM-assigned Bloom’s taxonomy labels, and an explanation panel of LLM-generated weights labelled as SHAP values. Forty-one students at one institution in Pakistan completed a task requiring use of all three features, followed by a self-report battery (ResQue, SUS, UEQ-S and three unvalidated feature scales).

Mean feature ratings lay on the agreement side of the midpoint (3.62–3.80 on 5-point scales), but reverse-worded items correlated negatively with their scales, indicating acquiescent responding. Exploratory correlations linked explanation-panel perceptions with perceived trust (rₛ = 0.47, 95% CI [0.18, 0.68]) and Bloom’s-label perceptions with perceived usefulness (rₛ = 0.51, 95% CI [0.23, 0.71]). Perceived usability was below benchmark (SUS = 56.6 vs. 68), and preference relative to Google Scholar was evenly split.

Inspection of the source code showed that the sliders only reorder a fixed set of generated articles. The findings describe perceptions of an interface prototype, not recommender performance. The authors thank the undergraduate students of IMSciences, Peshawar, who participated in this and prior studies in the research program.

The authors also like to thank the Office of the Associate Provost for Research at United Arab Emirates University (UAEU), United Arab Emirates, and, in part, Prince Sultan University (PSU), Saudi Arabia. The authors also thank the undergraduate students of IMSciences, Peshawar, who participated in this and prior studies. Institute of Management Sciences (IMSciences), Peshawar, 25000, Pakistan College of Computer and Information Sciences, Prince Sultan University (PSU), Riyadh, 12435, Saudi Arabia Engineering Requirement Unit, College of Engineering, United Arab Emirates University, Al-Ain, 15551, UAE The authors declare no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.