大規模言語モデルの「自分をよく見せる」回答バイアスを 定量化し、抑制する心理測定法を開発 ――比較型測定により、心理尺度を用いたAI評価の信頼性を高める――

2026-07-06 東京大学, 神戸大学

図1：本研究の枠組みの概念図。

＜関連情報＞

• https://www.u-tokyo.ac.jp/content/400291561.pdf
• https://arxiv.org/abs/2602.17262

LLMにおける社会的望ましさに基づく回答の定量化と軽減：望ましさを考慮した段階的強制選択式心理測定学的研究 Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

Kensuke Okada, Yui Furukawa, Kyosuke Bunji

Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics  Published:2026

DOI:https://doi.org/10.18653/v1/2026.acl-long.1865

Abstract

Human self-report questionnaires are increasingly used in NLP to benchmark and audit large language models (LLMs), from persona consistency to safety and bias assessments. Yet these instruments presume honest responding; in evaluative contexts, LLMs can instead gravitate toward socially preferred answers—a form of socially desirable responding (SDR)—biasing questionnaire-derived scores and downstream conclusions. We propose a psychometric framework to quantify and mitigate SDR in questionnaire-based evaluation of LLMs. To quantify SDR, the same inventory is administered under HONEST versus FAKE-GOOD instructions, and SDR is computed as a direction-corrected standardized effect size from item response theory (IRT)-estimated latent scores. This enables comparisons across constructs and response formats, as well as against human instructed-faking benchmarks. For mitigation, we construct a graded forced-choice (GFC) Big Five inventory by selecting 30 cross-domain pairs from an item pool via constrained optimization to match desirability. Across nine instruction-following LLMs evaluated on synthetic personas with known target profiles, Likert-style questionnaires show consistently large SDR, whereas desirability-matched GFC substantially attenuates SDR while largely preserving the recovery of the intended persona profiles. These results highlight a model-dependent SDR–recovery trade-off and motivate SDR-aware reporting practices for questionnaire-based benchmarking and auditing of LLMs.