[세미나] 전문가 초청 세미나 (Sr. Researcher Xiaoyuan Yi, Researcher Jing Yao @ MSRA)

● Title: Value Compass Benchmarks: Towards Comprehensive, Generative and Self-Evolving Evaluation of LLMs' Value Alignment

● Speaker: Sr. Researcher Xiaoyuan Yi @ MSRA

● Time : 15:00 - 15:45, May 30th, 2025

● Location: 경영관 지하 1층 33B101호

● Language: English speech & English slides

Abstract: 

As LLM-based generative models become increasingly integrated into human life, it's essential to assess their potential risks and societal impacts. Beyond risk-specific benchmarks, evaluating the value orientations reflected in LLMs offers a holistic lens for diagnosing potential misalignments and understanding how they align with the preferences of diverse user groups. However, value evaluation faces validity and informativeness challenges: how to ensure that assessments accurately capture an LLM’s underlying values and yield insightful and informative results. To address these challenges, we propose Generative Self-Evolving Evaluation, which leverages LLMs’ generative capacity and Psychometrics theory to dynamically and adaptively probe their value boundaries. Our method automatically generates novel, value-evoking items to avoid data contamination and ceiling effects, enabling a more faithful investigation of models' values. Building on this framework, we present the Value Compass Benchmarks, an online leaderboard offering a comprehensive analysis of the value orientations of 33+ popular LLMs.

Bio:

Xiaoyuan Yi, Senior Researcher at Microsoft Research Asia. He obtained his bachelor’s and doctorate degrees in computer science from Tsinghua University and mainly engaged in Natural Language Generation (NLG) and Societal AI research. He led the development of one of the most famous AI poetry generation systems in China, which has millions of users from 100+ countries. He published 30+ papers at top-tier AI venues and won honor such as the Tsinghua University Supreme Scholarship, the Xinhua Net The 10 Most Influential People on the Internet, Best Paper Award and Best Demon Awards of the Chinese Conference on Computational Linguistics, Rising Star Award of IJCAI Young Elite Symposium, Rising Stars in Social Computing by The Chinese Association for Artificial Intelligence and so on.

● Title: From Universal Value Alignment to Customized Alignment for Large Language Models

● Speaker: Researcher Jing Yao @ MSRA

● Time : 15:45 - 16:30, May 30th, 2025

● Location: 경영관 지하 1층 33B101호

● Language: English speech & English slides

• Abstract: 
• As Large Language Models (LLMs) more deeply integrate into human life, aligning them with universal values like helpfulness, harmlessness and honesty become insufficient to satisfy diverse users across cultures and communities. Therefore, it is crucial to customize the alignment of LLMs for improving user experience and mitigating social conflicts. Despite considerable advancements in recent years, there lacks a clear discussion about what goals we should customize LLM alignment and what key challenges lie in this field. To bridge this gap, we made a comprehensive survey to figure out this task and shed light on the inherent challenges. Along this direction, we first delve into cultural alignment and address the data challenges. Existing approaches for cultural alignment faced two key challenges. (1) Representativeness: They fail to fully capture the target culture's core characteristics with redundancy, causing computation waste; (2) Distinctiveness: They struggle to distinguish the unique nuances of a given culture from shared patterns across other relevant ones, hindering precise cultural modeling. To handle these challenges, we introduce a novel cultural data construction framework. Extensive experiments demonstrate that our method generates more effective data and enables cultural alignment with as few as 100 training samples, enhancing both performance and efficiency.
• 
  
• Bio: 
• Jing Yao is now a researcher at Social Computing Group in Microsoft Research Asia. She received her M.S. degree in Computer Science from Renmin University of China in 2022, and a B.S. degree in Computer Science from Renmin University of China in 2019. She joined MSRA in July 2022. Her research interests include responsible AI, large language model alignment, trustworthy recommendation and information retrieval. She has published some academic papers on top-tier international conferences such as Neurips, ACL, SIGIR, WWW, NAACL, CIKM. She serves as a program committee member for several conferences such as Neurips, ICLR, ACL and SIGIR.