PrivacyAlign：用于LLM智能体的情境化隐私对齐技术
PrivacyAlign: Contextual Privacy Alignment for LLM Agents
摘要
代表用户行动的AI代理会不断做出各种决策。为了让用户信任这些代理，这些决策必须符合用户的真实需求。隐私问题正是代理决策中需要协调的重要问题：代理所发出的每一条消息、每篇帖子或每次工具调用，都涉及到关于何者适合分享、与谁分享以及在什么条件下分享的情境判断。由于这些判断取决于社会期望和规范，人类的判断不仅能够识别隐私侵犯行为，还能帮助定义这些行为。虽然现有的研究在训练和评估过程中依赖不可靠的替代指标，但我们将人类判断置于代理隐私协调的核心位置。我们引入了PrivacyAlign这一数据集，该数据集包含1,350个样本和3,516条详细注释，这些注释来自599位不同的注释者，适用于当前LLM可能出现的各种情况。我们利用这个数据集来确保代理的协调训练和自动化评估都符合人类的隐私规范。基于这些注释，我们首先证明，让LLM的评判者参考人类注释和解释来处理同一提示下的响应，可以使其判断更加可靠。接着，我们引入了基于注释的奖励模型，利用这些注释来评估新的响应，结果显示，使用这种奖励机制训练的小型开放权重代理能够更紧密地符合人类的隐私规范，在PrivacyAlign以及现有的代理隐私评估指标上取得了显著成绩。
English Abstract
AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.