Auditing the Objective
A technical report on making the objectives of engagement-driven recommendation systems explicit, measurable, and auditable.
Topic
Recommendation systems that are built to maximize engagement can end up amplifying outrage. The report asks what an auditor should examine when a system is designed that way.
Motivation
Its central question: how can recommendation objectives and their consequences be made explicit, measurable, and auditable?
Approach
- A multi-objective reward that puts predicted engagement, independently estimated content quality, and an outrage indicator into one explicit specification.
- A slate-level ranking filter designed to limit high-outrage exposure.
- A comparison of exploration strategies (UCB1 and quality-aware Thompson Sampling), with a forced-exploration floor.
- A 22-item audit checklist in five domains, with worked examples.
Main discussion
The report connects contextual bandits, fair ranking and constrained learning with a practical checklist for examining a platform’s disclosed ranking documentation and logged behavior. It also covers subgroup calibration, feedback loops, and how content producers adapt to ranking rules.
Limitations
- Quantitative results come from synthetic simulations, not measurements from real platforms, and have not been independently reproduced.
- Mathematics alone cannot decide which objectives a platform ought to optimize.
- Usefulness depends on estimator validity, subgroup coverage, and access to documentation and logs.
- The report repository does not include an executable reproduction package.