Thoughts and Lessons for Planning Rater Studies in AI

With the goal of deploying generative AI systems, rater studies are becoming increasingly common and important. This means more and more researchers and engineers face the challenge of actually planning and conducting rater studies for AI systems.