The model doesn't have preferences, the model expresses the most likely preferences that the training data would have expressed (humans).

This feels like throwing a tennis ball at a brick wall then interpreting the angle and direction that the ball travels as it bounces off as the tennis ball or wall's preferred direction of travel.

You are the one who threw the ball. You are measuring your own preferences.