The referee must keep up with the fastest player, for the whole match. The players take turns to score points, and spread the load between the team. The referee not only has to keep up, but never take their eye off the ball. Not only that, if a team loses, they can get the blame.
If you can't play become a referee? More like the other way around.