The math:
More mice means more mistakes.
In 1-shaman rounds, there are 2 mice in a 3 mice room, and 3 mice in a 4 mice room. Suppose each farmer has a 90% success rate [10% probability of failure], and that it takes only 1 mistake to slow down the room.

In a 3-mice room, the probability of flawlessness is 90% * 90%, which is 81%. This means the failure rate is 19%.
In a 4-mice room, the probability of flawlessness is 90% * 90% * 90%, which is 72.9%. The failure rate jumps to 27%.
In a 6-mice room, the probability of flawlessness is 90% * 90% * 90% * 90% * 90%, which is 59%. The failure rate jumps all the way to 41%.

Mistakes cost seconds.
The shaman can revive and save the mice, yes. But that costs time. Even if each item only costs 0.5 seconds, a single mistake would require 2 items [Personal teleporter+Chief's food, Ambulance+Easy victory]. That is one whole second per mouse. Considering that shaman rounds take about 10 seconds to complete, adding a whole second makes the room 10% less efficient than standard, which means instead of being able to gather 1000 cheese in 5 hours, we only gather 900.

And the more mice there are, the more mistakes. Which means more mice = more seconds = less cheese.