But it looks like nearly every bee initially leaves the hive
Noted elsewhere, but not the most clever statement. Check out at what age proff seeley reckons bees start to fly. That might mean that up to ten percent of the colony suddenly abandon all the brood and fly off before they have previously even left the hive or got to the stage of being airworthy?
Needs a bit more thought than that. Does all foraging cease on the day of the swarm? If not, what happens to those bees out foraging?
We know that large numbers of bees are idle before the swarming date - those that recognise or notice these 'pre-swarming' indicators would agree, I hope. So who is to say that the leavers are not mainly pre-determined and a lot of other bees are just carried out with the rush, to return soon after. There would, of course, be some border-line cases of bees that might or might not go and so would 'change' their minds or be a bit undecided.
Bees are simple creatures but as a super organism certainly not stupid. Think here, also, of ants, where the flying ants all leave umpteen nests, in the same area, at around about the same time. Not just pure luck, or is it?