I had almost exactly the same situation as yourself earlier this year, the colony left on the stand was fine , no detrimental effect from the other bees begging their way in and they're doing fine now on double brood.
Like you I had only 5 hives and was reluctant to lose 20% of my colonies but...