Why block the hole? your bumbles will cause no problems.
Certian bumbles will use the same nest area year after year, there nest is never going to be big and I can't see there will be problems or damage to your Cavity wall..
If the bumbles nest is well out of people walking past I would leave them alone.
We moved our honey bees out of the garden and I'm wanting to build a honey room where they were, I had noticed that there is a bumble nest in the concrete plinth where the hives were stood, so now my honey room is on hold and we are using the area as a wildflower garden instead untill the autumn.
it's quite a special thing and lucky to have them nesting in your garden /wall pollinating your flowers and fruit ,i think the blackcurrant and raspberries are going to do well this year in the garden.
.