Not many people look at my bottom (it's been said that it 'looks like a bulldog chewing a wasp' when I'm walking), so they wouldn't see the sign.
It doesn't matter who's bees they actually were. Anyone can sue you for whatever they want. If they decide it was 'your' bees then it's you, or your insurance company, that has to fight your corner in court. The cost of the court case could easily be more than the amount of the damages you're being sued for.
If someone who has been badly stung, perhaps even had an anaphylactic reaction, thinks you have no legal defence and no reliable witnesses they might just take a chance at court action for some compensation. That's just the kind of society we live in.