I use a sub-domain (e.g. @m.mydomain.com) for my catch-all and this hasn't happened to me. There are various easy ways for spammers to find out about domains, but sub-domains can remain relatively obscure.
Or worse, using <randomchars>@somesubdomain.example.com as the SENDER address on spam to others. I once had spammers find a subdomain that accepted wildcard emails, and the backscatter was just insane. Had to spend a whole day trying to make a list of valid <usernames>@ on that subdomain to whitelist to put an end to it. (Not easy if you haven't already been keeping track of which addresses you've handed out throughout the years)
I have a setup like that, I get more spam because of that of course but Google is really good at filtering it. And especially in conjunction with priority inbox, it's a breeze (and very convenient).