I suspect it's automatically a criticism if the words coming out of your mouth are "I only want to work with smart people" or on interesting problems, or whatever. I further suspect people will say more or less precisely that without realizing it. "oh, I didn't mean to call you stupid, you're just not as smart as me, that's all."