I'm not sure who came up with that term first, Bob Black (of The Abolition of Work) or the anonymous graffitist who wrote "Je suis marxiste, tendance Groucho" on a wall in Paris sometime in 1968. But it's a sentiment that's strong in me: I wouldn't belong to any club that would have me as a member.
It's not that I think group identities, or people who take them on, are wholly bad, but that I have an inherent distrust of the way labels distort viewpoints and terminate thinking. If I tell people I'm a registered Democrat, that allows both of us to believe certain things about my worldview, some of which may well be wholly wrong. But we don't have time or energy to laboriously document all of those things between us, so we let them stand until one of us opens our mouth and knocks them down ("What do you mean, you supported the Gulf War?").
I felt reluctant to explicitly identify myself as a Zen Buddhist even after studying it for several years, for a variety of reasons. For one, I wasn't a vegetarian (although I try to eat less of the red stuff as time goes on). For another, I have not yet fully reconciled many of my feelings about what it means to assume such a label; I wasn't fond of the idea of assuming the label and then being criticized for not doing this or that thing commonly associated with Buddhism.
Most people have completely mistaken ideas about what Buddhism is, let alone Zen, and so by just telling them "I meditate" or what have you, I avoid a lot of tiresome explaining. I suspect a lot of this is just me not having condensed down things far enough; I suppose I could do a two-minutes-standing-on-one-leg explanation that begins something like "Contrary to public belief, Buddhism isn't the philosophy that life sucks..." and works its way out from there.
I guess in the end, yes, it falls to me to explain myself, especially if I'm taking a point of view that isn't necessarily part of whatever constitutes a mainstream of thought. But if I don't have to do it, I'd rather not bother. I'm not sure everyone I meet is obliged to know my entire political, moral, social, and spiritual compass right off the bat anyway.