I keep running into the sense that many people outside of SF&F just find the whole thing fundamentally silly.
I suspect that's because they are so used to it being associated with either the types of stories that are nothing but juvenile wish-fulfillment (Renata Adler said much the same thing about 2001) — or, maybe more importantly, where the audiences for such material themselves exude an attitude of juvenile wish-fulfillment. I suspect it's the latter: they wouldn't be caught dead with the rest of those ... nerds.
What passes for SF in the eyes of the mainstream is typically on the juvenile side, or rather hews close enough to the juvenalia that it can't help but be associated with it. The really good stuff — the adult, intelligent work — falls away from the public eye for the same reason most stuff in any other category succumbs to the same fate: because, in the end, it's not easily classified. It transcends its container and becomes its own thing, and when faced with that most people's easy systems of classification and recommendation break right down.
Most people aren't critics, let alone fans. They most often read because something is available, rather than because they seek it out. Note that this doesn't mean they're morally inferior to fans or critics; that's like saying a baker is morally inferior to a sushi chef. Each occupies a different spot in the same ecosystem. Some people just want to read — hell, I know there are days when I don't want to do anything except veg out with a copy of the latest Vampire Hunter D book.
Some time ago people likened the original breeds of Unix to a virus, one that spread because of its adaptability to various architectures. Liekwise, stuff that is most easily talked about by the broadest swath of readers tend to spread the fastest amongst them. If you tell someone "I can't really describe this book but you'll love it, just read it," more often than not that produces bewilderment and not curiosity, and since some of the best and most ambitious works tend to produce that sort of reaction, it's no wonder they don't get a lot of traction.
So if there's a reason so much of SF is seen as being kinda nerdy and goofy, perhaps it's because that's the variety of it that spreads most quickly and readily.