When I was younger — I'll say high school, since that seems about the right mental timeframe for thoughts of this caliber — I had the belief that science fiction and fantasy could be all of the things literature always seemed to be trying to be but holding itself back from being.
It wasn't until a little later on that I found the reason for this holding-back was, for lack of a better label, a fear of social ostracism. No one, or at least no one who cared about their credentials as a Serious Writer, wanted to be caught dead writing such juvenile stuff. It was the unseriousness of it that offended them, or rather, the idea that the only way a story could embody serious intentions was by being what people could label outwardly as serious, because how the hell else were they going to know what was what?
"Unserious" doesn't mean silly. Some of what appear to be the silliest things have pretty gravid underpinnings: Blazing Saddles has more to say about race relations than any ten po-faced "message" films. "Unserious" is when you're not walking it like you talk it, when you're more worried about the posture you're striking than the work you're doing.
This is a grave charge to fling around, so I will try not to make a Molotov cocktail out of it. What I see is a split between two kinds of writing: one where you take the business of entertaning the reader seriously (and, often enough, succeed), and one where you want to do more than merely entertain. The first is difficult enough and the second is a noble calling, which maybe by itself explains why we want so often to believe the two are at odds.
Think about it. If we let ourselves believe there really is something Deep and Meaningful in "Duck Amuck" (for my money, as profound a short film as anything, oh, Stan Brakhage did, maybe even more so), then by extension we have to grant the same significance to anything else in roughly the same territory, and if he gets up, we'll all get up, and then it'll be anarchy!
I've never been able to fathom this kind of parochialism, which is why I suspect it's slowly going out of fashion. Plenty of people with turf to defend are wincing at the idea of bestowing cultural significance on Harry Potter or The Hunger Games, if only because they sincerely believe there are other and better things worth reading first. This ignores the fact that it's entirely possible for something to be culturally significant and be worthy of study for reasons entirely apart from its quality as art, something MST3K proved in spades.
Regular readers will remember that a big part of why I found myself fond of both SF&F and litfic, but also dissatisfied with each, was because each had things the other could have benefited from immensely. We do have litfic that uses SF&F's counterfactual leaps of fantasy, and SF&F that benefits from litfic's human insight, but again I feel the problem is that we talk of the issue as if we were mixing two disparate ingredients from separately-labeled bottles. These ingredients were distilled and stored apart from each other a long time ago, when originally they were of the same primordial soup. The point is not to turn the clock back to a happier time (which most likely never existed except in our projections), but rather to find what we thought we gained by storing these ingredients separately.
We like our stories to have easy labels — this one's for the spy-novel fans, that one's for the Sookie Stackhouse-loving crowd, etc. — but that sort of labeling may have more to do with the ways we teach ourselves to enjoy our arts than because of any inherent benefit in labeling them that way. We should strive to take more of the work we enjoy across the spectrum, put it side-by-side, and see what's in common rather than look for differences. Not because we're trying to confuse one with the other, but because it's worth seeing what lessons each has to teach that the other might partake of.
We don't have to feel like we're doing damage to the canon to read what's new with an eye towards how we find deeper meaning in it — emphasis on we — rather than how what we read can be elevated onto some pedestal. The minute we put anything on a pedestal, it becomes fodder for being knocked off, anyway. Let's think more about the art in ourselves, rather than ourselves in the art.