"No one in a novel by Virginia Woolf ever filled up the petrol tank of her car. ... No one in Hemingway’s postwar novels ever worried about the effects of prolonged exposure to the threat of nuclear war."
Said Ballard, who wrote "science fiction" the way Kurt Vonnegut did: only because other people insisted that was what he was doing. Ballard was most widely recognized outside of that category, in part because he'd done enough things to avoided being pigeonholed as "just" an SF author. Not that such things still don't happen, only that back then escaping the SF ghetto was all but impossible once people saw you had that as your mailing address, so to speak.
Ballard encountered no small amount of rejection for attempting to cross borders so freely. One of the most infamous stories in all of publishing is how Ballard's The Atrocity Exhibition had its first print run recalled and destroyed by its own publisher, when the head of the company actually read the book. Total Destruction To His Mind ensued.
The fact that it was SF per se wasn't the trigger; it was the way Ballard was trying to explore much of the same couterfactual territory found in SF by way of surrealism and horror. It was the same sort of transgression that forced some people to think of Naked Lunch as failed science fiction because they couldn't think of any other way to assess it. The same went for Crash, Ballard's most widely-known book for many of the same reasons Anthony Burgess's biggest seller was A Clockwork Orange.
Some folks resent it when a self-styled literary author tries to write SF, because it often means both ends of the deal get fumbled. The "literary" side of the story translates into "no plot; characters I can't stand; artsy writing for its own sake", and the "SF" side of the story becomes "a patina of SF ideas sprinkled on top of the goings-on without being given the close investigation and internal rigor that SF demands as a literary form". Ballard was not, in my opinion, one of the offenders; he wrote literary SF that did justice to both sides of that equation (his novella "The Voices of Time" is a great example of this). But there are plenty of people who see what Ballard did and assume that because he could get away with it, so could they.
Never mind that this wasn't about merely getting away with something. Ballard was trying to reconcile the realities of modern life with the forms of modernist fiction, to develop a way of looking at the increasing incomprehensibility of the former through the lens of the latter. The people who read modern highbrow fiction were among the most benumbed and bewildered by modernity, and the fiction they chose to read in turn didn't choose to reflect on their problems except in a jejeune or hand-wringing way. It wasn't that Woolf or Hemingway had nothing to say to a modern reader, but that their fiction had no real comprehension of the idea that we could end up living in to what to them would have been a science-fiction novel. Ballard was one of the few authors who got that feeling right, and who sensed that a new age would need new literature to go with it, whether or not that literature would be well-received at first.
Most of this we have come to accept in one tacit form or another — that a literature about the modern world needs to take into account its feeling of counterfactuality, of things feeling science-fictional and what that does to us as people. (See also: Dick, Philip K.) The downside of this approach is how we have a whole surfeit of books today that think being "modern" means mentioning Twitter or the Khardashians, or including other topical references that will make the material seem sadly dated in ten years. It's a little hard to read American Psycho today, with its farrago of brand names and point-in-time drop-ins, without feeling like it was found in a time capsule. So much of the blather about "relevance" in literature winds up being about its activity as a camera, merely snapping pictures of the current moment, and not as an X-ray.
SF has the raw fearlessness to see the world's counterfactuality, but often lacks the psychological insight and the person-to-person empathy that makes it more than just a what-if statement. Litfic offers plenty of the latter, but often at the expense of that very fearlessness, maybe out of fear of seeming ridiculous — we worry that to suddenly talk of outer space, or vampires, or what have you, will set the audience to giggling — or rather, will set the audience we want to reach to giggling. That tells me a good fusion between SF and litfic won't be a matter of appealing to either one as a market, but appealing to enlightened readers wherever they might find themselves.
Not long ago I picked up a copy of Thomas Mann's The Magic Mountain, one of those classic books I keep promising myself to read. (I'll be starting it soon, I swear.) A book like this would be difficult to write today, not just because we no longer think of novels as being useful tools of social diagnosis (we want our art to be nice and harmless, especially our "serious" art), but because the mindset that created and received such a book has all but vanished. It would be impossible to return to such a mindset; we can only go forwards, not back, and we can only read something like The Magic Mountain today by keeping half a mind towards the time and the manners it confronted. But that, in turn, makes it all the more timeless.
SF has a hard time being timeless, because it's rooted that much more deeply in the moments of its creation. And a lot of that is our own fault — something that Ballard might have twigged on as a sign of how an age's own greatest insanity is invisible to itself. What sort of madmen would develop a literature of speculating freely and openly about things that might well be unthinkable or unspeakable, a literature that serves as a kind of sandbox for us to rehearse the future — and then relegate such literature to the status of mental cotton candy, junk for popcorn-munching theater-goers, psychic void-filler for straphangers? Ballard could scarcely have invented a better story himself. Or a more horrifying one.