Jack Finney (he of Invasion of the Body Snatchers fame) once wrote a story called "I'm Scared", where an unnamed narrator describes a slew of cases collected by him over the years in which time itself seems to have become put out of joint. Finney wrote "science fiction for people who don't read science fiction" — the sort of thing that was marketed to slicks like Collier's and The Saturday Evening Post back when a person could make quite a decent living as a short-story author.
"I'm Scared" wasn't my first introduction to SF, time travel, or Finney (I'd seen Body Snatchers on TV). It was, I think, my first introduction to SF-that-is-not-SF, at least in written form. Michael Crichton, if memory serves, came after that, as did Kurt Vonnegut. It also wasn't until sometime in high school that I ran into Vonnegut's own words about SF-or-not-SF and his unease at being lumped in with it. As Frederik Pohl once mentioned, Vonnegut had visited the Milford Science Fiction Writers' Conference and walked out of there with a bad taste in his mouth. "From them on," Pohl wrote, "he distanced himself from science fiction in every way he could — except in what he wrote."
Pohl being an SF writer himself, I had to wonder how accurate that was, but it meshed at least partly correctly with the public image Vonnegut cultivated. He didn't want to be seen as someone just grinding out cheap escapist thrills for kids of all ages. He was more comfortable as the crotchety gadfly, the canary in the coalmine that keels over to let you know how very, very bad things are getting. I suspect he reacted to the SF label as a way to render his work harmless — it's just science fiction, the way other things are just comic books or just movies. (Who was it that said that genres are essentially reading instructions?)
Ray Bradbury, by contrast, didn't seem to care about the labels one way or another. I don't recall him having a problem with Fahrenheit 451 being received as "SF": it was, after all, serialized in its original novella form in Galaxy in 1951, and later serialized in its current version in Playboy three years later. (O tempora, o mores: who today even thinks of Playboy as the starting point for up-and-coming fiction writers anymore?) What mattered most, it seems, was that the book was out there being read. The fact that it was one of the most widely misunderstood works of its kind was not a product of it being or not being "SF": Bradbury was not attacking censorship per se but the death of the imagination in the mass mind, the creeping erosion of attention spans and curiosity, the way censorship is all too often an after-the-fact gesture committed on a populace incurious enough to allow it to happen at all. ("There is more than one way to burn a book", as he put it.")
Does it really matter whether 451 is, or is not, SF? Or A Clockwork Orange, or 1984, or Brave New World, We, Kallocain, Dune, More Than Human, The Stars My Destination? I suspect it matters most to SF fans, because it allows them to claim ownership of something which has already been granted a stamp of approval by literary society at large. Many of their own favorites still languish in the SF ghetto, and only escape when the right combination of circumstances propels it to mainstream attention. Dick and Heinlein and Asimov were both lucky enough to be enshrined in Library of America editions, but Sturgeon and Delaney and many more are still waiting to be so blessed. Do they need it, or is that just the cherry for their particular sundae?
The more I think about it, the more I realize there are two different questions being asked. The first revolves around legitimacy as granted by literature and literary society at large. That society reaches down every so often — and maybe not all at once, but over time — and plucks something up into its number: a Heinlein here, a Dick there. They have looked down and seen something worthy of broader recognition, for whatever reason.
The second question is to what degree any given work of SF can step out on its own and be seen with its labels scraped off. That requires the presence of things which a genre is usually invoked to justify, excuse or ignore the lack of: credible plotting, rounded characters, a coherent view of life.
And even there, I find just as much "regular" literature wanting in its own way. People accuse SF of being redolent of incredible (in the sense of "not to be believed") plotting or flat characterization or questionable sociopolitical theories — all of which are criticisms that have been and continue to be levied against everyone from Jorge Luis Borges (is there a single character in any of his works apart from the author himself?) to Faulkner to Theodore Dreiser.
So in the end, it's not even about genres: it's about the strength of any one work by itself — a standard that we apply to "conventional" fiction even if we don't think we do. This means finding ways to look past labels, to appreciate what critic and author Dale Peck described as votes cast in the form of individual authors or individual books. Mainstream high- and middle-brow fiction has long struggled with its own bondage to form and function, its slavish inward trendiness and mutual-appreciation societies. And yet despite all that great individual books continue to be written, hauled up by enthusiastic readers out of tiny press runs or boneheaded criticism and allowed at last to fly.
SF is equally hidebound in its own way, and in both its inward- and outward-facing aspects. We are not, by and large, prepared to believe that Orson Scott Card's novelization of James Cameron's The Abyss could be a literary work, but whatever you think about the source material or the author, that was the spirit he approached it in, and it shows. Maybe we also grind our teeth at a book by a "mainstream" author being dubbed SF without our say-so (the most recent victim of this is, I think, Colson Whitehead). But if we want our stuff to be called mainstream, or literature, then I suppose we have to live with the indignity of having the reverse done to us. And if we do, is that really such an indignity?
We've got a lot of catching up to do in this respect. We only recently got over our indignant harrumphing about comic books — sorry, sequential art — even long after work was already being done in that realm which deserved permanent enshrinements of one kind or another (e.g., Osamu Tezuka). And while at last report Harold Bloom still has a self-important bone in his throat over Harry Potter, the folks who have actually read the books can and do make up their own minds about their lasting importance.
The real work to be done is not in the form of publications in journals or even bringing older titles back into print, although those certainly help. It's in cultivating the right kinds of reading habits. That will in turn, I hope, cultivate even better kinds of writing.
Because from what I can tell, it's readers who know this best, even when literary authorities claim not to. According to Amazon, those who purchased Invasion of the Body Snatchers also purchased Alberto Moravia's The Conformist, Alan Warner's Morvern Callar, and Flannery O'Connor's Wise Blood. The cynic in me says that's only because all of these were also made into movies, but some part of me wonders more openly than that.