... an academic writer is trying to fill a niche. Now, the niches are getting smaller. Academics may write for large audiences on their blogs or as journalists. But when it comes to their academic writing, and to the research that underpins it — to the main activities, in other words, of academic life — they have no choice but to aim for very small targets. Writing a first book, you may have in mind particular professors on a tenure committee; miss that mark and you may not have a job. Academics know which audiences — and, sometimes, which audience members — matter.
One of the first things you get drummed into you as a writer — that is, if you plan on doing it as anything but a hobby — is to know your audience and aim for it. Too much of the time this becomes "people who read X", instead of "people who are interested in X". I wonder how, or even if, a non-fiction book like Gödel, Escher, Bach would be able to find an audience today, because of how unlike previous treatises on its subjects it was and is.
The same applies to fiction, too, of course. Because profit margins are so tiny for any kind of publishing, fiction doubly so, everyone seeks to maximize their take by reproducing — if only provisionally — what someone read last week. It's less expensive to market a maverick book, because you don't have to spend nearly as much time explaining to people what the blasted thing is. And given that most people barely have time to read anyway (you skimmed this while waiting in line for your coffee, right?), it's something of a losing proposition to ask for anything more from the apparatus created to extract value from words.
Some parts of the publishing market are scrappier and more inventive than the rest; they hew more closely to the indie comics scene than anything else. But too much of what both the indie comics scene and the indie publishing scenes produce are weird for weird's sake — different for the sake of difference, not different for the sake of being genuinely better. I suspect that in turn comes down to how writers are cultivated and appreciated.
Maybe what we need is more of a single-tent approach — a publishing house that also has a literary organ for criticism, one where the critics are entirely separate from the publishing side of things, and can speak honestly (and have the audience speak back). That might in time not only create better books, but create an atmosphere of discovery for them for readers, too.