Earlier my friend Steven Savage and I were reminiscing about the way literary SF in the 60s and 70s had enjoyed a burst of maverick creativity. I felt the reason why written science fiction was such a bowl of gorp* at the time revolved around a few things. Most of them were market conditions.
After stuff like 2001 came and went, it started to become clear to publishers that there were tons of young people with tons of disposable income who wanted to read science fiction, so they started putting out most anything that fit the bill. After Star Wars, the dam really burst, but even before then there was the sense of an unmet need.
In an environment like that, it was possible for experimentation to take place — in big part because neither the public, nor the readers, saw it as "experimentation". It was just one more of the many ways SF would manifest itself. The crazy, as Steven likes to call it, was a symptom, not a motivational force.
Provide people with a space in which they can do crazy things and not be punished for them, and you'll get innovation. Some of it will be stupid; some of it will be intriguing; some of it will be fantastic. For bonus points, add some kind of positive-reinforcement feedback loop. But the base conditions for fostering creativity all include some kind of propensity for risk-taking, where the cost of failure is noticeable but not crippling. In other words, it helps to have a disincentive to fail, but it shouldn't be so high that one isn't also encouraged to stick one's neck out.
Writing a book is one of the least risky ways to experiment. It costs nothing save for one's time and the materials — and most of the time, the materials are something you already possess without any additional capital outlay. But when a publisher or a distributor comes into the picture, that's where the risk also comes in, and they are all determined not to stick their necks out one millimeter more than needed. Who blames them? Publishing is such a marginal business, even with the overhead of printing and fulfillment out of the way thanks to e-books, it's a miracle anyone can make a dime. And so thought turns instead to making as many dimes as possible. Just to be safe.
I don't think there's any way to inherently decouple the risk from the reward; that's utopianism. But I would like to see more of the risk assumed in ways that makes it far less difficult for any one party to bear. Kickstarter-type arrangements shift the risk towards the consumer rather than the producer, but at the cost of attracting lopsided audiences. Instead of pitching the project to a publisher or an agent, you're pitching it to a lay audience — one even more easily swayed by the obvious and the immediate moment, and not attuned to what needs to be done to outlast the moment.
But most of why any kind of crazy flourishes is because there's some kind of audience for it, one cultivated quite deliberately. “Hollywood is not based on talent,” film historian David Del Valle once said. “It's based on being in the right place at the right time and having within you whatever that is that somebody wants.” I think he was also talking about publishing.
* a bunch of fruits, nuts, and flakes.