[Note: I'm skipping a rule because I feel Rule 4 in Hoyt's list is essentially the same as Rule 3.]
You shall not commit grey goo. Grey goo, in which characters of indeterminate moral status move in a landscape of indeterminate importance towards goals that will leave no one better or worse off is not entertaining. (Unless it is to see how the book bounces off the far wall, and that has limited entertainment. Also, I’m not flinging my kindle.)
This one barely needs comment from me, except perhaps in the form of perspective, in which I may digress a bit. Bring coffee.
I grew up with, and to a degree still have, tastes in reading that are a good deal more tolerant of experimentation than I sometimes think is right for me. I read Naked Lunch and Last Exit to Brooklyn and a great many other books I knew would never be anyone's idea of "fun" reading — in part because at the time what I thought was "fun" was pretty out of phase with other people's idea of "fun". I later found out I was not alone in this respect, and I felt cheered by that discovery.
The more I read, though, the more I grew disenchanted with the idea that the less constrained fiction was by stylistic precedent and conventionality of form, the better it was. The theory went that it was automatically better because the author was trying something new, you see. I know full well this is a justification I've used to this day to explain my own affinity for something: it worked for me because, well, the author wasn't just content to do the Same Old Thing. If the story consisted of a bunch of mopes bumping hopelessly through a gray, barren landscape of anomie and heartless indifference, it was okay as long as the writing was really nifty. Surely that counted for something?
Well, yes, it did and does. Clever writing apart from content counts for something, in the same way that showing up on time for class counts for something. But so does handing in assignments on time, and participating along with the rest of the class, and so on. A work is not exceptional merely because it breaks with convention. That only means it breaks with convention, which by itself is not an act of creation but merely one possible route towards same.
The biggest reason to break with convention in terms of plot or storytelling is because there is a need to do so that is not satisfied by any other means. We assume a story that throws us into grey-goo-land has a higher reason for doing so, and if we find some hint of that higher reason, it becomes easier to forgive them their trespasses.
That's the theory, anyway — and the problem is expecting readers, even intelligent and well-read ones, to side with the theory over the practice.
Authors who opt for the experimental road first are too often cheating themselves of the chance to create something accessible, out of an unearned contempt for the accessible. They are rightfully hesitant to write agitprop, but they've run too far in the other direction: by trying not to hit the reader over the head with a message, they've retreated into the quietism of saying that there's nothing to be said at all, and their work will stand as an embodiment of that.
It's one of the flimsier excuses that gets waved around: because modern life is aimless, pointless, meaningless, suffused with noise, etc., then our literature deserves to be like that too, because that would only be an honest reflection of it.
There was a time when I almost believed such claptrap. I don't anymore, in big part because I've come to a different understanding of book-as-reflection-of-the-times. You are always only going to be able to write the book that can be written in your moment in time, from your point of view. There's no sense in forcing the point.
The real issue, though, is how they are creating a false dichotomy, whether or not they know it. A good book does not have to be "hard", any more than a bad book has to be "easy". Some of the most riveting and insightful writing I know scores high on the Flesch-Kincaid readability spectrum. But I know repeating this won't make it any more true for the writer who has already made up his mind about what constitutes real writing as opposed to that popular pap. He is driven to break from the pack, time and again, and for that I salute him, for I have the same impulse. I suspect any writer does. What I worry about is whether in breaking from the pack he has simply charged over a cliff — whether he has, in the words of another, "sinned against his talent".
I do not imagine that anyone who writes in ways that "break the rules" imagines that they are simply writing to break the rules. They always believe there is a perfectly good ulterior motive for breaking rules, whether in the form of leaving out all punctuation or making their hapless characters trudge through Grey Goo Land. You're never going to talk Cormac McCarthy into using quote marks, because he's convinced not using quote marks for his dialogue is somehow better. Whether or not you feel he can get away with that is in big part a matter of taste and acclimation.
Over and above, that, though, is why this whole game of rule-breaking has become more important to some writers (whether or not they seem to admit it) than anything else. I suspect, again, it has more to do with standing out at all costs — at looking or sounding "writerly" — than it has anything to do with actual innovation or invention. If they abandon all pretense of telling an interesting story in favor of stranding us in "grey goo", in their minds that's just the price paid for trying to walk their own path. They tell themselves, it is better to tell people the painful truth than give them pretty lies. But this is about as charitable to the audience as a doctor telling a patient right before his yearly checkup, "Well, you're going to die anyway, but here goes."
Part of why I am so dismayed about this is because it is nearly impossible to talk about it without sounding like a Philistine. I am not against speaking the unvarnished truth when it must be spoken out loud. I am not against innovation in form in writing as such; I doubt I'd be a fan of Virginia Woolf or Notes from Underground or, heck, the plotless bile of Journey to the End of the Night (one of many books I have an ambivalent relationship with) if I didn't.
What I am against is the ways such things are easily abused as motives: how unvarnished "truth" at the expense of compassion — for characters, audience or humanity in general — comes all too often at the cost of accuracy, and in turn truth itself; that it is okay to think entertaining an audience is a form of lying to them, or being complicit in their self-deception, or some variant on that formula; that innovation in form is the ultimate goal worth aspiring to for a writer; that any writing that doesn't innovate is worthless; that innovation in form can be used to excuse any other deficiencies (especially deficiencies of story, theme, character, content, etc.); or any number of other such variations. I am against the idea that it's always better to err on the side of elitism; that if you build it, no matter how insular, an audience will find it (even if you do your best to keep them out in the first place); that obscurity is a hallmark of integrity.
But I also don't believe for a second any writer is going to cop to being an obscurantist. Every writer I have ever met has their reasons. Lord, do they ever have their reasons. I'm no different, either: I have perfectly well-formed justifications for every silly indulgence and po-faced blunder I've ever put to paper. What I would like to believe (although I'm going to have to leave this to others to decide if I pull it off or not) is that I know how to look back on what I did and get past the worst of the excess and self-indulgence, learn from it, and roll it forward into something that at least meets the audience halfway.
Let's not kid ourselves. Every author wants his work read by the widest possible audience — yes, even the people who flip their noses up and declare they are not in the best-seller business. Their real wish, impossible as it is, is for the masses to be brought around to their line of thinking: how better to solve the whole problem? But as Franz Kafka was once alleged to have said, if it's a fight between you and the world, bet on the world. Break the rules if you absolutely must, but it's harder to go wrong if you write a story.