"There ought to be a story about X" is as powerful a motivator to create something as there is likely to be. If you don't see what you want out there, that's a good sign it's crying out to be created. That's also a sign at least one other person might want to see it too. Not everyone has the wherewithal or the resources to write a story, so your work may end up speaking for others.
Most writing projects peter out because of boredom or bad time management. Work up the discipline not just to stick with a project, but to know as quickly as possible whether or not a given project is something you want to stick with. How much is there in it for you?
I don't like to use the word discipline because that word has been tainted to mean something to most people more like "punishment", but that's not it at all. Discipline is something you impose upon yourself, not others.
"Happy people have no stories," as a refrain in a song once put it. And even happy people want something. Everyone in your story, at every time, should want something — even if they don't know it at the time — and that wanting should inform the story and give it direction and purpose. Gulliver Foyle in The Stars My Destination wants to find out who left him to die in space, and transforms himself into a superhuman Renaissance Man to do it. Macon Leary in The Accidental Tourist wants nothing more than to overcome the grief of losing his son, but doesn't even know it until a chirpy pet store employee wakes him up to it gradually. The end of desire is the end of, if not all stories, at least the one you're writing.
It should always be clear what a given scene is about and to what end. Every sentence should advance the action somehow; it should tell us something we didn't know before, show us something we didn't realize, illuminate things with new light. If you come back to your own work and find yourself reading it two or three times to figure out what you meant, or say to yourself "they'll know what I mean," that's a sign of mud on your story. Wipe it off.
Writing is rewriting is revelation. If a story is hinting to you that it needs to be different in order to really work, go there and find out what happens. If you see a thread hanging off a sweater, it's very hard to not pull it. In a story, you can and should pull it, just to see how things unravel. You can always dump your changes if they lead into a dead end. But often they lead to revelation.
We all know this one, but I think the reasoning behind it is not always as clear. One does not want to be hopelessly in love with one's own language that you defend it against all changes, even those that make the work as a whole stronger. It's OK to have a sentence here or there that calls attention to itself, as long as it doesn't muddy the work or bring it to a halt. But always be on guard against defending your work from change it badly needs.
Every difficulty in your story has the seeds of its own resolution within it. I insist that this is the case without exception. In the early stages of working on Flight Of The Vajra I found myself flummoxed by a problem with the story's milieu. How could a religious institution, founded as a bulwark against relentless futurism, survive when what it offered to the problems of death and decrepitude simply couldn't stand up over the centuries? Then I thought, maybe that is the story — that the Old Way, as I called it, is fast discovering it has nothing to offer humanity as it has become, and has to adapt or die, as all organisms do. The difficulties in your story are the story.
If you know in your bones you can't have a happy ending, don't give in because it'll tick off some undefined reader out there. I struggled with this one for a while before realizing I was only ever writing for myself first and everyone else second. This is not an excuse to shun the audience or have contempt for them, only to know that they came to you for the ride they are about to take. If you give them a happy ending that feels right in the moment but is false in retrospect, they're going to sour on you even if they don't know it. Stanley Kubrick wrote Paths Of Glory with a fake happy ending to give to the studio, then turned around and shot the real one, the heartbreaking one, because he knew in his bones it was the one that made sense for that story.
It took me plotting several novels to realize stories are not about plots but feelings. You enter them knowing and feeling nothing, and along the way you are encouraged to feel certain ways about what goes on. Every time I worked out a story, I always told myself: how do I want everyone, both the audience and the characters, to feel about this when it's over? The two parties don't even have to feel remotely the same things. A superb trick, one I have yet mastered, is to give the characters a happy ending, but to have the audience know this is anything but. But your story should be an emotional arrow fired at an emotional target of some kind.
One of the nice side effects of thinking of a story in this way is it gives you a wider range of emotional flavors to suss out, become familiar with, and work into your stories. Not everything is about happy and sad alone. When people first begin to cook they often have little to no awareness of all the subtleties that go into good cooking or how to produce them. They know only the flavors that come out of a box or a salt shaker. By the time they're comfortable in the kitchen, their tongues have become experienced.
Dale Peck once wrote that fiction doesn't discover truth, but invents it and dispenses it as it sees fit, which is why it's called fiction in the first place. But even the invented truths of fiction come from the truths of things outside of fiction. I would go further and say they must come from such places.
I once noted that we have no trouble with the idea that Superman can fly and project laser beams from his eyes, because he still has many attributes we associate with human behavior: his respect for his parents, his love of Lois, his unwavering devotion to defending the world from both itself and others. The less Clark Kent there is in Superman, the more we waver at accepting him. Even R2D2 and C3PO (and BB-8!) in Star Wars are worth identifying with because they have human traits. Even Kafka's nightmarish stories obey aspects of human truth: that our fears are not rational, that we can justify anything if we want to (and we always want to), etc.
But a story about people whose behaviors do not correspond to human reality, even one that has no fantastic elements, is going to be a tougher sell unless you dig out the truth from it. We would accept Superman without blinking, but we would have a harder time with a man who cheerfully sells off his children into slavery. That would require an extraordinary effort to be made comprehensible; in fact, I'm sure there's a story in it somewhere. But many novice writers suffer from not knowing how to observe or record human behavior in a convincing way, and don't always realize it.
Some of this also relates to politics, in the sense that a writer's political leanings tend to manifest in their work whether or not they put them there consciously. When most of us say they don't like politics in fiction, they mean politics they don't like, and I think this goes for everyone regardless of their leanings. But I think the deeper truth of this is most of us just resent it when a political viewpoint of any kind is artlessly shoehorned into a story. Some part of us knows it's just plain bad writing, and gets turned off. A political thriller can be interesting, but even it has to maintain respect for everyone involved as people. And a work of art that has a political alignment has to start by being art and end by being art, like Picasso's Guernica.
I also believe (you don't have to join me on this) that most people know on some level when the politics in question is on the wrong side of history. But I would be just as bored by a story that is leftist propaganda than a story that is reactionary propaganda. A story that departs from talking about people to talk about politics, that eschews the specific human case for the generic political one, is more often than not just a weak story. Nineteen Eighty-Four would be a mere tract if it were not also invested with a strongly constructed story, one about a character we cannot help but feel invested in as the walls, already close around him, close in even further. There but for the grace of Big Brother go we.
How you embody your point of view is everything. I was no fan of Heinlein or his politics, but I never got the impression he used his writing as a way to legitimize his view. It was the other way around: his view informed his work, and was also far from the only thing that informed it. Philip K. Dick had a worldview 180° out of phase from Heinlein's, and he couldn't help but let that influence his work. But he too didn't make his work into mere polemicism, just of the leftist sort. He had bigger things in mind — maybe too big for him to survive, but that's another issue. If a writer is to have any politics at all informing their work, it ought to be the politics of being on the side of other people, whatever and whoever they are. Fiction's mission is to sympathetically take the side of the human (or near enough to human) condition.
When we talk about truth in a story, we are not talking about the marshaling of facts, although getting your facts straight matters. (I had to rewrite a story after the last minute because I forgot that you can't pay cash for an airline ticket anymore.) We are talking about the feeling you get when you say to yourself, "Yes, it is like that, how come I never noticed before?" and the feeling you get when you say to yourself, "Maybe it could be like that." Those two kinds of truth are the ones most worth striving for in any story, because we get them so rarely anyway.
New York City
Other Lives Of The Mind