I imagine a fair number of people reading this are familiar with Shin'ya Tsukamoto's Tetsuo: The Iron Man, that grotesque little masterwork of Japanese indie cinema — "like H.R. Giger and Yukio Mishima had each other's body-horror babies," as a friend of mine once quipped. What's odd is how Tsukamoto's follow-up, Tetsuo II: Body Hammer, had a bigger budget and better production values, and yet is far inferior on almost every other level.* Thomas Weisser's capsule description of the two films in one of his Japanese cinema guidebooks provided the single best sussing-out I've seen to date of what went wrong: Tsukamoto had "replaced the nightmare with insanity."
Here's what I mean by this. The first movie had the same dream-logic quality of David Lynch's Eraserhead, except at something like one hundred times the speed of that film. There wasn't a rational explanation for what happened; it just unfolded in its own way and with its own mission. A rational explanation would have been — how I love this phrase — cutting the drum open to see what made it go bang. (The live-action Silent Hill film made this mistake, too.)
The second Tetsuo, though, had an explanation, but it didn't matter. Instead of being in the grip of someone else's dream logic, we were just witnessing someone's descent into aberrant behavior, and once the movie set up the basic metaphor there wasn't much to be done with it. (It is fun to look at, though, and Tsukamoto made creative use of the sterile Futurist look of the Tokyo of the time.)
The problem with insanity as witnessed from the outside, instead of being experienced empathically from the inside, is that it's very difficult for it to be experienced as anything but a record of aberrant behavior. Insanity by itself is a self-limiting subject; unless your point of view is larger than the mere depiction of the behavior, after a while there isn't much to say except, gee, that poor fellow sure needs help. Just showing someone being insane isn't all that enlightening, and after a while not even all that interesting. (I won't go into the way mental illness has been romanticized in popular culture; that's worth a whole screed unto itself.)
Nightmares, on the other hand, are personal and intimate. Seeing someone else's nightmare depicted well can feel like we are having the nightmare on our own. Making the audience share the feeling of madness is far harder to pull off, although I've seen a few productions that did accomplish it. Clean, Shaven comes very close indeed. Ditto perhaps Memento, although strictly speaking the character in question was not insane but mentally defective ... although even that's questionable by the end/beginning of the film.
In the end, I guess it comes down to the two not being interchangeable. The nightmare worked better for what Tsukamoto was trying to do; when he turned it into a story of madness, he started to miss the point.
* The third Tetsuo movie is an even bigger letdown.