I’m a plotter rather than a pantser, I’ve tried both, and planning ahead works best for me. I still find plenty of new material as I write, so I’m not against discovery writing at all. Writing with a plot in mind leaves less room for my story to go off on a tangent. With a firm backstory in place, even one readers don’t see, I avoid plot holes and stay focused on where I want the characters to end up.
There is still a lot of room for characters to wiggle out of the scenes I have in mind. They’re slippery lil buggers, I’ll give them that. With my outline as a guide and the backstory in mind, I have all the tools I need to corral their unruly tendencies.
Readers never have to know how close I came to going off the rails either. They don’t have to immediately know why X instead of Z happened. Even if the question nags at them as to why the story went left instead of right, they’ll understand eventually. That is the important thing, the backstory is for the author.
The reader may crave the backstory. If I’ve written the scenes well. Rest asured, I will dole it out to them over time. But giving them the whole picture at once not only overwhelms the reader. It slows the story and creates big chunks of information that doesn’t serve the story at the moment. In Fantastic America, magic waxes and wanes over long period of time. There is a great reason why, but readers don’t need to know it… Yet.
The best method I’ve found for giving the reader backstory is to let the characters uncover it, so the reader learns it at the same time. This enables the story to progress, the character and reader to learn, and highlights the emotions of the characters in a way the audience inherently identifies with. Win, win, win.