From rules-lite dungeon crawl to rules-heavy railroad

Looking back across the years and all the various editions of D&D (and other tabletop RPGs,) you can't help but notice how the hobby has evolved and changed.  Two of the most obvious ways are in the volume of rules and the style of play.  Where once there were unscripted dungeon crawls run with a system described in a rule book hardly thicker than pamphlet-size, now there are carefully scripted and paced adventures geared toward telling specific stories,and rule sets comprising many hundreds of pages spanning two or three volumes or more.

Rules expansions have been driven by a few different factors.  One of the earliest was the need for a uniform set of rules for tournament play.  Beyond that, though, I think once players and GMs master a rule set, there's a natural tendency to explore adding more complexity.  Sure, modeling combat with only attack and damage rolls works, but here's a nifty sub-system for stuns and knockouts, and one for disarming, one for grappling, and one for fighting from horseback.  Here's one to codify exactly what characters' non-class-related skills are and what effects they can achieve.  Here's another to model the effects of fatigue, hunger, thirst, and sleep deprivation on physical and mental performance.  Here's twenty-seven new variant character classes.  And so on, and so forth.

Sometimes these things add interesting choices to the game, which as far as I'm concerned is about the only legitimate reason for rule expansions.  Sometimes they're bolted on in a (misguided, I believe) attempt to increase realism.  In either case, they add complexity, against which the benefits of the new rules are often not adequately weighed.

Players and GMs alike seem to gravitate toward ever-increasing rules-bulk.  Perhaps on some level, the "game" aspect of a role-playing game impelled us to try to fit all the myriad possibilities promised by RPGs into a more familiar game format.  You might house-rule your Monopoly game, for example, but it's still governed by objective rules.  At no time would it be considered appropriate for a player to try something not specifically authorized by the rules, nor for the Banker to make an ad hoc ruling to determine the outcome. 

It's true that within the framework of a rules-lite system, a character can try almost anything, but in a system laden with subsystems, modifiers, and conditional rules, the player doesn't have to rely on the GM's ruling.  These are the odds that the action you want to try will succeed; this is what happens when you succeed, and this is what happens when you fail; it's right there in the book!  The GM is absolved of the responsibility of making rulings which might anger players or provoke arguments: This is what happens, not because I say so, but because it says so right here in the book!

Railroading, at least in my experience, has somewhat different roots.  When I first learned to play the B/X game, I had as my template the module B2: The Keep on the Borderlands, which was an admirable example of a setting without a particular story.  What the PCs did there, and how the inhabitants reacted to them, was the story.  This is true of almost all the early modules for non-"advanced" D&D (i.e. Holmes and Moldvay/Cook.)  Look at the B-series modules up until about B5 or so.  Look at X1: The Isle of Dread.  Even when there's a backstory to the setting, it's meant to provide information to DM and players, and to explain why the setting is currently the way it is, not to dictate an outcome.

Somewhere along the line, though, adventure modules shifted in focus.  Instead of a backdrop against which a game group could generate their own stories, the core of the module was a story crafted by the module's author, and any NPCs and setting elements were there to serve that story, and also unlike those earlier modules, they very likely would never be used or visited again.  These adventures featured programmed, scripted events to drive the plot forward at least as often as they featured map-based encounter keys.  Instead of moving from room to room in a dungeon, the party moves from event to event in a timeline or a set of possible timelines.  While the spatial matrix of the dungeon constrains the absolute freedom of the players to some degree, the narrative matrix of an event-driven adventure limits them in much more profound ways.  Sometimes an adventure seemed to consist of nothing but a series of set-piece encounters, each with the purpose of providing the clue that leads to the next one.  Occasionally, the outcome of an encounter might influence which branch of the matrix the story would proceed upon, but almost invariably the branches converge again at the same climax anyway.  

What drove the change from the spatial structure of the dungeon or wilderness crawl to the temporal/narrative structure of the story-based adventure?  My guess is Frustrated Novelist Syndrome: module authors wanted to show off their storytelling chops rather than just provide game groups with some interesting scenery to chew up in any way they choose.  As reading material, many of these modules are wonderfully engaging.  They're full drama and intrigue, intricate plots set in motion by spectacularly detailed and backstoried antagonists.  There are clues both subtle and explosive weaving every plot thread together, and careful pacing to build tension slowly to a thrilling climax.  And you think to yourself, THIS is what I want my players to experience! 

Perhaps DMs were writing scripted, event-based adventures on their own initiative before the slew of published modules that enshrined the form as an RPG staple, but in my case at least it never occurred to me to do it any other way than the Caves of Chaos/Isle of Dread template until I read some of those magnificent railroads, and I admit that I caught the bug.  

These things represent, to some degree, independent trends in RPG evolution, but they're also connected in some interesting ways.  For instance, an airtight set of rules can facilitate the running of a railroad.  The GM can more easily anticipate what players will do and how often they'll succeed, and players are conditioned not to try anything that's not explicitly endorsed by the rules.  A complex set of tactical rules can also give a player some measure of agency, or at least the illusion of it, when engaging with the various set piece encounters in a railroad adventure.  He may have little or no control over where the story goes between those set pieces.  It may be a foregone conclusion that he and his allies will defeat the ogres blocking the bridge and go on to meet the baron, but with enough flashy character abilities and combat options, he can at least show off his system mastery and beat those ogres with panache!

Both also stem, in large part, from a lack of trust and a corresponding need to exert control.  The players don't trust the DM to make impartial rulings.  The DM doesn't trust his own judgment under pressure.  The DM doesn't trust the players to accept rulings.  The DM or the players or both don't trust in each other to make an exciting, memorable story through spontaneous interactions at the game table.  Ultimately, whatever the merits or demerits of rules-intensive game systems and story-driven adventures, they fail to solve those issues.

What does this say about the OSR, then?  Is there more to it than just nostalgia?  Perhaps a backlash against one or both of these trends, which, while they both clearly have their beginnings in the halcyon days of TSR D&D, are generally scorned by self-identified old schoolers these days?