Non-Revolutionary thought in progress–complex game settings aren’t the problem (and I’m not going to go into how I think they should be expressed for maximum gameability, because that’s a whole separate thought process). You either like them or you don’t, but there isn’t a right or wrong.
The problem is, where is that complexity housed?
If you have a campaign setting that presents a world, and then products that make individual areas even more detailed, if you are interested in that specific area, that’s still only “two layers deep.” Main setting book and regional sourcebook.
However, if you have a faction book that also adds significant details to that region, and then maybe an adventure that will be assumed to be canon in its events as well, things have gone four layers deep. All of those books have significant details on a region, but only two of those books have an obvious connection to the region, with two less obvious connections.
I think I understand how this happens. Working with data now for my job, I have to stop and think about how I want to design a report. There are times I think I have accounted for everything I want to display, but I realize I can’t show part of my significant data without going back to the beginning and adding something to the overall structure. It’s a pain, but once the structure is in place, adding new data is going to be simpler, because I have the structure defined.
However, when people are being creative, they often don’t think about end users and structures. Because they aren’t thinking in terms of what should or should not be included in specific product, three months after a setting book comes out, an adventure comes along and levels a region, and three months after that, a new sourcebook assassinates a key ruler and invalidates even more of a setting.
For maximum “buy in” to a setting, my structure at the beginning would be that the core setting book is a snapshot of the setting that will be true until a new core setting book comes out. Regional books might add to details, but won’t contradict anything presented. Adventures can have huge, huge stakes, but they aren’t assumed to have happened in a campaign unless you use them at the table.
If you want to put out a new setting book five years later, cool. That’s the new default, and its really easy to point people to the new product and have it be your new standard. It also makes it easier for people to say “no, I don’t like that they advanced the timeline and destroyed X.” Regardless, it’s an easy, clearly defined transition point, instead of an aggregated mess that becomes more difficult for new people to navigate.
If you have novels, set some rules for what they are in your grand plan. If they aren’t canon to the setting, make it clear. If they are, set some limits. Have them only happen before the date of the “snap shot” in your core setting book, and make sure they don’t create any new major events you didn’t already define in the core setting book. If you want the King of Peaceful Land to have been a doppleganger all along, don’t do that in a novel, wait until you do your setting update book five years from now. You can explain the aftermath of what happened, or you can throw in some plot hooks for how that will matter for the PCs.
If you have things that happened in the past involving a legendary team of mercenaries or adventurers, that’s a great place for a novel that gives details and personalizes that legendary team. Don’t move the timeline forward, introduce new heroes, and have them do all the important stuff in the setting that will now “count.” Fill in details of the past in a consistent manner with what has been presented and leave the future as an open canvas, at least until you stake your next point on the timeline with a new core setting book.
I’ve never published a setting, so maybe I’m full of crap. However, it seems like if you keep these ideas in mind, and have some rules in place ahead of time, even a complicated setting gets less daunting to manage.
Keep everything that is in your core setting book true until you are ready for a new core setting book
Add detail in sourcebooks, but don’t contradict what has already been stated
Avoid absolutes where possible–knowing there are only X number of temples in a city can trip up a GM that already started using a region in their campaign, but saying the major temples are X, Y, and Z is fine
If you have a fiction line, don’t let it do what you wouldn’t let it a sourcebook do
If you have a fiction line, set it before the “snapshot” of the current setting book’s timeline
Don’t assume events in an adventure are canon for your setting–if you want to “canonize” the adventure later, do it when you update your core setting book
Don’t let a sourcebook add detail to an aspect of the setting that it wouldn’t obviously be concerned with–if you detail an organization, don’t add significant details to a bunch of cities that organization is active in, especially if those cities have already been detailed
Full disclaimer, I just recently finished listening to the audiobook of the Kobold Guide to Worldbuilding. I don’t think any one essay presented exactly what I elaborated above, but lot of them said large portions of what I mentioned above, my brain just kind of needed to mash together some of the bits that were common to many of the essays together and sprinkle in my own, less brilliant flavor.