There's a difficult balance between "future-proofing" and "over-engineering". I'm not sure which is actually worse. Given how often I've seen a client happily run a prototype of some code in a production environment for years, I tend to err on the side of thinking future-proofing being a bit of a waste of time. This is especially true if there isn't a clear roadmap that states when something is going to change, and the cost of changing isn't already committed to the project.
IME the best architectural designs are usually the result of heavy retrospective refactoring, not up-front design.
Thus future proofing isn't just a waste of time, it's actively counter productive. The predicted requirements are rarely correct although the extra code remains a liability.
> IME the best architectural designs are usually the result of heavy retrospective refactoring, not up-front design.
Right. And this applies to other disciplines too. "Heavy retrospective refactoring" is exactly what a writer does each time they go back and make edits to a piece. Connections between elements such as characters, themes, writing style, and dialogue are felt out only by emitting draft after draft.
Even this comment went through many edits to get into this shape. These two sentences alone have been rewritten ten times.
In my experience there's also value in imposing a framework once one has done enough emitting of raw material. For code this might be looking for ways to extract common logic out of several similar functions. For writing this might be considering how two characters have "interacted" so far and what other details ought to fill out their relationship (without considering whether or not all those details will manifest overtly in the work.)
This is where experience comes into play though. Not every bit of software is totally unique and built on first principals. If you're working at some place and built similar software at a previous place, part of the up-front design is going to be driven by retrospection on previous projects.
But wait a minute - shouldn't most of one's future-proofing be making sure you CAN easily alter a design later if need be (including better documentation), especially in the ways you can already see change coming?
Cleaning up existing code will make it easier to alter designs later and that's something you should continually be doing as you implement features and fix bugs, but that's not pre-emptive architecture.
As I say elsewhere here, I've killed projects assuming that. Pulling out bricks from the bottom and replacing them is sometimes so hard, you really have a dead project.
