I love this idea, but the practicality of it all does depend a lot on the language used to describe the diagrams. I have seen impressive "single file" LaTeX articles, where the figures were carefully crafted in Tikz, and the source code of the figures was all but readable.