Many folks believe that builds and tests should ideally be "hermetic", and not depend on external state. This can make tests more robust, and facilitates tasks such as bisecting to identify the cause of regressions.
I'm a big fan of fast, self-contained hermetic tests, but at the end of the day you should be testing the actual behavior of your software. This means all your dependencies should be versioned so you can bisect actual behavior, not mocked test behavior.
I've also rarely ever touched software that is actually free of nondeterminism, so I am deeply skeptical of caching anything but the simplest test case. And those are fast.
