The runtime doesn't know all that much, though. All it has is a single instruction flow, that it can extract fine-grained parallelism from and try to speed up further via speculation. Nothing whatsoever about other work that may be scheduled in when the processor is stalled by memory, other than via SMT. Nothing about priorities or coarse-grained dependencies among work units. So there's a lot of parallelism that's left on the table, and a lot of speculated work that might just be wasted.
If we are talking about JIT, yes, it does, for it instruments the runtime, gathers the information about hot code paths and performs the in-place optimisation. Think of the profile guide compile time optimisation having been carried over into the runtime.
