Ultimately, if we do transparent per process memory encryption, then we can let the CPU do all the speculation it wants, but the result will be gibberish. And it's a lot easier to do a simple key switch than doing a full TLB flush. Of course, it probably doesn't do much against/for the timing attacks (side channels).