The trouble is that the memory representation is fixed and observable in many cases. If you have arrays of primitive or repr(C) types, those are guaranteed to have the same contiguous layout as an array in C would. In particular, you often need to pass pointers to those things to actual C code. Copy types are similar; even if you can't necessarily assume all the details about their layout, you can copy them byte for byte and assume the result is valid. But that doesn't work if the compiler secretly inserts an owning heap pointer that needs to remain unique. The compiler could try to deduce when the memory layout is actually observed in practice, and do whatever it wants in cases where it can prove the difference doesn't matter, but I feel like that would end up making everything more complicated rather than less.