You sure do - the variation between devices is the manifestation of the ability to control layout depending on the recipient's display's resolution and size.

If anything, a PDF designed for A4/letter is going to be cumbersome to read on a (probably rather small) 640x480 display.

The sad state of PDF rendering on (most) e-book readers should be evidence enough.