Blaming the X architecture for slow drawing and windowing operations is a common mistake. Most of the latency that's introduced into operations that affect the user's perception of GUI "speed" is introduced by toolkits and inadequate 2D acceleration by the driver infrastructure, not the X protocol. It's true that the experience is poor on many configurations, but the upside is that there have been great advances on fixing these issues in the recent years (check out the new "flicker-free" GTK for an example).