Ah I could not remember which of xmonad or awesome was doing that. Mission Control bird's eye view makes workspace management admittedly much less confusing than either one.

It could be hidden behind the existing — and checked by default — option "Automatically rearrange Spaces".