Middle classes losing out to ultra-rich