Trickle Down Attention Doesn't Work