If we only had 20 the witnesses outside the top 20 would have an even harder time getting votes.
Until folks only vote the top 20 in attack scenarios and spread their votes outside the top 20-30 routinely we should probably keep them, or the witness list could shrink to just the top few.