When scaling a base layer blockchain, the first thing to achieve is a low-fee, high-performance layer—a data availability layer with very low-cost transactions.
There are factors to consider when having a group of witnesses. If you have too many witnesses, you run into a few problems. First, it is hard to have 100 super high-quality, reputable witnesses. No matter what way you twist it, there will always be witnesses that are stronger and better than others. The goal should be to capture the "top" quality witnesses to ensure the greatest performance. Why settle for the 100th best witness with you can cut it off to the top 20? Having 100 witnesses would mean many miss blocks, many go down, and violate swings in who is in consensus and who isn't. The more witnesses you have, the slower the decisions and upgrades to the chain becomes. However, none of this pales in comparison to the worst problem that arises from having 100 consensus witnesses, which is the ease of sybil attacks. Asking a human to research 20 witnesses is doable but stretches the limits. Asking a human to research 100 witnesses is a fool's task and cannot work well in practice. There isn't enough bandwidth for governance voters to keep up with that many witnesses.
One of the strongest elements of governance voting is reputation. It is the only way to get into consensus. Reputation is by far the biggest defense any voting system has. By making 100 witnesses, reputation goes out of the window, it ceases to matter and will become easily sybil attacked. By having 20, reputation is crucial in getting in and out of consensus. It is the stairs up and the elevator down.
Having 20 witnesses is trimming the fat, giving you the fastest performance and security. Keep in mind, when saying 20 consensus witnesses (21st rotates in and out to keep backups honest) is not the end of it, if any or even all of the witnesses in consensus went down, they would be immediately replaced by backup witnesses. The more witnesses that go down, the more incentive for backups to emerge as the competition to get into consensus is lowered. The backups earn coins also, which is important to have incentivized backups ready at the drop of a hat.
I am not someone who comes to you and speaks only theory, I have been helping run a witness for years and have slowly clawed our way into consensus, as it should be, nothing here should be easily gotten. I can speak directly from practice as well.
If you want a scaleable chain that offers efficient transactions at low cost, you need a "smaller" number of censuses witnesses that run the chain—The starters on a football team, for example. But the backups are just as important and can be overlooked, which is a shame because they are just as important. In effect, there is no limit to the number of witnesses because, in theory, backups can be infinite. There is a smart parameter to protect the chain from slow performance and sybil attacks. Without the parameter, you end up with 1 "witness pool" in charge, or if you add a parameter such as 100 witnesses, the chain becomes dramatically less secure and efficient.
I can expand into much greater detailed on this subject, but I find bitsized, direct to the point blogs can be useful for many to absorb the content better.