First off, thanks for putting time and effort into actually understanding this stuff. We need more smart people like yourself actually researching and asking questions.

Secondly, I think many of the confusions you have here are in understanding the basics of blockchain technology in the first place. DPOS (in many ways) is blockchain 2.0, and I think you're confused on a lot of blockchain 1.0 concepts, based on my reading here.

Maybe this can help: Understanding Blockchain Freedom — Episode 002: What Is Cryptocurrency? (specifically, the 17 minute Blockchain 101 video I refer to there)

Or maybe just start with the source, the original 9 page Satoshi white paper: bitcoin.org/bitcoin.pdf

Many of the questions you ask here revolve around something that is not technically possible on a blockchain. If a node of the blockchain doesn't follow the agreed upon protocol, everything it does gets rejected by every other node due to the nature of what a blockchain is. It's almost like saying, "What if someone tried to send an email with custom SMTP commands?!?!" Every other SMTP server would reject the email and not send it.

Examples:

Suddenly, a bad transaction is introduced.

That's not possible. If a witness node tried to add a "Send X funds from account A to account B" without the proper cryptographic signatures for the accounts involved, the transaction would be immediately rejected by every node following the correct protocol.

I'll give some more examples where I think you may not yet comprehend how a blockchain works. There is no central authority determining who can do what. It's all based on the agreed protocol that all participants in the network have to follow. In order to change that protocol, a hardfork is required and that means a majority of block producers have to agree to the code change and by doing so define the new protocol everyone has to follow.

What happens if all of the block producers collude

And do... what? Introduce a hardfork that includes a rule which says "all balances can be drained by Hive account@uberbrady"? Clearly, that rule would not be agreed to by stake holders and all those witnesses would be voted out as block producers. If required, the blockchain would be rewound and any transactions following the rule stake holders in the network didn't agree with would be rejected. Digging into the history of STEEM, you can find an example where a hacker injected a JavaScript hack and stole a bunch of funds. Something like this was done to protect those users and return those funds.

What hidden 'centralizations' are happening that could make it so that one entity an control who ends up producing blocks?

The only potential concern I see here is how the Hive account@steemit account (and many other Steemit, Inc related accounts with high Steem Power, such as Hive account@ned) could, in theory, vote in 11 witnesses to fork the chain in a direction the majority of other stake holders didn't agree with. Yes, it's technically possible, but it's also economic suicide. By doing so, they would destroy the value of the STEEM token as investors and stake holders would quickly lose trust and exit the platform. Hard fork 17 is an interesting example where the community disagreed with some changes Steemit, Inc wanted to make so the witnesses rejected the fork. 18 then went forward with modifications the stake holders could agree with.

The top N witnesses by total approval are selected. By who? How?

By Steem Power stake holders, as defined by the blockchain protocol. Again, the very nature of a blockchain involves hashing all the history up to that point. That's what makes it immutable. All nodes following the agreed protocol ensures no block producer could just do whatever it wants. It's all built into the protocol. You're familiar enough with protocols to understand how if they aren't followed, it doesn't work. You may have a personal opinion on how HTTP should work, but all the browsers would still follow the agreed upon protocol and ignore your opinion.

The witnesses are then shuffled, and .... By who? How?

Again, by the protocol. It's all open source so you could dive into the steemd C code to find the actual algorithm used here if you wanted to.

which aren't elected witnesses - to produce blocks?

Not possible. Only witnesses can produce blocks, based on the protocol.

Observers need to be constantly looking at the entire blockchain

As I mentioned in my tweet, we currently have ~30k active users. If even one of them saw their post or vote or comment disappear, there would be hell to pay and the blockchain would store a history of who produced blocks at the point that transaction was created and why it was not included in that 3 second window block. Something like that would be absolutely damning to a witness and they would be quickly voted out if they happened to be running a custom version of the steemd code which censored certain transactions.

That said, it's a valid point that we do have to keep track of. If, within a 3 second window, a transaction isn't included, that would be bad. Once it's included, due to the nature of how blockchains work, it can't be removed as that would invalid the hashes of every transaction after that and that entire chain would be rejected by every node on the network.

people trying to somehow 'magic in' new currency

Again, not possible. It's kind of funny that you suggest it might be possible. Blockchains don't allow this. :)

there is always perfect agreement about the results

Again, that's the revolutionary genius of a blockchain. In this case we absolutely can agree. In fact, we have to agree. History on the blockchain is immutable. All votes are part of the blockchain as you can see on block explorers like steemd.com (or run your own).

Where is the vote stored?

On the blockchain.

How do we prevent double-voting?

That relates to the Byzantine Generals Problem which is one of the key innovations of blockchain technology (read the original bitcoin white paper for more on that).

How do we prevent sock-puppet voting?

This is why it's Stake-Weighted voting. It's not just a vote count. It's all about the amount of Steem Power the voter has. It costs money to obtain Steem Power. There are many very brilliant systems in place here to fight against Sybil attacks.

What's to prevent a corrupt witness from bribing people

Nothing but free market choice and reputation. Funny thing is, it already happened when Hive account@jerrybanfield first started his witness campaign. He started sending money to people asking for votes and rewarding those who voted for him. Quickly, the community responded and many posts were written about this behavior. He apologized and has since changed his approach to realize he needs to provide real, tangible value to the community if he wants to maintain a high ranking as a witness. He's shifted tactics to spend money on advertising campaigns and commit his witness rewards to that effort.

how do we know that that app isn't corrupted or compromised?

By tools like https://steemd.com/, https://busy.org/, https://chainbb.com/ and many, many others. Some people just run their own front end locally on their laptops. I see that as the future, personally. We could all run our own custom Condensor via the open source project here: https://github.com/steemit/condenser (some are already doing this)

custom URI protocol -steemit://uberbrady/blog_post_slug

That's being actively discussed right now, actually. Tools like Vessel are going to use them.

when everyone is actively competing

Witness are still competing here as well, they are just cooperating with the users to get votes. In contrast, POW competes with users by incentivizing the miners to increase fees which hurts usability.

Geez, that was a lot to cover. I'm tempted to make this a root post so I could get paid well for the effort, but I don't want to call you out like that to my 5k+ followers, so I'll just leave this as a comment. I hope you found it helpful. :)