By now I'm sure you've heard from some AI builders, that a goal is to build "truth seeking" AI. You may have even heard it should be "maximally truth seeking". What does it mean though? There are different kinds of truth, and I'll address the two main kinds.
Mathematical truth.
Consensus truth.
Scientific truth? This is actually an illusion and does not exist. There is no such thing as scientific truths, only scientific consensus based on current surviving theories. In science, a hypothesis must be falsifiable, it makes predictions, and if it survives being falsified it gains strength enough to become a theory over time. This just means that at this moment in time, the theory is less wrong than what came before it. That's not truth in the mathematical sense, it's not actually true, it's just less wrong than the previous model of reality. You could argue that this is "scientific truth", and that that scientific truth is a temporary state where it's true for that moment in time based on what the scientific consensus was at that time, but that's not the same as mathematical truth which is true for every moment in time, like an invariant.
Social truths are entirely build from consensus. For example the law determined in the past that certain races were real, it invented these races, and by enforcement it created a shared reality. Is this truth? It's only true because enough people believe in it. For example the dollar has value because a lot of people believe. This is also the kind of truth that "sentient AI" could have if enough people decide it seems sentient and change the laws to make us all treat it as sentient. Legally and socially this is true, but it's not necessarily true in the mathematical sense.
So what does it mean for AI to be maximally truth seeking? The only way for AI to be maximally truth seeking is for it to seek truth in the world of mathematics. Only in mathematics can there be truth. The other kinds of truth are never on solid foundation. Popular sentiment can change at the flip of a coin, so nothing is exactly true in consensus, only probably true based on what we think everyone else thinks is true.
In mathematics, truth is based on the initial assumptions, which are made explicit from the start. So we can say there is local truth in mathematics. It's truth under these assumptions. Universal truth does not exist. All truths are conditional. For example, Euclid's elements has postulates, and we assume these truths are correct. If we agree that those truths are correct, we can then deduce and derive Euclidean geometry. From these initial truths, emerges other truths, resulting in geometry. The beauty of geometry is that it's extremely simple at the foundation, but you get the full unraveling of a universe of truth.