I don’t think you can have true sentient AI until the machine can feel pain and this is why: In humans mirror neurons have evolved to allow us to feel someone else’s pain which is unpleasant but useful in society. We have developed morals to avoid pain in ourselves by preventing or reducing pain in others.

If the machine cannot feel pain how would it know right from wrong? From the programmers subjective opinion? Even with unlimited data points there will always be the possibility of the Trolley problem. Is it simply a numbers issue and the choice is to go with the fewest casualties? What if the next Einstein was in the casualty group? Is that a data point the machine would take into account? Morality is a messy business and easy solutions are hard to come by.

I think the best we can hope for is to reduce the total amount of pain of all living things but especially in people as suggested by Sam Harris in “The Moral Landscape.”