Background
Autonomous systems are rated from Level 1 to 5. Five (and four?) being the ones that don't require some human oversight. Currently consumer level autonomy is advertised as Level 2. Level 3 is primarily reductions in the number of situations where the human is required to step in. Level 4 is human optional but the car may 'pull over and stop' if it's confused requiring human rescue.. possibly creating problems.
The levels describe the expected capability of the AI.. but there is no levels or clear boundary or limits to human comfort with the idea once it gets started.
The Familiarity With Aliens Problem
Once you experience being driven by an AI that is highly capable in one area it's very easy to trust it, and very natural to grow that trust faster into areas that the AI's actual abilities don't cover. This is 'projection' or tendency to 'anthropomorphize'.. assuming because it acts human (how we want it to act), it will continue to act human in other ways or situations. With driving there's no safe way to know you've gotten too trusting, unless you experience the bad situation BEFORE you trust the AI to deal with it. But your trust will grow faster than the number of experiences you have. We as humans can't stop that.. it's inevitable. We're going to need help.
As AI drivers get better Human comfort grows and therefore human participation in the system inevitably gets worse over time. Even if the AI doesn't improve...the human comfort with it's abilities will grow.. Humans do get too comfortable, plus they get tired, plus sometimes they just make a bad decision about when to be aware. There's no feedback for being a good 'backup' driver until something bad happens.. and the longer AI prevents something 'bad' from happening.. the less likely the human will be paying attention when it does..
The Safest Cars
What cannot be argued.. is that if autonomous driving is on, and the Human is fully active monitoring the driving.. you have the full safety of both redundantly covering each other.. and autonomous cars are safer in every way.
This is the 'safe' design of level 2 promoted and required by Tesla,. GM, Nissan, Cadillac, and others.. and there can be no question it is safer if used as instructed by the car.
Where it gets unsafe.. is when the humans decide their awareness is optional. This happens unpredictably based on their individual history and personality and anecdotal experience and increases in probability with each AI success. Ironically the more good autonomy happens, the more unsafe the human backup system becomes.
The only thing about AI that advances faster than AI ability, is the human assumption they understand AI and can predict it.
AI success breeds reluctance to put effort into being 'the backup'. No human wants to be the backup that's never needed. That's where we check-out mentally. But that's what all agree is our 'safety solution'.. the human backup. It may not fail often.. but it will fail very predictably.. and there's maybe AI solutions to that too. Help the human feel necessary.
Driving the Humans
I think perhaps partially autonomous systems could include intentional 'quirks' to keep the human backup 'turned on'. Autonomous cars can easily make unpredictable moves in a safe situation to shake off any build-up in laxness in the human driving components... until level 5 autonomy stops requiring human oversight... AI would be safest by encouraging / rewarding the human brain for it's attention.
The perfect synergy and safety would be if the human is hands on and showing awareness... the car knows the human is helping and drives perfectly and never misbehaves... the safety is the total of the best abilities of both 'drivers'.. and the human driver is both empowered, and freed from the mundane bits ( the natural step up from cruise control ). This is co-piloting .. this is AI & Driver symbiotic autonomy levels 1,2 & 3.
If the human is seen to get regularly 'too comfortable' the car attempts to disrupt the overconfidence. If you've ever traveled late at night maybe you've had the conversation where the driver says to the passenger. "I need you to talk to me to keep me alert". This is the AI version of that... More than 'rest a hand on the wheel please'... There's a more lasting , sub-conscious level awareness that needs to be stirred up.
Some potential 'wake up the back up' behaviors.
- take a turn / exit that doesn't significantly impact travel-time.
- ride a lane marker or ping-pong a bit when no cars are near-by if driver is getting inattentive.
- instead of flashing a 'hands on' warning... brake-check the car if the driver isn't attending (and there's no following vehicles)
- Switch to the 'teen-driver' model, harsher accelerate, braking and steering corrections .. if driver is getting too comfortable and not monitoring.
- Just straight up refuse to engage AP if the same driver has been on the road for 12 hours and is increasingly nodding off / hands off.
The problem of course is wake up the human.. without startling the human so much that they initiate a panic maneuver themselves?
A Happy Future Together.
The goal of course is that Autonomous improves until it's actually as safe as it feels like it is.. and we all get to relax and ignore the way we want to...
The food for thought is... until level 4 or 5 happens should Autonomous systems do more 'interacting' to make humans still feel necessary?..
- Should they 'act' unpredictable but not unsafe occasionally to self-correct the humans sub-conscious?
- Should Autonomous driver assist include disrupting human driver expectations to improve safety?
- How can AI reward the human backup for it's attention?
- How can AI interact with the human driver at the sub-conscious level?
- Is there anything your car could do when you aren't aware. . that you wouldn't 'learn' to ignore.. but wouldn't cause you to disable the AI