In an earlier post I described my daughters 6th grade science fair project to examine the exploration/exploitation trade-off in reinforcement learning. That post covers more details of how the robot is learning and the software and hardware we are using to control the robot. This is is just a quick video of one of the trials showing the robot learning with a small amount of exploration that decays over time. In this case, the probability of randomly taking a move each step (epsilon) is 10/(10+t) where t is the step number. If the robot does not take a random move it uses its Q table (see the previous post) to take what it has learned is the best move for the state it is in. Each trial consists of 2000 steps. At the beginning of the video t is such that there is around an 80% chance of taking a random move. After 1000 steps (the middle of the video) the chance of a random move has dropped to less than 10%.
The robots goal is to learn to maximize its total future discounted reward. Each step of the trial, the robot selects an action and then gets a reward. If the robot moves forward it gets +1. If it turns right or left it gets +0.5. All other actions (rotating or moving backwards) get -1.0. Hitting a wall gives it -5.0.
To sense its environment, the robot has only one touch sensor tied to the front bumper and one ultrasonic sensor. The ultrasonic sensor can read distance from 0 to 255 cm, but to keep this simple for my daughter, the distance is rescaled to 0 to 4. After a few trials and some follow-up tests with the ultrasonic sensor, we realized that the sensor cannot detect the wall if it is at a very low (acute) angle to the wall. At that point, it returns that it is 200+ cm from the wall. As a result, the robot tends to learn a conservative strategy of simply going in circles when it is far from walls. Under perfect sensor conditions, this would not be optimal as it will only return .5 per step, or 100 for 200 steps vs. what should be closer to 200 under an optimal go forward then turn before the wall strategy. However, the problems with the ultrasonic sensor, possibly in combination with the various parameters we have set, are making it difficult to find and maintain that strategy.
Another problem here is that due to the sensor issue and how the distance is discretized, the robot's world is only partially observable. The robot would benefit from adding a memory of recent states, but that would greatly complicate the Q table.
The robot could also benefit from moving to a learning approach that uses continuous states and actions, but that requires a shift to much more complex algorithms and underlying technology, such as DDPG (Deep Deterministic Policy Gradient). That's not going to be something my 6th grader can explain to others.
Very rough code is available here: https://github.com/tjohnson250/ev3rl
To support this post please upvote, follow 
Presearch: Earn Cryptocurrency By Searching the Internet
Honeyminer: Start mining bitcoin in 1 minute
Proud member of: