I did nothing but play with Lego. Eventually I had children of my own, who had a nice Lego collection themselves, but nothing you’d need machinery to sort. That changed after a trip to Legoland in Denmark.
I noticed adults at the park buying Lego in vast quantities, despite its high price. Even second-hand Lego isn’t cheap, sold as it is by the part on specialized websites, or by the boxed set and in bulk on eBay. I noticed that bulk unsorted Lego sells for roughly €10 per kilogram (about US $11/kg), boxed sets go for €40/kg, and collections of rare parts and Lego Technic pieces (the sort used to build complex mechanical creations) go for hundreds of euros per kilo. Consequently, there exists a cottage industry of people who buy new sets and bulk Lego and manually sort all the pieces into more valuable groupings.
I figured this would be fun to get into. I put in some eBay bids on locally available large lots of Lego and went to bed. The next morning, I woke up to a rather large number of congratulatory emails from eBay sellers (eBay lesson one: If you win that many auctions, you are bidding too high).
And so, after I picked up my winning lots of Lego, my garage was stacked top to bottom with crates and boxes—about two metric tons, all told. Sorting this by hand was never going to work. I decided to build something that would scan and sort each part accordingly.
The first problem was that feeding Lego from a hopper mounted above a conveyor belt is surprisingly hard. I’ve yet to find a configuration so wide and deep that a random assortment of descending Lego could not spontaneously form a pretty sturdy bridge across the opening. Consequently, I use a slow belt to pull parts up out of my hopper before dropping them onto a much faster transport belt. This belt moves parts past a $30 magnifying camera connected via USB to a PC for identification.
Once identified, the parts have to be moved from the transport belt to the correct bin. After some experimenting I settled on putting air nozzles next to the belt. A well-timed puff knocks the part into the desired bin.
The biggest challenge of course was identification. First, I had to capture a good image of each piece. There were all kinds of gotchas here. For instance, parts may be longer than a single image frame, parts can be a color that is extremely close to that of the background, and so on. It was several weeks before I could reliably stitch and crop scans automatically, so that I was generating images containing one complete piece of Lego.
Then came the really hard part: identifying each piece. Lego bricks come in thousands of distinct shapes and over 100 colors (you can roughly tell how old someone is by asking them what Lego colors they remember from their youth). Initially, I tried to classify this zoo with the OpenCV computer vision library. Using things like contour matching and circle detection, the system could tell the differences among the basic Lego bricks, but not much more than that.
Next, I tried Bayesian classification: I chose distinguishing features and built software detectors for those. I came up with around 18 features, which included things such as the height of the part, whether or not it had any holes, how many studs were visible, and so on.
Building and testing the detectors took quite a while, but eventually I was able to identify pieces with impressive accuracy. But the system was too slow to keep up with the machinery. After a few other failed approaches, and six months in, I decided to try out a neural network. I settled on using TensorFlow, an immense library produced by the Google Brain Team. TensorFlow can run on a CPU, but for a huge speed increase I tapped the parallel computing power of the graphics processing unit in my US $700 GTX1080 Ti Nvidia video card.
TensorFlow has an arduous learning curve, but eventually I was pointed to Keras, a Python language library by François Chollet. Keras acts as a wrapper for TensorFlow and makes it much easier to use, especially after looking at Jeremy Howard and Rachel Thomas’s excellent starter course on machine learning.
I started building my neural net system in earnest. Within several days I had the sorter working and handling more than a few classes of parts for the first time.
The next step was to get a training set for my neural network that was large enough to make working with over 1,000 types of Lego pieces possible. At first this seemed like an insurmountable problem: I could not figure out how to photograph and label enough sample parts by hand. Even the most optimistic calculations had me working for six months or longer flat out.
Then I realized I didn’t need to make the training set all by myself. The machine takes and labels images. All I need to do is spot the ones where the computer was wrong and relabel the image correctly. As the neural net learns, there are fewer mistakes, and the labeling workload decreases.
The first day I managed to label a starter set of about 500 assorted scanned pieces. Using those parts to train the net, the next day the machine sorted 2,000 more parts. About half of those were wrongly labeled, which I corrected. The resulting 2,500 parts were the basis for the next round of training. Another 4,000 parts went through the machine, 90 percent of which were labeled correctly! So, I had to correct only some 400 parts. By the end of two weeks I had a training data set of 20,000 correctly labeled images.
Some classes are still underrepresented in the training set, so I need to increase the number of images for those. I’ll probably just run through a single sample batch consisting of nothing but those parts. Once the software is able to reliably classify across the entire range of parts in my garage, I’ll be pushing through the remainder of those two tons of bricks. And then I can finally start selling off the results!