Self-Driving RC Car
Skilled Used: Deep Learning, Reinforcement Learning, Computer Vision, YOLO, PID Control, Algorithms, Simulation
I took a project-based class with the final goal to have a self-driving RC car controlled with Reinforcement Learning. I was in a team of 3 for the semester long project. Our physical environment was Duckie Town, but we were not allowed to use their simulator or any other code associated with Duckie Town. We had various milestones including PID Control (Simulator + Physical Car), Reinforcement Learning Control (Simulator + Physical Car), and Road Sign detection (using YOLO).
Implementing PID control was fairly straight forward. We used color thresholding and segmentation to find all of the yellow marks on the input image (road centerline). Then, we used the horizontal position of the closest yellow "blob," as the PID setpoint. This worked surprisingly well in both simulation and hardware and was robust to image noise. The image of the car driving with PID control is below.
We also tried keeping a buffer of yellow "blobs," masking out certain parts of the image, different color spaces, but in the end the simple approach worked the best for PID control. In reality, it was just a proportional controller as any derivative or integral gains made the system more unstable.
pid_control.mp4Reinforcement Learning was a huge challenge for our team. We initially selected the DQN algorithm and spent a lot of time attempting to get it to work. We also tried PPO and SAC. We even tried stable-baselines to make sure that our implementation wasn't incorrect. Here's a list of other methods that we tried to get good results with Reinforcement Learning Control:
Reward Shaping (Sparce vs. Dense Rewards)
Image Preprocessing (Adding noise to training, only looking at certain colors, different color spaces, resizing the image)
Network Input (Image buffers, black and white images, locations of yellow center, location of road lines, etc.)
Network Architecture (more/less layers, hidden nodes, activation functions, etc.)
Training Methods (Randomize start locations, randomize map seed, maximum number of training epochs)
Action Space (Depending on the algorithm we could allow turns from -30 to 30 degrees in discrete or continuous steps)
In the end, we got decent results in simulation, but our reinforcement learning controller never really worked on the hardware. This was disappointing because we spent a lot of extra time to make it work, but it was not uncommon in the class. Our final controller is our PID controller and it works great (see final video below).
model_inloop.mp4This is a short video of one of our models. The car is represented by the short red and green lines that move around. As you can see, the model quickly gets stuck in a loop in the lower right hand corner.
While driving on the physical course, there were a total of 14 objects that needed to identify and play a short sound once they were detected. Mostly signs, but there is a railroad crossing with arms that go up and down, so it was critical that our detection worked well to avoid damaging the car or environment. We used YOLO with the tiny weights for extra speed. This allowed us to check for signs up to 15 times per second. Our sign detection worked great and there is a demo in the full video below.
