MakPilot v1.0

MakPilot V1.0 is an AI that I trained to play Euro Truck Simulator 2. For me, the objective with this project was to get the AI to drive the truck straight on the highway without traffic, as a first step with this project. It performed much better than I had originally expected it to, however I do think there is room for improvement in some areas. But overall, I'm happy with how it turned out considering this is only the first iteration. I was inspired to do this after watching Sentdex's video on how he trained a neural network to drive a car in GTA V in a similar fashion.

I used Python and Java for the development of this project. Python was used to build and train an implementation of the Convolutional Neural Network (AlexNet) while I used Java for gathering training data.

---

To get started, I had to get training data with which I would train the neural net I would build. I created a Java Swing application that allowed me to do this. Anyway, in order to train the network, I needed features and labels. In my case, the features (inputs) would be the section of the road in front of the truck in the game, and the labels (outputs) would be the key I pressed.

The Java program captured the section of the screen which contained the road (230x120 image), and also recorded the key I was pressing using a global keyboard and mouse listener. It stored the image in a directory containing all my training images, but also stored the filepath string of each image in a CSV file, along with the string representation of the key I pressed.

Once I had finished gathering this data, I realized that out of the ~100,000 images I had collected, around 90K were from when I was pressing the Up arrow, while I had around 5000 images each for pressing the Left and Right arrows. So, I needed to balance my dataset as well, which meant selecting 5000 samples for each label and combining them to form a resultant dataset of approximately 15000 images. Had I not done this, my model would have failed quite miserably.

Next, I needed to preprocess this resultant dataset, because what's a CNN supposed to do if you only give it a multi dimensional array of strings? So, I wrote a script that read each filepath from the CSV as a numpy array using OpenCV's imread() function, and appended it to my list of inputs, which I would later convert to a numpy array. It also took each label and encoded them into valid output labels that can be used to train the CNN using the one-hot encoding method.

This process took considerably long, due to the large number of samples, and the large dimensions of the images. So, I ended up with two arrays: an "X" array which contained all of my training inputs, and a "Y" array which contained all of my training outputs.

However, when it came time to save the arrays as a .npy file, my PC froze every time I tried. My machine was simply not capable of saving and loading HUGE datasets and arrays like the ones I had created. I did, however notice that it was capable of saving arrays with up to 3000 samples. So, I decided to splice my array into chunks consisting of 3000 samples each, which I would then reload individually in my training script. With that, I was done with generating and preprocessing the training data.

---

Now, I needed to create and train a Convolutional Neural Network model in Python. I used the Keras library with Tensorflow to accomplish this, because I felt as if it was the perfect tool for beginners in deep learning like me. So, my first idea was to create a Keras implementation of the VGG16 convnet. After I had created the model and started training it, I noticed that it was taking an extremely long time while producing terrible, terrible results in terms of validation accuracy.

So, firstly, rather than training the model locally on my machine, I decided to make use of Colab Notebooks on Google Drive, and train my model using a GPU that they would provide. But even that still produced mediocre results while training, so I decided that the problem lay elsewhere. This led me to make two significant changes: 1) Use the AlexNet architecture instead of VGG16, 2) Resize the image to a smaller size (184 x 96) to reduce training time. After adding these modifications, I noticed that training the network became much easier, and I could do more epochs to be able to attain a higher validation accuracy of around 80%, which is not bad at all considering the number of training samples I had.

---

\Now, all that was left was to test the model. So I wrote a python program that loads in our trained model, reads the area of the screen in the game using the PIL image libary, gets predictions from the loaded model and sends the corresponding keystrokes to the game. And that was it!! I had created my very first machine learning project!

---

In terms of the model's performance, I think it performed much, much better than I had expected. When I first saw that the model was 80% accurate, I couldn't help but underestimate its performance, as I imagined it to make absolutely random turns everywhere, wreaking havoc upon the road in every way possible. On the contrary, it actually turned out quite decent, especially considering the relatively low number of training samples I had. Shockingly enough it performed pretty well during the night time as well.

Still, that isn't to say that there is no room for improvement. Occasionally, the AI would repeatedly hit the truck towards the highway median, and the overall performance seemed to decrease whenever the roads were a ligher color and there was low contrast between the road lines and the surface of the road itself.

But yeah, for the record, I see this as an absolute win!!