While working with AWS and attending various AWS conferences I noticed what looked like remote control cars racing around a small track but without a human ‘driving’ them. I discovered this was known as Deepracer. It looked intriguing but I didn’t imagine this interest would extend to getting involved in a work based way.
That changed when I was at AWS:reInvent in January. There was a talk by @Chris_L_Suter on how the NHS Business Services Authority is using Machine Learning powered by AWS to process prescriptions. He revealed they had been playing with Deepracer internally and that he had set up a Public Sector league for Deepracer.
For those who have not come across Deepracer before, AWS describe it as:
“…the fastest way to get rolling with reinforcement learning (RL), literally, with a fully autonomous 1/18th scale race car driven by reinforcement learning, 3D racing simulator, and a global racing league. Developers can train, evaluate, and tune RL models in the online simulator, deploy their models onto AWS DeepRacer for a real-world autonomous experience and compete in the AWS DeepRacer League for a chance to win the AWS DeepRacer Championship Cup.”
Defined at https://aws.amazon.com/deepracer/
And to clarify reinforcement learning (RL) :
“Reinforcement learning is the training of machine learning models to make a sequence of decisions. The agent learns to achieve a goal in an uncertain, potentially complex environment. In reinforcement learning, an artificial intelligence faces a game-like situation. The computer employs trial and error to come up with a solution to the problem. To get the machine to do what the programmer wants, the artificial intelligence gets either rewards or penalties for the actions it performs. Its goal is to maximize the total reward (…) By leveraging the power of search and many trials, reinforcement learning is currently the most effective way to hint machine’s creativity. In contrast to human beings, artificial intelligence can gather experience from thousands of parallel gameplays if a reinforcement learning algorithm is run on a sufficiently powerful computer infrastructure.”
Taken from ‘What is reinforcement learning? The complete guide’
Experimenting with AWS Deepracer at Invotra
I was keen to apply what Chris had revealed and, as I enjoy a good competition, this seemed an ideal opportunity to introduce machine learning concepts to the wider company.
We set up a lunchtime meeting on a Friday and invited anyone from the business who was interested. The first few meetings were spent discussing RL concepts before moving on to experimenting.
One of our first challenges was getting our head around the idea that we were not writing code that says for example:
Drive in a set direction
On the straights do x speed
On the corners do a quarter of the speed on the straights
Instead, with RL, we had to write a reward function. What would this look like? AWS exposes various parameters to you including:
Is the car on the track?
How far from the centre link of the track is the car?
What speed is the car doing?
With these parameters, you then want to come up with the best reward function.
So I could write a really simple reward function of
If car is on track give maximum reward
With enough training, the model would probably have enough ‘learning’ to get around the track. However, to get to this point is going to take a lot of training as we aren’t giving the model any other context.
If it zig-zags all over that track but stays on the track the model is still going to receive maximum reward and we want to get around the track as quickly as possible.
So we need to provide the model with more context so that it is rewarded when it takes the racing line and archives the best speed.
Hopefully, that gives you a basic idea of the shift in mindset that is required to approach Machine Learning problems.
How far have we got with AWS Deepracer at Invotra?
We have come a long way from where we started! Initially our model was not even able to get around the track, and now, each week we are shaving a second or two off our track completion time.
We also celebrated coming third in the Deepracer Public Sector League!
Final Thoughts
I’m really pleased that we have been having a go with Deep Racer and taking a foray into the world of machine learning. We have had staff from across the business involved and have enjoyed exploring the different ideas that have been brought to the table.
With regular offers to take advantage of , why not have a go yourself!
Talk to us
To learn more about Invotra, please talk to us.
