Small Planet is a New York based development shop that builds apps and digital experiences for companies like FanDuel and Disney. Recently, they’ve started a Machine Learning Lab to do take their projects to the next level with AI, Core ML, and TensorFlow.
We came across Small Planet after finding their awesome Pinball Wizard project where their team trained a neural network to play pinball. Not satisfied with simply playing a digital pinball game, the Small Planet team hooked an iPhone 6 up to an Onion Omega2 IoT computer so that it could manipulate the flippers of a real machine.
They collected a bunch of training data from the iPhone’s camera and used that to train a network which was then loaded executed on the phone using Core ML.
You can check out all of the code on Github and see other projects by the team on their website.
This week, we caught up with Rocco Bowling, CTO of Small Planet, and asked him a few questions about the project.
What’s your background and how did you come up with the idea for a pinball playing iOS app?
I am a computer scientist with 20+ years experience. I have a very broad history in the industry, doing everything from real-time visualizations of large data sets at the NSA to independent game development. I’ve worked on everything from embedded devices to supercomputers, from mobile devices to the Microsoft Surface Table.
I’ve developed apps that have taken home Apple Design Awards and I’ve published a few duds too. I enjoy everything from graphics to AI and I love optimizing code for performance and solving complex problems. For the past eight years I have been the CTO of Small Planet where I lead a talented group of developers as we solve a mix of very interesting problems for our clients.
Every year after WWDC a few of us at Small Planet get together and brainstorm ideas under the premise of “what cool shit can we do with the new stuff announced at WWDC this year?” We then present those ideas to Gavin Fraser (CEO at Small Planet) and together we pick something worth exploring between our normal client engagements. As you can imagine, this year many ideas were generated around AR and ML.
I put forward the idea of making an iOS app that could play a real pinball machine. The idea had appeal because it was something no one had done yet but it relied on areas of machine learning which have been well researched. This was important as no one at Small Planet had experience with machine learning yet, so this would be an ideal opportunity to learn.
What’s does your tech stack look like and what tools did you find helpful?
For the iOS app itself we used XCode and Swift. We started pinball while iOS 11 and Mac OS High Sierra were still in beta, and the early choices for Core ML development were limited. After some quick experiments with using TensorFlow directly, we settled on using Keras, the python deep learning library stack for our ML training. At the time it was one of the few stacks compatible with the coremltools. Recently Core ML has added support for TensorFlow directly and we will be experimenting with that.
By coincidence my existing computer was already well equipped to handle ML so I have done all of my training locally. Quinn made use of cloud-based solutions like AWS and Paperspace to access GPUs and leveraged Jupyter Notebooks for training and documenting his process.
What was the hardest part?
The concept of an iOS app playing pinball is comparatively simple and can be built using existing ML research. However, the difference between implementing these ideas in a fully-controllable, simulated environment and implementing it in the real world are rather vast.
For example, to use reinforcement learning one might need to have the AI play millions of games in a sped up simulated environment; attempting use the same method on a real pinball machine would take a very long time. It’s the difference between using ML to play Mario Kart and using ML for self-driving cars. We are still solving these problems for phase 2 of pinball.
Do you have any advice for other developers who are looking to get started with machine learning?
Don’t let the idea of machine learning scare you into not trying it. As with most topics in computer science these days, there is a wealth of information freely available to get you started. Just pick a manageable idea and go for it!
Comments 0 Responses