In the past few years we’ve seen many startups and even mature companies coming up with new mobile apps or features powered by machine learning and AI. These features require some heavy, real-time processing by neural networks.
The potential killers of these ML-powered experiences? Data roundtrips for inference, the cost of backend servers to support millions of devices, concerns surrounding user data privacy.
But luckily, there’s a way to solve these issues: Mobile Machine Learning. By performing machine learning inference on edge devices, developers and engineers can count on better performance, increased reliability, faster speeds, and less power/memory consumption.
That’s all well and good in theory, but what are the available tools to make it happen? Luckily, we have several wonderful capabilities for AI on the edge in terms of both hardware and software.
Both of the leading mobile OS makers have come forward with their own implementations for running ML models on their platforms.
On top of that, there are a number of good solutions that help manage, customize, and protect TensorFlow Lite and Core ML models to make it easier for developers and engineering teams to integrate AI features in their apps.
All of these libraries aim to provide better developer experiences while working on AI-powered apps.
As a mobile app developer, one has to make the critical decision of selecting one of the available libraries. This requires a detailed analysis of what all of these libraries are offering.
In order to make your decision easier and more data-driven, I’ve analyzed some of the basic aspects/features of these available libraries.
We’ll be looking at the following tools:
Firebase ML Kit:
ML Kit is a mobile SDK that brings Google’s machine learning expertise to Android and iOS apps in a powerful yet easy-to-use package. Being backed and used by Google for a number of products in its family of apps, Firebase has a real edge when it comes to reliability.
Pre-trained models — 5 CV & 3 NLP Models
ML Kit comes with number of pre-trained models to get you up-and-running quickly with some commonly-used AI features. Here is the list [as of June 2019]:
- Computer Vision: Text Recognition, Face Detection, Barcode Scanning, Image Labeling, & Object Detection & Tracking.
- NLP Models: Language Identification, Translation, & Smart Reply
Battle tested
Firebase ML Kit is heavily-used in Google products and are making use of it on really massive scale. Given that it’s in the Firebase ecosystem, integrating ML Kit in your existing apps or in your libraries might bring some real headaches—of the Google Play Services versions.
Community & Support — Codelabs & Tutorials
The Firebase community has grown at quite a huge pace in recent months, and as more and more apps are adopting AI features, the number of tutorials and other content will continue to rise. There are also Google groups for Firebase and ML Kit.
Besides this, there are some cool Codelabs for both iOS and Android. There are some awesome tutorials for using ML Kit by Harshit Dwivedi, Britt Barak, Joe Birch, and Miguel Beltran, among others. (There are some other great articles on Medium, as well)
Support for Custom Models — TFLite Only
You can use your own existing TensorFlow Lite models with ML Kit, which acts as an API layer to your custom model, making it simpler to run and use.
Updating Custom Models — ✅ In Background
ML Kit comes with support to configure the conditions under which your app automatically downloads new model versions: when the user’s device is idle, is charging, or has a Wi-Fi connection. This provides a smooth user experience for apps that retrain models quite frequently.
Performance Monitoring — Using Firebase Performance
There is no defined performance monitoring tools for your models deployed through Firebase, but you can measure it using Firebase performance monitoring APIs.
Model Security — ❌
As mentioned in the docs for ML Kit custom model integration, regardless of how you make your TensorFlow Lite models available to ML Kit, ML Kit stores them in a standard serialized protobuf format in local storage.
In theory, this means that anybody can copy your model. However, in practice, most models are so application-specific and obfuscated by optimizations that the risk is similar to that of competitors reverse engineering your code. Nevertheless, you should be aware of this risk before you use a custom model in your app.
Something Extra — ☁️Cloud APIs
ML Kit gives you both on-device and Cloud APIs, all in a common and simple interface, allowing you to choose the ones that fit your requirements best.
The on-device APIs process data quickly and will work even when there’s no network connection, while the cloud-based APIs leverage the power of Google Cloud Platform’s machine learning technology to give a higher level of accuracy.
Clarifai:
Clarifai is primarily focused on computer vision applications for helping businesses solve complex tasks with the help of AI. Clarifai uses deep convolutional neural networks (CNNs), a subfield of machine learning, to understand objects in images and videos.
Their ML models are powered by the Clarifai Object Recognition Engine (CORE), which adds millions of images to its repository on a daily basis and retrains models to provide the most accurate results.
In essence, they leverage all the images coming in from different customers to make their CORE better —thus improving all models built on top of CORE.
Pre-trained models — 13 CV Models
Apparel Recognizer, Celebrity Image Recognition, Color Recognition, Demographics Detection, Face Detection, Face Embedding, Food (along with ingredient) Detection, Moderation & NSFW, Texture & Pattern Recognition, Travel Attributes (pool, garden, balcony, etc), Wedding Attributes Detection
Battle tested — Trusted by number of companies
Clarifai has a wide customer base that includes a number of large-scale companies. Their computer vision solution helps companies in better organizing a user’s content on-device, checking for unwanted nudity in user-generated content, and for diagnosing human ear problems from images.
Community & Support — Custom Community & Stackoverflow
Calrifai has an active forum where developers can post queries and issues they’re facing while working with the Clarifai API or SDKs. You can also get solid help if you search on Stack Overflow.
Support for Custom Models — Clarifai Runtime Only
Clarifai gives you the tools you need to train, search, and label your data from your images and videos. They have a UI to train and explore models along with client libraries available in a number of programming languages.
Skafos — iOS Only
Skafos is a tool for deploying machine learning models to mobile apps and managing the same models in a production environment. Built to integrate with any of the major cloud providers, users can utilize AWS, Azure, Google, IBM, or nearly any other computational environment to organize data and train models. Skafos then versions, manages, deploys, and monitors model versions running in your production application environments.
Pre trained models — None
Skafos doesn’t have any pre-trained models. However, you can use any Core ML or TF Lite model with their SDK.
Battle tested — Used by Apps in Store
Apps in production that using Skafos for ML model delivery and on-device inference include The Aeditor and Filament for Thingiverse. Both mentioned are quite happy with the performance of the solution so far.
Community & Support — Active Slack Community
Their blogs can help you in integrating Skafos with a number of existing model training environments. Skafos has a good and helpful Slack community with their team of experienced, dedicated data scientists and software engineers to get you through the development of your machine learning apps using Skafos.
Support for Custom Models — CoreML & TF Lite
Skafos supports both Core ML and TensorFlow Lite models. Once you add them to Skafos, app versions are automatically version-controlled by Skafos when saved, can be loaded into remote training environments for evaluation or retraining, and can be deployed to mobile devices for on-device inference.
Updating Custom Models — ✅ In Background right from Dashboard
Skafos allows mobile teams to update on-device AI models remotely — no need for app updates. This makes it easier for developers to ship improved models more frequently.
Something Extra —Integrations & Automated Version Controlling
Skafos allows you integrate with a number of popular data science tools and frameworks such as GCP, AWS, Azure, and IBM Watson, along with your local environment. Using their Python SDK from training environments you can save updated model versions and then add them to your Skafos models.
Skafos automates version controlling of your models as you add them to Skafos models. This allows developers to easily ship and roll back models according to their performance, tracked via the Skafos dashboard. It also allows model tagging, which can be used for the various purposes discussed above.
Numericcal — Android Only
Numericcal is more geared towards data scientists/machine learning engineers who are looking to get their work implemented on edge devices. Their platform allows developers to ship the fastest model implementation without additional development effort.
Pre trained models — None
Numericcal primarily focuses on optimizing and delivering your own models.
Support for Custom Models — Own Format
The platform provides tools to help you reach your implementation goals quickly and effortlessly.
Performance Monitoring — 📈A Step Ahead
Their performance monitoring tools measure on-device speed and power usage to help you evaluate and compare models across hardware platforms. But it doesn’t ends there—going an extra mile, it helps you pinpoint performance bottlenecks and focus your model optimization on layers that matter the most.
Updating Custom Models — Over-the-Air Deployments
Numericcal allows you to seamlessly update models from their dashboard, which means as your model matures, you can ship it right away to the desired set of devices.
Something Extra
Numericcal claims that their runtime is better than TensorFlow Lite. There’s even a small test they shared to support this claim.
I’ve tried to give you a comprehensive overview of all of these frameworks that are doing some really cool stuff for end-to-end mobile ML application development.
Let me know if you think I should add some more to this list or if I’ve missed out on some information that might be useful for you in making your decisions.
Discuss this on Hacker News & Reddit.
Comments 0 Responses