Organizing mobile machine learning projects with the Fritz CLI

When the dust settled on a recent mobile machine learning project, we had accumulated 392 different model checkpoints. With numerous architectures to test, dozens of hyperparameters to sweep, and multiple on-device formats to support, models piled up quickly.

Staying organized and creating efficient workflows were the keys to success. We knew we needed to make this process easier—so we developed a set of command line tools to help.

The Fritz CLI is a suite of utilities designed to improve mobile machine learning workflows from any terminal or Jupyter Notebook. Here are a few specific capabilities the CLI offers:

• Easily convert and push models from your training scripts directly to mobile devices for testing.
• Instantly deploy new versions of a model to your mobile app without ever leaving your Python workflow.
• Integrate with popular training framework like Keras so the latest version of your model is always available in your app to test instantly.
• Store metadata with each model so you never lose track of hyperparameters or experiments.

To show you how it works, let’s train some models for use in a mobile app and use Fritz AI to keep track of everything along the way. We’ll cover:

1. Installing and configuring the Fritz CLI and Python library.
2. Training a simple Keras model to recognize MNIST digits and converting that model to mobile formats like Core ML and TensorFlow Lite.
3. Uploading individual model checkpoints to Fritz AI using the CLI.
4. Automatically creating and uploading new versions of models using the Fritz Keras callback.
5. Using the Fritz CLI to view new versions of our model and deploy a new version over-the-air to users.

If you want to follow along with an interactive version of this tutorial, check out our runnable Colab Notebook.

Let’s dive in!

Installation and Setup

The Fritz CLI is installed via pip :

Once installed, you’ll need to configure the CLI to connect to your account. If you haven’t already, you’ll need to sign up for a free Fritz AI account.

To configure the CLI, login to your account and select Training from the menu on the left. In the section labeled Python Library Config you’ll find a command that’s been pre-populated with your API and Project ID. Copy this command and run it in your terminal.

The Fritz CLI will now be configured for this account and project.

Viewing all projects

To verify everything is working, let’s list all available Fritz AI projects. In Fritz AI, you can organize your applications and models using projects. A project can contain multiple apps and models. All models within a project are available for deployment to applications in the same project. By default, Fritz AI creates a project when you create an account, but you may create others later on. You can view all of your projects by running

Training a model

Now it’s time to train a model. To keep things simple for this tutorial, we’ll be using Keras to train a very small neural network on the MNIST dataset. We’ll also convert this Keras model into Core ML and TensorFlow Lite formats for on-device inference.

import keras
from keras.datasets import mnist
keras.backend.clear_session()

(x_train, y_train), (x_test, y_test) = mnist.load_data()


def build_model():
    input = keras.layers.Input((28, 28, 1))
    out = keras.layers.Conv2D(16, 3, strides=2, activation='relu')(input)
    out = keras.layers.Conv2D(32, 3, strides=2, activation='relu')(out)
    out = keras.layers.Flatten()(out)
    out = keras.layers.Dense(10)(out)
    out = keras.layers.Activation('softmax')(out)

    return keras.models.Model(input, out)

model = build_model()

model.compile(
    keras.optimizers.Adam(lr=0.001),
    loss=keras.losses.sparse_categorical_crossentropy,
    metrics=[keras.metrics.sparse_categorical_accuracy]
)

batch_size = 32
model.fit(
    x_train[:, :, :, None],
    y_train[:, None],
    epochs=5,
    batch_size=batch_size,
    validation_split=0.05
)

The code above loads the MNIST dataset, creates a small neural network using Keras, and trains the model for a few epochs.

We also need to convert our Keras model to mobile-friendly formats, which is shown in the code snippet below.

# Convert to mobile formats
import coremltools
import tensorflow as tf
import tempfile

def convert_to_coreml(model):
    return coremltools.converters.keras.convert(
        model,
        input_names=['input'],
        output_names=['digit']
    )

def convert_to_tflite(model):
    # save the model to a temp file so we can
    # convert it.
    keras_file = tempfile.mktemp()
    model.save(keras_file, include_optimizer=False)
    converter = tf.lite.TFLiteConverter.from_keras_model_file(keras_file)
    return converter.convert()

mlmodel = convert_to_coreml(model)
tflite = convert_to_tflite(model)

model_name = "mnist_cnn_lr001_batchsize128"

# Keras
model.save(
    model_name + ".h5",
    include_optimizer="False"
)

# Core ML
mlmodel.save(model_name + ".mlmodel")


# TensorFlow Lite
with open(model_name + ".tflite", 'wb') as wfid:
    wfid.write(tflite)

Saving a model checkpoint and uploading to Fritz AI

​Now that we’ve trained a model, let’s get it into Fritz AI for safekeeping. One way to do this is to save the checkpoints to our local drive, then upload them via the Fritz CLI using the fritz model upload <path> command. We’ll add a few hyperparameters to the filename to help us keep track of things.

In a terminal, we run:

Upon successful upload, the Fritz CLI will display model information such as name, format, and size along with a Model ID and Version ID you can use to reference the model in the future.

Listing all models uploaded to Fritz AI

We can see all the models we’ve uploaded to Fritz AI by running:

All three of our models are now safely stored in Fritz AI, where we can share them with the rest of the team and deploy them to Android or iOS apps.

Interested in mobile development powered by machine learning? Sign up for the Heartbeat Newsletter and join the largest community focused on mobile ML.

Uploading new model versions to Fritz AI automatically during training

Most machine learning projects aren’t finished after just one model iteration. Let’s keep training our model and experimenting with different hyperparameters. For example, let’s try increasing the batch size of 256 to improve convergence.

Uploading individual checkpoints via the CLI is handy, but we’re going to keep iterating on these models, and it would great if new versions were uploaded automatically during training along with configurations and metadata. Fritz AI can do that as well.

Here, we’ll make use of the Fritz Python Library to create a special Keras callback that will not only upload our Keras checkpoints, but will also convert models to Core ML and TensorFlow lite for mobile usage. Newly-trained checkpoints will be uploaded to Fritz AI as new versions of the models we uploaded by hand, so that all of our experiments are grouped nicely together.

Note that all versions of a model must have the same format, inputs, and outputs. If you change the model inputs or outputs, you’ll need to create an entirely new model. Fritz AI makes it possible to swap model versions on devices in real-time, and if inputs or outputs change, your application code will need to be updated accordingly.

import fritz
import fritz.train

# Fritz needs to be configured first. Calling the fritz.Configure() method will
# read the credentials we setup for the CLI earlier.
fritz.configure()

# Create the callback

# Start by defining a training configuration and storing it as metadata
metadata = {
    "learning_rate": 0.001,
    "batch_size": 256
}

# Specify model ids so new versions of our model get associated with the
# original checkpoing uploads from above.

model_ids = {
    fritz.frameworks.KERAS: "<YOUR KERAS MODEL ID>",
    fritz.frameworks.CORE_ML: "<YOUR CORE ML MODEL ID>",
    fritz.frameworks.TENSORFLOW_LITE: "<YOUR TENSORFLOW LITE MODEL ID>"
}

# Specify our conversion functions so keras checkpoints
# get turned into Core ML and TFLite models each time
converters = {
    fritz.frameworks.CORE_ML: convert_to_coreml,
    fritz.frameworks.TENSORFLOW_LITE: convert_to_tflite
}

fritz_callback = fritz.train.FritzSnapshotCallback(
    model_ids_by_framework=model_ids,
    converters_by_framework=converters,
    output_file_name="mnist.h5",
    metadata=metadata,
    period=10 # Save after every 10 epochs and at the end of training
)

When constructing the Fritz Keras callback, we’ve:

1. Told Fritz AI to upload checkpoints as new versions of previously uploaded models.
2. Provided a conversion function for each mobile format we wish to support.
3. Attached metadata related to the training configuration to each model.

Now lets train a new model.

keras.backend.clear_session()
# Retrain the model with our new configuration and callback
model = build_model()

model.compile(
    keras.optimizers.Adam(lr=metadata['learning_rate']),
    loss=keras.losses.sparse_categorical_crossentropy,
    metrics=[keras.metrics.sparse_categorical_accuracy]
)

model.fit(
    x_train[:, :, :, None],
    y_train[:, None],
    epochs=5,
    batch_size=metadata['batch_size'],
    validation_split=0.05,
    callbacks=[fritz_callback]
)

Inspecting model versions with the Fritz CLI

The Keras callback has automatically uploaded three new models to Fritz AI: Keras, Core ML, and TensorFlow Lite checkpoints. Let’s use the CLI to verify that everything ended up in the right place.

We use the same Model ID from the callback function to list all versions of a model:

We can see that there are currently two versions of the model in Fritz AI. The original we uploaded manually, and the one uploaded by our callback. If we get the details of the most recent Version ID, we’ll see that additional metadata has been added.

Fritz AI has automatically captured the loss and accuracy values of our models at the times they were checkpointed as well as the metadata on our training configuration.

Deploying a new model with the Fritz CLI

Now that we’ve trained a new version, let’s push it out to all of our users. As noted by the command above, our new model is version number 2. We can set this to the active version shipped to users by running:

Summary

Lets recap what we did:

1. Installed and configured the Fritz CLI and Python library.
2. Trained the first version of a Keras model on MNIST.
3. Converted the trained Keras model to Core ML and TensorFlow Lite models.
4. Uploaded individual checkpoints to Fritz AI using the CLI.
5. Used a Fritz Keras callback to automatically load newly-trained versions of models to Fritz AI.
6. Used the Fritz CLI to view new versions of our model and deploy a new version over-the-air to users.

As you’ve seen, the Fritz CLI can be used both in a terminal or Jupyter notebook so you can organize, search, and manage models without needing to jump between applications and tools. Iterate faster and never lose track of an experiment again with Fritz AI.

If you want use an interactive version of this tutorial, check out our runnable Colab Notebook.