Real-Time Style Transfer for Android — Transform your photos and videos into masterpieces

Have you tried photo editing apps like Prisma where you can apply artistic styles to any of your images? Creating filters or features like these takes months of effort — development teams tweak and optimize style transfer models to create unique experiences that millions of users love. Now, with Fritz, you’ll be able to add the same features to your app in minutes.

Behind the scenes, Style Transfer uses a neural network that takes the style of one image and applies it to the content of another. For example, take a picture of Government Center in Boston and apply the style of Vincent van Gogh’s Starry Night.

Original Image (left) | Starry Night Style applied (middle) | Styled Image Result (right)

Unlike other solutions, Fritz has optimized models to work in real-time on many devices. In order to build out Style Transfer, we’ve trained 11 different styles from Claude Monet’s The Poppy Field to Picasso’s Les Femmes d’Alger. We created a simple-to-use API that allows any developer to integrate style transfer into their apps. All the processing is done on-device and in real-time using mobile machine learning. In this tutorial, we’ll walk through how to integrate this feature into your own apps.

Setting up a Fritz Account (5 minutes):

To get started, first make sure that you’ve signed up for a Fritz account and created a new project. Projects help you stay organized by grouping multiple flavors of the same app together. For example, you might have an iOS and Android version of the same application or different builds for development, staging, and production. All of these variants should be contained in the same Fritz project.

Next, register an Android app to the project. If you don’t have an app yet or want to get started with style transfer quickly, you can use our camera app on GitHub. Whatever you choose, just make sure the package name (e.g ai.fritz.camera-app) that you’ve provided matches the one in your build.

In your app/build.gradle. Notice the applicationId.

Android app to create in the webapp. The package ID must match the applicationId shown above for your application.

Configure the SDK (10 minutes):

Now that we’ve successfully registered an app, you’ll need to add the SDK to your app. After you’ve set up the SDK, you’ll be able to use any Fritz feature (image labeling, object detection, style transfer).

First, make sure to add the Fritz repository. This will allow you to download the necessary dependencies:

Edit your project level **build.gradle** file to include the url to the **fritz-repository**

Add the Fritz dependencies to your app/build.gradle file and re-sync gradle.

dependencies {
    implementation "ai.fritz:core:1.2.1"
    implementation "ai.fritz:vision-style-model:1.2.1"
}

In your app/AndroidManifest.xml, add internet permissions if you don’t already have them enabled (don’t worry, the model runs directly on the user’s device) and add the API key for your app. (To find your API key in the Fritz webapp, go to Project Settings > App you created > Actions (click on the three vertical dots) > Show API Key).

Finally, configure the Fritz SDK in your application or MainActivity.java onCreate method.

Using Style Transfer (10 minutes)

With the Style Transfer API, you’ll be able to transform your photos. There are 4 simple steps to follow in your camera code (in our sample app, see MainActivity.java to follow along):

First, get a FritzVisionStylePredictor by specifying the output style you’d like to achieve (you can see the different options in the official documentation). In this tutorial, you’ll use Starry Night.

2. Next, convert the image you’d like to transform into a FritzVisionImage object. You’ll pass this into the Style Predictor.

If you’re reading a Bitmap, you can create a new FritzVisionImage object like so:

FritzVisionImage fritzImage = FritzVisionImage.fromBitmap(bitmap)

If you’re reading the input directly from the camera (media.Image object), you’ll want to first get the image rotation which depends on the device rotation and the camera orientation sensor. The cameraId identifies the active camera being used on the device (front, back, etc).

After you’ve determined the image rotation, create a FritzVisionImage object with the Image and rotation value.

3. Run prediction on the input image to get a new FritzVisionImage object with the style applied.

Runs inference in the background. After running prediction, styledImage will have the same size as the original input (previewWidth x previewHeight). Since we want the styledImage to take up the entire viewport, we can use the **FritzVisionImage** **scale** method to fill up the screen.

4. Finally, display the new image directly on a canvas or access the Bitmap directly if you intend to save it.

Here’s the result on a live video:

For the full solution, take a look at StyleActivity in the Heartbeat GitHub sample. Also, for additional options that you can set, check out the full Android documentation.

Our goal is to make it easier for developers to get into mobile machine learning. With the Fritz SDK and the Style Transfer API, you can now take any image and transform it into a work of art. We can’t wait to see what you’ll create! Try it out and let us know what you think.

If you enjoyed this post and want more skimmable bytes of mobile machine learning delivered directly to your inbox, sign up for Heartbeat’s newsletter.