Video Star: Creating custom artistic styles for music videos with mobile machine learning

Frontier Design’s Video Star is a freemium video editing and creation app with a rich, deep toolkit that allows users to become the stars of their own music videos. From transitions, to blur and color effects, to adding 3D objects and text, and beyond, Video Star has a seemingly endless trove of fun capabilities to explore.

And when powerful tools like Video Star combine with the boom of creative platforms like Instagram and TikTok, it quickly becomes clear that there’s never been a better time to be a content creator.

With 4 million monthly active users and a flourishing community of creators, Video Star represents one major player in a secondary market of creativity tools that empower users to produce immersive, stylistic, and unique visual content that’s instantly shareable on popular social channels.

But co-founder Charlie Hitchcock knows that to keep users engaged and coming back, he and his team need to regularly develop new features and effects.

Adding ML-powered video effects: Style Transfer

One particular visual effect Charlie and his team considered adding for a long time was style transfer, a machine learning-powered image manipulation technique that overlays the artistic style of one image onto another image or video clip.

But there were considerable roadblocks in adding this kind of feature, including cost and consistency of performance — two common hurdles when implementing ML models on mobile. To overcome these hurdles, Charlie and his team decided to seek a solution in which the style transfer effects would be embedded on-device.

“While images can be uploaded to a server for processing, with the results then downloaded, we believed that good, small models could be effective and fast on current mobile phones,” Charlie noted. “It not only would help us avoid the ongoing server costs and maintenance, but it also let our users implement style transfer effects even when not connected to the internet.”

Even with these potential savings and increased accessibility, there remained the problem of performance hiccups when working with video.

“Style transfer has been around for several years but has mostly been applied to still images,” Charlie continued. “But naive use of standard style transfer techniques results in very noisy, busy, and unsatisfying results when applied to the frames of a video clip independently.”

Creating and managing new styles with Fritz

Facing these dilemmas, Charlie and his team turned to Fritz to seek a lasting and reproducible solution that would guarantee a seamless user experience by allowing more efficient and higher-performing style transfer models to run on-device.

To do this, the Fritz engineering team worked hand-in-hand with Charlie and Video Star to implement stable style transfer — a special model training method that reduces flicker and noise in video.

“Fritz was able to adapt previous [style transfer] work to the phone, producing stable style transfer for videos,” Charlie said. “The results are fast, and the stability makes the clips a joy to view.”

To capitalize on this unique capability, however, Charlie and his team needed a way to easily create new style transfer effects and manage the models that power them.

With custom model training notebooks developed by Fritz, the Video Star team has been able to integrate the creation of new styles into their workflow. And with the ability to add tags and metadata to each custom style developed with Fritz, the team can easily manage their distribution.

This easy customization has led to a growing catalog of effects and the ability to monetize that development — 4 styles are available in the app’s free version, while more than 40 styles (and counting) exist in the app’s Art Studio, which has monthly, quarterly, and yearly subscription options.

Video Star has proven that on-device machine learning has the potential to differentiate mobile creativity tools in a rapidly-expanding marketplace. But to truly maximize the potential of this technology, developers will need intuitive tools and advanced support in order to continually deliver seamless, delightful experiences to their users.

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *

wix banner square