Colorizing Images in an iOS App Using DeOldify and a Flask API

Computer Vision — iOS

Finding old black and white photos and imagining what people, landscapes, and objects looked like in color has probably already crossed the minds of many.

While it is possible to manually colorize black and white photos using image editing and retouching software such as Photoshop, the manipulation, which is only available to insiders and professional image editors, can be time-consuming and tedious.

But all is not lost for lay users. Thanks to artificial intelligence and machine learning, it is now possible to colorize any image in black and white with a few clicks.

Be careful, however, in performing such manipulation and bringing your old photos back to life—these online platforms require you to upload your photos. While most say they do not keep any uploaded snapshots, the user has no other choice but to trust these declarative indications.

Also, not all online image coloring services deliver the same results. The rendering of some colorized photos may be of better quality on one platform than on another and vice versa.

The algorithms used being what they are, keep in mind that the colorization of an image using artificial intelligence is based on different approaches and techniques. But the final image colors do not necessarily provide an accurate representation of the actual colors as they would have been in the photo at the time.

In this article, I’ll create an API that will process a black and white image and propose a colour version. The API is using DeOldify, which is a deep learning project created and maintained by Jason Antic and Dana Kelly. Then, I’ll create an iOS application that will consume the API.

Overview:

Build the API and run it on Colab
Create the iOS application
Witness the magic
Conclusion

I have included code in this article where it’s most instructive. Full code and data can be found on my GitHub page. Let’s get started.

Build the API and run it on Colab

DeOldify

DeOldify is an image colorizing package created by Jason Antic, and it provides access to multiple models for different uses. We’re going to use the simple image colorization model.

To understand how it works, you can read the technical details in the readme section of the repository.

ngrok

ngrok is a small, free application that allows you to, for example, share a website under development on your local machine with anyone around the world.

If you’re a web developer or an infrastructure manager, this tool is useful because it gives you the ability to expose your local web server. Indeed, ngrok will allow you to make public your work under development on your local machine. Once you launch the ngrok application, it will create a tunnel and will provide you with an address such as abcd.ngrok.com.

Flask API

Flask is a Python web application micro-framework built on the WSGI library of Werkzeug. Flask can be “micro”, but it’s ready for use in production for a variety of needs.

The “micro” in the micro-frame means that Flask aims to keep the kernel simple but expandable. Flask won’t make many decisions for you, like which database to use, but the decisions made are easy to change. Everything is yours, so Flask can be everything you need and nothing else.

I prefer to use a library called Flask-RESTful made by Twilio that encourages best practices when it comes to APIs.

Create a class DeOldify
Create a method post
Parse the request (from the iOS application) and extract the base64 image string
Decode the base64 string and save the image in the directory using a native base64 Python module.
Perform the colorization and save the output image
Encode the image to a base64 format
Send a dictionary with final base64 image string

When you start the API, you will get the ngrok link, which is the API entry point

Build the iOS Application

Create a new “App” and make sure you choose Storyboard as User Interface.

Now we have our project ready to go. I don’t like using storyboards myself, so the app in this tutorial is built programmatically, which means no buttons or switches to toggle — just pure code.

To follow this method, you’ll have to delete the main.storyboard file and set your SceneDelegate.swift file (Xcode 11 only).

With Xcode 11 and 12, you’ll have to change the Info.plist file like so:

You need to delete the “Storyboard Name” in the file, and that’s about it.

Change the SceneDelegate with the following code:

func scene(_ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions) {
    guard let windowScene = (scene as? UIWindowScene) else { return }
        
    window = UIWindow(frame: windowScene.coordinateSpace.bounds)
    window?.windowScene = windowScene
    window?.rootViewController = ViewController()
    window?.makeKeyAndVisible()
}

Create View Controllers

We need two ViewControllers:

ViewController():

This is where we’ll set our application entry point and set the buttons to either take a picture or import a picture from the photo library.

ColorizeViewController():

This controller will be used to select the image in order to send it to the API, as well as receive the API callback.

Setup ViewController():

Logo Image at the top of the view

Instantiate a UIImageView object for the logo image using the “Image Literal” function
Enable auto layout

let logo: UIImageView = {
    let image = UIImageView(image: #imageLiteral(resourceName: "default").resized(newSize: CGSize(width: screenWidth - 20, height: screenWidth - 20)))
    image.translatesAutoresizingMaskIntoConstraints = false
   return image
}()

Buttons

The ViewController() has two buttons—one for the “Camera” and the other one for an image “Upload” feature. I also created a custom button called CustomButton() to increase code reusability (also available in the GitHub repository).

Only the button title and target change, the rest is identical:

Instantiate a CustomButton() object
Enable auto layout
Set the title string
Add an icon
Add a target that will lead to ColorizedViewController()

lazy var openCameraBtn : CustomButton = {
   let btn = CustomButton()
    btn.translatesAutoresizingMaskIntoConstraints = false
    btn.setTitle("Camera", for: .normal)
    let icon = UIImage(named: "camera")?.resized(newSize: CGSize(width: 45, height: 45))
    let tintedImage = icon?.withRenderingMode(.alwaysTemplate)
    btn.setImage(tintedImage, for: .normal)
    btn.tintColor = #colorLiteral(red: 0.892498076, green: 0.5087850094, blue: 0.9061965346, alpha: 1)
    btn.addTarget(self, action: #selector(buttonToOpenCamera(_:)), for: .touchUpInside)
    return btn
}()

lazy var openToUploadBtn : CustomButton = {
   let btn = CustomButton()
    btn.addTarget(self, action: #selector(buttonToUpload(_:)), for: .touchUpInside)
    btn.translatesAutoresizingMaskIntoConstraints = false
    return btn
}()

Set the layout and add all the elements to the main view

Add all the elements to the ViewController’s subview
Set up constraints for each element

fileprivate func addButtonsToSubview() {
    view.addSubview(logo)
    view.addSubview(openCameraBtn)
    view.addSubview(openToUploadBtn)
}
fileprivate func setupView() {
    
    logo.centerXAnchor.constraint(equalTo: self.view.centerXAnchor).isActive = true
    logo.topAnchor.constraint(equalTo: self.view.safeAreaLayoutGuide.topAnchor, constant: 20).isActive = true
    
    openCameraBtn.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
    openCameraBtn.widthAnchor.constraint(equalToConstant: view.frame.width - 40).isActive = true
    openCameraBtn.heightAnchor.constraint(equalToConstant: 60).isActive = true
    openCameraBtn.bottomAnchor.constraint(equalTo: openToUploadBtn.topAnchor, constant: -40).isActive = true
    
    openToUploadBtn.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
    openToUploadBtn.widthAnchor.constraint(equalToConstant: view.frame.width - 40).isActive = true
    openToUploadBtn.heightAnchor.constraint(equalToConstant: 60).isActive = true
    openToUploadBtn.bottomAnchor.constraint(equalTo: view.bottomAnchor, constant: -120).isActive = true
    
}

Setup ColorizedViewController():

Output Image

Instantiate an empty UIImageView object
Enable auto layout
Select the content mode — I chose .scaleAspectFit
Enable masksToBounds in order to clip any layer bit outside the view boundaries

lazy var inputImage: UIImageView = {
    let image = UIImageView()
    image.translatesAutoresizingMaskIntoConstraints = false
    image.contentMode = .scaleAspectFit
    image.layer.masksToBounds = true
    image.clipsToBounds = false
    return image
}()

Save the colorized image in the photo library

Instantiate a CustomButton() object
Enable auto layout
Set the title string
Add an icon using SF Symbols
Add a target that will trigger a function to save the image in the photo library

lazy var saveImage: CustomButton = {
    let button = CustomButton()
    button.translatesAutoresizingMaskIntoConstraints = false
    button.addTarget(self, action: #selector(buttonToSaveImage(_:)), for: .touchUpInside)
    button.setTitle("Download", for: .normal)
    let icon = UIImage(systemName: "square.and.arrow.down")?.resized(newSize: CGSize(width: 35, height: 35))
    let tintedImage = icon?.withRenderingMode(.alwaysTemplate)
    button.setImage(tintedImage, for: .normal)
    return button
}()

Dismiss button

It’s the same logic as the other buttons—you just need to change the target function to dismiss the ColorizedViewController.

Setup the layout

func addSubviews() {
    view.addSubview(inputImage)
    view.addSubview(saveImage)
    view.addSubview(dissmissButton)
}

func setupLayout() {
    inputImage.topAnchor.constraint(equalTo: self.view.safeAreaLayoutGuide.topAnchor).isActive = true
    inputImage.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
    inputImage.widthAnchor.constraint(equalToConstant: view.frame.width).isActive = true
    inputImage.heightAnchor.constraint(equalToConstant: (inputImage.image?.size.height)!*view.frame.width/(inputImage.image?.size.width)!).isActive = true
    inputImage.leftAnchor.constraint(equalTo: view.leftAnchor).isActive = true
    inputImage.rightAnchor.constraint(equalTo: view.rightAnchor).isActive = true
    
    saveImage.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
    saveImage.heightAnchor.constraint(equalToConstant: 60).isActive = true
    saveImage.widthAnchor.constraint(equalToConstant: view.frame.width - 40).isActive = true
    saveImage.bottomAnchor.constraint(equalTo: dissmissButton.topAnchor, constant: -40).isActive = true
    
    dissmissButton.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
    dissmissButton.heightAnchor.constraint(equalToConstant: 60).isActive = true
    dissmissButton.widthAnchor.constraint(equalToConstant: view.frame.width - 40).isActive = true
    dissmissButton.bottomAnchor.constraint(equalTo: view.bottomAnchor, constant: -100).isActive = true
}

Handling the API callbacks

The API expects a dictionary of type [String: String]—the key being “image”, and the value is the image in the format of a base64String (which is a string also).

I’m using Alamofire, which is a widely-used Swift package for handling HTTP networking with Swift. Install the package using your preferred method (I used pod).

Convert the UIImage to a base64 encoding with no compression ratio—we keep the original image as is.
Create the parameter that will be used to send the POST request value to be encoded into the URLRequest.
Perform the request using the Alamofire request method. Pass the API entry point, the type of method (POST in our case), and the parameters.
Handle the API response result. If successful, we’ll parse the API response as a JSON object and convert the image from a base64 format to a UIImage object.
Update the UIImageView with the colorized image:

func colorizeImages() {
    let imageDataBase64 = inputImage.image!.jpegData(compressionQuality: 1)!.base64EncodedString(options: .lineLength64Characters)
    let parameters: Parameters = ["image": imageDataBase64]
    
    AF.request(URL.init(string: self.apiEntryPoint)!, method: .post, parameters: parameters, encoding: JSONEncoding.default, headers: .none).responseJSON { (response) in
        
    switch response.result {
        case .success(let value):
                if let JSON = value as? [String: Any] {
                    let base64StringOutput = JSON["output_image"] as! String
                    let newImageData = Data(base64Encoded: base64StringOutput)
                    if let newImageData = newImageData {
                       let outputImage = UIImage(data: newImageData)
                        let finalOutputImage = outputImage
                        self.inputImage.image = finalOutputImage
                        self.colorizedImage = finalOutputImage
                    }
                }
            break
        case .failure(let error):
            print(error)
            break
        }
    }
}

Witness the magic

Conclusion

The results are great when the original images are in a good state. But sometimes the model proposed by DeOldify has a tendency to make everything purple.

You can also use a different model to turn black and white videos into a colorized version. There is the possibility to extend the API to accept videos and return a colorized version of the original one.

Thank you for reading this article. If you have any questions, don’t hesitate to send me an email at [email protected].