Implementing a Natural Language Classifier in iOS with Keras + Core ML

An iOS Swift, fully offline Natural Language Classifier (NLC) for implementing local in-app intent understanding

IBM Watson NLC and Conversation services (as well as many other NLU cloud platforms) provide a Swift SDK to use in custom apps to implement intent understanding from natural language utterances.

These SDKs and the corresponding NLU platforms are super powerful. They provide much more than simply intent understanding capability — they also detect entities/slots and provide tools to manage complex, long running conversation dialogs.

However, even for the most basic NLC inference, these SDKs depend on network connectivity, as the NLC model is run in the Cloud.

iOS already provides very efficient Text-To-Speech and Speech-To-Text APIs (SFSpeechRecognizer and AVSpeechSynthesizer) fully capable of working offline on iPhone/iPad devices. By using Core ML models to run NLC and NLU algorithms on the device, we can provide similar functionality without relying on cloud inference.

I thought it would be helpful to implement an open source project to implement basic intent understanding functionalities, directly importing datasets of intents and sample utterances designed with those popular NLU cloud platforms.

SwiftNLC

SwiftNLC is a Natural Language Classifier based on Core ML / TensorFlow integration, capable of running offline on iOS/watchOS/tvOS devices.

This project is available on GitHub at:

SwiftNLC is composed of different sub-projects:

  • Importer: A Swift multi platform console app to import intents and utterances from different formats (could run on macOS and Linux)
  • SampleDatasets: A folder for JSON files containing intents definitions and sample utterances
  • Embedder: A Swift macOS console app to prepare the word embedding encoding using NSLinguisticTagger. It must run on macOS and not on Linux, as Apple NSLinguisticTagger is not part of the public multi-platform Foundation Library
  • ModelNotebook: A folder containing Jupyter Notebooks for implementing the Deep Neural Network Classifier model using Keras/TensorFlow API and exporting it using the Apple CoreMLTools Python library
  • Wrapper: A Swift wrapper to the auto-generated Core ML model to simplify access to the Core ML Classifier model and prepare/encode the utterances to use for prediction
  • SwiftNLCTestClient: A simple test iOS application to play with the wrapper and the Core ML model

Detailed instructions

Importer — Swift script to import from Watson and others

This sub-project contains Swift code (to be executed on Linux or macOS environments) to import a file containing the dataset used to train the NLC model.

The idea is to provide different importers for different file formats in order to import datasets from existing NLU/NLC/Conversation platforms such as IBM Watson, AWS Alexa/Lex, Google Dialogflow, etc.

The first importer implemented is for the IBM Watson Conversation service. Watson uses a JSON “workspace” file containing intents with several sample utterances, entities (slots), also with sample utterances, as well as a tree for complex dialog management of long-running conversation. This project is solely about NLC, and it only import intents and sample utterances from this file.

Usage example:

Generated dataset.json example:

Sample Datasets

This folder contains sample datasets with intents and sample utterances from different sources:

  • Watson/WatsonConversationCarWorkspace.json in the Watson subfolder is an export of the standard sample workspace for demoing IBM Watson Conversation service on the IBM Cloud.
  • PharmacyDataset.json (no need to import)

Embedder — Word one-hot encoding implemented in Swift using NSLinguisticTagger

This sub-project contains Swift code (to be executed on a macOS or iOS environments) to import a JSON file containing the dataset to be used for training the NLC model.

This importer uses Apple Foundation NSLinguisticTagger APIs to analyze and tokenize the text in the sample utterances, creating a word embedder. In particular, it outputs a one-hot encoding for stem words, a corpus of documents, and a class of entities to train the data and prepare the TensorFlow model, as well as for inferencing the model with Core ML.

Usage example:

This command produces the following files in the current folder: bagOfWords.json, lemmatizedDataset.json and intents.json

ModelNotebook — Instructions to create TensorFlow model

This is a Python Jupyter Notebook using the Keras API and TensorFlow as a backend to create a simple, fully connected Deep Network Classifier.

If you’re new to the Keras/TensorFlow/Jupyter world, here are step-by-step instructions to create the ML model using Keras/TensorFlow and export it on Core ML using CoreMLConversionTool

  1. Download and install Anaconda Python:

2. Create the Keras, TensorFlow, Python, and Core ML environment:

This environment is created based on the environment.yml file for installing Python 2.7, TensorFlow 1.1, Keras 2.0.4, CoreMLTools 0.6.3, Pandas and other useful Python packages:

NB NLTK is only needed for the createModelWithNLTKEmbedding Notebook used on initial testing.

The final createModelWithNSLinguisticTaggerEmbedding does not use NLTK, as the word embedding is implemented in Swift on the Embedder module using the NSLinguisticTagger API.

3. Activate the environment (Mac/Linux):

4. Check that your prompt changed to:

5. Launch Jupyter Notebook:

6. Open your browser to:

To create a basic model with Keras/TensorFlow and export it with CoreMLTools, open createModelWithNSLinguisticTaggerEmbedding in your Jupyter browsing session and execute any cells in order to create, save, and export the Keras Model using Core ML exporting tools.

The basic Core ML model will be saved in the current folder.

Keras / TensorFlow model consideration

As discussed above, the Swift Embedder project is generally responsible for massaging the dataset, and responsible in particular for creating the one-hot word encoding for stem words from sample utterances and for creating a corpus of stemmed, encoded documents to be used for training the model.

For this reason, the Python code to create the model is simplified with very little pre-processing, and it quickly starts to create the TensorFlow model used for learning.

This model is a simple, fully connected network that receives as input an array of embedded 0’s and 1’s for each sample utterance. It uses ‘ReLU’ as an activator for the first and an internal hidden layer, and then at the end, as this is a multi-class classifier, it uses ‘softmax’ to emphasize the winner prediction.

The training of the model is even simpler, as per definition the intents/utterances dataset used per input is very limited, and there’s no space at all for creating validation and testing sets.

It basically trains the entire dataset with a sufficient number of epochs to obtain maximum possible accuracy.

Once the Keras/TensorFlow model is trained, this is easily exported to Core ML using the Apple CoreMLTools Python library:

Core ML Swift Wrapper and Word Embedding preparation

Once the exported Core ML model is imported in a client app to predict the intent from a new utterance, it will need to be encoded using the same one-hot embedding logic used for preparing the training dataset.

The wrapper folder contains the following Swift files to implement the one-hot embedding using the NSLinguistic tagger and to easily wrap access to the Core ML model. This simplifies the annoying creation of MLMultiArray parameters.

Lemmatizer.swift

import Foundation

class Lemmatizer {
    typealias TaggedToken = (String, String?)

    func tag(text: String, scheme: String) -> [TaggedToken] {
        let options: NSLinguisticTagger.Options = [.omitWhitespace, .omitPunctuation, .omitOther, .joinNames]
        let tagger = NSLinguisticTagger(tagSchemes: NSLinguisticTagger.availableTagSchemes(forLanguage: "en"), options: Int(options.rawValue))

        tagger.string = text
        
        var tokens: [TaggedToken] = []
        
        tagger.enumerateTags(in: NSMakeRange(0, text.count), scheme:NSLinguisticTagScheme(rawValue: scheme), options: options) { tag, tokenRange, _, _ in
            let token = (text as NSString).substring(with: tokenRange)
            tokens.append((token, tag?.rawValue))
        }
        
        
        return tokens
    }

    func lemmatize(text: String) -> [TaggedToken] {
        return tag(text: text.lowercased(), scheme: NSLinguisticTagScheme.lemma.rawValue)
    }
}

BagOfWords.swift

import Foundation

struct BagOfWords: Codable {
    let sortedArrayOfWords: [String]

    init(setOfWords: Set<String>) {
        sortedArrayOfWords = Array(setOfWords).sorted()
    }

    internal func binarySearch(_ word: String) -> Int? {
        var lowerIndex = 0
        var upperIndex = sortedArrayOfWords.count - 1

        while true {
            let currentIndex = (lowerIndex + upperIndex) / 2
            if sortedArrayOfWords[currentIndex] == word {
                return currentIndex
            } 
            else if lowerIndex > upperIndex {
                return nil
            } 
            else {
                if sortedArrayOfWords[currentIndex] > word {
                    upperIndex = currentIndex - 1
                } 
                else {
                    lowerIndex = currentIndex + 1
                }
            }
        }
    }

    func embed(arrayOfWords: [String]) -> [Int] {
        var embedding = [Int](repeating: 0, count: sortedArrayOfWords.count)

        for word in arrayOfWords {
            if let index = binarySearch(word) {
                embedding[index] = 1
            }
        }

        return embedding
    }
}

SwiftNLCModel.swift

import Foundation
import CoreML

class SwiftNLCModel {
    lazy var bagOfWords: BagOfWords = {
        return try! JSONDecoder().decode(BagOfWords.self, from: Data(contentsOf: Bundle.main.url(forResource:"bagOfWords", withExtension: "json")!))
    }()
    
    lazy var intents: [String] = {
        return try! JSONDecoder().decode(Array<String>.self, from: Data(contentsOf: Bundle.main.url(forResource:"intents", withExtension: "json")!))
    }()
    
    var lemmatizer = Lemmatizer()
    
    func predict(_ utterance: String) -> (String, Float)? {
        let lemmas = lemmatizer.lemmatize(text: utterance).compactMap { $0.1 }
        let embedding = bagOfWords.embed(arrayOfWords: lemmas)

        let model = SwiftNLC()
        
        let size = NSNumber(value: embedding.count)
        let mlMultiArrayInput = try! MLMultiArray(shape:[size], dataType:MLMultiArrayDataType.double)
        
        for i in 0..<size.intValue {
            mlMultiArrayInput[i] = NSNumber(floatLiteral: Double(embedding[i]))
        }
        
        let prediction = try! model.prediction(input: SwiftNLCInput(embeddings: mlMultiArrayInput))
        
        var pos = -1
        var max:Float = 0.0
        
        for i in 0..<prediction.entities.count {
            let p = prediction.entities[i].floatValue
            if p > max {
                max = p
                pos = i
            }
        }

        return pos >= 0 ? (intents[pos], max) : nil
    }
}

Sample iOS App

Once imported, the Swift wrappers files above, using this SwiftNLC Core ML model, will be easy to execute, as in following code:

    let model = SwiftNLCModel()

    @IBAction func go(_ sender: Any) {
        if let prediction = model.predict(commandField.text!) {
            intentLabel.text = "(prediction.0) ((String(format: "%2.1f", prediction.1 * 100))%)"
        }
        else {
            intentLabel.text = "error"
        }
    }

More details about how to use Core ML in an iOS app are provided in this tutorial:

Future

As I mentioned, this open source project doesn’t aim to replace much more evolved NLC and NLU platforms such as IBM Watson NLC, Conversation, and others.

At the moment, it only supports basic intent understanding and, importantly, it uses a very simple, fully connected neural network with a limited one-hot word encoding technique.

It works quite well and, of course, it works totally offline!

In the future, I’ll extend this project to support more complex embedding methods, such as Word2Vec. I’ll also adopt a more powerful RNN model to try to detect entities/slots.

Discuss this post on Hacker News

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *

wix banner square