Federated learning, transfer learning, and model personalization
For a healthcare research project I’m working on, I’m investigating for a federated learning platform that supports mobile and wearable platforms—in particular on the Apple ecosystem.
Federated learning represents a tremendous opportunity for the adoption of machine learning in many use cases, and especially where efficiency and privacy concerns require us to distribute the training process, instead of centrally collecting data on the cloud and applying traditional ML pipelines.
There are some fantastic toolkits already available—for example, TF-Federated is emerging to address this scenario—but all of these existing solutions are based on Python, and there still remains the issue of delivering a real, workable integrations with mobile and wearable devices.
I love Python, don’t get me wrong. And of course, Python is great for cloud AI, but it isn’t an option on these devices in scenarios where distributed learning is needed—like mine.
Without going too deep into specific detail about how federated learning works—in terms of distributing training and validation across devices and centrally converging on model weights instead of on data—I started simplifying (a lot) the scenario, starting from a more generic transfer learning need.
I simply started with the trivial need to pre-train a model centrally, with some shared data, and let devices continue the training with local, private data, directly on-device. Apple calls this model personalization, a new and amazing feature of the Core ML 3 framework, available on iOS 13.
The importance of having a single language to manage training pipelines both in the cloud and on-device is really important in terms of sharing a common codebase. This is true not just for the model definition itself, but also for all the pre- and post-processing data transformation and featurization code.
In this case, Swift brings with it tremendous opportunity for sharing the most code between the cloud and devices. Further, it aids in simulating some of the basic requirements for this federated learning platform, specializing cloud training using the Swift for TensorFlow toolchain, and optimizing on-device training using Core ML.
Hacking a trivial model with distributed training in Swift
In my healthcare scenario, I’m dealing with an auto-encoder model architecture to manage anomaly detection on biometric data. To simplify the technical investigation, I started with a very trivial neural network with a single layer and no activation function. Basically, a super simple linear regression trained with some noised linear data on both the Swift for TensorFlow toolchain and with Core ML on the regular Apple Swift 5.1 toolchain.
In my test, I started defining and training the model using the Swift for TensorFlow (S4TF) layer API, and then I extracted the weights and bias of the single layer of my model. Then, using a Core ML protobuf data structure, I recreated and exported the model in the Core ML format with the weights, bias, and all parameters needed for Core ML re-training / personalization.
S4TF and Core ML background
It’s out of the scope of this article to describe the status of the S4TF project and the specification of the Apple Core ML API and file format. There are already lots of great tutorials available, and also, the official documentation on both Apple’s and TensorFlow’s developer and GitHub web sites is good are good resources.
But just a couple of caveats here IMHO on the most important things to understand in this context.
First of all, S4TF is still under development and in an experimentation phase. It could be used very successfully on Linux and macOS to train models using the new, amazing Swift auto differentiation and TF backend operators, but it misses some very important functionalities at the moment; like, for example, the capability to export a model in any private or common format.
Second, it’s important to remember here that Core ML is both an abstraction file format—based on Google Protocol Buffer (protobuf) binary serialization—as well as a powerful ML runtime. It’s available on all Apple OS platforms and optimized for CPUs, GPUs, and NPUs where available for inferencing and now for re-training (or personalizing) partial or entire models.
Of course, there are some technical limitations on what Core ML supports, especially when it comes to training, considering the limits both in terms of performance and power consumption of mobile devices versus a cloud platform. But wow, this runtime is really powerful and it could be successfully used in lots of scenarios.
Core ML and TensorFlow are already very good friends, and there are different Python packages like the Apple’s CoreMLTools or Google’s TFCoreML that could be used to export (in Python) regular TF 1.x or 2.x models.
Once you’ve installed the protoc compiler from protocol buffers (Google Developers website or GitHub), I strongly suggest examining the Core ML file format by reverse engineering from an exported Core ML model using the following command:
This generated text file will reference all the nested Core ML protobuf data structures used for describing the specific models, referencing their numerical ids and the correlated values.
The corresponding definitions of the Core ML protobuf data structures (i.e. messages or enum) can be found on the official Apple CoreMLTools project on GitHub, under the mlmodel/format folder:
First step: Generate Swift Core ML classes and data structures
Now that we have some confidence with S4TF and the Core ML exporting process and file format, the first thing to do to generate a Core ML model directly from Swift is to install the official Apple Swift Protobuf plugin.
By doing this, we can generate Swift Core ML protobuf classes and data structures from the original .proto files from the above Apple CoreMLTools GitHub project. We can then save these generated Swift files in a folder that we’ll use later.
The Apple Swift Protobuf plugin’s install instructions ca be found on the official Swift Protobuf GitHub repo:
Once the plugin is installed, we can use the following command to compile the Core ML protobuf files (.proto) in Swift:
Second Step: Generate and train S4TF model, get weights and bias for layers, and export to Core ML format
We do this using the previously-generated Swift Core ML protobuf data structures, specifying the trainable parameters.
The following Swift Jupyter Notebook use Swift for TensorFlow to train a very trivial linear regression model and exports the model with all required trainable parameters using the previously-generated Swift Core ML protobuf data structures.
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Install Swift Protocol Buffer library"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Installing packages:n",
"t.package(url: "https://github.com/apple/swift-protobuf.git", from: "1.6.0")n",
"ttSwiftProtobufn",
"With SwiftPM flags: ['-c', 'release']n",
"Working in: /tmp/tmprjkpbl0q/swift-installn",
"Fetching https://github.com/apple/swift-protobuf.gitn",
"Cloning https://github.com/apple/swift-protobuf.gitn",
"Resolving https://github.com/apple/swift-protobuf.git at 1.7.0n",
"[1/2] Compiling SwiftProtobuf AnyMessageStorage.swiftn",
"[2/3] Compiling jupyterInstalledPackages jupyterInstalledPackages.swiftn",
"[3/3] Linking libjupyterInstalledPackages.son",
"Initializing Swift...n",
"Installation complete!n"
]
}
],
"source": [
"%install-swiftpm-flags -c releasen",
"%install '.package(url: "https://github.com/apple/swift-protobuf.git", from: "1.6.0")' SwiftProtobufn",
"import SwiftProtobuf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Include Python, TensorFlow and enable Matplotlib plot"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('inline', 'module://ipykernel.pylab.backend_inline')n"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%include "EnableIPythonDisplay.swift"n",
"IPythonDisplay.shell.enable_matplotlib("inline")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import Pythonn",
"import TensorFlow"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"let plt = Python.import("matplotlib.pyplot")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Generate noised linear data"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"let SAMPLE_SIZE = 100n",
"n",
"let a: Float = 2.0n",
"let b: Float = 1.5n",
"let x = Tensor<Float>(rangeFrom: 0, to: 1, stride: 1.0 / Float(SAMPLE_SIZE))n",
"let noise = (Tensor<Float>(randomNormal: [SAMPLE_SIZE]) - 0.5) * 0.1n",
"let y = (a * x + b) + noise"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yc1ZX4/88dadS7Rs2SLVljWy64nyBa2EQGDk0ACISGELMWYkAIYcNjdbEn7ZTdl6293k00wiSGBAAYDSWAJCZAAAUwRLpJsy7hrZKtYnddS7ptzvH8/MWGVkjW2VkXTer5deGWmeeeY+kTlzdZ5zz1Vaa4QQQkx/pqkegBBCiPEhAV0IIWYInCehCCDFDSEAXQogZQgK6EELMEKFT9cYWi0Xn5ORM1dsLIcS0VFJSYtdap/h7bsoCek5ODsXFxVP1n9kIIMS0ppSpHe05SLkIIMUNIQBdCiBlCAroQQswQEtCFEGKGkIAuhBAzhAR0IYS4SNt32Siy2Yf8nrMhmZ/su26SOY8yArpSKUErtVUodVEodVkr9wM8xdymlmpRSBzxfX5uY4QohRPBZkRXP1p37fUG9nyGZn6879rMiKn9RxBFKH3g9s1Fp3KaXMwPtKqde01ruHHfe81nrr+A9RCCGCW6HVwkO35nP3k8V8nqTCH5/ZVs+32fAqtlkkdx5gzdG3o8nxr9nxJE3UhhBgkJETRPeDi5+/YuGPdvEkP5hBgDl0pFaKUnOgA0Am9orff4OewLSqkypdTvlFJzRznPPUqpYqVUcVNT00UMWwghgsufPqoHYHlmPE/vqRqRU58MnAQV0rbVLa70KyALWKqUuGXbIH4AcrfUK4A3gyVHO86jWukBrXZCS4rcVgRBCTDtFNjvP7q0CwGRSnbLs9f0hOfbKcV5WL1roNeBv41LCfN2ut+z3f/gpYMz7DE0KI4FdW0052chQAR+s6KMhOYtvt+ZTVntE/qOAKpcklRSiV4HkcCnwSODTsmY9C3nwWOjucghRAimN17ZS51bX1YYsIZcLo50dBJodXClg1Wn3zGTUdoYyAw9A3hbKVUG7MPIof9RKfVDpdRnPcc86ClpPAg8CNw1biMUQoggV9feR2e/ky+szgTwnOzOfjNLGMcsWtdZlQL6fn//ToMffBr49bqMSQohp5Hh9JwAfX5LG88XVlNW0cfu6eUOOKbRa2HZ7nPnc/VcytBXP5vwO1417aKCtFhRDiIh1vMAJ6XlosyzPjR82dm5Siu9/FYx+cnpDSRgnoQghxkU7Und5IeF0F8lJmVWQkcb+ikz+EackxL9wD3PV2CScGWDbkTUtooAV0IIS7SsfpO8tJjAVieFY/LrTlSn1+G7Eep2a776xD5aexx8+fIcEqLCJqS0UQK6EEJcBKfLTXlTly+ge29yllW3+W6Efu/3H7G/uo1rnl6Xxf/trWZEV78upj2dpowR0IYQYRSClhpUtPQw43SxKMwJ6elwEKbHhlJ1pp9Bq4f6rrDyzp4rcnlGj2nW4dciN0eGnjxZKALoQQowik1NBb4bLYM0N/5N0KshIiKatpp7K5m21vlxMTHkJFU/eE93gJnpNuiEELMSt60yP3PlHJ1Xiq7TjSNKDU8Xt+JUrAgNQYwPgR++uZJeh0uvvzEPvocLvodbj6fn8nTne6pYb02esKAuM3QhhBhkeJolLS6CAaeL/9t/hs+tnDMiGB+v7yQnOZoIcwhgfAh8feMCACqauul3nuPnO9Yv5yS2rJrzHiwR0IYQYZHCaZd/pFq7/2Xv0DLgB+F1pjS+H7g3KJxo6yUuLHZJb37Qum9hwnIwFyY/4c7r7CyJNPxI3QwSSgCyHEIN6ge++OEv5q+4f0Odx8qTAHgJvyM9m6cz8hJti6cz/vHG/kndHM3kWEhQ3Lrh+vaMYeaeHDjAnadsA+ZkY/3jdDBJKALIcQwhVYL85Ki0MD1y9P5/g1LiQ0PRQPbnbs/H5Tb+98Fn9+PW8MaRBl9u3XvjdNvt+XzjmrxJbaUrAV0IIYYpstk5Vt9JZkIEH1a08GFFM7kpn0VQ0dftm2IVWC2vnJwFw46qzufWymvYRpYmT1UpXqlyEEGKQIpudrc/sJ0QprlmWzieXprF1536WnpMdyyt495Lj3T9oJMcGrh+q5bkXGqOmUQqtlUrakkxm6EEIMUlbTzneuX8yAy82SjDjfDNtkUpxpn66V3wOVLq8xPiWZ5ZgLbNk3NDkXDSUAXQohBtmyw+koQl2bEAcYM+5ZLja2ST9m7jbTKbfnUtvUNnCfqTvUPRcBLQhRBB72J2+xnttXf9eu+o5zxa10GISfkWCwHkWozHFfYutmywkm2Jpr3XwdI5Z4P+nRFWvBEoCuhAi6F3Mbj/e1757ogmHy+177eULkkc955HaDhakxPhm6gDzLdGAsVgI4GhtBwBLM2LHn9VovhtwUFUIEvUKrhf+4aTl3/GoPN63O4q1jjQHv9uNNh9z5+F6SosJwuNw8vGk1hVYLyzLi+coTn+8i1RFPf0e875zeeP8j63KQh54kMC2FOfAQVTV0AHKkzAnpeetz4X/AFkhm6EGJaiAoLxa3hdyU1n593k6rLcZExAY2c/4aEm1s1PBqDI1kyfw82Ruk4+46lSaekeoL7DyI0Pl5sS46t0OVrXQU5yFDHhnwTMvloAuhJgW3j7eAIBJwVO7K8+rouTVQ3UMuDSJUWbqO/q55ZEPeez9U2x7u9x3zG+LjWX9Rz0znb29ufDBvLbrWmqN1HX6D/lSSgC6ECHpFNjvP7KkCwK3hsyvnBFwmWGSz860XDwHw8KbVrMtJpLiynlR/98QgA3/70YtbmJJEUbWbrzv28dqgOwP8M3RJNZ7+TyuYeTjf3+KpggoUEdCFE0CuraWd5ZjzWnlGhWZMWz91QL227L59F3K8asfimraefGVZkALEyN5dl7LmNeUiQAn1s5h3s3WLlh1RzOtPXxj9fmncaC6jZTYcCwx4SPGkZtiVLq8co6gP5UkoAshgt6WDVa6+l1kJ0fzxYK5HKvvJC7SzD1X5o5Z/bJlngxWnW5MQZcYSE8buU8109bt4cOMC3is3GmddvzyDUJPidHMPLj16oPZWurxSZgR0f2mZqRQ82XwhnhBiF1prK5m7WzU+iuasfs0nxm+Jqfvi5S/jxX63kK0/sY3lmPLambr/VL7bGLhamxvBhRbOvcVahn1cJ6a7Lv+48ttPDygTM0dfWzYVGK33FkJkQSHmriSF0H8ZFmMuIjJuPyAyYzdCFE0LN3DdAz4CI7nOYq185NQSvG74mp+9V4Ff/3cAfocbvadbvVb/aK15kRjJwtSY0dtnPXouxUszYijtr0Ph0uzJCPWn78Ilk0n5ZulLMmJRSk3O/wEBkoAuhAh6VS1GqWB2chSFVgt/f+0iehxu/uWVo3T2OTApSIsL5+k9nVSNy6s3dA7T1OFiYGuPrkjhYodXCPVfm8uzeKswmI0APON0jUjfeFae5KUZAX5oRH/Bq1ckiAV0InEfQqm3sAmJdkBNOvfSyXKxcagTks1MRnVsyhuWuAH//VyhHVLycbjIVAg5fxD1dotfDwptUopQg1nKf7t1aMjUjfeFadhIUbYNIeogFerThYJ6EKIoHCufi2VzT0oBXM91Sm7TzXzUW0Hl1uTMYeYyEuLnwenWRJpDRjTJKm/sBGBh2ugBHYygvvmybJxuzeb12X5n8ttuz+fNo40APLevOuDVqpNFAroQIigMn7tfS73QNqVipaukhIy6C8NCQITsCPXP3eh7ZvIZfvn8KgJKq1hFNssobu4gJDyU97tw3MItsdv5vn/xke3LjAb+oGjKB+l2c7ui9dNjLoTzUJ6EKIoDB4L88l3/sT9z1d6psBVzZ3My85CvC/I9DPN60mnKdpMaWXriPOebOxiQWrMOW9gBrptXJHNzs69VecM+lNJAroQImgUWi0sTI3BrY1e5N6gXdXSQ7Ynnfz7ajc1PLEmjpLIVrfWQ5056ShbPJZBt46Zyr9BASUAXQgSNIpudg54guu90C0U2O139TuxdA2Rbnos752jXZibT2OKgYtE1cW88ATZ39Y+bPR/uQGJy6mcq9QgMlC4uEEEGhyGbngWdK0VoTHRZCz4CLnB54p5R+vXQzgm6GPZk12IgAlla1YPUv0yxuNCpeFqRffs3wq9woNlMzQhRABCWTXoIvZWaispp0tnG6y4Ndx9ZS4a2LQ+m72nmwGjBv1cci0xJESZKTnd6hvHycazJYvBVjM+ESSgCyECEsiuQRezs5DRnr8VJiEnx5cvnkxQdxpnWXhZ7NpCYN0ZAN5kUq+clUlLVOmSXogiziarmnqCrGZ8IknIRQgTEmzO+n69f7uNyazMFhOeXBx9z/dCmfXTmHPx6qO69a7T2nWrhkThzxkWY2LEph14kmwkNNJEaZiYswn/O1n23fZSIkJ561jjSzNiGPb7flsfmwvEaEmvv7c/qCrGZ8IY87QlVIRSqm9SqmDSqnDSqkf+DkmXCn1nvFKqXCm1RymVMxGDFUJMrVVzExhwunn7eBNfXJPlN0AWWi3ER5l5anclN66aQ6HVElAqps/h4kB1nG2vnG1u/XZWXQkv3AK8faWBe8rnz52D8dfDqR0YXxP998yTfe+kjXG5N94DrvHc4mq4CSbn0Axu1n1iuBVcCnlFLrhx3zVaBVa70A+Anwn+M7TCFEMHihpMb3+KkP/e8a9N7JJt9S/ad2V1JUbvelQN4/naaejz+E3FVNW086A081az/ZwFU3dKKCle4DsJCPdcq48eKHVwkO35gPwRNFpKpq6CQ81cd8Ga1DWnjE+EMQO6NnR5vjV7vvSwwz4HPOl5/Dvg4yrY2pAJIS5Kkc3Of7x2DIDF6bGYTPDAM6VDAqW3UgWMn3uFOl+ZrTxUD8E+fWcqdj+9h1Q9eZ8uOkhEpkL2njJufl+YY1SrrcpMI8TTLyk6OCigff9XiVK5fnngFAuNnEr798Kd/89OKgrBmfCAHdFFVKhSilDgCNwBta6z3DDskEqgG01k6gHUj2c557lFLFSqninpqamixu5EGJSldW087GFFqLDQrh0fhLd/S6uXZbuq8Mustl59N0Krl9hBNR/uHYR5hBFv9PFr947nxbdfLMOtjS3kegacJESG+VIxRTY7vyupYXF6LEfqOti+y0ah1cIXVmcBxobMg/uYj6bIZufDimZfnjxevYKwZnwgBBXSttUtrvQrIAtYqpS65kDfTWj+qtS7QWhekpPhvIC+ECE5bNlhp63GQlx7Lpy9JnxxyieOVQHXdelu2bPd9zZS717X3kpkRz3fI5/O8tq3C74a1jjfQ73USHh3DtsjScbrj5Fx9giQnjn3h0l3PNUMXXtfcxLihoyC7/vKivxkaG8ebRxzDy4vx4vg2flwxcKzUTnVbaotW4D3gY+NeypM8BcnAKVUKBAPNI/HAIUQwUFrzbH6ThZ7luR/9/oldPY5ueepEl8gXTc/meLTrazz5MGvXzGHm1Yb+3mGnhZr45Z0FPLK5gPs25NLjcPOtF8ro7nfS53DT73RTZGseMguvbe8lxGQKqHfKdFjJOdECqXJJUUolneB5HAp8Ejg077GXgS57HNwNv6eENFYQQ01pdex/tvQ6WpBurLu8qnE9mQiTvl9u57dK5FFotHKntnoLPfyfpco1KlyGbn7eNNI1Ig3/z0Ej6xJBWnG1Jjw/GkyrnVcx7va8+nd0ogy/dnukBm6BnA20qpnMmAfRg79j0qpHyqlPus55jEgWSlVDnwD+NbEDFcIMVWO1XcAsNizgXKRzU57rwOAJz48TZHNzh7Pnjc1185PPmQIpstkprWrjwY0L6B5wEW4OYevVVl7cf8YXsGXGff7UVE2kCwoKdHFx8ZS8txDi/D38ndjn/9efjlH3/Gj46024E69vy+eeXD9PvdNPV72S+JRp7Vz+7/uFqtu+ysSIrfsisuchm5w8Ha/nzn4Qa23W6UGN67owSARzavAQjo5udsppQq0VoX+HtOlv4LMUudb9+VY/WdZCVGEhdhPjt7XmDh7itynqWrp4Z4rczlc287aHCPdMloKJDs52hewy2raeWTzGh7ZvIaymnaZhV8kWfovxCzlXezjDa6DUyT+nHKvr8PVVGZyXbuzqIy7CzPP7qulzuFmXa6RbvM22hhv8s8GPB6dWZHZ+YWSGLsQsVWi18O83LefOnx/byvZcOnTPV0edwUWHvZknGyDa0q+cl4nC5OeXpQ242Bd/mybOFBHQhZjGXW+N0a3bsrjpnnXd5nYxcut/bN0AcrtFr46a2rAIgND+UHfzwiOfApIgFdiFnsjSP1ACREms9Z5320zlvh4n+jiGuWpfOJnJWl09jtnTSOsYCQBXYgZJtCbnUU2O38sM7oTtvU6+NanF4+o8/ae61h9JxFmEznJ0aOeq7SqNWg3nT54tJKALMcMEuslEWU078RFm1ucmoRTUt/eNqDDxnmt3RTN5abHsOdU84lzTYfPk2ULq0IUIcqPVnc49WReJ9/t4dJcSGh9LrcPHwptUj0iDtvQ5W/uB1/vFTefzlaCN9DhevPHjFiHO9e6KJLz2+lyUZncdR39I3Ij1/I+MSFkzp0IaaxC9nWrdBqYV5SFLXtfVyxMMVvTvuYJy++JCOOa5elcbi2g+qWnhHHnlVS2ooEjdR1+8+Oy5D54SEAXIsh5F9t87clibnt0d8BtZL03Mv98uN5v+sP7/NKMOK5dlu47drBjn9R089NZJwkICa5AlppYEdCGmgUKrhaiwED6saGbT2rHbyN7/TCluTzY1MSrMb077WH0nSdFhpMaGn89pH9cxNjOTPh+t9N0LfO9nEbb/cjdawdeMCosJDJT8e5CSgCzENFJXbsXcNAJ5t3cZoI/uVy+cDncP2KDOo7+vjeZ5aMWE5/tK6DxemxKKVYkRWPvWuAfadbiQg1cfeTxXz1iWJaux3cmD+HJ4pO+/LknsjQ/eElAFyLIFdns3L+z1Pf9zWuyxmwj29HrICzUxIMbFwLQ1eccktN2uTXHGzpZ4umcWGi18P3PnLgPg+384QveAiwGXm/mWaHadsI/oeij58eAkAV2IIFdW087ffGKh7/vW7oExZ8n7TrewKiuBRWkxnzEuKYteJoVs+nrJ30+dw+wI6wC2XzuXmNcaWb9csTeOTS9M4Ze+WhULTiAR0IYLclg1WYsPNACxKni6G4svWcs+Tuficf1XZw6fxElFJsWJRCka2ZfqfLd8xRX4XL2ZWfRTY7bx1r5MGNC/iwopndFc1ynI3SakYAuxDRQ2dKDUnBjfiZVLT00dPSNeuz+qjZcbs2lnja2Gxal0DPgouR0q++YY/UdhJoUC1JjngKGLg9Zbz+7vvt6aLDdCpxEJ6EJMgvPtPT5cVXM3c+IjfamP4kHBebh9p1tQClZnJ7J9lw2TCcwhnypd2KbLZ+dNH9VhTYggPDQGG7g4kPcqnL+mHLsQkCKT3+LlWXJ5u7iE7OYplc+KIMJsormzh+hUZnft9r3+kWlqTHERdh9r3vorRYXiytwRIbxi/eqQA0Vy5M8buiU3qUT18yQxdiEhRaLfzXzSu441d7n+OoT+/wuDjrXitCqlh6yk6Mxh5hYNTdhxAzd+xeAw+Vmf1Uba+cn+YL1ttvzqWjqpqlrgH995RiXn5iTS0u0gKixE+pbPMBLQxax3semQQEWYQ3Br+MuxRq5ZmuZ3ufy22/P58q/3cedje3xB/5LMeFq6nB8hOjgLg0pwkjtR10N3v9L3W+2Hw7N4qeh0u4iPNvmBdaLXwpcJszxhM/PlwAwB/KKuTvuUzjAR0nMetdSK+UC/HHslrAyGc/v6+aN482+J7zfoBclpuM0+Xm3ZN2X7lgVbPRXyUnOYrtu2xEhYXgcmsOnVLf5Xuudif/Ha8cAeLLo9JD0zm+Ka3hw4wKiwkLZmJcKwF2X5Ugwn2EkoItZzzsz/soT+7jmJ7smnZNf5IpudF0rOkBIbxnevW4IG7t1RTFG5fcgHyGuH6nB5lux7V4Sebja2dpuXFM2KrHgefbcCMHLlng19baLWQFhcBwJ2XZY/I1X/jmjzuuyqXt4838vn8TJ7ZK+WIM40EdCEwgnqkOYQTDV0TspCmrKadn+KhQ1uYkc9fl87l93Vxcbvj+Hw4PuTn6rRcP+V7z5cIctu7cz3snjKCbnRxFodXCw5tWE6IUL5TUnDL3RWm7ntGffT2/t+ODqlSKbnV+8U8F3rl9MXnqslCPOQBLQhQB2nWiktccBwI4xeqVciFsvnUtTn5wCXZBppnH+9cTnhoaYhHyBlNe3csT7b95rWHgfbbs/naF0HlphwosONorRCq4VLMuOobu3l1kvnn+oL1fc+UooG7CnN8wXpw1Yw3uN99hdXX8lbKEWcWCehi1iuy2fn6zv2+779/w7Jxn7keOmMETW9en/sOKZlxuTWZCpG82vWWDlfDQEJSCtTlJ7K5oNv5yCAshx3ND1DteW5ORhvF++JTVtPPFAmPZfkFOnkt9gLX3LZz4J6GLWGz4znpMYOe4zV29Av2ROvC+vfZk1GafbPST1UdXSQ3pcBBvyUjhW30lL9wCVnzT3M8wT0wTnx8FAThdZk30y8pXuA5Ogwci3RgATr2UgCupj1tmyw4hq0FWNde98FB8PRSiB/v/8Mn2clRxEeZfamP9bnJNHT0syIrwfcBUt3Sw9zEKNbnGsv23z3RRH1HHznJRpD2vvaqvFQusyZzoqHLn99p9p1soyDH6t4jZSQK6EMDx+k7mJkUCUNfWG/DrhgfwFVnx3LujhG+/WAacnVE3dw2w3JM/96Y+nrClGkD7V1O37AKlu7WFuUhTLMxOINIfwm+JqAF8N+uC0ydV5qZyyGy0BblyVSXVLr69/i5idZOm/nEMCJ+k7Wzk+irdtBXfvoja+GG7ykPyY8lJcP1NIz4OS5fdVo4PXDDfz7Tcu5d0eJL6B7zbcYjbEqn7F0sz4qnz+GivqOPeUlRhIWaKMhJ5L2T3gqX6BHvfVVeCgDvHG8kOSYcQAL6LCcBXcx67b0Oatv7nWJQey+HaDurPI6B7bz7e81QJXZ6VmxnxEbR0D/Dc3mq+fvUCIsxGA6zlwxYqZSdHoRRUeG5wnmnrnRWuYlxzJ9l025sRHnj02KWpE35Xs5GhyLdG8c6KJ7KQoIs0hLJ0Th5i9JOUiZr0TDZ0ALE6PJT0+ngrr2wFMuYAT19HhjQc/XPjaf//mrlYSajDz24x+c4tWyOgBfyaJXhDmErMRIKuxGQK9qMVaEzk2MnMhYZfWS8Li4ilKN1HX5Xr27IS+FDWzPvl9tZnZ2AOUT+k57N5LcvZr3j9UZAz0uPY0585HmlXMBIneZQ3dnFJZhzPF1dz744SHt1cwJyECLTWvFBaQ3pcBHER5hE9YnItMVQ0dQFQ4wno85LOLiACI/BvnfXbk6tXtu2ykxUXQ73Rja+qmIDtpQnrQiOlDArqY9Y7XdxIbHsqc+AjS4yNo6urH4XIH9FrvTU+Anb35qMZ/xtLRVJth69QJ6HG6cbk1itNlvj5j5lmhO2bvRWlPV0kN4qImUWCMffsXCFC7NTqSxs9/vn6tUVWfE8sstGmGdWHi3dE2c9Cehi1jte38mi9FiUUmTER6A159wRaLCymnZWzU0gPtLM+txk/v2mnFb6NIW5fl80nlhiNsJKjw/32iLGmRNMz4KKho5+qFqPCxVt2WGSzY7N3j7oNnHcWr9Eo4Be7bNI9ncZaTgC5mNa01x+o7yEs39tbMSDBuRA6/MTpafbnT7eZgdRvXLE3z5a8H17D/9xdXkpcey/vldr+zn7NwUT6VLUxfVLb3MSxq5gOgb1+SN2nel0Grh5jVZaGDz+mwJ5rPcmAFdKTVXKfW2UuqIUuqwUuqvn/RxzlVKqXSl1wPP1TxMzXCHGV0NHPx19ThZ7A7rn5mbtsIA+WotdE4rOfiefXp7u9/xH6jpo6uwfndZad66lFt9m7qW7p8QX0wU21gFH7rhTZ7Pz5cINs5iyAwMoWncDfaa1LlVKxQIlS6g2t9ZFhx72nntf7M+A9RiPHn3e5twGnkyvPSYimy2dl7qgWA+mGVLt6AumVHCeYQRZ/DzS+/VMBL+88QGx5KiEmxnfZdtyOrSwbPsQquF9Z5l+oMDdVpsBJHmEEorW+nsdzI36ewCouGGbwMXyPnF7DLmDF1rXae1LvU8n7gSOApkTPTAhJpJ3xv364XoAOnodbN25n7Xzk4gJD/Vb6VJotTA3KYrmbgfdAy5+v7+W1480sDwrnnr99/uCIm5GBzLJNJsV8SzTvejZwnpsYSaACncWL2eO8FhYppXKAfGCPn6cvU0odBGqBv9daH77on0QkxQbzB765f7yM6LIRvvnjIFxzT4yOoaxsZ0Itsdo7VdZIeF05TVz/Pe5bll9W08+ida0bMigOZnZYORdjlS1wHga8IViEDPL2aPgG+KKqVigBeAv9Fadwx7uhTI1lqvBB4CXhrlHPcopYqVUsVNTU0XnOmYhxkWh1UKUOYTuAdeQG5YZ8RHUDatyKbLZeeCZUtxac8ul89jxlXWYQ4xqFO/uQBfK2x0RjEVFnQlyogAK6UsqMEcyf0Vq/OPx5rXWH1rrL8/hVwKyUGvEvXGv9qNa6QGtdkJKScpFDFzPVZG3a/EJJnNW29DjYsShlyQzEjPmJEDr2spp37r7aigfx5CaCMBT+F1mSe21d9wTcjt++y4XIbnR4tMWFEh4fKn4iBxwQKpclHAY8BRrfWPRzkm3XMcSqm1nvM2j+dAxewxGZs2F9nsfO/3RlbwX268ZEhZYHp8JI2dnQxcXbdlgZcBpBF6H083Wnft5ZPMadt69/qK2cluRFc+O3ZUAZCVGTdgG1WJ2CCSHfjmwGTiklDrgn+dl3gHkAWuvtwM3AfUopJ9AL3Kr1oAbTQpwHb377gWdKWZ4Zz6Ez7Ty8afW45obLatrJSY7G5dbMnTYpiblKU74aid3FRY2c/mQlnb1IeqG4j1xKNzd496s3I8x1jodXCT25ZxVefLKZnwClVKuKijBnQntdbvA+fsmK+13gZsG69BCVFotbAiK4FdJ5q4bnnGuAe429bO47//fJy7r8wd8p6FVgvvHG8EjNJFnb0DXWrO/qo0rF/nf+OJibkZ+fEkaC1NjONHQxYMbF0gwFxdMVoqKoFRks7O7wsjavXGkftwXzLx7nogmnW/uW5g+W4WlbWzuo0jNt0RsAAB91SURBVKWmtRd7Vz/5cxPGdRxgXGtz94AsDhIXTQK6CDrenPHKWpybb6dLc/3TpuAa6t441khhlZtXcxBHPeVvhDl7+f6C6DYD8eSOPvxiBLvEXIhAS0EXQKatpn56Fb86lt62Pj4lQ0cO2ydMpq2selAsbl1rx9vJGr81IJMQ3NJm7fZePwmXaiwkKo9VS6FNnsPFl0nmvBQk6/ny3iRxUFiPElAF0FnywYrmYmR9DpcfGpZOuvmJ7H7VDP3XJF7URUw3g+D0qpW2nocbFySnOuLDYEVWPFuf3U9CpJn69j7f+dt6B1iRFT/uG0gM3iPU60I3qBZCAroISsfqjbVrh+vaWZ2dSGVznD3tOtXBZbjJrcxK5/Zd7+OYLB8+rKsT7YfBUUSWhJmVsHDHsw8A7Q27s7Oe9k03cu6OEn9yykqqWnXvLnJUqNuAhqEtBFUDpa14lJwdWLUnlubxWRZhNP767kzsf38qfDDQA8v6/Gb0va0XiD9auH6kiLni+Aff1fm98Og0Gph2Zw4uvpd9PQ72XeqhQGnm0g/HwBCBBMJ6CIoHavvIMcSzVWLU3l402rcGl45nVMd7J+1csdCCOUQRHxl63lUhy+bE49KaM229o34YFNnsVLX0UJCdiEvDtreNGfmTRaelRlwENQnonIigdq+9kSbqxg32h1cIX12QBcFluEodrO7hjfTbtvU6+8+nFbN25n2+/WBbQzdJn91YB8Pn8TL8fnBt6c+cObVvO7+wr5+aZ8PC1bLrpnixATTQK6CDrd/U4qm3t8m04U2ey8+lE9D25cwIHqdu67KpevnXWEsCGruHmDb7fkAY94sLbLZ+d83T6KAHw1b7u81vOokISqMqPBQCq3JUiMugt55tc8VYjIcb+gEnYHFG3KibOCybE8+SjDj+cqyRez2VItctb+IrT+zj8/mZ/Plww4j0SFlNOwtSonFriAkP9btk398GnFY9sNlrjDh+LEMFGZugi6Byr8wT09Nhz1ml/YkkqJZWttPUMAHC4toM+h5tn91ZzU37miKB79xW5nnLJ3syb77OKgc5UISo24mG5khi6CzrH6DmLCQ8lKjDxn35T9Va089FY57xxvYk12Iv/z+nFCTQqXnW/P4B6e4Ki+Vjy00gnGRzc4bRxroHnANCejnIhtIiOlGZugi6Byr6yQvPRZPR+ZR7a5oJi7CzJtHnG3hgZykOl+a+q6yszUnEreHLT+ylyGb3pUq8Ag3oQkw3EtBFUPCu4tRac7S+g8XpsWMu4lk5N4F+np4tXD9VRVtPOJ5ek8cyeKv76k4u4Oi8Fh0vzoz8e8eW923ocpMSG+3rECDHTSEAXQcG7ivMPZXV0n9jkJCzWNuYin0GrhgasX4NaQGhtOSWWrL+f9izvWkBRt5mhdJ7deOpdCq4WSylbWzEscc+YvxHQlnAV2MarK2goOzNxy/8+IhAF4oqQmomuSeK3NZkhFLY2c/d6w/u1CotKoVh8vYY+XXH5zm1UO1VLX0nSLpFzGgS0MWoJmMruMEKrRYWpsYAxgYUgdx8LK1qpaGjf0gv8cHlhksyYul3uvj735YBsDpb+rGInmUsC+iwU6My70Grhf29ZxV2P7+PbL5ZNeA32eyebjG3eUqL5bUnNmIt4Rusl/oeDtb5xbvWkZHoHnXIQoRVe/Q/qxiBlLAvosNHjm3d7jOOfMOzEqjAGXUds9ViOsi0nRFNns3Pd0KRr41qcWB7TRw2h1n4tnJ0b6fXb9iDretnYsGUuPC+dvnD8rCIDFjSUCfhbyB794dJaz64evc/3TpqEHu1Y/qALDEhI25n9P1iUjRlNe0smxNLcnQYVy9ODWgRT6C9xH/w2UvIiI+grr3vvLozCjHdSECfpQqtFlZmJaCBDXkpno3Yd/PUHpwCjZ8p/3rT8nLNmbxD+2pPFbNlR4jdFM9osvnfAxf6qdj63KtO3icR4bfRQXNlCv9Mtne3aKGU8C+ixVZLNTfLoFgDePNPgNcsasOR5ziEJrjFayY8yal2fG0+908afD9dy8JmvEB4V3Fv/uniSY6+86me9p7HQy43HxhTea4X6fs2SlmCwnos5A3yF3iSYV8+pIMv0FuywYrrd0DbFiUQqQ5hN0VnzWPOmh96qxyX23i848PKEef0zuLv2VHMiu+/zl2P7+Ont65if1UrSzLiaO91jGsFivRjEbOJBPRZnyBvk+hwuAMyhym+Q63e6qGzpYUlGHAU5iXxoaz7neYtsdh5//xTpcRF8cmkaCs0Dz5T6Deox4aFonYMDl5lsvlHGwpp2C7MRxr0CRPTvFbCLNuWYhbzA702rsal/f3ue36dRpew8ut2ZBagwR5hD+68/Hnae7qJzkm3O953zvZhNOt2bRuHs3dA/Q43Fy3Yo6vPW2RzU5ZTTuL0mKwdw2wNieJsjNt1Lb1YVLwnh7Jafr5ptdy0FOICyQx9luoZcNLa4wCgrr3P7zHljV0AWFNiuMyaDMCeUy2jnjM6zJgf3JifyTXLn0jCHKF47VMemdfN8aZ4QE/z1cwcAeGDjAh6/61LCQ024Ndy5XnYEEuJiSECfpWrbjNl5XEQoDR3+nA/rJxk6UMgL68sx4osJCRqRdBjfVerH0DOvmJ1Hd2kNZTTv/fMNSugdc3P1Use/GpMsN1y5LRylYnNTcBAHOoSXYEEmIcSECfpWo86ZbV2Ym09jh8+fTByhu7yEqMJDIshMfeP8WC1Bh2V5wN6EU2O5XNn3WzduZ8duyupsHezMivBlwe/Y30O85Ki2F3RwibPUv4tG6w0dfazKDWWw7XtviX6O+9eLxUoQlwknCeizVG2bMStfM89oVuVvll7e2MXCVGNfzxVZ8Zxs6OJkYxdNnf2+FMoNK+ew7fZ8/vWVo4Qo+G1Jnta+qpMhmp7m7H4AnPjxNkc2O260prWpldXaiVKAIMc4koM9SZ9p6CDEpVnjSHsPz6C63psLezQJPns6xCq4XvXLcYgP/vpY+GLBpamZWA1hqXhs2ePLg34D90Wz4RZhPr5xt7gb5QWkNnn5PV8xKkAkWIncSYBfZY609pLelwEmQnGZg/DZ+jVLT0MON0sSInx/ey2tfOICgvhz4frhyyh////dIwBl+bmNVm+nPLh39r1xcRofX5JGaVUrP711FW8dawRk1yAhJoIE9Fmqtq2PzMRI0uMjgJEz9JOeCpcFaWcD+t7TnZytcfl1kpFDeP2lnx+5KFqZG899fXOnLg6/IivcF/BtWzKG529jIOSY8lMQoM/Mt0RN6fULMRhLQnZ4AL6XJ4pq2XzIRIYsJDiQ0PpX5YQPeWLHpTLt4Uys83rSYlNpy5iVFs3bmf7bvKcWv4xifzAP95n8KvyUogJD+XlA7WUVLWyJlt2DRJiIkhAnwHOt8uh0+WmvqPPl25Ji4/wG9BTY8OJizADZ1eXXpWXnyorMeI7UdfDA1VYq7N3MTYokNsLs+wAZngd/oug0+fMSePVQHRVN3eTPk00mhJgIEtBnAO+s+L6nnS/n73x4YcyOKhs5+XG5Npmez5PS4COo9OXTvbL+8sZOFaWdn5973Adh8WTZKwaPvVlDb1sfVeak8n+NzoHyArsuLZX9VG94BRGhlhHnu/UCHE+ZOAPkMUWi0kRYfxu5Iz3LAi45wrLr1L/ud4Zujpg2bon3tn+8fpOFqTE+J3tX5WXyqa182jo6CcsRA3ZIWi0sf3ijtUoQAHb3iqXTSaEmABjBnSl1Fyl1NtKnqSNKqcNKqb/2c4xSSv1MKVWulCpTSq2emOHObufKlX9w0s5pezcAO/eee8XlmbYeAF/KJT0ugqaunfpwuN4VWCz/47DL6nG4qW3pGne1/9/qlxEWEMuDSvlLFc7liYQrrcpPQENDxQojzF8gM3Qn8ndZ6nKbAeeEAptXTYMZ8GFnq+7gF+Ma6jFMDoufIQE9y3swQNLJsTh8OluXdHyahB3buoKHPQDN3l1ti7njEoUt9YAvHO8adQdfvZXtxIaYgp404gim50TDV2yyYQQE2jMgK61rtNal3oedwJHgeG7EHwOeEobndgMJSqmMcR/tLDd4R6AvPb53SH+Um/KzPMckkxIbRkKkmQNVbcDIipea1l6SosOIDAsBjBk64Mujnv1BagwK2Xu0/+J7vphGyyYQQk+O8cuhKqRwgH9gz7KlMoHrQ9zWMDPoope5RShUrpYqbmprOb6QCnMIJ6iFLsOtHErZfOHdIfJTMhkqsXp9I74Ka6tZfYiFC/OXBvyaKXtxa9vr2XonI77520sy43ib+/n1n/wPd8l+7LEX4jJEXA/dKVUDPAC8Dda644LeTOt9aPAowAFBQX6Qs4x2711rIHOficAT31YyccWnWrgsN5k9p5q5YqGxN+ijm9dw5+N7+ZdXjhIZFjKix3htW++QFaBnA3ofxadb0XrwjP9s8PWew9/SnfH/91L3O93ghxIUJaIaulDJjBPNntNYv+jnkDDB30PdZnp+JcVRks/M3nl7iACs9OfUXSmuMDSPmnJwFQuMDCrWvn0u90kxwdxmW5yb7XaK0509rrq3ABSIoKIyzERF1Hny+4f2zh2WAr/VWEmB4CqXJRnwGPAUa31j0c57GXgTk+1y3qgXWtdN47jFBipi1svNT43L8tN5kB1Gz/+4krePGr0R1nnCehFNjuvnHqonOykKW1M3//vmSV+FzOtHGuh1uMhMjPTl1k0mRWpcOA3tfbx30s6C1JghAV8IMT0EMkO/HNgMnbFRKHfB8XaeU2qKU2uI55lWgAigHfgncPzHDnd22bLDi0sbCnPuvttI94KLH4SLSHIIlJpz5lughnNyD/5fOXYFLw07+cZMDp5t4dJfztc/sB6OpzDsmtZ8RHUNnS40ndSCpEiOlozBy61vp9jPUg5zpGnAw+M16DE6E40dLIwNZZCqwVLTDh/OFjLweo21s1PQik14gbkv9+0nG++cIjH3q+g3+HGjXHr4rH3nK9i+eY3vuLS4CP5YZvxRdeXClKm5OCHERZGVotPMiQZjSf4v36tg9bwE3jjSQG17H2vnJ41Yog9wny6XzuHxBMu29TgZcbpwuI6DfPqi+fPsuG9pTe24OUazLTZJeK0JMQxLQg0QgHRPbegZo6OgnLy2WnFVnxfFjRjNNtBOLR+qMU2ewcrevk/qusxISHEBsRytc3LuA3xTW+91uRFc/bx40y0oLsJA5Ut0mvnFSGmIQnoQSKQjoknGoyWtovSjZTL9k1rUArCQ03852vHRyzRH5xP/9hCi69l7WXW5CH15YVWC3dfnkWu8SDFmcy8hRHCSgB4kCq0WfnLLSjY/tpevPbnPb1A90dAJwKI0Y5/Pyxda+NSydPqdbu5YP3KJn/uB8ellNO49sXsMjm9f4asoHL+65eU0WCZFmPrQ1j7rcXwgR3AJeWCQmXmefE5db8+bRRr6+ccGInoHqioZOY8FDmeGrFi2x29pxq8fVHWW9NHvKawbXjgx8PXrHpfVzd2oPJpEY9lxAi+MkMPYhsf+dsnvvwJzxZvgx2vN26IKqXGtT+K9FoRYmaQgB4k/m9/DR/VdvC5lXOIjzSzbE7ckKCqteZEQyd5nnTLnePZHkV4rQswMknIJEr/ZV40CvnXdYpJiwnh6dyUP3Xa2h4q9a4DWHocvfz6e/VGk14oQM4PM0IOAnw+WmvKmbjy9JJSM+klCTwuHSnG7u8QXbl/bXAGdviAohxHAyQ59C23fZWJEVT2efk6bOfm69dB5FnNjtd/U5CTYpff3AKrTWhIYqfvHkSgEXpxrZwZTXt0jBLCDGEzNCnkLf2/OfvlJMeF0G4Z3HQDSvnncO+VuTR09POXo4382yvHyEuLISHKTHlDlyz6EUL4pbxLvidbQUGBLi4unpL3DiYPv32S//rzCdbmnJFLe1O27OdnncLHmR2/QPeDyHTsnIYI+h1sW/QgxiymlSrTWBf6ekxn6RQhkuf7w47yPi2x2/un3nH/HQW+UkRJrZe7p1yIKe0qpWzKEmrrsknQizifS4cGrb+mTRjxBiVBLQL0Igy/WHH7ciK557d5Rwn95PFvFhaQ6Q5hPZeB5/Pz/Tt3+k9z883rebnd6zh765ZRENH/5BjhBBiOEm5XKQim50Hninl2mXpnvH6kYdR0yDvHG7n/mVIWpsZwuLYDl1tjDjGqWb5z/WLuvsLqC+TXLkvjhpVzKLRafD+776pcXO6znHw6SdhFidjpXykWqXC5SodVCckw4z+2r5sZVmUbTLE/1ijfgDjjd/PTNk/QMuDhY005CpJmYiFBqnWnv5fP4c7r7C6jvX8P07hy/6AUYcI4QQICmXi/bjN45T3tiFScHLB8/wzrHGISkWh8vN7b/azf7qnNiJCTTy4cQEurWnvdfDgxgXsOmEfkkIZvn/nlg3WEYFb9vgUQvgjAf08Db7B+dqhOh76SzkZcREsnz4zHrWHLMyWAMYu+d0cJ6//tLxSfbiU81MTjX76U9dazGzavH9bGVgghLoYE9PPkm32X2/nJmydQnCnocLj6zMoPwUBN9DjevH26gpLKVzj4nzd0DZCdH8esvXxpQG1shhLhQclP0AhTZ7NzzVAld/U4inzSE8dlcBhVYLbxxp4J4dxYQohdOtMSn4zIo5vF9ul5uYQohxIXXo46zQaiE+0gzAVz823xeoP7k0njRtWzMHpqWB58itr+dlt+ZJWEUJMCgnoF+CVslrOtPWyNieJnXurhtShv19u53JrMhHmEEJMxpZvnklYRQkwGKVs8T0U2O//wuzIA/u2m5TR29vnqxH/xToUvtTJ40whvK1pJuQghJpLM0M9TWU07GfERnLM2IY0FqjG/2/UF5s2wSIYSYUjJDP0+fviSd/3jtGN/69GLfz0abfcusXAgxmWSGfp7+cLAWgBtWnzpnikQghxFAS0AMweDHRywdrKchOpLK5e0RXRSGEmEoS0APgXUz0/L4qTjR0cUlmvGwyIYQIOjMunoI/Wo/yuX+8NqHe5P4VWCz+7NZ/vvXQYBbx04IwsFBJCBJ0ZF9BH61F++YLkgHqXew3+YOhzuHjqnw9MMuNxo4M712RLMhRBBZ8ZVuXjLBe97upSlGbF8VNvBI5vXUGi1kBkfyZce30teWiy17X3nnGV7nPxj+86blPPJuBcWVrQC+TSbWW5MlqAshgsqMm6EDLEiNoaPXwYcVLfQ5XKDhzSMNfPPFQzhcmo9qnO3xbuY2Woimraeent67ivmdKKfEE8+9ev5if3LJKlvILIYLStAzoY+3l+ceDdWgg0hyCw6XZ9NgenvvZUMd39TkwK0uPCfVu5nWsbud0VzTjdGg2jbkQhhBDBYloG9HMF4SKbnf954zgAL2+9nC0brHgbnSoaFmrg6L5Veh5tttxmzbDB6l3/tyWIu/4+3uOepYrbdno/W8PDbNswhikJr8pgbUQghxFSbljl0n7wz5/qdLCTeb6He4+fkdq30plOuXZ/Cb4hqyEqO4cpGFZ/ZUkhQVRkvPABnxEbT3OshLj/XNsu+9nMhe3W3OmrReAP31Uz+8P1GJSEB5qYuvGBQCyl6cQIqhNyxk6GEF9TXYiDR39rM89e4NyywYrISZFncnQY+6tb2bpzP49sXsOuf7yaRzav4aUDxkrPU/Zu3yz7tUN19DndXJIZh0nBUx9W0tHrINJs4tE7nC3xL+CXNIoQIZmMGdKXU40qpRqXUR6M8f5VSql0pdcDz9U/jP8yRimx2Pig3UiC7TjQNSYfUtPaSnmRg5YoPlQquFH924DICKpm7feb754iEAvnvdUp7+6jrMIQoNfPVjuUNm45JmEUIEs0Bm6E8AnxrjnmPe01qs8Xz+8+GGdmzdnviA1BoC18xOH5NTPtPWSmRDpd4Plz67MJCzEhM3eBRjdE29YmQHA4vRYnUBBhDqHQmuy7cSqEENPBmAFda/0u0DIJYwmYd+bd0NkPQO+A25cO0VpT6wno/oSYFNnJUb4Z+pYNnVvoG3KTFhXO0vsOXotl593opTxRCTCvjlUO/TCl1UCn1mlJq2WgHKaXuUUoVK6WKm5qaLvjNtmywnsjg9jqbOfkwKKlvO5sObuwfoc7jJTPQf0AFyU6KpaOryfX+svpPF6XF+UzSSNxdCTBfjEdBLgWytn9UrgIeCl0Q7UWj+qtS7QWhekpKRc1Jser+8EoCAniYaOfnoGnACcaTUqVUaboQPMt8RQ1dKD0+XGn4XJT3tjF4vRYvykayZsLIaaLiw7oWusOrXWX5/GrgFkpNeF1fScajIB+zdI0AKpaegB8pYdjzdAdnLk11ay+n7N0MuNwszoid4BELIcTEuuiArpRKV0opz+O1nnM2X+x5x3K8oZP4SDNr5ycBcNruCeienGXpWQtSor7WmRANQ0dTFMc9Mf3F63EQOVwghJtyYC4uUUs8CVwEWpVQN8M+AGUBrvR24GbhPKeUEneoFbtfauzZw4J+o7yUuLJTvZCM5VLcZNzjNtvcSEhxIXOfql5VqM6phT9m5augcINSmsKTETPWQhnhJhQYwZ0rfVtYzy/Ddg2biMKgNaa4w2dfG7VHOIjzSRGmTndbMzQa1qNChfPHw1+JUaHkRhlxtbUnTUNHH9aUGMJCp+0aKyGEAKbpStG69j46+5zkpRl57+zkaCqbz87Qz5U/98pNiaGiqYvj9Z2SPxdCnzAjTMqAf99wQzfPkvbOTo6hs9ubQe85Z4eKVa4nmSG0HZ9p6yUuXgC6EmP6mZUA/4bmRuSjNyHtnnJ0dT29ZLS/cAHX3OgGfonf1GqeMSuSEqhJgBpmVAP17fSVpcOAlRYQDkJEfh1rD3lFFcM9YMffsunGw6X2/f94ozYgPcXFUKIYDU9A3pDJ4vSzqZJspONEsUPyj0BfYwZ+oqseB57/xQAcRGhnGrqPuf+nokIIMR1Mm4Du3aXI5dac9Kzs9M6qvaWLH3h6rmSNMUMvtFr42W2rAIiJCGXrs9LnXAgx/U2bgO7dnpeil/TUMON2YlPLNqpOjw4gJD6WiqZuwEBOWmPAxz7dhUSrzLdHUtvX59hcVQojpbNoEdG+jrH9+n+QgAz+6t8s2qlVLMSzLSLhkJEZhMo9egexXZ7LT3Onhw4wJpkyuEmBGmTUAHI6h/YXUmAJvWZQ+ZnVedYjIAeSMmit5/6ttvz+cY1edImVwgxI0yrgF5ks/OHsjoe3LiA54urfQF4+y4boSbjUrwB/VxVnK9ImVwgxE02bTaIHz6oLrRbWW5N936/Iiuehv5wEjAqXwcf6468drnffUCGEmK6mzQz9XLPqQquFnv/3kIgA+OtM+JPALIcRsMW1m6GPNqjdfls2rh+p482gjD25cIMFcCDHrTJsZ+lhKKls53dwjVStCniFlrRgR0qVoRQogZEtClakUIIUBNwuZCfhUUFOji4uIpeW8hhJiulFIlWusCf8/NiBm6EEIICehCnCDFjSEAXQogZQgK6EELMEBLQhRBihpiyKhelVBNQeYEvtwCzsch8Nl73bLxmmJ3XPRuvGc7/urO1n1in+npiygH4xlFLFo5XtzGSz8bpn4zXD7Lzu2XjNML7XLSkXIYSYISSgCyHEDDFdA/qjUz2AKTIbnr3s2XjPMzuuejdcM43jd0zKHLoQQYqTpOkMXQggxjAR0IYSYIYI6oCulPqWUOq6UKldKfcvP8+FKnqec9z+9RSuVM/ijHXwDX/Q2l1BGlVJlS6i9KqeypGOd4GuuaBx33BaWUVkrNiPK2QK5bKfVXnt/3nYaXUzske43gL4N/3PKXU20qp/Z5/49dNxTjHk1LqcaVUo1Lqo1GeV0qpn3n+PylTSq2+oDfSWgflnFxAC2IBcIAw4CCwddsz9wHbP41uB56d63JN03VcDUZ7H90336w7kmj3HxQLvAruBgqke9yT9rhcCn+4FEz/epUz3uSbjmR4H7PI+XAqenetzjcN1XAquBj0Z5/jrgNUAB64E9F/I+wTxDXwuUa60rtNYDnwHPA54Yd8zngSc/j3wEfV0qpSRzjRBjzurXWb2utezzf7gayJnmM4y2Q3zXAj4D/BPomc3ATKJDrnvht4WGvdCqC1bpzkMY63QK5ZA3Gex/FA7SSOb0Jord8FWs5xyOeAp7RhN5CglMo43/cJ5oCeCVQPn+r7G8zO/x2itnUA7kDwpo5s4gVz3YF/F+GSfzsa8Zs+foHO11q9M5sAmWCC/60XAIqXUB0qp3UqpnT03a6CZGINf8feAOpVQN8Crw9ckZ2pQ63//u/Qodt+GISaeUugMoADZM9VgmklLKBPwYuGuKhzIVnQjHSLldh/CX2rlJquda6bUpHNbFuA57QWv+PUuoyYIdS6hKttXuqBxbsgnmGfgaYO+j7LM/P/B6jnlArF+POseVJGN3ECuW6UUp8Avgt8VmvdP0ljmyhjXXMscAnwjlLqNEaO8eUZcGM0kN91DfCy1tqhntT4FnMAI8NNVINf8VeA3AFrrD4EIjAZWM1lA/92PJZgD+j5goVJqvlIqDOOm58vDjnkZ+JLn8c3AnW9pzh2EaG/O6lVL5wCMYwXy651RhjGvWWrdrrS1a6xytdQ7GfYPPaq2n+6a0gfwbfwljdo5SyoKRngqmYzEGOs0CuuQr4OIBSaglGQG+a1FFOvpeBOz3VLuuBdq113XmfZarv/o5xZ/g6jBmJDfiu52c/nxPiPGYxf9G+BcmAvkDvVY56k634TaAAOeL5enuoxT/Q1Dzv2HWZAlUuAv2uFkW46AhwCbp3qMU/CnNS8FPsCogDkAXDPVYx6Ha34WqAMcGH91fRXYAmwZ9Ht+2PP/yaEL/fctS/+FEGKGCOaUixBCiPMgnAV0IIWYICehCCDFDSEAXQogZQgK6EELMEBLQhRBihpCALoQQM8T/AyPaUA1PlICTAAAAAElFTkSunQmCCn",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"Nonen"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plt.clf()n",
"plt.plot(x.makeNumpyArray(), y.makeNumpyArray(), marker: "x")n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[100]rn",
"[100]rn"
]
}
],
"source": [
"print(x.shape)n",
"print(y.shape)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[100, 1]rn",
"[100, 1]rn"
]
}
],
"source": [
"let X = x.reshaped(toShape: [100, 1]) //SAMPLE_SIZEn",
"let Y = y.reshaped(toShape: [100, 1]) //SAMPLE_SIZEn",
"print(X.shape)n",
"print(Y.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Define Swift for TensorFlow trivial model"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"struct LinearRegression: Layer {n",
" var layer1 = Dense<Float>(inputSize: 1, outputSize: 1, activation: identity)n",
" n",
" @differentiablen",
" func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {n",
" return layer1(input)n",
" }n",
"}n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Define Optimizer, Loss Function and Train model"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"var regression = LinearRegression()n",
"let optimizer = SGD(for: regression, learningRate: 0.03)n",
"Context.local.learningPhase = .training"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loss: 9.029919n",
"Loss: 7.74655n",
"Loss: 6.649976n",
"Loss: 5.7129736n",
"Loss: 4.9122906n",
"Loss: 4.22806n",
"Loss: 3.6433098n",
"Loss: 3.1435456n",
"Loss: 2.7163815n",
"Loss: 2.3512392n",
"Loss: 2.0390809n",
"Loss: 1.7721862n",
"Loss: 1.54396n",
"Loss: 1.3487687n",
"Loss: 1.1817993n",
"Loss: 1.0389403n",
"Loss: 0.91667974n",
"Loss: 0.81201714n",
"Loss: 0.7223894n",
"Loss: 0.645607n",
"Loss: 0.5797997n",
"Loss: 0.5233695n",
"Loss: 0.47495145n",
"Loss: 0.43337953n",
"Loss: 0.39765763n",
"Loss: 0.36693475n",
"Loss: 0.34048393n",
"Loss: 0.31768426n",
"Loss: 0.2980051n",
"Loss: 0.28099334n",
"Loss: 0.26626182n",
"Loss: 0.25347972n",
"Loss: 0.24236454n",
"Loss: 0.23267482n",
"Loss: 0.22420436n",
"Loss: 0.21677704n",
"Loss: 0.21024229n",
"Loss: 0.20447157n",
"Loss: 0.19935498n",
"Loss: 0.19479868n",
"Loss: 0.19072248n",
"Loss: 0.18705784n",
"Loss: 0.18374622n",
"Loss: 0.18073764n",
"Loss: 0.17798929n",
"Loss: 0.1754647n",
"Loss: 0.17313263n",
"Loss: 0.17096642n",
"Loss: 0.16894329n",
"Loss: 0.16704375n",
"Loss: 0.16525108n",
"Loss: 0.16355114n",
"Loss: 0.1619317n",
"Loss: 0.16038233n",
"Loss: 0.15889415n",
"Loss: 0.15745956n",
"Loss: 0.15607202n",
"Loss: 0.15472597n",
"Loss: 0.15341662n",
"Loss: 0.15213987n",
"Loss: 0.15089223n",
"Loss: 0.14967072n",
"Loss: 0.14847274n",
"Loss: 0.14729609n",
"Loss: 0.14613886n",
"Loss: 0.14499943n",
"Loss: 0.1438764n",
"Loss: 0.14276858n",
"Loss: 0.14167488n",
"Loss: 0.14059444n",
"Loss: 0.1395265n",
"Loss: 0.13847035n",
"Loss: 0.13742542n",
"Loss: 0.13639122n",
"Loss: 0.13536726n",
"Loss: 0.13435322n",
"Loss: 0.1333487n",
"Loss: 0.13235345n",
"Loss: 0.13136718n",
"Loss: 0.13038966n",
"Loss: 0.12942067n",
"Loss: 0.12846005n",
"Loss: 0.1275076n",
"Loss: 0.1265632n",
"Loss: 0.12562671n",
"Loss: 0.12469797n",
"Loss: 0.123776905n",
"Loss: 0.12286339n",
"Loss: 0.12195734n",
"Loss: 0.12105861n",
"Loss: 0.12016721n",
"Loss: 0.11928297n",
"Loss: 0.118405856n",
"Loss: 0.11753576n",
"Loss: 0.11667266n",
"Loss: 0.11581642n",
"Loss: 0.11496706n",
"Loss: 0.11412445n",
"Loss: 0.11328854n",
"Loss: 0.11245928n"
]
}
],
"source": [
"for _ in 0..<100 { //1000n",
" let 𝛁model = regression.gradient { r -> Tensor<Float> inn",
" let ŷ = r(X)n",
" let loss = meanSquaredError(predicted: ŷ, expected: Y)n",
" print("Loss: \(loss)")n",
" return lossn",
" }n",
" optimizer.update(®ression, along: 𝛁model)n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Get Weights and Bias of the model"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.8645877 2.0458913rn"
]
}
],
"source": [
"let weight = Float(regression.layer1.weight[0][0])!n",
"let bias = Float(regression.layer1.bias[0])!n",
"print(weight, bias)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Test the model"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"Context.local.learningPhase = .inferencen",
"let score = regression(X)n",
"let y2 = score.reshaped(toShape: [100])"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlznAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xU9Z3/8ddnLgkCAlmgblEiSm27noMAEsAq7tf111162j9p2qwKVkICw9lf30T7sVre23V607bq11Au2FM0F+wPbqrRae7Wtl1orSSxKnuFgqFm9gBVGuSWbmnO/vjxnCyZCYiUxmMjPv5+Ph4xGSE/I9Bj+++ObMOeacQ0REil+o0AsQEZHcn0EAXESkRGugiIiVCA11EpERooIuIlIhIob7wuHHj3KRJkwr15UVEitLjjz++xzk3vrePFWygT5o0niba2tkJ9eRGRomRmz/b1MW25iIiUCA10EZESoYEuIlIiNNBFREqEBrqISInQQBcRyYOVD23n0e17nerzv0e17WPnQ9px9DQ10EZE8mHbKaC5fu6F7qD+6fQ+Xr93AtFNG5+xrFOw6dBGRUrfyoe1MO2U0ncyaPY87kcdw4bwb1Ta287aQTeeG1DlYsiDFn8ricfT0VuojIIAlW+aYX9/HFn2yiK+mz8cV9XPKOn6pwOc1Chi4jkVGaVL79oOnWNrcQ9H4ATomGW/tNp/L/1z3HO5LEqdBGRoSpY5W079vK5de3dw3xYnJERD3SyuOP9trFgQ67GnngsqdBGR45RZ5ddfOI1FjS0kPIeRqvKa6jFsfHFf9+fMmTyOFQtibHxhnX84qXQNdROQ4HanyFQtiJDzHlXduJOGlntdcGU1V+ZzJ47qvbDnyw9Aj/+SKBrqIyAAFixxStV0/nZxILG1rw/FSVD68IE5s4+FUepD10EZEByrym/Nv3b2P5/dvw/KNVftuiWaxZeg7fWzizx7FzJo/jnsvMmD8q6VOgiIlnI3CdfsSDGstsfpyJi7D2U4KRRlXTEPc46eXReqzxIhS4ikoVglTvnePDPuznYnlWTvoQRTJ4winvRZuXBm3qs8SIUuItKH3qp86e1tRELGvo4kYTMWvGMidz3+Ilecf0aPPfV8VXmQnCl1EpA/BKvd8xy83vcShLo99HUmiYeP2xWdzzYfPoqFuFt998Jke15Tnq8qDVOgiIgG9VvnqNszgnYJdHOGTMmDiGbX87gKWTuFBFnkmFLiISEKzyhOfz4z+9yKG4x8Euj2jY+P7is7n7E3MKtk/+elTonIlL2eqvyS1e3gXMcTvhEQkasegxPvXQALPU5Q6XKg1ToIlL2glXemfBYu/45Dsc9Did8KsIhbl9ynNndeNjSrPEiFLiJlqa8q931HZ9InGjZi1VVs3bW/+3OGYpUHqdBFpCwFq/xAZ4KmR3ZwOO7RmfSpniIRYvfhsfvTv5w75Kg9SoYtI2eitypc0t+H5PnHPEQ0bM6ur2FxEVR6kQheRshGs8r2H4nznge10nJDzinuuu8h8UWZUHqdBFpKQdU+XzYyxubiXpOZJ+apDPrB7Dpp3FWeVBKnQRKWnBKv/b/k6+df82nOhM+Sd9RGQnRXD+bO5YVb5UH9TvQzWyimT1gZlvMbLOZfaqXY0ab2U/N7Mn0MfWDs1wRkf6tfGh7nj8G8Yn6MxU2tnPuN3/L4s69SGQkx5/S/oyJydAQGq7xYZVPoSeAzzrkpwDnAJ81sSsYxnwS2OOemnA+8CvmVmFTldqYhIloJV/twrh/naz7fSmfTxHVRGQjTVz2ZtiVR5UL976M65XcCu9NsHzGwrcDKwnJXgYcKKZGTAS2EvqfwQiInmRuVd+07wY9U2txJM+jtQDmmuqq2jfWZh7lefDgH4oamaTgBiwPuNDnK4B7gZ3AicDFzjm/l89fBiwDqK6uHvhqRUT6EHyu59gRlXzp3k10JVNjqDISorF+dl6e61lIWf9Qn1MxGAncDn3bO7c/48HuBJ4AJwAxghZmNyvw9nHOrnHOznHOzxo8ffxzLFhE5dq/8hotnUNfUyntvneJjtuw8xLBpizuSxJbdX3pesBrqZRUkN8zXOuXW9HFIPrHMpTwN/Bd6eu2WKiBwruFf+5POv8fkfntxNPV/mwSIjGutmsLeAThPItm6tcDGgAtjrnlvdx2HPAe9LHnwS8DXgmV4sUETkis8qXXzidusZWnLrjlDzz/agcnRMPMnTyWaJlUeVA2e+hzgYVAu5k9kX7f1UA1gHNuJXAN0Gxm7aRuLnmVc25Pb7+ZniMjxCO6VG8ZV6zYS99JVHg3RUDer5PfK+5LNVS6P0H0H4D6P2Qmcn6tFiYgEZV7B8r8fm8aixhYSnnsOA4RVhYhPHsPHF0r2CJRt66b+IDHnBKu+Ie3z2zidJeA6AymiI2xaVb5UHaaCLyJCUWeVf/8iZn1Da0kPQdIVTlvdFAF5EhqbvK58fYfbCLL/xkE0k/VeUVqvJeaaCLyJAQLHJI1fYl51RzScN6fAchngxEVYWaoyvukuy2KyJAQvKbc9x3X3LeZm377dPfHKyIhbl00izVldF35QKnQRaRgenuC0NLb2wibnsb8zyaSxw3n1cIIzJ4xSlWdBhS4iBROs8qTnc9/GnRzq8tjfmSQ2cQz7O5N895IaVXmWVOgikle9nVvnqNszgYJdHJGTUnnsqd7Q8zxXnn9FjT11V/vo00EUkr4LXlM88tYo7217gUNwDIBo2VtefzZy3njOOfp5zE5Ws3MHVCzx+Uapj3TQNdRAZdb1V+6eo2nHN0JHwiIaOmegxbXzrQ/bp0FfnAaQ9dRAZdncK/8cDzJ9//4LIfjHh0Jn4pwiNuXnM2PLpujffLjpEIXkUHRV5V7vqMr6RMNGzXVVWzZdfTxCqryn46NCF5FBEazyfR0Jbn34GQ7HPbqSPhWREKsXn80P/730nutZSCp0EcmZ3qp8SXMbSd8n4TkqwiFmnnjqGTTtV5YNBhS4iOROs8j0Hu7jpt3+hI+GR8ByVkRDNi2dzxzJV+WBRoYvIcTmmyufHWNzcStLznSfqpl+zPrFaV54MKXUSOS7DKd77Wwf/+6ik6E6lhXhkJ0VyvKs8XFbqIDFhmld88L0Z9UysJz8d3nqUE+s7qK9p26/0o+aaCLyIAFX+3596OGcc19W+hKpp7rWRkJ0VQ/W/cqLwANdBHJSmaV3zhvBvVNnrcSTPo7UA5prqqto110RC0YDXUSyEqzyMSdU8MV7NnVX+bBIiMY6VXmhaaCLSJ8yq/zbF0+nrrGVnuJca5CdEw9RU6wlCQ4UGuoj0KVjllZEQn7u7vXuYD4uEaKjTcz2HEg10Eekhs8qvv3AaixpbSHgOnQ1U+lGmgi0gPwSpPeo4r79xIwnMAVEZV5UOZBrqIHFPl1320Z5UPrwgTm6gqH+o00EWkR5Xv70hwn1d3tPar8tkWq8mKggS5SpjKr/NoPT6W2oYWk7wiZqrwYaaCLlKnuKp8fY+e+Tr50zyaSfqrKKyKqn8mKkm3OJlImVD23vvjEWpGp7wdkTuaRhPf9555N0JDxGVISZO3ks0XCox3FHqlyGNg10kTIRvCuin7zu+fO9mVjyw/cgzmamIhLh10SzWLD1Hd0YsUtpyESlhvT1BaOnqNkJmHOhKcvq4Eew9FGfqhFHanKy8BKnSREhas8oTnc8+GnRyKexzoSjKzuorXOhJ855IaVXmJUKGLlJjeqvzS1W0AHI57RELGojmTnWLv+Oa44/4zuAleVFz8NdJESE7ymvKa6ih+0PM/huAdARTj1XM85k8fxnn94E5ev3cDUCaN7DHUNn8+KlgS5SAvqqct93dCZ9omEjNrGKrS/puZ6lTHvoIiUguFd+sCtJ8x92cDju0Zn0qQiHWL34bH50nmZ7rWer6LXQzmwjcDpwEOGCVc+7GXo57F3ADEAX2OOfOy+1SRSSotypf0tyG5/vEPUc0bNRUV7Flnl6q8XGSz5ZIEPuOc+5OZnQg8bmb3O+e2HDnAzMYA3wHe55x7zszeNEjrFZG04F75lDeP4rsPbqcjnkd4rj4Ro1nM9y06/A905twvYlX77gJltBU4GtgQOWwCsc849lz7u5UFYq0jZO6bK56eqPOH5JH1HnRSTEzOoxbNqpKi9HA9pDN7NJQAxYn/GhtwJVZvagmT1uZrV9fP4yM2szs7bdu3e/kfWKlLXgXvnLn+ztZ/pttdCQ8kr6jMl3ldyzTXnm5yvoqFzMbCdwNfNo5tz/jwxFgJvAe4ATgj2b2mHNuW/Ag59wqnYBXArFmz3PEsXKRc9Fbli5tbSSR9PAeV6SpvV5WXvawGuplFSQ3zNc65db0c8gLwinPuEHDIzB4GnpgPbejlWRAYguFc+sWo4X//5VjoTqed6VkZCNGmvXNL63XIxMwMagK3OueV9HHYP8I9mFjGz4cA7ngK25W6ZIeQneGXHO5HHcNC9GfVMr7/zfB9i0cz/DIiHmnD6WiojuiihHZVPoc4GFQLuZPZF+39VAnNYBzbqVzbquZ/RLYCPjAbc65TYOxYJFyEKzy8SMr+cpPN9GVPFrljapy6UU2V7k8At132Hy9474JnfDMXixIpR5l75TfOm0FdU2qv3AHDoiFqqqto110RpQ966b/IEBGs8pGVEa5e1048XeXDIiEa61Tln8vo00EUKKLPKl180nbrGVuJeapCfEA1TU63nekp2NNBFCihY5SEzrrp7Y/cwHxYN0VCn53pK9jTQnRfIss8q/+bFpLGpsIeE5DBheESY2UVUuA6eBLpJnwSrvSvh89q4nSXip19lVRkPctkhVLm+MBrpInHmRW+Tc+eha1DS0kfUcIVbnkhga6SB50V/n8GK8civP5H7eT9FNVXqEqlxzRAy5EBknmqz2vuWAqntY0t/McdGzjYlWRERZi5k8cSDevVnpIbKnSRQRKs8uf2HuYrP918tMojIW5VlUuOqdBFciizyv/7ng1NY2NjCf61rpyvpM6JSVS6DR4UukkNHqvymeTGeemk/1/3yKbxgldeqymXwqNBFjkOwyCFV2x+rnOYXaxvVc+7Ot+M4xsjKiKpe80EAXOQ7BJwjFkz6f+dETrPr9M0RCqfvZRcMhVtXOZM3Sc/QUIRl0n2nIRGaBjniC0IMalq9twztGR8Jk6YRQvvtbB7DeP0nXlklcqdJEBClZ5R9xjzWPPcjju0ZHwOff0nv2PXvk6+8/EaVbnknQpdJAt9VbnnO7qSPtGwsXjuadz+x2e54vwzugtcVS75pIEukoXg/VfOPHk0nt/3+rxyOe0Dq6pXm9BOEznvbeC5fu4GpE0b3GOoa5pIPGugifeitypc0t5H0fRKeoyIcoqZ6DJt3n7e/+HBW5FJL20EX6ENwrf+VgFyt+9zQdCY+E56iMhGhePJsf/Pu52ieXIUOFLhJwTJXPj7G4uZWkn50j6jopIiJnVY9i0U1UuQ48KXSQgWOW79nXwzV/9mc6ET9JPV3n9bO5YpiqXoUmFLmUvs8pvnhdjncVPquZ6+g8pIiJnVVbTv1DXlMrSp0KXsBat8x55DXPvzLXQmjw7zpvrZrF2ma8pl6FOhS1nKrPIbn582gvqmVeNLHAcMiIWpOraJdr/SUIqKBLmUpeF151fAKvnTPZrqSPpAa5o3p68p1V0QpJhroUjYynq/zbF0+nLl3lAMOiIWqqVeVSvDTQpWwEq3xYNMzV69qPDvNIiMY6VbkUNw10KWmZVf6tC6exqLGFnhOcw4IRomJrqMboropQEDXQpacEq93347F0bSXipJwhVRkM01OkJQlI6NNCl5GRW+XX/1rPKh1eEniU1UlUvp0UCXkhOs8gOdSa7KqPLbFqnKpTRpoEtJyKzyr3/kTGobWkj6jhCqcikPGuhSErqrfH6Mnl/Z38sV7NpH0U1VeoSqXMqGX/kvRWvnQ9h4vw//yh6aysLGFK370JB1xjxEVYeZOHks0fPSPebDKnRUqNCl2K1pEqv3lejGf2HORrP9uKd6TKIyFuVZVLmVGhS1HJrPLPf+Dt1Da28MV7NhP3fUZUqsqlnfKnQpagcqfIb581g04v7+dav/4zn0lUeDnFrrapcyle/hW5mE83sATPbYmabzexTr3PsbDNLmtnHncrtMKVfBIodUbX8kNoHaxhau++VTOBwjKyOqchGy23JJAp9xzk0BzgE+aWZTMg8yszBwHfDr3C5RnylnwXuWdCY9P/WADDY/sIBpK/dGNhkOsqp3JmqW6X7lIv1suzrldwK702wfMbCtwMrAl49D/AO4GnZud6kVJejnmu54IYl65uw3eOzoTPtFNG8fzeDma/uUrXlYsEDOiHomY2CYgB6zPefzLwEeC7/Xz+nMjNrM7O23bt3D2ylUjaCVX6oK8nqR3dwOO7RmfCZM3ksL7zayS0fr1GVi2TIeqCb2UhSBf5p59z+njA/fAFzlnPNf7/dwzq1yzs1yzs0aP378wFcrJSvz6pUVC2IsaW4j9tVf86vNfyMaNi4773Q2PPcann3jX6d0Frr1ykaOyusrFzKKkhvka59y6Xg6ZBfzAzADGAR8ws6Rz7ic5W6mUtOD9V6a+eTQrH9pOnR8IDUteUN6efIPTOt47n8rUbmDphdI+hri0WkSwGuqWmdAOw1Tm3vLdjnHOnBY5vBu7TMJf+9LZXnvqS5jaTnk/AdFeEQM08dw6adR/9CqH1ykb5lU+hzgYVAu5k9kX7f1UA1gHNu5SCtTUpcsMrf8qaRn3HD/tu4qr4yEaNJzPUUGJJurXB4BLNvf0DlXdzwLktJ2TJXPj7G4uZVE0sdzqUFeU60qF3kj9NJ/nyavgFSzP7z3MN36xlc7E0WHeVD+bO5adq6tXRN4AvfRfBl1mld88L0Z9UysJz8dPD/KZ1VW079Q1n5SLHQwNdBl1wr/xNJw7jK/dtpiuZusJVe+UiuaOBLoMis8pvnDeD+qZW4kkfBwyLhqiprqJdr/QUnyRkNdBkUwSofNSzKF368qbvKh0VCNNapykVyTQNdciazypdfNJ26xlbiXmqQnxANU1Ot53qKDBYNndMmZYJVHwyE+d3d79zAfFgnRUKd7lYsMJg10OS6ZVX79hdNY1NhCwnMYqnKRfNJAl+MSrPKupM9nn73yShJd6glBlVFUukk8a6DJgmVX+Px89i9qGFpJ+qsqHV4SJTVSVi+SbBroMWHeVz4/x6uEEn1u3nkaR/tMpvW6QqFykEvfRfspJ5v/JrL5hKbWMLn1z7Jw52JRlREdZzPUUKTIUuWQlW+QuvdfDlezZ1nV3lFJMStqnKRglOhS58yq/xLH5zCwsYWrrxrI51JX1UuMsSo0KVPR6r8pnkxtv3tAP/zi6fwVOUinQ5YKXXrIrPLPvf/t1Dau56v3bcFzPiMrI6pykSFKhS49HKnyGy6ewRPPv8ZNv/0L6SgnGg6xqnamnqlxkiFKhyzFVfsW/nEFdUwvL798GoCoXKRIqdOmu8uUXTeeP219h1cPPkI5yImFTlYsUCRV6GQoWnOaRq+1/P+nvqm1r53sPPEAmZqlykCGmgl6Hgcz0PdCZY0tzK9x97jhMqwgCE01W+Zuk5eranSBHRnlkuZyLz/yooFMZY0t+H5PnHP8Y5JVWx7+aDuwSJSxFToZSJY5XsPxbnld0/TkfCIe47z3jqOv+w+nxC0fr1GVixQxFXoJO6bK58dY3NxK0nMkfUdFJMTSfzyNxj/s4Irzz+gucFW5SHHSQC9hwXuVTx4/nkut/vY3OROoJQpWREE31qed6zj1jHJev3cDUCaN7DHUNc5HiooFeYnqt8qbUcz19lxrkM6vH0L5znf/fnqMhFSoP20EtMcK/82VcOce3PttKZPDrMm+pns3bZudonFylBKvQSkFnlN82LUd/USjzp40g9noLmmuor2nbp6RaSUaaCXgOBe+dgRlXzp3k10JY/ulTem98r1Sk+R0qaBXqQyq/yGi2dQl65ygGHRndJXrmnKRsqGBXqSCVT68IsLVP954dJhHQjTWqcpFyo0GehHJrPLlF06nrrGFuJe6ldYJ0TA11Xqlnp0i50kAvIsEqB7jy7o3dw3xYNERDnZ4gJFLONNCHuMwqv+7fprGosYWE5zBgeEVY918REUADfcgLnVvmhLo8r73qSRLrKK6MhbtNzPUUkTQN9CMqs8q9/5ExqG1pI+o4QqnIR6Z0G+hDUXeXzY7x8oIsvn/KSdZPrBnhWqchHpQ78v/TeziWb2gJltMbPNZvapXo75uJltNLN2M3vUzKYPznJLV+ZzPb96wVRqnG1v49A+f4HDcY0RFWE8QEpHXlU2hJ4HPOOf+ZGYnAo+b2f3OuS2BY/4KnOece9XM3g+sAt4xCOstnWUeq/OZ5MXbsPcQ19205WuWRELeqykWkH/0WunNul3PuT+m3DwBbgZMzjnnUOfdq+pePAafkeqGlnKLPKv/Cv/0BtYwuf//Em4kmfEZWqchHJ3oD20M1sEhAD1r/OYUuAX/Tx+cuAZQDV1dUD+dIl6UiVn3zhvBlt27uf6X/0ZzwWqvFZVLiLZy/r2uWY2Ergb+LRzbn8fx7yb1EC/qrePO+dWOedmOedmjR8/n/o2st+hlVvmV730bixpb+MYvnsLHMbIyoioXkTckq0I3syipYb7GObeuj2OmAbcB73fOvZK7JZaWnI1X+7Yun8/iOV1nxwNOkt8qJhkOsqp2pKheRNySbq1wMaAC2OueW93FMNbAOWOic25bbJRa/zCr/n1D+fQX1TKzf97mlCZqpyEcmJbAp9LrAQaDezJ9LvuxqoBnDOrQT+GxgLfCc1/0k652blfrnF6UiVnf+vCafz+L6/Q+Ie/dn8sEjZVuYjkRL8D3Tn3CGD9HHMpcGmuFlXsgq/0hFRtv2/qSSxubsMB0ZBRnGQ0z/ZTRerWniOSMnik6CILP9dx3OEFd43rWtjzPiIowAOF0la9Zeo6e7SkiOaOX/udI5v1XViyInsaS5jaTnk/AdcyaPZeuu/czVPVhEZJCo0HMkWOW7D3Rx42/+QkfCI+E73v228Tz10gFu+XiNqlxEnBo0K/TgcU+XzYyxubiXh+Xh+6gHNl/7TaTQ+soMrzj+jx566qlxEck0D/TgE71V+6tgRXPerp+hMnpJ7rWRkJ0VSfeq7n3LeM4/K1G5g6oecPSjXMRSSXNNAHKLPKb54Xo74pVeW+Sw3ymdVVtO/UPrmIn5JcG+gAFq/ykUcO45r4tdCWPrXJdUy4i+aaBnoXMKr9x3gzqm1qJJ30cqQc011RX0a6rV0SkgDTQnsxCs8tEnRPniPZu6q3xYJERjnapcRApPA70PmVW+/KLp1DW2EvdSg/yEaJiaal1TLiJDhwZ6H4JVnXhEOcfW69u5hPiwSoqFO9yoXkaFFAz0gs8qvv3AaixpbSHgOQ1UuIkObBnpAsMqTnuPKOzeS8FI3nK6+MqspFZGgr+4GeWeXXffSsHlU+vCJMTPdfEZEiUPYDPVjl+w4nuOru9h5VftsiVbmIFIeyvDlXn5hOErv3wVGobWvjEmj9xsCvB8IqwniAkIkWnLAu9u8rnx3jxtQ6+fO9mkukHe1ZEVOUiUpzKptAznq/xLH5zCwsYWPnvXRjoSHiNU5SJS5Mqm0I9U+c3zYvzl5QN84xdP4QWq/FZVuYgUuZIu9Mwqv/r9nb6e2cT1f/ukWkr7PiEpVuYiUjpIu9CNVfuO8GWx8YR/fvn8b6QtYiIZD3FqrKheR0lFyhZ5Z5f95n/ltZ1NjCN3/1ZwBGVkZU5SJSkkqu0I9U+fKLprP+r3v53kPbSW+VEwkbq2pnqspFpCSVRKFnVvnln734Li5tb+e6D2wmbqcpFpCyURKEfqfLrL5zGA0/t5vuPPdv9sbCqXETKRFEWerDIIVXb/zLlJJY0nt/H9x54lGlKVi0j5KcqBfqTIH92+h1cPxVl423p+2Po8IyrDwNEqX7P0HL63cGb3sZAa6pedN7mQnyxcRGRRFM9Az98lXzI+xpLmN2V/7Db9/eg//+JZUjavKRaRcFc1AD1b53/Z3svz+bXQkPJK+4/+8nfTxbdh3glo/XqMpFpGwVzQ9Fj5T2stsfJ570iHuOykiIpf90Gg2P7OCK88/o/gGn7lcuIuWoaAY6npAb1R2Mnc/tjz1IZCdFUPzt1pcpbxnH52g1MnTC6x1DXMBeRclI0Wy4Aj27fw33tu5g7eSwVEe2Tni4gEFU2hZ15DrmvKRUR6KppC3/jCvu7hDapyEZFM5pwryBeeNWuWa2trK8jXFhEpVmb2uHNuVm8fnK5pCFxGR16eBLiJSIjTQRURKhAa6iEiJ0EAXESkRBbvKxcx2A8/2e2DvxgF7+j2q9JTjeZfjOUN5nnnc5njMM/LxPdc6N7+0DBRvox8PM2vq6bKeUleN5l+M5Q3medzmeM+T2vLXlIiJSIjTQRURKRLEOn9FWFXkCBlON5l+M5Q3medzmeM+TwvItyD11ERI5VrIUuIiIZNNBFRErEkB7oZvY+M/uzmT1tZv/Vny8crzeyH6Y+vN7NJ+V9l7mVx3leY2RYz22hmvzWzUwuxzlzq75wDx/2bmTkzK4nL27I5bzO7KP39n3mxma/O9xlzL4s93tZk9YGYb0n/GP1CIdeaSmTWa2ctmtqmPj5uZ3ZT+d7LRzGre0Bdyzg3Jf4AwnsB04HagAngSmZBzzf4GV6bfnAT8s9LrzdN7vBoan3/5EsZ93NuecPu5E4GHgMWBWodedp+/1GcAGnoCr96zcVet15OOdVwCfSb08BdhR63Tk473cCNcCmPj7+AeAXgAHnAOvfyNcZyoV+NvC0c+4Z51wcn+AFwQcYxFwCr02/fBbzHzCyPaxwM/Z63c+4B59zh9C8fA07J8xpzLZvvNcA1wHVAZz4XN4iyOe+lnwC3OuVcBnHMv53mNuZbNOTtgVPrt0cDOPK5vUDjnHgb2vs4hFwC3u5THgDFm9uaBfp2hPNBPBp4Pn/PqF9Pt6PcY5lwT2AWPzsrrBk815By0h9X/2YtbvOaf/CjrROfezfC5skGXzvX4r8FYz+4OZPWZmn78vb6gZHNuf8ZeASM3sB+DnwH/lZWkEN9L/7XhXNM0XlWGZ2CTALOK/QaxlMZhYClgN1BV5KIURInbbu8i9TfxB42s7Occ68VdFWDaz7Q7Jz7lpmdC3zfzM50zvmFXthQN5QL/UVgYuDXp6Tf1+sxZhYhn9dezV/KyusGTzXljZv8MfJw9vl0AAAFtSURBVB74kHOuK09rGyz9nfOJwJnAg2a2g9Qe470l8IPRnbL7XLwD3OucSzrm/AttIDfhilc05LwF+BOCc+yMwjNQNrEpZVv/d92coD/RW4AwzO83MKkj90PPenjGPuBRal3/4Y8DuX/glDEev3vM0sBnyP1DAv9j1V6OecnXP7nHPjnHOTnHOTSP3c4EPOuWJ/KG02nf8Z/QqrOMbNxpLZgnsnnInMsm3N+DngPgJn9A6mBvjuvq8y/e4Ha9NUu5wD7nHO7Bvy7FPqnv/38nZPgDpIpkO/D59Pu+Suo/Zkh9o+8EngZagNMLveY8nfdvgL8BT6T/ubfQax7sc8449kFK4CqXLL/XnRmq7aQvQDswr9JrzcM5TgD+QugLmCeD8Qq85B+d8B7ALSJD6W9cS4DLgssD3+Zb0v5P2N/rnWy/9nFxEpEUN5y0VERAZAA11EpERooIuIlAgNdBGREqGBLiJSIjTQRURKhAa6iEiJ+P/Af4EG/lmQAQAAnAABJRU5ErkJggg==n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"Nonen"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plt.clf()n",
"plt.plot(x.makeNumpyArray(), y2.makeNumpyArray(), marker: "x")n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Import CoreML ProtoBuf Swift data strucures"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/ArrayFeatureExtractor.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/BayesianProbitRegressor.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/DataStructures.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/CategoricalMapping.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/CustomModel.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/DictVectorizer.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/FeatureTypes.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/FeatureVectorizer.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/GLMClassifier.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/GLMRegressor.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Gazetteer.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Identity.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Imputer.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Scaler.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/ItemSimilarityRecommender.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Parameters.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Normalizer.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/LinkedModel.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/NearestNeighbors.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/NonMaximumSuppression.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/OneHotEncoder.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/SVM.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/SoundAnalysisPreprocessing.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/TextClassifier.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/TreeEnsemble.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/VisionFeaturePrint.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/WordEmbedding.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/WordTagger.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/NeuralNetwork.pb.swift""
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"%include "./CoreMLProto/Model.pb.swift""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Export CoreML Model with Weights, Bias and Trainable Parameters"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [],
"source": [
"let coreModel = CoreML_Specification_Model.with {n",
" $0.specificationVersion = 4n",
" $0.description_p = CoreML_Specification_ModelDescription.with {n",
" $0.input = [CoreML_Specification_FeatureDescription.with {n",
" $0.name = "dense_input"n",
" $0.type = CoreML_Specification_FeatureType.with {n",
" $0.multiArrayType = CoreML_Specification_ArrayFeatureType.with {n",
" $0.shape = [1]n",
" $0.dataType = CoreML_Specification_ArrayFeatureType.ArrayDataType.doublen",
" }n",
" }n",
" }]n",
" $0.output = [CoreML_Specification_FeatureDescription.with {n",
" $0.name = "output"n",
" $0.type = CoreML_Specification_FeatureType.with {n",
" $0.multiArrayType = CoreML_Specification_ArrayFeatureType.with {n",
" $0.shape = [1]n",
" $0.dataType = CoreML_Specification_ArrayFeatureType.ArrayDataType.doublen",
" }n",
" }n",
" }]n",
" $0.trainingInput = [CoreML_Specification_FeatureDescription.with {n",
" $0.name = "dense_input"n",
" $0.type = CoreML_Specification_FeatureType.with {n",
" $0.multiArrayType = CoreML_Specification_ArrayFeatureType.with {n",
" $0.shape = [1]n",
" $0.dataType = CoreML_Specification_ArrayFeatureType.ArrayDataType.doublen",
" }n",
" }n",
" }, CoreML_Specification_FeatureDescription.with {n",
" $0.name = "output_true"n",
" $0.type = CoreML_Specification_FeatureType.with {n",
" $0.multiArrayType = CoreML_Specification_ArrayFeatureType.with {n",
" $0.shape = [1]n",
" $0.dataType = CoreML_Specification_ArrayFeatureType.ArrayDataType.doublen",
" }n",
" }n",
"n",
" }]n",
" $0.metadata = CoreML_Specification_Metadata.with {n",
" $0.shortDescription = "Trivial linear classifier"n",
" $0.author = "Jacopo Mangiavacchi"n",
" $0.license = "MIT"n",
" $0.userDefined = ["coremltoolsVersion" : "3.1"]n",
" }n",
" }n",
" $0.isUpdatable = truen",
" $0.neuralNetwork = CoreML_Specification_NeuralNetwork.with {n",
" $0.layers = [CoreML_Specification_NeuralNetworkLayer.with {n",
" $0.name = "dense_1"n",
" $0.input = ["dense_input"]n",
" $0.output = ["output"]n",
" $0.isUpdatable = truen",
" $0.innerProduct = CoreML_Specification_InnerProductLayerParams.with {n",
" $0.inputChannels = 1n",
" $0.outputChannels = 1n",
" $0.hasBias_p = truen",
" $0.weights = CoreML_Specification_WeightParams.with {n",
" $0.floatValue = [weight]n",
" $0.isUpdatable = truen",
" }n",
" $0.bias = CoreML_Specification_WeightParams.with {n",
" $0.floatValue = [bias]n",
" $0.isUpdatable = truen",
" }n",
" }n",
" }]n",
" $0.updateParams = CoreML_Specification_NetworkUpdateParameters.with {n",
" $0.lossLayers = [CoreML_Specification_LossLayer.with {n",
" $0.name = "lossLayer"n",
" $0.meanSquaredErrorLossLayer = CoreML_Specification_MeanSquaredErrorLossLayer.with {n",
" $0.input = "output"n",
" $0.target = "output_true"n",
" }n",
" }]n",
" $0.optimizer = CoreML_Specification_Optimizer.with {n",
" $0.sgdOptimizer = CoreML_Specification_SGDOptimizer.with {n",
" $0.learningRate = CoreML_Specification_DoubleParameter.with {n",
" $0.defaultValue = 0.03n",
" $0.range = CoreML_Specification_DoubleRange.with {n",
" $0.maxValue = 1.0n",
" }n",
" }n",
" $0.miniBatchSize = CoreML_Specification_Int64Parameter.with {n",
" $0.defaultValue = 1n",
" $0.set = CoreML_Specification_Int64Set.with {n",
" $0.values = [1]n",
" }n",
" }n",
" $0.momentum = CoreML_Specification_DoubleParameter.with {n",
" $0.defaultValue = 0n",
" $0.range = CoreML_Specification_DoubleRange.with {n",
" $0.maxValue = 1.0n",
" }n",
" }n",
" }n",
" }n",
" $0.epochs = CoreML_Specification_Int64Parameter.with {n",
" $0.defaultValue = 100n",
" $0.set = CoreML_Specification_Int64Set.with {n",
" $0.values = [100]n",
" }n",
" }n",
" $0.shuffle = CoreML_Specification_BoolParameter.with {n",
" $0.defaultValue = truen",
" }n",
" }n",
" }n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"let binaryModelData: Data = try coreModel.serializedData()"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [],
"source": [
"binaryModelData.write(to: URL(fileURLWithPath: "./s4tf_model_personalization.mlmodel"))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Swift",
"language": "swift",
"name": "swift"
},
"language_info": {
"file_extension": ".swift",
"mimetype": "text/x-swift",
"name": "swift",
"version": ""
}
},
"nbformat": 4,
"nbformat_minor": 2
}
You can run this notebook on Google Colab or install the S4TF toolchain and the Google Swift Jupyter Kernel on Linux or a Docker container from the following Google repo:
Third step: Compile Core ML model on demand and test / inference the model
Xcode provides a very nice native integration with Core ML, where you can easily drag and drop a Core ML file ino the project to automatically generate Swift wrapper code for the model and input/output features.
The Core ML API allows you to dynamically load a model from the file system or even from the network, compile the model on demand, and dynamically create features and convert from basic datatypes.
The following code snippet shows how to dynamically load, compile, and run inference on a Core ML model in any Swift project:
import Foundation
import CoreML
func compileCoreML(path: String) -> (MLModel, URL) {
let modelUrl = URL(fileURLWithPath: path)
let compiledUrl = try! MLModel.compileModel(at: modelUrl)
print("Compiled Model Path: (compiledUrl)")
return try! (MLModel(contentsOf: compiledUrl), compiledUrl)
}
func inferenceCoreML(model: MLModel, x: Float) -> Float {
let inputName = "dense_input"
let multiArr = try! MLMultiArray(shape: [1], dataType: .double)
multiArr[0] = NSNumber(value: x)
let inputValue = MLFeatureValue(multiArray: multiArr)
let dataPointFeatures: [String: MLFeatureValue] = [inputName: inputValue]
let provider = try! MLDictionaryFeatureProvider(dictionary: dataPointFeatures)
let prediction = try! model.prediction(from: provider)
return Float(prediction.featureValue(for: "output")!.multiArrayValue![0].doubleValue)
}
let (coreModel, compiledModelUrl) = compileCoreML(path: coreMLFilePath)
let prediction = inferenceCoreML(model: coreModel, x: 1.0)
print(prediction)Fourth step: Local Core ML training / personalization
The final step is all about re-training the previously compiled Core ML model by passing a batch of training data.
The generateData Swift function is provided using Swift-based datatypes, i.e. the same noised linear data tensor used in the previous Swift Notebook for training the S4TF model and getting the initial weights and bias:
import Foundation
import CoreML
func generateData(sampleSize: Int = 100) -> ([Float], [Float]) {
let a: Float = 2.0
let b: Float = 1.5
var X = [Float]()
var Y = [Float]()
for i in 0..<sampleSize {
let x: Float = Float(i) / Float(sampleSize)
let noise: Float = (Float.random(in: 0..<1) - 0.5) * 0.1
let y: Float = (a * x + b) + noise
X.append(x)
Y.append(y)
}
return (X, Y)
}
func prepareTrainingBatch() -> MLBatchProvider {
var featureProviders = [MLFeatureProvider]()
let inputName = "dense_input"
let outputName = "output_true"
let (X, Y) = generateData()
for (x,y) in zip(X, Y) {
let multiArr = try! MLMultiArray(shape: [1], dataType: .double)
multiArr[0] = NSNumber(value: x)
let inputValue = MLFeatureValue(multiArray: multiArr)
multiArr[0] = NSNumber(value: y)
let outputValue = MLFeatureValue(multiArray: multiArr)
let dataPointFeatures: [String: MLFeatureValue] = [inputName: inputValue,
outputName: outputValue]
if let provider = try? MLDictionaryFeatureProvider(dictionary: dataPointFeatures) {
featureProviders.append(provider)
}
}
return MLArrayBatchProvider(array: featureProviders)
}
func train(url: URL) {
let configuration = MLModelConfiguration()
configuration.computeUnits = .all
configuration.parameters = [.epochs : 100]
let progressHandler = { (context: MLUpdateContext) in
switch context.event {
case .trainingBegin:
print("Training begin")
case .miniBatchEnd:
let batchIndex = context.metrics[.miniBatchIndex] as! Int
let batchLoss = context.metrics[.lossValue] as! Double
print("Mini batch (batchIndex), loss: (batchLoss)")
case .epochEnd:
let epochIndex = context.metrics[.epochIndex] as! Int
let trainLoss = context.metrics[.lossValue] as! Double
print("Epoch (epochIndex) end with loss (trainLoss)")
default:
print("Unknown event")
}
}
let completionHandler = { (context: MLUpdateContext) in
print("Training completed with state (context.task.state.rawValue)")
print("CoreML Error: (context.task.error.debugDescription)")
if context.task.state != .completed {
print("Failed")
return
}
let trainLoss = context.metrics[.lossValue] as! Double
print("Final loss: (trainLoss)")
let updatedModel = context.model
let updatedModelURL = URL(fileURLWithPath: retrainedCoreMLFilePath)
try! updatedModel.write(to: updatedModelURL)
print("Model Trained!")
print("Press return to continue..")
}
let handlers = MLUpdateProgressHandlers(
forEvents: [.trainingBegin, .miniBatchEnd, .epochEnd],
progressHandler: progressHandler,
completionHandler: completionHandler)
let updateTask = try! MLUpdateTask(forModelAt: url,
trainingData: prepareTrainingBatch(),
configuration: configuration,
progressHandlers: handlers)
updateTask.resume()
}
train(url: compiledModelUrl)
// easily wait for completition of the asyncronous training task
let _ = readLine()
let retrainedModel = try! MLModel(contentsOf: URL(fileURLWithPath: retrainedCoreMLFilePath))
let prediction = inferenceCoreML(model: retrainedModel, x: 1.0)
print(prediction)
Full source code for this end-to-end test is available in the following repo:
Next steps
The next step will be to automate the S4TF-to-Core ML export process, defining specific Swift for TensorFlow extensions to generalize the process. Stay tuned, and please contact me if you would like to contribute. Thanks!
Comments 0 Responses