OpenCV is a powerful tool for doing intensive computer vision operations in real-time, even on resource-limited mobile devices. Throughout a couple of tutorials, we’re going to create an Android app that applies various image effects. Some of these effects might use just 1 image, while others might use more than 1 image.
This tutorial will discuss horizontal and vertical image stitching. The sections covered in this tutorial are as follows:
1. Preparing OpenCV for Android
4. Saving Images to the Gallery
5. Uploading Images from Android to a Flask Server
The source code for this tutorial is available in this GitHub project. Let’s get started.
Preparing OpenCV for Android
Despite being essential for running the code in this tutorial, we won’t discuss linking OpenCV to Android. The reason for this is that we’ve covered this in details in a previous tutorial titled A Guide to Preparing OpenCV for Android.
It is highly recommended to go through this tutorial because it covers all the steps required for making OpenCV functional for building Android apps, from building an Android Studio to building a simple app.
If you don’t want to read the previous tutorial at the current time, then at least download the Android Studio project built in the previous tutorial so you can get started with OpenCV quickly. The project is available on GitHub.
Image Stitching
Stitching images is a technique that stacks multiple images together to create a panoramic image. Stitching has different styles. For example, images might be stitched horizontally so they appear side by side. Stitching can also be done vertically, stacking images on top of each other. Stitching can also produce 360 panoramas, spherical panoramas, and more.
Stitching 2 images has the challenge of determining the best points at which the 2 images can be stitched. So you have to search for such points in the 2 images and then use them as intersection points. The stitched image should make a user feel that it was captured as a single image. When stitching is done poorly, it’s clear that there are cuts in the connection between the different images.
In this tutorial, we’re going to handle the most straightforward way of image stitching—concatenating 2 or more images together horizontally or vertically. The next section discusses horizontal stitching.
Horizontal Image Stitching
Horizontal image stitching aligns 2 images side by side. This tutorial uses the hconcat() method in OpenCV for this purpose. This method is available in the org.opencv.core.Core class. The header of this method accepts 2 arguments, as given below.
The first argument is a Java List of OpenCV Mat arrays collecting the images to be concatenated together horizontally. The first Mat array in the list appears at the leftmost side of the stitched image, and the last Mat array in the list appears at the rightmost side of the stitched image.
The result of the concatenation is returned in the second argument, which is an OpenCV Mat where all images are concatenated horizontally. Note that you have to create an empty Mat to accept the result.
Here’s an example, assuming that there are 3 resource images with IDs im1, im2, and im3 to be concatenated horizontally.
The first few lines decode the resource files using the BitmapFactory.decodeResource() method, which returns the results as Bitmap images. Note that there is an instance of the BitmapFactory.Options class to define the decoding options.
The only option defined is that the inScaled flag is set to false. The reason for this is that the decoded images are rescaled to match the graphics system scaling of the device. To force loading the images in their original size, set inScaled to false.
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false;
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Mat dst = new Mat();
List<Mat> src = Arrays.asList(im1, im2, im3);
Core.hconcat(src, dst);Because hconcat() accepts a list of Mat arrays, then each Bitmap image must be converted into a Mat. Fortunately, OpenCV supports a method named bitmapToMat() to convert a Bitmap into a Mat. It’s available in the org.opencv.android.Utils class.
The header of this method is given below. It accepts 2 arguments. The first one is the Bitmap to be converted into a Mat. The second argument is the destination Mat in which the result will be saved.
Note that you have to prepare an empty Mat and feed it to the bitmapToMat() method to receive the result of the conversion. This is similar to the hconcat() method. It’s normal to see that the variable to be returned from an OpenCV method is fed to it as an argument.
In the above code, there are 3 Mat arrays created, which are named img1, img2, and img3. They accept the result of the conversion of the Bitmap images.
After preparing the Mat arrays, next is to insert them into the List of Mat arrays. The above code creates a list named src for that purpose.
Finally, the 3 Mat arrays are concatenated horizontally using the hconcat() method, and the result is saved into the dst Mat.
The 3 resource images are shown below.
The result after running the previous code is shown below.
After understanding how this works, let’s build a complete Android app that’s fed 3 images and returns their horizontally stitched image.
Horizontal Stitching Implementation
The XML layout of the main activity is listed below. It contains a LinearLayout as the root view. It holds 2 child views. The first one is a Button. When clicked, a method named stitchHorizontal() is called for stitching images. The second view is an ImageView, which displays the stitched image.
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical"
tools:context=".MainActivity">
<Button
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:text="Stitch Images Horizontally"
android:id="@+id/stitchHorizontal"
android:onClick="stitchHorizontal"/>
<ImageView
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:id="@+id/opencvImg"/>
</LinearLayout>The activity window is given in the next figure.
After discussing the implementation of the activity XML layout, next we need to discuss the implementation of the activity itself. The activity contains the method stitchHorizontal() that will be called when the Button view is clicked. The summary of this method is as follows:
- Preparing the images to be stitched together.
- Stitching the images using the stitchImagesHorizontally() method.
- Saving the stitched image using the saveBitmap() method.
Let’s start discussing the implementation of this method, which is listed below.
As discussed previously, the method starts by decoding the resource files and returning them as Bitmap images. The Bitmap images are then converted into Mat arrays using the bitmapToMat() method.
public void stitchHorizontal(View view){
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false;
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Bitmap imgBitmap = stitchImagesHorizontal(Arrays.asList(img1, img2, img3));
ImageView imageView = findViewById(R.id.opencvImg);
imageView.setImageBitmap(imgBitmap);
saveBitmap(imgBitmap, "stitch_horizontal");
}There’s a method named stitchImagesHorizontal() that does the actual stitching. It accepts a List of Mat arrays as input and returns the stitched image as a Bitmap image. The stitched Bitmap image is then displayed on the ImageView. Finally, the returned Bitmap image is saved in device storage using the saveBitmap() method.
The implementation of the stitchImagesHorizontal() method is listed below. It accepts a list of Mat arrays as an argument. These arrays represent the images to be concatenated.
The result of this method is converted into a Bitmap image using the matToBitmap() array. Note that there is a variable named imgBitmap that’s an empty Bitmap image that holds the result of the conversion. Finally, the imgBitmap variable is returned to the stitchHorizontal() method.
Bitmap stitchImagesHorizontal(List<Mat> src) {
Mat dst = new Mat();
Core.hconcat(src, dst);
Bitmap imgBitmap = Bitmap.createBitmap(dst.cols(), dst.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst, imgBitmap);
return imgBitmap;
}Saving Images to the Gallery
The implementation of the saveBitmap() method is listed below. It accepts 2 arguments—the Bitmap to be saved and the file name as a String. Because there might be a previous image saved by this name, the name is appended to the current date and time.
The final filename is returned in the fileName String variable. After that, an instance of the FileOutputStream class is created. This instance is what will write the image in the storage. It isn’t yet initialized but will be later.
In order to save the image, we have to know what path to use. The path used for saving the images is returned using the Environment.getExternalStoragePublicDirectory() method. This method doesn’t only return a directory but a public directory that can be accessed by the user and other applications without having the device rooted.
There are different folders in the device to accept the saved images. For example, the DCIM folder accepts the images captured by the camera and the Download folder accepts the downloaded files.
I chose the Pictures folder for saving the images created in this tutorial. To specify that the path of the Pictures folder is to be returned, Environment.DIRECTORY_PICTURES is passed to the Environment.getExternalStoragePublicDirectory() method as an argument. The path variable accepts the path of that folder.
void saveBitmap(Bitmap imgBitmap, String fileNameOpening){
SimpleDateFormat formatter = new SimpleDateFormat("yyyy_MM_dd_HH_mm_ss", Locale.US);
Date now = new Date();
String fileName = fileNameOpening + "_" + formatter.format(now) + ".jpg";
FileOutputStream outStream;
try{
File path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES);
File saveDir = new File(path + "/HeartBeat/");
if(!saveDir.exists())
saveDir.mkdirs();
File fileDir = new File(saveDir, fileName);
outStream = new FileOutputStream(fileDir);
imgBitmap.compress(Bitmap.CompressFormat.JPEG, 100, outStream);
MediaScannerConnection.scanFile(this.getApplicationContext(),
new String[] { fileDir.toString() }, null,
new MediaScannerConnection.OnScanCompletedListener() {
public void onScanCompleted(String path, Uri uri) {
}
});
outStream.close();
}catch(Exception e){
e.printStackTrace();
}
}Note that images will be saved in the root directory of this folder. I prefer saving them in a subfolder within the Pictures folder to hold our images. The subfolder is named HeartBeat as given in the line below. The saveDir variable now holds the directory of this folder.
Note that if this folder is not created, then a FileNotFoundException will be thrown. To make sure that the HeartBeat exists, the following if statement is used. If the folder doesn’t exist, it will be created. Nothing will happen if it’s available.
After making sure the directory in which the image will be saved is available, next we need to concatenate the directory saved in the saveDir variable to the fileName variable, as given in the next line.
After that, we’re ready to write the Bitmap image according to the next 2 lines. At first, the instance of the FileOutputStream class is set to save the data into the file directory specified in the fileDir variable.
Then the Bitmap image data is written to that file using the compress() method of the Bitmap class. The compress() method accepts 3 arguments. The first one is the compression technique, which is set to JPEG. The second is an integer representing the quality of the saved image. 0 means the least quality, and thus the smallest size. 100 means the maximum quality, and thus the maximum size. The third argument is the instance of the FileOutputStream class.
At this time, the stitched image is created and saved. The last thing to mention is that the MediaScannerConnection.scanFile() method is used to list the saved image in the device gallery. Without it, the file won’t be listed in the gallery. You can still navigate the device folders to locate the images.
After discussing the implementation of the 3 methods—stitchHorizontal(), stitchImagesHorizontal(), and saveBitmap(), here’s the implementation of the main activity class named MainActivity.
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.media.MediaScannerConnection;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.support.v7.app.AppCompatActivity;
import android.view.View;
import android.widget.ImageView;
import org.opencv.android.OpenCVLoader;
import org.opencv.android.Utils;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import java.io.File;
import java.io.FileOutputStream;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import java.util.Locale;
public class MainActivity extends AppCompatActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
OpenCVLoader.initDebug();
}
public void stitchHorizontal(View view){
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false; // Leaving it to true enlarges the decoded image size.
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Bitmap imgBitmap = stitchImagesHorizontal(Arrays.asList(img1, img2, img3));
ImageView imageView = findViewById(R.id.opencvImg);
imageView.setImageBitmap(imgBitmap);
saveBitmap(imgBitmap, "stitch_horizontal");
}
Bitmap stitchImagesHorizontal(List<Mat> src) {
Mat dst = new Mat();
Core.hconcat(src, dst);
Bitmap imgBitmap = Bitmap.createBitmap(dst.cols(), dst.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst, imgBitmap);
return imgBitmap;
}
void saveBitmap(Bitmap imgBitmap, String fileNameOpening){
SimpleDateFormat formatter = new SimpleDateFormat("yyyy_MM_dd_HH_mm_ss", Locale.US);
Date now = new Date();
String fileName = fileNameOpening + "_" + formatter.format(now) + ".jpg";
FileOutputStream outStream;
try{
// Get a public path on the device storage for saving the file. Note that the word external does not mean the file is saved in the SD card. It is still saved in the internal storage.
File path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES);
// Creates a directory for saving the image.
File saveDir = new File(path + "/HeartBeat/");
// If the directory is not created, create it.
if(!saveDir.exists())
saveDir.mkdirs();
// Create the image file within the directory.
File fileDir = new File(saveDir, fileName); // Creates the file.
// Write into the image file by the BitMap content.
outStream = new FileOutputStream(fileDir);
imgBitmap.compress(Bitmap.CompressFormat.JPEG, 100, outStream);
MediaScannerConnection.scanFile(this.getApplicationContext(),
new String[] { fileDir.toString() }, null,
new MediaScannerConnection.OnScanCompletedListener() {
public void onScanCompleted(String path, Uri uri) {
}
});
// Close the output stream.
outStream.close();
}catch(Exception e){
e.printStackTrace();
}
}
}After clicking on the Button, the stitched image will be created, displayed in ImageView, and saved in the /Pictures/HeartBeat directory. The app screen is shown below.
Note that we used the MediaScannerConnection.scanFile() method to list the saved image in the gallery. The next figure shows that the HeartBeat folder is successfully listed in the gallery. The saved image is available within it.
The details of the saved image are listed below. You can see that its name contains the date and time it was created. Its size is 1920×570. If the sizes of the 3 individual resource images are equal, then the width 1920 will be divided by 3, and thus the size of each one of them will be 640×570. This is because the concatenation is horizontal. Vertical concatenation will be discussed in the next section.
Uploading Images from Android to a Flask Server
Note that we have saved the image in the Android device storage. If we need to send it to our PC, we can reuse a project created in a previous tutorial, titled Uploading images from Android to a Python-based Flask Server.
It creates a Flask server that accepts an HTTP request with an image file to be uploaded. There’s an Android app client that selects the image and builds an HTTP request using the OkHttp library. You can find the source code of both the client and the server in the Part 1 folder of its GitHub project.
At this point, the project for building an Android app that stitches images horizontally has been discussed. Next, we’ll slightly modify the project to allow vertical stitching.
Vertical Image Stitching
There isn’t much difference between vertical and horizontal stitching. The primary change is using the vconcat() (vertical) method rather than the hconcat() (horizontal) method.
Here’s the activity XML layout of the new application. A new Button is added that calls a method named stitchVectical() when clicked.
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical"
tools:context=".MainActivity">
<Button
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:id="@+id/stitchHorizontal"
android:text="Stitch Images Horizontally"
android:onClick="stitchHorizontal"/>
<Button
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:id="@+id/stitchVertical"
android:text="Stitch Images Vertically"
android:onClick="stitchVectical"/>
<ImageView
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:id="@+id/opencvImg"/>
</LinearLayout>The screen of the app is shown below after adding the new Button.
The implementation of the stitchVectical() method is listed below. There are 2 changes compared to the stitchHorizontal() method:
- Calling stitchImagesVectical() for vertically concatenating the images rather than stitchImagesHorizontal().
- Passing stitch_vectical as the file name for the saveBitmap() method rather than stitch_horizontal. Note that the saveBitmap() method won’t be changed.
public void stitchVectical(View view){
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false; // Leaving it to true enlarges the decoded image size.
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Bitmap imgBitmap = stitchImagesVectical(Arrays.asList(img1, img2, img3));
ImageView imageView = findViewById(R.id.opencvImg);
imageView.setImageBitmap(imgBitmap);
saveBitmap(imgBitmap, "stitch_vectical");
}The implementation of the stitchImagesVertical() method is listed below. The only change is using the vconcat() method rather than hconcat().
Bitmap stitchImagesVectical(List<Mat> src) {
Mat dst = new Mat();
Core.vconcat(src, dst); //Core.hconcat(src, dst);
Bitmap imgBitmap = Bitmap.createBitmap(dst.cols(), dst.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst, imgBitmap);
return imgBitmap;
}The complete code of the MainActivity class is listed below.
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.media.MediaScannerConnection;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.support.v7.app.AppCompatActivity;
import android.view.View;
import android.widget.ImageView;
import org.opencv.android.OpenCVLoader;
import org.opencv.android.Utils;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import java.io.File;
import java.io.FileOutputStream;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import java.util.Locale;
public class MainActivity extends AppCompatActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
OpenCVLoader.initDebug();
}
public void stitchVectical(View view){
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false; // Leaving it to true enlarges the decoded image size.
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Bitmap imgBitmap = stitchImagesVectical(Arrays.asList(img1, img2, img3));
ImageView imageView = findViewById(R.id.opencvImg);
imageView.setImageBitmap(imgBitmap);
saveBitmap(imgBitmap, "stitch_vectical");
}
Bitmap stitchImagesVectical(List<Mat> src) {
Mat dst = new Mat();
Core.vconcat(src, dst); //Core.hconcat(src, dst);
Bitmap imgBitmap = Bitmap.createBitmap(dst.cols(), dst.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst, imgBitmap);
return imgBitmap;
}
public void stitchHorizontal(View view){
BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false; // Leaving it to true enlarges the decoded image size.
Bitmap im1 = BitmapFactory.decodeResource(getResources(), R.drawable.part1, options);
Bitmap im2 = BitmapFactory.decodeResource(getResources(), R.drawable.part2, options);
Bitmap im3 = BitmapFactory.decodeResource(getResources(), R.drawable.part3, options);
Mat img1 = new Mat();
Mat img2 = new Mat();
Mat img3 = new Mat();
Utils.bitmapToMat(im1, img1);
Utils.bitmapToMat(im2, img2);
Utils.bitmapToMat(im3, img3);
Bitmap imgBitmap = stitchImagesHorizontal(Arrays.asList(img1, img2, img3));
ImageView imageView = findViewById(R.id.opencvImg);
imageView.setImageBitmap(imgBitmap);
saveBitmap(imgBitmap, "stitch_horizontal");
}
Bitmap stitchImagesHorizontal(List<Mat> src) {
Mat dst = new Mat();
Core.hconcat(src, dst); //Core.vconcat(src, dst);
Bitmap imgBitmap = Bitmap.createBitmap(dst.cols(), dst.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst, imgBitmap);
return imgBitmap;
}
void saveBitmap(Bitmap imgBitmap, String fileNameOpening){
SimpleDateFormat formatter = new SimpleDateFormat("yyyy_MM_dd_HH_mm_ss", Locale.US);
Date now = new Date();
String fileName = fileNameOpening + "_" + formatter.format(now) + ".jpg";
FileOutputStream outStream;
try{
// Get a public path on the device storage for saving the file. Note that the word external does not mean the file is saved in the SD card. It is still saved in the internal storage.
File path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES);
// Creates a directory for saving the image.
File saveDir = new File(path + "/HeartBeat/");
// If the directory is not created, create it.
if(!saveDir.exists())
saveDir.mkdirs();
// Create the image file within the directory.
File fileDir = new File(saveDir, fileName); // Creates the file.
// Write into the image file by the BitMap content.
outStream = new FileOutputStream(fileDir);
imgBitmap.compress(Bitmap.CompressFormat.JPEG, 100, outStream);
MediaScannerConnection.scanFile(this.getApplicationContext(),
new String[] { fileDir.toString() }, null,
new MediaScannerConnection.OnScanCompletedListener() {
public void onScanCompleted(String path, Uri uri) {
}
});
// Close the output stream.
outStream.close();
}catch(Exception e){
e.printStackTrace();
}
}
}After clicking on the button that stitches the images vertically, the result will be displayed on the ImageView as given below.
The details of the vertically stitched image are shown below. Its name uses the word vertical rather than horizontal. Its size is 640×1710, where the height value (1710=3×570) is the sum of the heights of the 3 images.
Conclusion
This tutorial built an Android Studio project for implementing the easiest way to stitch 2 images horizontally and vertically. The images are concatenated together horizontally using the hconcat() method and vertically using the vconcat() method. The stitched images are saved in a directory, which is listed in the device gallery.
As an extension to this project, in the future we’ll discuss a more advanced way of image stitching in addition to implementing new image effects.
Comments 0 Responses