Working with Face Mesh in Lens Studio

Make masks that mimic facial movements and attach objects to the mesh with the help of face mesh.

Face mesh provides a 3D mesh that mimics the user’s facial movements. You can add 3D masks using the face mesh which makes your lenses more interesting. In this article, we will create a mask using the material provided in the asset library. Then, we’ll attach objects to the mesh so that they can move with the user’s face movements.

Let’s get started!

Adding a Face Mesh

Open Lens Studio and create a new project. On the left upper side, there is an object panel. Click on the + button and select face mesh. This adds a head binding and face mesh.

Now that we have the face mesh, we can add the material of our choice to the mesh. Click the asset library at the top left section and select materials. In this case, we will use disintegrate, wood, and car paint materials. Click on the import button to download them to the resources panel.

With the face mesh selected, go to the Inspector panel and drag the disintegrate material as the material for the face mesh. The user’s face gets covered with the disintegrated material. The disintegrated material can be customized which allows us to modify it depending on the outcome we want.

Customizing Material

To customize the material, select it in the Resources panel. In the Inspector panel, there are settings that can be modified.

Under the graph parameters, there’s a number of properties that you can change. The base color is the first color of the material that shows. The metallic option has a slider that lets you adjust the intensity of the metallic sheen of the material. Under roughness, when the value is 0, the material is shiny and smooth and when the value is 1, the material is rough. The animate option when enabled animates the material using the base color and the edge color. The noise frequency is how much of the disintegration occurs. The higher the value of noise frequency, the more the material disintegrates.

Adding a Sphere

The next step is to add a sphere that will hold objects that we will attach to the mesh. To add a sphere, go to objects>mesh>sphere. Select the sphere in the Objects panel and in the Inspector panel, select wood as the material.

The changes can be seen once you select wood as the material. Next is to change the settings of the wood material. To do that, select the wood material in the Resources panel. In the Inspector panel, there are settings that you can play around with to get the output you want.

You can change the position of the object in reference to the mesh by selecting the sphere and adjusting the scale in the transform settings in the Inspector panel or in the 2D scene.

Attaching Objects to Mesh

To attach the sphere object to the face mesh, we will use the pin to mesh component. Select the sphere in the Objects panel, and in the Inspector panel, click add component>pin to mesh. In the pin to mesh component, select the face mesh as the target. The sphere object now moves with the user’s face.

In the pin to mesh settings, set the orientation to position and direction which pins both the position and direction and also enabled vertex normal. The offset position allows you to set the position of the object off from the mesh. In this case, I set the values for x,y,z as 3.

So far it looks like the image above.

In this case, I added another sphere object with a different material.

Add sphere in the Objects panel and in the Inspector panel, select car paint in the material option. Adjust the settings of the car paint material in the Inspector panel. It’s fun to use and adjust the parameters to get the desired output. Adjust the size of the object once you select the sphere that holds the object and adjust the size in the 2D scene or the transform settings in the Inspector panel.

Then pin it to the face mesh by adding a pin to mesh component. Use the offset position to set the values off from the target mesh.

The final result of pinning the objects to the mesh is shown in the image below.

Conclusion

You have seen how a face mesh is used to make masks that mimic the user’s facial expressions. You‘ve also seen how objects can be pinned to the mesh to create some cool results. Face mesh allows you to exercise your imagination. What can you create?

Fritz

Our team has been at the forefront of Artificial Intelligence and Machine Learning research for more than 15 years and we're using our collective intelligence to help others learn, understand and grow using these new technologies in ethical and sustainable ways.

Comments 0 Responses

Leave a Reply

Your email address will not be published. Required fields are marked *

wix banner square