Sparrow WebGL Devlog 6: GLTF Animations & Vertex Skinning

Let's add some movement

18.06.2023 - 12:53
Last week, when I added 3D models from glTF files to my WebGL engine, I only implemented static models. However, we can all agree that static models aren't very interesting. To make things a bit more exciting and dynamic, I worked on glTF animations and vertex skinning this week.

Animations

As mentioned before, glTF is a comprehensive format, which includes the ability to import animations. In glTF files, animations can change the translation, rotation, and scale attributes of specific nodes. In the simplest case, the file only has a single node with a single mesh, so the animation is directly applied to the node. However, it's also possible to have much more complicated node structures with hierarchies. In this case, the transformations depend on the transformations of the node's parent and we have to traverse the whole hierarchy up to the root node.

The animations use a keyframe system with a time input and a vector or quaternion output depending on the animated property (translation and scale use vectors, while rotations use quaternions). The files I exported from Blender seemed to export every frame even though there were only a few keyframes in the Blender timeline. However, the default Blender framerate is 24 fps, which is much less than a typical 60 fps WebGL application or game. This means we have to implement keyframe interpolation. glTF supports three interpolation methods: Step, Linear, and Cubic Spline. Step interpolation is trivially easy as it's just the nearest keyframe. Linear interpolation for vectors isn't much more complicated either, but it requires special attention for quaternions. For them, we have to do spherical linear interpolation instead (often called slerp). I have previously implemented quaternions and slerp in C++, so it was mostly just converting the code to JavaScript. I haven't added cubic spline interpolation yet, because the default bezier interpolation from Blender gets exported as linear interpolation. I'll do it when I encounter a glTF file that uses it (*important project gets delayed because I didn't do it now*).

With the ability to load and parse animations from glTF files, I was able to create this simple test animation in Blender and export it to WebGL:

Vertex Skinning

Having the ability to render animated objects that move or rotate is nice but not very useful for WebGL applications or games. You could always import a static mesh instead and do the translations, rotations, and scaling directly in WebGL. This is where vertex skinning comes into play. If you aren't familiar with it, this is the technique that is used to animate characters and creatures.

The idea behind vertex skinning is very analogous to real life. The core of every animated character is a skeleton, which is a bunch of connected bones, for example, the spine connects to a shoulder bone, which connects to an upper arm bone, to a lower arm bone, and finally a hand bone. You can do this with more or less bones depending on the level of detail your animations needs to have (you could add bones for every hand and finger segment for example). On top of the skeleton is the actual mesh or skin as it is called for vertex skinning. This is just a normal triangle mesh.

The bones of the skeleton are used to pose and animate the model. But how does the mesh know how it needs to deform when a bone is moved? When creating such a model in Blender or another 3D modeling program, it has to be weight painted. This means every vertex of the mesh gets a weight for every bone that describes how much the movement of the bone affects the position of the vertex. Weight painting models is very tedious and not a lot of fun.

All of the extra data required for animated models has to be exported to the glTF file. Unlike other file formats, there isn't a special section for the skeleton data. Instead, the bones are encoded as normal nodes without meshes attached to them. As mentioned above, the nodes can have hierarchical structures, which are used to describe the parent-child relationships of the bones. The glTF skin section stores the inverse bind matrices and all meshes that are part of a vertex skin have two extra attributes: The bone indices and their weights.

I haven't explained the inverse bind matrices yet. When you rotate or scale something in 3D, the origin is very important. When you move your arm, it rotates around your shoulder. In 3D, we can only rotate around the origin, which means to get a correct arm rotation, we have to move everything to the origin, rotate, and then move everything back. This is what the inverse bind matrices are used for. They store the transformation that moves everything to the origin and the animated transformation matrices of the joints are used to move it back.

Implementing animated models and vertex skinning is always a bit tricky, especially when you are using a new format. There are a lot of small details and every one of them has to be correct for the final animation to be correct. Because I have never worked with glTF files before, I had to figure out how everything is stored and is intended to be parsed, but after a few annoying mistakes, I got it working.

The bug that took me the longest to find was that glTF stores the bone indices as unsigned bytes, which makes a lot of sense because they are small numbers. So when I loaded the indices I stored them as a Uint8Array and then forgot about it. But when I copied the shader code from my C++/OpenGL vertex skinning implementation, it used a normal float vec4 for them. GLSL doesn't auto-convert the numbers when they are sent to the GPU, so trying to fill a float array with bytes makes the animation glitch out. I also had the multiplication of some matrices in the wrong order in one case, which was another tricky one to find.

---

Animated vertex-skinned models are a big step for my WebGL engine. It tends to be one of the more complicated features of a game engine and even though my implementation still needs a lot of work, it's great to have the basics already done.

---