I’ve encountered a slight problem with encoding 31 textures into ASTC format. The thing is that usually in all our apps geometries are batched and merged to reduce the amount of draw calls. So typically we use a couple of large texture atlases for all geometries. In this scene however geometries were not batched and I decided to keep this as is since the total amount of draw calls is still rather small (around 30). So this resulted in 31 textures which were to be converted into ASTC format.

So usually I just encoded textures with some ASTC block sizes (4x4, 6x6, 8x8 and 10x10) and manually picked the one which looked good enough for my liking. While this was OK for a handful of textures it was quite a cumbersome process for 31 ones. This needed some automation, and fortunately I’ve encountered this wonderful tweet about quality improvements of Spark realtime texture encoder (make sure to check it out, the project is fascinating by itself). What was interesting in it for me is the usage of a tool called SSIMULACRA2. This open-source tool compares two images and gives it a quality score. And the algorithm is specifically tuned to look for both user-perceived and blocky compression-related artifacts in the images! That’s exactly what I needed!

Over the weekend I’ve created a bash script (bash is not my native language so I’ve used ChatGPT to help me here and there) and tested it. The algorithm of the script can be described with this pseudocode:

for each input file  
	for each ASTC block size from the lowest to the highest quality  
		encode image and save its decoded image  
		compare quality between original and decoded images  
		if quality is within threshold encode with this block size

You can optionally specify SSIMULACRA2 score but the default is 70 which can be described as “there are some compression artifacts if you look for them specifically”.

There’s additional logic for the detection of mipmaps - these must be encoded with the same compression so only the highest level is tested for quality.

There’s a limitation of the SSIMULACRA2 tool - it doesn’t validate images smaller than 8x8 pixels so these are encoded with fallback 4x4 block size.

Also I’ve played around with bash output a little so it is not so silent while working - it informs how it processes the current image, and shows scores for the processed images:

This saved not only my time - I didn’t have to check the quality of each file manually - but file sizes too! Because of my laziness I checked manually only a subset of ASTC block sizes but this script relentlessly runs through all 14 blocks supported by ASTC, choosing the best suitable one. And for some smallish textures with quite uniform images even 12x12 block size was good enough.

You can check the source code of the script here on the GitHub and adapt it for your needs. Also it has an imagemagick script to create mip maps.

However, because of its simplicity it does have a couple of shortcomings and for some reasons doesn’t provide access to all internal things under its hood. One of these minor inconveniences is the way it initializes multisampled render targets.

To understand why this is important let’s explain how Metal handles MSAA. It supports multiple ways of implementing it:

1. You can have a multisampled render target and then resolve it to on-screen render target automatically.
2. You can use this multisampled render target with custom resolve shaders.
3. On supported hardware, you can omit mutlisampled render target and resolve it automatically directly to the final render target. This will still use multisampled render target but it will be memoryless.
4. The same approach but with tile shaders to resolve (apply custom tone mapping, etc).

You can find a more detailed explanation of these methods with a sample XCode project in this official Apple documentation article

What is of a particular interest for us is the memoryless multisampled render targets. They are very efficient since they are transient and reside only in (extremely fast and tiny) temporary tile memory of GPU. Because of this they don’t use main memory allocations and don’t add up to precious VRAM access bandwidth.

Here is the typical 4x MSAA rasterization process with default render pass created by MTKView:

And here is the same one but using efficient memoryless render target:

Basically, the only difference is that we substitute transient multisampled render target with the memoryless one and this results in a huge improvement in memory allocation and bandwidth. Please note that according to the Metal Feature Set tables memoryless render targets are not supported by old devices. Namely, Intel-based Macs don’t support tiled rendering and cannot use them. But if you target shiny new Apple-silicon devices then you definitely must use them because they are so extremely efficient.

The thing is that (possibly for a better support of all hardware) MTKView initializes MSAA only with classic in-memory render targets — multisampled one for the rendering and and the final one for resolving and presenting result on the screen.

In the aforementioned official Metal MSAA example you can find a proper way of initialization of memoryless MSAA resolve but it doesn’t use this handy MTKView — instead there’s quite a lot of glue code to make it work.

However I’ve found a hacky yet relatively simple and perfectly working way of initializing efficient memoryless MSAA resolve using the default MTKView wrapper view. Let’s take a look at what configuration options MTKView does provide. Obviously there’s a sampleCount which will initialize MSAA render targets. Also there are depthStencilPixelFormat and depthStencilStorageMode fields. And you can change depth+stencil to use memoryless storage too by setting depthStencilStorageMode=.memoryless, which also saves a lot of RAM usage and bandwidth in case you don’t need depth information of your frames. Here’s a typical MTKView initialization code (for an Apple-silicon GPUs, which support memoryless textures):

_metalView.depthStencilPixelFormat = .depth32Float
_metalView.depthStencilStorageMode = .memoryless
_metalView.preferredFramesPerSecond = 60
_metalView.sampleCount = 4 // hard-coded 4 samples but you can query max available samples for GPU and set it accordingly

That’s cool, let’s also switch color render target to the memoryless mode too! Unfortunately for the color render target there is only colorPixelFormat available (typically set up automatically) and there is no colorStorageMode. So there’s no easy way to just set it up to use memoryless MSAA mode.

Still there’s a relatively simple way of switching it to the memoryless mode after it has been initialized!

The thing is that Metal API allows you to change the MSAA resolve texture of the current render pass. The descriptor of this render pass is provided to you by MTKView and obviously it is pre-initialized with in-memory texture. So all you need to do is on the first frame you draw to create a memoryless render target and substitute the default resolve texture with the new one.

// New memoryless MSAA texture
var textureMsaa: MTLTexture?

.................

func yourCodeToDrawStuff() {
    // Before rendering, create and replace MSAA resolve RTT.
    do {
        let resolveTexture = view.currentRenderPassDescriptor?.colorAttachments[0].resolveTexture
        if resolveTexture != nil {
            let width = resolveTexture!.width
            let height = resolveTexture!.height
            if textureMsaa == nil || textureMsaa?.width != width || textureMsaa?.height != height {
                // Auto-purge the old unused resolve texture
                renderPassDescriptor?.colorAttachments[0].texture?.setPurgeableState(.volatile)
                textureMsaa = try create2DRenderTargetMemoryless(width: width, height: height, pixelFormat: .bgra8Unorm, metalDevice: device)
                textureMsaa?.label = "Main pass RTT"
            }
        }
    } catch {
        fatalError("Cannot create MSAA texture: \(error)")
    }
    // Use new memoryless texture
    view.currentRenderPassDescriptor?.colorAttachments[0].texture = textureMsaa

    // Do you rendering here as usual
    .....................
}

func create2DRenderTargetMemoryless(width: Int, height: Int, pixelFormat: MTLPixelFormat, metalDevice: MTLDevice) throws -> MTLTexture {
    let descriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat, width: width, height: height, mipmapped: false)
    descriptor.textureType = .type2DMultisample
    descriptor.sampleCount = 4 // Yes I use hard-coded 4 samples here too :)
    descriptor.usage = [.renderTarget]
    descriptor.resourceOptions = [.storageModeMemoryless]
    let result = metalDevice.makeTexture(descriptor: descriptor)
    if result != nil {
        return result!
    }
    throw RuntimeError("Cannot create texture with pixelFormat \(pixelFormat) of size \(width)x\(height)")
}

This simple trick effectively substitutes the auto-created in-memory color render target with the memoryless one.

There is one important step to do — you must set the purgeable state to volatile for the old unused render target in order for it to free memory. Otherwise even if it won’t be used it will still keep a large amount of memory allocated for it. This is an extremely powerful and easy-to-use feature of Metal API which I love — if you don’t use some resource, API can get rid of it for you automagically. You don’t have to manually delete them as in OpenGL.

Here are some final memory usage comparisons on a MacBook Air M1 with a full-screen 2560x1600 render target:

First, a default approach — MTKView with 4x MSAA in-memory resolve texture:

This multisampled texture uses 78 MB of memory which is being accessed (both write and read) on every frame!

And here is the memoryless one:

Notice the 78 MB texture is now listed in the unused resources. It actually uses 0 bytes, only listed as a “dormant” 78MB resource which could be re-allocated in case it will be reused again. This can be confirmed in the Activity Monitor. Before optimization:

And after:

Now my app uses just under 80 MB of total RAM instead of 150! This is a good result for a full-screen 3D app — actually it about as much as just two stock MacOS calculators! (Yes you can check it yourself — Calculator uses ~40MB of RAM which seems to be a bit excessive).

Hope this little tutorial will be useful and will make your Metal app more memory and power-efficient!

Scene has a distinct stylized look with no textures used except for animated characters — knights and birds. To add depth to the scene, a simple linear vertex fog and real-time shadows are applied.

Total scene polycount with both static and dynamic objects is 95k triangles.

Static geometry

All static objects in the scene are merged into 2 large meshes to reduce the amount of draw calls. These assets don’t use textures, instead vertex colors are used. These colors are stored as indices which allows for an easier customization of color themes. Shaders accept up to 32 colors set via uniforms.

The stride for static objects vertex data is 12 bytes — 3 FP16 values are used for position, 3 normalized signed bytes for normals, and 1 unsigned byte for color. 2 unused bytes are used for 4-byte alignment of data:

I also tried using a more compact packed GL_INT_2_10_10_10_REV type for vertex positions to fit data in 8 bytes. Unfortunately, its precision was just not enough for this purpose. This data type provides precision of roughly 1 meter per 1 km. And since the scene uses quite large geometries batched into 2 meshes this precision was clearly not enough.

Shadow maps

Lighting in the scene is not baked — shadow maps are used for all objects to cast dynamic shadows. Shadow maps have no cascades since the scene is rather small. However, the detail of shadows is adjusted in a different way. The light source FOV is slightly adjusted per each camera to have more detailed shadows for close-up scenes and less detailed for overviews.

Shadow map resolution is 2048x2048 which is sufficient to create detailed enough shadows.

To smooth out hard shadow edges, the hardware bilinear filtering of shadow map is used. Please note that OpenGL ES 3.0+ / WebGL 2.0 is required for this. OpenGL ES 2.0 supports only unfiltered sampling from shadow maps which results in boolean-like comparison whether a fragment is in shadow or not. Hardware filtering is combined with 5-tap sub-texel percentage closer filtering (PCF). This results in smooth shadow edges with a relatively small amount of texture samples.

I also considered a more expensive 9-tap PCF which improved image quality in case of unfiltered shadow texture but with the filtered texture improvement over 5-tap one was negligible. So according to the golden rule of real-time graphics programming “it looks good enough”, the final filtering used in the app is 5-tap PCF with hardware filtering.

Here you can see comparison of different shadow filtering modes:

Performance

To improve performance, a couple of optimizations are used.

First is a quite typical, simple and widely used one — shadow map is updated at half framerate. This is almost unnoticeable since nothing in the scene moves too fast. For cases when the camera and light direction is about to switch to a new position it is rendered at full framerate to prevent 1-frame flickering of shadow map rendered with the old light source.

The second trick is that the PCF is not applied for distant fragments — instead a single sample of shadow texture is used. It is impossible to spot any difference in image quality in the distance because shadows are still hardware-filtered but the performance and efficiency are improved. But isn’t it considered a bad practice to use branching in the shaders? Yes and no. In general, it is not so bad on modern hardware — if used properly and not in an attempt to create some all-in-one uber-shader. Actually, it is quite often used in raymarching where it can provide a measurable performance improvement by branching out empty/occluded parts. In this particular case branching helps to save one of the most critical resources on mobile GPUs — memory bandwidth.

So how can we test if this branching actually improved things or only unnecessarily complicated shaders?

First, let’s perform a static analysis of shaders. To do this, I use the Mali Offline Compiler tool. Here are results:

Non-optimized:

Work registers: 25
Uniform registers: 16
Stack spilling: false
16-bit arithmetic: 66%
                              FMA     CVT     SFU      LS       V       T    Bound
Total instruction cycles:    0.31    0.14    0.06    0.00    1.00    1.25        T
Shortest path cycles:        0.31    0.11    0.06    0.00    1.00    1.25        T
Longest path cycles:         0.31    0.14    0.06    0.00    1.00    1.25        T

FMA = Arith FMA, CVT = Arith CVT, SFU = Arith SFU, LS = Load/Store, V = Varying, T = Texture

Conditional PCF:

Work registers: 21
Uniform registers: 16
Stack spilling: false
16-bit arithmetic: 68%
                              FMA     CVT     SFU      LS       V       T    Bound
Total instruction cycles:    0.31    0.22    0.06    0.00    1.00    1.50        T
Shortest path cycles:        0.17    0.11    0.06    0.00    1.00    0.25        V
Longest path cycles:         0.31    0.20    0.06    0.00    1.00    1.25        T

FMA = Arith FMA, CVT = Arith CVT, SFU = Arith SFU, LS = Load/Store, V = Varying, T = Texture

So according to these results, the new version is no longer texture-bound in the shortest path and still has the same cycles for the longest path. Also, the number of used registers is reduced. Looks good on paper, isn’t it?

But of course both versions of shaders perform identically in the Android app on my Pixel 7a — it always runs at stable 90 fps. So to see if GPU is less loaded, let’s run Android GPU Inspector on two versions of app and compare some metrics from profiles:

As expected, it doesn’t affect overall GPU cycles much but reduces load on texture units. As a result, the GPU now is less busy — it consumes less power and has more free resources to smoothly render home screen UI on top of live wallpaper.

Animation

All animated objects in the scene are animated procedurally. No baked skeletal or vertex animations are used. The simple shapes of these objects allow them to be animated in vertex shaders relatively easily.

I have found some inspiration for procedural animations of rats in the “King, Witch and Dragon” (https://torchinsky.me/shader-animation-unity/) and fish ABZU (https://www.youtube.com/watch?v=l9NX06mvp2E). Animations in our scene are of course simpler than the ones in these games because animated objects have stylized boxy look and therefore movements are also stylized and simplified.

Knights

Scene has 16 knights with each model made of just 48 triangles.

Animation is done in the KnightAnimatedShader.ts. Let’s take a look at how it animates the model. To do this, first it needs to detect vertices belonging to different body parts of the knight. But the vertex data doesn’t have a special “bone id” attribute for this. This cannot be done by testing vertex positions because some body parts overlap. For example, the head has 4 bottom coordinates identical to the body. Some texture coordinates also overlap so we cannot rely on them too as in the case of rats animation in the “King, Witch and Dragon” game. So I grouped the vertices for each body in the buffer, and the vertex shader determines body parts simply by comparing their gl_VertexID. Of course this is not optimal because it introduces branching but it is not overly excessive and it is done in a vertex shader for a very low-poly model. Model is grouped to have first body vertices, then head and then arms:

And here is the knight model with applied test coloring to visualize body parts:

Now that shader knows which vertex belongs to which body part it applies rotations to them provided via uniforms. Rotation pivot points are hard-coded in shader. You may notice that only the head and arms are animated. Because models don’t have separate legs they are not animated. Instead a bobbing is applied to the whole model to add a rather convincing effect of “walking”. The bobbing is simply an absolute value of the sine wave:

Birds

There are 6 eagles soaring in the sky, each model made of 70 triangles. They are flying in different circular paths.

Birds are rendered with EagleAnimatedShader.ts. Animations are done in the same way as for knights but they are simpler since only wings are animated and they are rotated synchronously. So only a single rotation timer is passed into the shader via uniform to control animation.

Flags

Scene has 3 different flags, all animated with the same FlagSmShader.ts. Technique is inspired by the wavy animation of the fish in ABZU. The simple sine wave is applied to vertices, reducing amplitude closer to the flagpole and increasing near the end of the flag. To correctly apply lighting, normals are also bent. To bend them the cosine wave of the same frequency is used since it is derivative of sine.

Wind Stripes

A small detail added to the scene as a last touch is mostly inspired by the Sea Of Thieves wind effect. In the Sea Of Thieves these stripes serve a purpose of showing wind direction to align sails so they are more straight. In our scene they are purely for the looks so they bend and twist much more.

Let’s take a look at the shader to draw them — WindShader.ts. It is even “more procedural” than the ones used for animated objects. It doesn’t use geometry buffers at all and generates triangles based on gl_VertexID. Indeed, as you can see in the source of its draw() method it doesn’t set up any buffers with vertex data. Instead it uses the hard-coded VERTICES array for two triangles declaring a single square segment. So if we need to draw 50 segments, we issue a glDrawArrays call with 50 * 3 * 2 triangle primitives. The vertex shader offsets each segment based on the gl_VertexID and tapers both ends of a final stripe using smoothstep. Then the coordinates of a resulting spline shape are shifted by an offset timer so they appear moving. Next, the shape is deformed by two sine wave noises in world-space. Color is animated to fade in and out, and offset is animated to move the stripe. All this results in a random snake-like movement of the stripes but keeps them aligned to the world-space path.

Here is a breakdown of rendering of a short 20-segments wind stripe with test coloring of segments to clearly visualize geometry:

Depth-only shaders

All objects in the scene except wind stripes cast shadows so they have to be rendered to a depth map texture using the light source camera. For this, the simplified versions of corresponding animated and static shaders are used. They perform all the same vertex transformations but don’t calculate lighting.

For example, let’s take a look at the KnightDepthShader.ts. It performs all the same vertex transformations to animate head and arms but does not calculate lighting based on normals. Even more than that, you may notice that its fragment shader is empty — it provides no color output at all. These are perfectly valid shaders in GLSL ES 3.00 since their only purpose is to write to the depth (shadow map) attachment.

Results and possible additional optimizations

Live web demo: https://keaukraine.github.io/webgl-stylized-castle/index.html

Source code: https://github.com/keaukraine/webgl-stylized-castle

Android live wallpaper app - https://play.google.com/store/apps/details?id=org.androidworks.livewallpaper.cartooncastle3d

Final web demo is ~2.2 MB which is not quite optimal because all objects are exported as 2 huge batched meshes. And there are a lot of repetitive objects like trees and cannons which are good candidates for instanced rendering.

Scene composition

Scene is aesthetically simple so it contains just four objects: planes, ground, clouds and wind:

Next, let’s take a look at what shaders are used to render models, and how geometries of these models are optimized.

Planes

Technically planes are rendered not as voxels (each voxel cube individually) but as a ready mesh, exported from MagicaVoxel. They are not simplified using VoxCleaner to use texture atlases and reduce polycount - I decided to use them as is because it will be easier to create alternate palettes, and anyways vertex data for planes have ridiculously small memory footprint.

Each plane consists of 3 parts - plane body, glass cockpit and rotating propellers. Plane is rendered using 2 shaders - a simple directionally lit PlaneBodyLitShader.ts for body and props and its variation GlassShader.ts for glass with stylized reflections.

The specifics of plane models allow vertex data to be packed using really small data types. All plane models are small and fit in the -127…+127 bounding box. And since vertices represent voxels they are always snapped to 1x1 grid. So I chose to store vertex positions in signed bytes which have just enough precision for the job.

The most interesting part however is storing normals and texture coordinates. They are packed in a single byte.

First, normals can be stored with just 3 bits. But how can it be enough to store this kind of information in 3 bits? Since voxels are cubes, they can have only 6 variations of normals. So instead of storing normals as vectors they can be represented with an index, and 3 bits is enough to store up to 8 variations. An array of actual values is hard-coded in the vertex shader. After 3 bits of byte are used by normals, we are left with 5 more bits which allow our models to have a 32-colors palette. And again this is more than enough for our case because stylized models use only 23 unique colors. Since the palette textures used by models are of 32x1 size, the V texture coordinate is omitted and hardcoded to 0. texelFetch is used to sample color from palette, and after combining it with directional lighting it is ready to be passed to fragment shader.

Please note that bitwise operators used to unpack normals and color indices are available only in OpenGL ES 3.0+ and WebGL 2.

Complete vertex data stride for plane and prop models is just 4 bytes, and it keeps original information 100% intact:

Byte with packed color and normals indices:

In a separate draw call, a glass with scrolling stylized fake reflection is drawn:

Glass is rendered without palette texture — its color is set via uniform. The texture passed to this shader is a mask for reflection. Its UV coordinates are calculated in the vertex shader based on model-space vertex coordinates. Of course, it is unfiltered for artistic purposes. Stride for glass models is the same but without texture coordinate for color — the whole byte is used to store normal index:

GlassShader samples texture using textureLod with 0 mipmap level. This is done to explicitly tell the OpenGL ES driver that we access the texture without mipmaps and to reduce some overheads. You can read more about this and some other texture sampling optimizations tricks in Pete Harris blog — https://solidpixel.github.io/2022/03/27/texture_sampling_tips.html

Also glass models have small polycount so they use unsigned byte indices which also reduces memory bandwidth.

Wind stripes

For wind stripes I decided to create a shader which doesn’t perform any memory reads at all. Wind stripe has a very simple geometry — a 100x100 units quad, stretched into an appropriately thin line by model matrix. Because of its simplicity, all this geometry can be hardcoded in vertex shader code. And it doesn’t use any textures as well — fragment color is passed via uniform. You can find the implementation in WindStripeShader.ts. It uses gl_VertexID to get position for a given vertex. When this shader is used to draw a wind stripe, no buffers or textures are bound.

Technically it even can be used as a “building block” to draw more complex shapes by issuing draw calls with different rotating/scaling/shearing its base hard-coded quad geometry but this will be too inefficient.

Terrain

Terrain textures are 256x256 tiling images. They are based on aerial photos with some GIMP magic sprinkled over them — contrast adjustments and colors reduced to just 10–12. This adds a more old-skool look to them and makes each texel more pronounced.

Shader to render terrain is in DiffuseScrollingFilteredShader.ts file. Let’s take a look at it.

It is a rather primitive shader which simply pans UV coordinates to create an illusion of moving ground beneath the airplane. However there is one additional thing it does, and it is texture filtering. You may be wondering what filtering is used here, ground clearly is unfiltered, it uses GL_NEAREST sampling! However, there is a custom antialiased blocky filtering used here. The thing is, that regular GL_NEAREST sampling produces a lot of aliasing on the edges of texels. This becomes especially noticeable at certain angles of the continuously rotating camera. The textureBlocky() function alleviates these aliasing artifacts while preserving that extra crispy old-skool look of unfiltered textures. Ground texture actually uses GL_LINEAR filtering and the textureBlocky() calculates sampling point to get either an interpolated filtered value at the edges or an exact unfiltered one from the center of texel for any other area.

The original author of this filtering is Permutator, and code is used under CC0 license from this shader toy — https://www.shadertoy.com/view/ltfXWS (you may find some deeper explanation of math used in this filtering technique there).

Here is a comparison (with 4x zoom) of a regular GL_NEAREST filtering vs custom blocky filtering. As you can see, both are pixelated but the latter one is not aliased.

One of the last additions to the scene is a transition between two different terrain textures. When you switch them, they don’t just toggle but instead a cute pixelated transition effect is used to smoothly switch between textures.

You can find a code for this transition in the DiffuseScrollingFilteredTransitionShader.ts file. Transition uses tiling blue noise texture for uniformly appearing square blocks on the ground. To make transition smoother, smoothstep() is used. However there is a commented out line with step() which makes transition more abrupt if you prefer it.

Clouds

Clouds don’t use this antialiased blocky filtering because they don’t rotate, are quite transparent and move relatively fast. This makes it quite hard to spot aliasing artifacts on them so they use the cheapest option available — GL_NEAREST sampling. Clouds use a custom mesh with cutouts where texture is empty. This significantly reduces overdraw compared to a regular quad mesh. Here it is visualized by disabling blending:

Result

You can see a live web demo here and if you like to have it on the home screen of your Android phone you can get a live wallpaper app on Google Play.

Source code is available on GitHub, feel free to play around with it.

As always, the web demo is heavily optimized for the fastest downloading of resources and Android app for the best efficiency and performance. The initial loading of the web demo is just 155 kB, and the size of all models and textures is 1.05 MB so you can fit this data on a floppy disk.

P. S.

These WebGL demo and Android app have been made during war in Ukraine despite regular power outages caused by deliberate destruction of country’s electric infrastructure. Please support Ukraine however you can and refrain from buying imported russian goods since taxes from these sales are used to support war.

Idea for this 3D scene comes from the magnificent Zhangjiajie National Forest Park in China. You can clearly see where the inspiration originates from - this majestic real life location also has grassy rock pillars covered in dense clouds and when observed from above, their bottom parts are disappearing in dense fog. To improve the magical feeling to the scene, we decided to make some rocks float mid-air. This additional inspiration comes from the map Gateway to Na Pali from my favorite game Unreal. This location has floating rocks in the distant background and is placed inside a huge floating rock itself. We decided to create a scene which would have a lot of similar floating islands densely packed in one area.

AI generated concept art

We also tried to use Stable Diffusion to generate some concept art for the scene in hope AI will hallucinate some unusual points of view or might incorporate some details which we may find fitting the scene. However, all images appeared to be virtually identical. AI created a series of rather dull images - the same rocks in the same fog without any additional details. We only used a couple of them as a reference for vivid sunrise color palettes which could as well be picked from any other sources.

Scene composition

To create this scene we’ve used and reused some stylized hand-painted 3D models from packs we’ve purchased quite some ago for our previous projects. No new assets were purchased for this project. Scene uses just 3 rock models, some generic ferns and trees, and birds flying in the sky.

Render order is the following: depth pre-pass, rocks, birds, sky, soft cloud particles.

Camera path and objects placement

To make an impression of an endless random scene there were 2 options: a true random scene and a looped generated path. The first option requires object placement on the fly in front of the camera which means these positions have to be transferred to GPU dynamically. So a better option is to generate a static looped path once and draw objects along it as the camera moves. You can find a function to generate a base spline in the positionOnSpline function in ObjectPlacement.ts file. It creates a circular looped path for the camera with oscillating radius. A couple of harmonics are applied to randomize the circle radius so it appears random but is still perfectly looped. Then, all objects are placed around this path - trees are under the camera, rocks above and to the sides.

Object positions and rotations are stored in typed Float32Array in the form of textures on GPU. drawInstances method in Renderer.ts renders objects visible only from a certain point on the spline. Because of scene simplicity there’s no need to use frustum culling - objects are drawn at a certain distance in front and back of the camera. This visibility distance is slightly larger than fog start distance so the new objects appear fully covered in fog and don’t pop. Instances are ordered front-to-back so when drawn they make use of Z-buffer culling.

Only rocks and trees models are placed this way alongside the camera path. Bird flocks use hand-picked linear paths to cover the whole area of the scene with minimal paths.

Here is camera path visualized with only a subset of objects rendered in its vicinity:

Fog cubemaps

Initial implementation used fog of uniform color which looked rather bland. To add more color variation from different directions (like sun halo) we decided to use cubemaps for fog. This allows great flexibility for the artist (my brother) - he can completely change the look of the whole scene by creating a cubemap and tweaking a couple of colors in the scene preset. Cubemaps were initially created as equirectangular images since it is easy to paint them. Then we used an online tool to convert an equirectangular source image to 6 cubemap faces, and a simple ImageMagick script to fix their rotations to suit our coordinate system (Z-up).

You can find cubemap fog implementation in static constants from FogShader.ts. All fog shaders use them. Final fog coefficient used by vertex shader for color mixing also contains the height fog coefficient.

In the web demo UI you can adjust different fog parameters - start distance, transition distance, height offset and multiplier. Also, changing the scene time of day is done by using a different cubemap texture and a couple of colors for each preset.

Interestingly, after implementing this I’ve found out that fog cubemaps are widely used in Source engine, and of course this technique has also been incorporated in some indie games too.

Grass on rocks

To make rocks less dull we also apply grass texture on top of them. This technique is commonly used to simulate surfaces covered by snow or soaked with the rain. Grass texture is mixed with rock texture based on vertex normal. You can play around with the grassAmount slider in UI to see how it affects grass spread on rocks.

Source code of shader which applies grass texture on top of rocks is in FogVertexLitGrassShader.ts.

Soft clouds shader

Clouds are not instanced but are drawn one by one because transformation matrices for these objects have to be adjusted to always face the camera. There’s not that many of them so this doesn’t add too many draw calls. Actually, if GPU state is not changed (no uniforms are updated, blending mode switched, etc.) then even non-instanced rendering is quite fast on modern mobile and desktop GPUs. For test purposes we had a quick and dirty visualization of a camera spline with non-instanced rendering of 5000 small spheres and it caused no slowdowns.

There’s also one minor trick in this shader. As the camera flies through clouds, they can be abruptly culled by a near clipping plane. To prevent this, there is a simple smoothstep fading applied to fade right in front of the camera. You can find code in FogSpriteShader.

Result

You can see a live web demo here and if you like to have it on the home screen of your Android phone you can get a live wallpaper app on Google Play.

Source code is available on GitHub, feel free to play around with it.

As always, the web demo is heavily optimized for the smallest data size and Android app for the best efficiency and performance. Web version uses WebP for textures which offer better compression than PNG, better image quality than JPEG and support alpha channel even with lossy compression. Mipmaps are generated for all textures. The total gzipped size of the web demo is just 374 kB so you can copy it to a floppy disk to show to friends who have no Internet :)

You can check out the live demo page and play with various configuration options in the top right controls section.

Implementation

Scene is composed from the following main objects: sky, ground, and 3 types of grass: flowers (each containing individual instances of leaves, petals and stems), small round grass and tall animated grass. To make the scene more alive, a sphere for glare and moving ants+butterflies are also drawn.

Draw order is the following: first objects closer to the camera and larger ones to use z-buffer efficiently, then objects closer to ground, then sky and ground. Ground plane has transparent edges which blur with the background sky sphere so it is drawn the last after the sky.

For sun glare effect we draw a sphere object with a specular highlight. It is drawn last over the whole geometry without depth test. This way everything is slightly over-brightened when viewed against the sun, and glare is less prominent when the camera is not facing the sun.

Tiled culling of instances

Grass and flowers are drawn using similar shaders with a common part in them being instanced positioning. These instanced objects get their transformations from the FP32 RGB texture.

All instanced shaders use the same include COMMON_TRANSFORMS which uses 2 samples from texture to retrieve translation in XY plane, scale and rotation. Please note that rotation is stored in the form of sine and cosine of an angle to save on rotation math.

Original transformation is stored in arrays FLOWERS, GRASS1 and GRASS2 declared in GrassPositions.ts. However, these arrays have coordinates for all instances of objects, they are not split into tiles yet. For this, they are processed using sortInstancesByTiles function. It creates a new FP32 array with rearranged positions+rotations and creates an array of tiles which specify instances count and start offset in the final texture used by the shader. This ready information is stored in the TiledInstances object. This function allows to split all instances spreaded across square ground area into arbitrary N^2 tiles. In both web demo and Android app all instances are split into reasonable 4x4 tiles area with 16 tiles. Tiles have a small padding which allows them to slightly overlap. This padding has a size of grass model so the instances placed at the very edge of tile won’t disappear abruptly when tile gets culled.

To visualize how instances are split into tiles, let’s imagine a sample area with 20 randomly placed objects which we would like to cull per tile. Let’s rearrange these instances into 2x2 grid with 4 total tiles:

Here is the structure of texture containing these objects, showing tiling and data stored in each component per instance:

Here instances for tile 0 have offset=0 and count=5, for tile 1 offset=5 and count=4, and so on.

This structure allows us to draw all 20 instances in 4 draw calls and cull them in batches per tile without updating any data on the GPU.

Culling of tiles bounding boxes on CPU is also relatively cheap. It is done on each frame and you can see how many tiles and individual instances are currently rendered in “Stats” section of controls.

And reducing grass density to scale performance is also really easy with this approach because inside each tile instances are random. All we have to do is to proportionally reduce number of instances per draw call (you can use density slider in controls to test it):

There are different instanced shaders for drawing different objects. Small grass and flower petals are the simplest ones - they use a simple diffuse colored shading. Dandelion stems and leaves apply specular highlights, and the shader used to render tall grass blades also uses vertex animation for wind simulation.

Random ants

To make the ground more alive, we draw some ants on it. They are also instanced - total 68 ants are rendered in 2 draw calls.

They move in circles with a random radius and center. They are drawn in two draw calls for clockwise and counterclockwise rotation within these circular paths. You can examine the math for positioning vertices in the shader’s source code.

It will be almost impossible to notice any animation on these pretty small and very fast moving objects so we don’t animate them at all.

Butterflies

No summer can be imagined without butterflies so we added them too. They are positioned similar to ants but a sine wave is added to their height. Each instance gets its color from texture atlas with 4 different variants.

On the contrary to ants, butterflies must have animation. To animate them we don’t use any kind of baked animation. Actually, a really cheap trick is used to animate it in vertex shader. It is animated by simply moving wing tips up and down. Wing tips are determined as vertices with high absolute values of X coordinate. Of course it is not a correct circular movement of wings around the butterfly’s body - wings elongate noticeably with higher movement amplitude, but this simplifies shader math and looks convincing enough in motion, as can be seen on this image:

Android-specific optimizations

As always with our web demos, they are optimized for the smallest possible network data size and the fastest loading times. So it doesn’t use compressed or supercompressed textures. The Android app is optimized for power efficiency so it uses compressed textures (ASTC or ETC2) depending on hardware capabilities.

To further improve efficiency it uses variable rate shading (VRS) on supported hardware.

And when the app detects that the device is in energy saving mode (triggered manually or when battery is low) it will reduce FPS and will use simplified grass shader without animation to significantly reduce power draw. Additionally, in this mode the app will use more aggressive VRS.

We’ve encountered general performance issues with rendering lots of instanced geometries on low-end Android phones - the bottleneck appeared to be vertex shaders. So when the app detects it is running on a low-end device it will render grass with slightly reduced grass density and without wind animation.

Failed implementations

Before implementing this tiled rendering pipeline a couple of more naive less performant implementations have been tried and tested.

Fully randomized

The very first version of grass was with fully randomized positioning of instances. It didn’t use texture to store pre-calculated random transformation for instances but calculated them in shader instead. This introduced more complexity in vertex shaders (random and noise functions have quite some math in them). Additionally, random values were different on different GPUs which made it impossible to finely hand-pick camera paths. Take a look at this photo where we tested this version on different devices - while the code is identical, placement of instances is different:

This version had no visibility calculation or frustum culling which also affected performance.

Per-instance culling

The first naive implementation of culling has been implemented per-instance. This version already used texture to reduce vertex shader math but each instance has been tested for visibility and then texture has been updated with only visible instances.

This worked just fine on PC and on high-end Android devices but proved to be way too slow on low-end phones - CPU took about 10 ms to calculate visibility of instances. Updating the texture on the fly with glTexSubImage2D() also was unacceptably slow - it took ~20 ms. For comparison, tiled culling takes ~1 ms of CPU time on low-end devices.

Final result

Total size of the demo page is just 741 kB so you can carry it on a floppy disk.

You can play around with different parameters on the live demo page. You can alter time of day, density of grass and other settings. Double click toggles free camera mode with WASD camera movement and rotation with right mouse button down (similar to navigation in viewport in Unreal Engine).

And as usual you can get source code which is licensed under MIT license, so feel free to play around with it.

Modern mobile Adreno GPUs by Qualcomm support Variable Rate Shading, and phones with these GPUs have been available since autumn 2021. Because our live wallpapers have to be power-efficient, we have got a test device with Adreno 642L to implement this feature in our apps.

What is Variable Rate Shading

The idea behind VRS is to rasterize a single fragment and then interpolate color between adjacent pixels on screen.

A good explanation of how VRS is implemented on Adreno GPUs can be found in the official Qualcomm Developer blog here. You can understand how simple it is by looking at this image from aforementioned blog post:

VRS is better than generic downsample of the whole frame because:

1. It preserves geometry edges (except cases when the shape is determined by discarding fragments).
2. Can be adjusted per each draw call — one object can be rendered at full detail while the other one will have reduced quality.
3. Can be applied dynamically to keep target FPS by gradually reducing image quality.

Implementation

On Snapdragon SoCs it is implemented with QCOM_shading_rate extension. Adreno GPUs support blocks of 1x1, 1x2, 2x1, 2x2, 4x2, and 4x4 pixels. Please note that some useful dimensions like 2x4 or 4x1 are not available because they are not supported by hardware.

To apply VRS to certain objects you simply make a call to glShadingRateQCOM with desired rate before the corresponding draw calls.

To disable VRS for geometries which should preserve details and be rendered at native shading rate, simply call glShadingRateQCOM with 1x1 block size.

One of the first apps we’ve added VRS support to is Bonsai Live Wallpaper. This is a good example because it has 3 very different types of geometries ranging from perfect candidates for VRS optimizations to the very unsuitable ones.

Let’s take a look at a typical scene from the app and how different parts of image can benefit from reduced shading rate:

The best type of geometry to be optimized by VRS is the one which is blurred and has small color variation between fragments. So, for sky background we apply a quite heavy 4x2 VRS which still introduces virtually no quality degradation, especially with constantly moving cameras.

On the opposite side of the scales is leaves geometry. On the screenshot below we applied 4x4 VRS to the whole scene to showcase the issue with alpha-testing. Please note that branches, while also using the same heavy 4x4 reduction in this example, have the same smooth and anti-aliased edges, clearly showing a benefit of VRS over traditional upscaling.

Needless to say, VRS is clearly not suitable for geometries with discarded fragments.

Also because VRS is applied in screen-space, it introduces significant distortions to transparent dust particles. Their size is comparable to VRS block and they start flickering during movement. I’ve noticed a somewhat similar rendering technique used in the COD:MW game on PC when enabling half-resolution particles — sparks and other small particles flicker way too much and look very blocky.

And somewhere between these two geometries lies the ground plane. This is where we apply 2x1 rate reduction. This results in OK image quality because there’s a larger color difference between adjacent vertical pixels compared to the horizontal ones.

Where VRS definitely shines is when it is applied to geometries with very little color difference between adjacent fragments, and Bonsai wallpaper has a stylized silhouette mode where fragments use literally single color:

Here we have 3 types of shaders:

1. Alpha-testing for leaves. We already know that we should not apply VRS to these geometries.
2. Solid black silhouette and ground. The heaviest 4x4 VRS introduces literally zero quality degradation.
3. For the sky gradient we use 2x1 blocks. Technically it would be perfect to have a 4x1 or even 16x1 blocks because gradient changes vertically and adjacent horizontal fragments have identical color but Adreno hardware supports only 2x1 ones.

All of these applied to the scene results in identical rendering (screenshots comparison found 0 pixels difference) and 1.5x of shading speed improvement.

Dynamic quality

All our wallpapers use some ways of reducing GPU load when the battery is low. Usually this is done by limiting FPS and omitting a couple of effects.

For more efficient power usage we apply stronger VRS to certain objects in low battery mode. Tree trunks are shaded with 2x1 blocks, sky and transparent effects (light shafts and vignette) are shaded with 4x4 instead of 4x2 or 2x2 blocks. This reduction of quality is still almost unnoticeable but reduces GPU load by additional 3%.

Performance gains vs quality tradeoff

You will be hard-pressed to find any difference between original and VRS-optimized rendering — color deviation is negligible, and blocky artifacts are really hard to spot. Only ImageMagick was able to show different pixels:

Both VRS-enabled and regular rendering pipelines result in steady 120 FPS on our test device (Galaxy Samsung A52s). So we’ve run a Snapdragon Profiler to analyze performance and efficiency of the optimized build. Here are the numbers:

Bonsai 3D Live Wallpaper, regular mode:

Bonsai 3D Live Wallpaper, battery saving mode.

Bonsai 3D Live Wallpaper, silhouette mode.

In the silhouette scene we don’t use different VRS blocks for regular and power saving modes because it already uses maximum block size and still renders the image identical to non-VRS one.

Long story short, we’ve improved rendering efficiency by approximately 30% with little to (literally) none image quality reduction.

You can find a live web demo here, and for persons sensitive to flickering lights a version without lightning is here. You can interact with it by clicking the mouse on screen — this will change animation. Also you can enter free-camera mode which uses WASD navigation by pressing the Enter key.

As usual, source code is available on Github.

And of course you can get an Android live wallpaper app.

Scene Composition

Scene is pretty simple so it doesn’t require any sorting of objects — carefully chosen hardcoded render order achieves minimal overdraw:

First, opaque (cloth is alpha-masked so it is also opaque) geometries are rendered. These animated objects use vertex animation with data stored in FP16 textures, so WebGL 2 is required for the demo. After rendering opaque geometries, writing to depth is disabled with glDepthMask(false) and then transparent effects — smoke, dust and ghosts are drawn over them with blending. Sky is also drawn at this stage. Because it is the most distant object, it doesn’t have to contribute to depth — it is basically treated as a far clipping plane.

Effects

That’s where most of the time was spent — thinking of, creating, tweaking and rejecting various effects for a really simple scene with literally a single character in it.

Every time I had an idea on how to improve a look I added it to the Trello board. Then I had some time to think about it — how will it fit the scene, how to implement it, etc. So here is a breakdown of all used effects.

First, soft particles are added to the reaper. Half of them rise upwards, half of them sink down from roughly the centre of the reaper model which fluctuates a little depending on animation. Of course to get the best visual appearance soft particles are used, hence the depth pre-pass. You can read about implementation of soft particles in one of my previous articles.

Then some flickering dust is rendered. You may notice that its brightness is synchronized with lightning strikes — usually dust slowly fades in and out but at lightning strikes it is more visible.

As a final touch, a rather heavy vignette is applied. This effect blends nicely with the gloomy atmosphere, helps to draw attention to the centre of screen and to visually conceal the bland void in the corners of the screen.

There are still a couple of effect ideas noted in my Trello board but I think that adding them will only clutter the scene without adding any more noticeable eye candies.

##Sky shader Sky is used to fill in the void around the main character. To add some dynamics and movement to these empty parts of the scene it is rendered with a shader which applies simple distortion and lightning to static clouds texture.

Let’s analyse the shader code. It combines three simple effects to create a dynamic sky:

It starts with applying colour to rather bland-looking greyscale base sky texture:

Then, waves from a small distortion texture are applied (a similar but more pronounced effect can be used for water ripples). Effect is subtle but does noticeably improve overall look:

And the final touch is lightning. To recreate somewhat realistic looking lighting which cannot get through dense clouds but shines through clear areas, brightness is increased exponentially — darker parts will get very little increase in brightness while bright areas will be highlighted. Final result with all effects combined looks like this:

Timer for the lightning strikes is a periodic function of several sine waves combined, clamped to range [0…2]. I’ve used a really handy Desmos graphing calculator to visualize and tweak coefficients for this function — you can clearly see that the “spikes” of positive values create short periodic randomized bursts:

Additionally, the sky sphere slowly rotates to make the background less static.

Ghosts shader

Ghostly trails floating around the grim reaper are inspired by this Unreal Engine 4 Niagara tutorial — https://www.artstation.com/artwork/ba4mNn.

Initial idea was to use a geometry in a shape of cut-out from the cylinder side and rotate it around the centre of the reaper model. However, my brother created a shader for a more flexible approach to use a single geometry which can be rotated at arbitrary radius and stretched to arbitrary length.

To achieve this, vertex shader changes the geometry of the original mesh. It modifies X and Y coordinates of the input model, bending them around the circle of given radius. Z coordinate is not getting additional transformations. It is responsible for scaling the final effect vertically. (World space is Z-up). Shader is tailored to work with a specific model — a tessellated sheet in the XZ plane (all Y coordinates are zero):

Later, geometry was optimized to tightly fit our sprite texture in order to reduce overdraw:

Based on the math of chord length, the X and Y coordinates of bent model are:

x = R * sin(theta);
y = R * cos(theta);

where theta = rm_Vertex.x / R, and R is a bend radius. However, theta is calculated differently in the shader:

float theta = rm_Vertex.x * lengthToRadius;

lengthToRadius value is a uniform, but it is not just a reciprocal of R — we can pass values greater than 1/R to get effect length scaled (because it essentially is a pre-multiplication of rm_Vertex.x). This minor change is done in order to eliminate redundant uniform-only math in the shader. Preliminary division of length by radius is done on the CPU and this result is passed into the shader via lengthToRadius uniform. I’ve tried to improve this effect by applying displacement distortion in fragment shader but it appears to be virtually unnoticeable in motion. So we kept the original simpler version with static texture, which is also cheaper for the GPU.

Reduced colours filter

Not implemented in the web version, but present in Android app is a reduced colours post-processing. This gritty effect perfectly fits the overall atmosphere and adds a right mood to the scene. It is implemented not as a separate post-processing render pass but is done in the fragment shader, so rendering is still essentially single-pass.

It is based on code from Q1K3 WebGL game https://github.com/phoboslab/q1k3, and I highly recommend to read a blog post about making of seemingly impossible Q1K3 — https://phoboslab.org/log/2021/09/q1k3-making-of.

Textures compression

Android live wallpaper targets OpenGL ES 3.0+ and uses efficient ETC2 and ASTC compressed textures. However, WebGL demo is optimized only for the fastest possible loading time. I really hate when some simple WebGL demo takes forever to load its unjustifiably huge resources. Because of this, a decision not to use hardware compressed textures was made. Instead, textures are compressed as lossy WebP. Total size of all assets including HTML/CSS/JS is just 2.7 MB so it loads pretty fast. Recently, our mountains WebGL demo has also been updated with smaller resources but it is still way larger than the Reaper one — it downloads 10.8 MB of data on initial load.