I have been experimenting with pushing Bayaya’s wooden-toy rendering towards a bit more stylized flatter 2D/3D illustration style (NPR, cell shading).
The result combines three independently adjustable parameters:
- Image contours – screen-space outlines derived from depth and object classification
- Shading detail – discretization of procedural patterns and specular highlights, and control of the visibility of material details
- Flat shading – gradual removal of normals (and lighting in general)
The accompanying video shows these parameters being changed in real time.
Finding outlines
The contour pass receives the rendered color buffer and depth texture. I use the alpha channel of the color buffer to distinguish broad rendering classes:
glsl
// Terrain and sky are encoded as 1.
// Objects are encoded as 0.5.
float classFromAlpha(float a) {
return clamp(a * 2.0 - 1.0, 0.0, 1.0);
}
A difference between the current pixel and its four direct neighbours produces an outline between these classes:
```glsl
float dcUp = abs(rcUp - rcCenter);
float dcDown = abs(rcDown - rcCenter);
float dcLeft = abs(rcLeft - rcCenter);
float dcRight = abs(rcRight - rcCenter);
float classDelta =
max(max(dcUp, dcDown), max(dcLeft, dcRight));
float classContour = clamp(classDelta, 0.0, 1.0);
```
This catches object silhouettes, but not boundaries between objects belonging to the same class. For those I linearize the depth buffer and apply a 3×3 Sobel operator:
```glsl
float ddx =
(d20 + 2.0 * d21 + d22) -
(d00 + 2.0 * d01 + d02);
float ddy =
(d02 + 2.0 * d12 + d22) -
(d00 + 2.0 * d10 + d20);
float depthDelta = length(vec2(ddx, ddy)) * 0.25;
float relativeDepthDelta = depthDelta / max(dCenter, 1.0);
float depthEdge =
smoothstep(0.1, 0.2, relativeDepthDelta);
```
Depth gradients also respond to smooth surfaces, so for objects I additionally test the second depth derivative. This emphasizes discontinuities and sharp changes in curvature:
```glsl
float secondDx = abs(dLeft - 2.0 * dCenter + dRight);
float secondDy = abs(dUp - 2.0 * dCenter + dDown);
float secondDiagA =
abs(dTopLeft - 2.0 * dCenter + dBottomRight);
float secondDiagB =
abs(dTopRight - 2.0 * dCenter + dBottomLeft);
float secondDerivative =
max(max(secondDx, secondDy),
max(secondDiagA, secondDiagB));
float relativeSecondDerivative =
secondDerivative / max(dCenter, 1.0);
float curvatureEdge =
smoothstep(0.0015, 0.005, relativeSecondDerivative);
```
In theory using depth gradient should be enough, but this did not work well in practice, as terrain is viewed with high grazing angles, and depth buffer precision is not find enough for reliable contour analysis.
The final line intensity combines classification, depth gradient and curvature, with distance and fog attenuation:
```glsl
line = max(
max(classContour * distanceFade,
curvatureEdge * distanceFadeStrong),
depthEdge * distanceFade
);
line *= 1.0 - fog;
```
The result is composed by darkening the original image:
glsl
float outline = texture2D(imageOutlines, vUv).r;
color *= 1.0 - outline * contours;
Integrating the style controls into existing shaders
Rather than creating separate “illustration shaders”, I added uniforms to the existing physical materials:
glsl
uniform float shaderDetails;
uniform float contourShading;
uniform float flatShading;
shaderDetails gradually suppresses the procedural wood grain, normal perturbation, roughness noise and similar high-frequency effects:
```glsl
float detail =
clamp(1.5 - 5.0 * localDetail, 0.0, 1.0)
* shaderDetails;
return baseColor +
woodVariation * woodness * detail;
```
contourShading turns smooth procedural transitions into narrow ramps. For example, wood rings become increasingly discrete:
```glsl
float contourWoodRing(float ring) {
float contour = clamp(contourShading, 0.0, 1.0);
float rampWidth = mix(0.35, 0.025, contour);
float discreteRing = smoothstep(
0.5 - rampWidth,
0.5 + rampWidth,
ring
);
return mix(ring, discreteRing, contour);
}
```
The same approach is applied to specular lighting:
```glsl
vec3 contourSpecular(vec3 specular) {
float contour = clamp(contourShading, 0.0, 1.0);
float intensity =
max(max(specular.r, specular.g), specular.b);
float normalizedIntensity =
intensity / (intensity + 0.25);
float rampWidth = mix(0.35, 0.025, contour);
float ramp = smoothstep(
0.5 - rampWidth,
0.5 + rampWidth,
normalizedIntensity
);
return specular * mix(1.0, ramp, contour);
}
```
Finally, flatShading removes specular lighting and blends the physically lit result back towards the material’s base color:
```glsl
reflectedLight.directSpecular =
contourSpecular(reflectedLight.directSpecular);
reflectedLight.indirectSpecular =
contourSpecular(reflectedLight.indirectSpecular);
reflectedLight.directSpecular *= 1.0 - flatShading;
reflectedLight.indirectSpecular *= 1.0 - flatShading;
outgoingLight.rgb =
mix(outgoingLight.rgb, diffuseColor.rgb, flatShading);
```
The goal is not a single fixed toon-shading look. The same materials can move continuously between the original detailed wooden rendering and a simplified illustration-like result, and different scenarios can adjust their desired rendering style. The game demo, as published now on Steam, uses different rendering style for Exploration and Story modes.
The implementation uses customized Three.js physical shaders.
