The vertex shader is relatively simple once the required conditions to the good work are met. Like any technological progress in the world of 3D graphics controllers, vertex displacement mapping has some little limitations which, if they are not taken into account, will transform your Ultra-Boosted 7800 GTX into an old S3 trio (do you remember...).

To achieve vertex displacement mapping, there are two main constraints:

Thus once the displacement map is loaded in floating point format with nearest filtering, displacement mapping becomes really simple as shown in the following vertex / pixel shader:

[Vertex_Shader]
	
uniform sampler2D displacementMap;

void main(void)
{
	vec4 newVertexPos;
	vec4 dv;
	float df;
	
	gl_TexCoord[0].xy = gl_MultiTexCoord0.xy;
	
	dv = texture2D( displacementMap, gl_MultiTexCoord0.xy );
	
	df = 0.30*dv.x + 0.59*dv.y + 0.11*dv.z;
	
	newVertexPos = vec4(gl_Normal * df * 100.0, 0.0) + gl_Vertex;
	
	gl_Position = gl_ModelViewProjectionMatrix * newVertexPos;
}

[Pixel_Shader]

uniform sampler2D colorMap;

void main(void)
{
   gl_FragColor = texture2D(colorMap, gl_TexCoord[0].xy);
}

We fetch the displacement map the same way as we do for texture fetching in the pixel shader: using the texture2D() GLSL function.

The Demoniak3D demo shows the deformation of a 80000-polygons mesh with a simple BMP image:

Fig. 4 - DEMO_displacement_mapping.xml

The nearest filtering mode is the only mode available in the vertex shader of today's hardware. But nothing can prevent us from implementing our own version of the bilinear filtering mode. Here is the function (adapted from nVidia's Cg code) that does the job:

#define textureSize 256.0
#define texelSize 1.0 / 256.0

vec4 texture2D_bilinear( uniform sampler2D tex, vec2 uv )
{
	vec2 f = fract( uv.xy * textureSize );
	vec4 t00 = texture2D( tex, uv );
	vec4 t10 = texture2D( tex, uv + vec2( texelSize, 0.0 ));
	vec4 tA = mix( t00, t10, f.x );
	vec4 t01 = texture2D( tex, uv + vec2( 0.0, texelSize ) );
	vec4 t11 = texture2D( tex, uv + vec2( texelSize, texelSize ) );
	vec4 tB = mix( t01, t11, f.x );
	return mix( tA, tB, f.y );
}

The use of this function is really simple. Just remplace texture2D() by texture2D_bilinear():

dv = texture2D_bilinear( displacementMap, gl_MultiTexCoord0.xy );

This code works fine in both vertex and pixel shaders.