Currently, we have a single Shader class that encompasses an entire shader pipeline, both handling GLSL and Cg shaders. In the case of GLSL, the shaders are compiled directly by the driver. This has a number of problems:
We cannot reliably provide preprocessor definitions, an oft-requested feature
Relies on driver side support for shader language(no Vulkan support for GLSL!)
Relies on driver to accurately check code, for code to be portable
Relies on driver to support GLSL version the user happens to be using
Assumes that a shader will only be used by one particular graphics API
Can’t know whether a shader compiles until it is used by a GSG
This means that it is currently impossible to use any but the most basic GLSL shaders in Panda3D without having to jump through hoops to make sure that the shaders work on every platform, with every driver, sometimes even needing multiple sets of shaders or custom preprocessing code.
Secondly, there are some issues with the monolithic design of the Shader class itself:
Shader has a lot of code duplication handling different types of shader stages
It’s hard to cache individual shader programs
Using a custom shader generator is not straightforward and requires modifying the source
There is no interface for passing compilation options
Cg is interwoven with the base Shader class, so we can’t split out Cg into a separate plug-in, which means it’s hard to exclude proprietary code from user programs without recompiling libpanda.
I propose that we fix these issues by splitting up the shader pipeline into a front-end and a back-end, and introduce an abstraction for the intermediate representation.
Shader pipeline
Front-end
The front-end ShaderCompiler is what turns shader code into an intermediate representation. We would offer a selection between different front-ends depending on the needs:
Khronos’ glslang or Google’s libshaderc(which uses glslang) is preferred for GLSL and HLSL, which compiles to SPIR-V.
Our own GLSL preprocessor can be kept as compatibility fallback if glslang is not enabled. The“compilation” here just means“preprocessing”; no validation can be done with this.
Cg shaders would use the Cg compiler as front-end.
For users who want to load SPIR-V directly, we would have a front-end that just validates it.
Using the glslang front-end for GLSL on all platforms ensures that shaders are compiled with the same compiler on all platforms; there are no more issues with some drivers being more lenient than others, or supporting a different set of GLSL versions. We can perform reflection, validation and any transformations on the resulting SPIR-V.
Since glslang supports HLSL, this gets us free support for HLSL, which is currently the most popular shading language in the industry. Since HLSL is very similar to Cg syntactically, we could use this opportunity to easily switch the shader generator to HLSL and rid us of the Cg dependency for shader generation. It may be that we can even compile Cg as though it were HLSL.
The front-end is not dependent on a particular GSG or driver and the compilation happens at shader load time, so if the shader fails to be compiled by the front-end, the application will know about it right away, the same way as with loading a model. Asynchronous compilation could be supported similar to loadModel.
We could introduce an CompilerOptions class(similar to LoaderOptions) to bundle various compilation options, such as search path for #include files, optimization options, and preprocessor definitions.
Intermediate representation
The front-ends produce a per-stage intermediate representation, stored in a ShaderModule class.
We could support several intermediate representations:
SPIR-V(preferred)
Preprocessed GLSL code(if glslang is not enabled). Only for compatibility.
Cg object
ARB assembly(hypothetical, if needed to support very ancient GPUs, OpenGL only)
The Shader class effectively becomes a container of ShaderModule objects; it does not contain the original source, since by the time that loader.loadShader returns, it will either have the intermediate representation or know that the shader has failed to compile.
This class solves several problems in one go:
One module corresponds to one shader stage, reducing code duplication currently in Shader
We can subclass ShaderModule for different implementations, fixing the fact that Shader currently handles all sorts of different languages, and making it possible to move CgShaderModule to a plug-in.
We can cache the ShaderModule to disk, similar to how we cache loaded .egg files or compressed textures.
It’s theoretically possible to eg. reuse the same vertex module in different shaders.
Shader pipeline
Front-end
Intermediate representation