CustomStencil

CustomStencil

#Overview

name: CustomStencil

The value of this variable can be defined or overridden in .ini config files. 1 .ini config file referencing this setting variable.

It is referenced in 11 C++ source files.

#Summary

#Usage in the C++ source code

The purpose of CustomStencil is to provide a way for developers to assign custom stencil values to objects in the scene, which can be used for various rendering and masking effects. This variable is part of Unreal Engine’s rendering system, specifically related to custom depth and stencil functionality.

CustomStencil is primarily used in the following Unreal Engine subsystems and modules:

  1. Movie Render Pipeline: Used in the MoviePipelineDeferredPasses for configuring custom stencil values during movie rendering.
  2. Material System: Referenced in material translation and expression evaluation.
  3. Rendering System: Used in custom depth rendering, mobile shading, and Nanite composition.

The value of CustomStencil is typically set on primitive components using the SetCustomDepthStencilValue function. It can also be modified through material expressions and shader code.

CustomStencil interacts with other variables such as:

Developers should be aware of the following when using CustomStencil:

  1. It’s part of the custom depth/stencil rendering system, which may have performance implications if overused.
  2. The availability and behavior of CustomStencil can vary depending on the rendering path (e.g., forward vs. deferred) and platform.
  3. For mobile platforms, there may be limitations on how CustomStencil can be sampled in shaders.

Best practices for using CustomStencil include:

  1. Use it sparingly and only when necessary for specific rendering effects or gameplay mechanics.
  2. Be aware of the performance impact, especially on mobile platforms.
  3. Coordinate its usage with other systems that may rely on stencil buffer functionality.
  4. When using it in materials, consider the material domain and any platform-specific limitations.
  5. In movie rendering scenarios, carefully manage the stencil values to achieve desired layering effects.

By following these guidelines, developers can effectively utilize CustomStencil for advanced rendering techniques while maintaining performance and cross-platform compatibility.

#Setting Variables

#References In INI files

Location: <Workspace>/Engine/Config/BaseEngine.ini:2904, section: [Engine.BufferVisualizationMaterials]

#References in C++ code

#Callsites

This variable is referenced in the following C++ source code:

#Loc: <Workspace>/Engine/Plugins/MovieScene/MovieRenderPipeline/Source/MovieRenderPipelineRenderPasses/Private/MoviePipelineDeferredPasses.cpp:588

Scope (from outer to inner):

file
function     void UMoviePipelineDeferredPassBase::RenderSample_GameThreadImpl
function     FStencilValues

Source code excerpt:

					: bRenderCustomDepth(false)
					, StencilMask(ERendererStencilMask::ERSM_Default)
					, CustomStencil(0)
				{
				}

				bool bRenderCustomDepth;
				ERendererStencilMask StencilMask;
				int32 CustomStencil;
			};

			// Now for each stencil layer we reconfigure all the actors custom depth/stencil 
			TArray<FString> AllStencilLayerNames = GetStencilLayerNames();
			if (bAddDefaultLayer)
			{

#Loc: <Workspace>/Engine/Plugins/MovieScene/MovieRenderPipeline/Source/MovieRenderPipelineRenderPasses/Private/MoviePipelineDeferredPasses.cpp:621

Scope (from outer to inner):

file
function     void UMoviePipelineDeferredPassBase::RenderSample_GameThreadImpl

Source code excerpt:

								FStencilValues& Values = PreviousValues.Add(PrimitiveComponent);
								Values.StencilMask = PrimitiveComponent->CustomDepthStencilWriteMask;
								Values.CustomStencil = PrimitiveComponent->CustomDepthStencilValue;
								Values.bRenderCustomDepth = PrimitiveComponent->bRenderCustomDepth;
							}
						}
					}
				}
			}

#Loc: <Workspace>/Engine/Plugins/MovieScene/MovieRenderPipeline/Source/MovieRenderPipelineRenderPasses/Private/MoviePipelineDeferredPasses.cpp:708

Scope (from outer to inner):

file
function     void UMoviePipelineDeferredPassBase::RenderSample_GameThreadImpl

Source code excerpt:

			for (TPair<UPrimitiveComponent*, FStencilValues>& KVP : PreviousValues)
			{
				KVP.Key->SetCustomDepthStencilValue(KVP.Value.CustomStencil);
				KVP.Key->SetCustomDepthStencilWriteMask(KVP.Value.StencilMask);
				KVP.Key->SetRenderCustomDepth(KVP.Value.bRenderCustomDepth);
			}
		}
	}
}

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/Materials/HLSLMaterialTranslator.cpp:7478

Scope (from outer to inner):

file
function     void FHLSLMaterialTranslator::UseSceneTextureId

Source code excerpt:

		if (!bSceneTextureSupportsDecal)
		{
			// Note: For DBuffer decals CustomDepth and CustomStencil are not available if r.CustomDepth.Order = 1
			Errorf(TEXT("Decals can only access SceneDepth, CustomDepth, CustomStencil, and WorldNormal."));
		}

		const bool bSceneTextureRequiresSM5 = SceneTextureId == PPI_WorldNormal;
		if (bSceneTextureRequiresSM5)
		{

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/Materials/MaterialExpressionHLSL.cpp:2119

Scope (from outer to inner):

file
function     bool UMaterialExpressionSceneTexture::GenerateHLSLExpression

Source code excerpt:

			if (!bSceneTextureSupportsDecal)
			{
				// Note: For DBuffer decals CustomDepth and CustomStencil are not available if r.CustomDepth.Order = 1
				return Generator.Error(TEXT("Decals can only access SceneDepth, CustomDepth, CustomStencil, and WorldNormal."));
			}
		}

		if (SceneTextureId == PPI_SceneColor && MaterialDomain != MD_Surface)
		{

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Public/MaterialSceneTextureId.h:65

Scope: file

Source code excerpt:

	/** Ambient Occlusion, single channel */
	PPI_AmbientOcclusion UMETA(DisplayName="Ambient Occlusion"),
	/** Scene stencil, contains CustomStencil mesh property of the opaque objects rendered with CustomDepth */
	PPI_CustomStencil UMETA(DisplayName="CustomStencil"),
	/** Material base, RGB color (GBuffer) */
	PPI_StoredBaseColor UMETA(DisplayName="BaseColor (as stored in GBuffer)"),
	/** Material specular, single channel (GBuffer) */
	PPI_StoredSpecular UMETA(DisplayName="Specular (as stored in GBuffer)"),
	/** Scene Velocity */

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Public/MaterialShared.h:2181

Scope: file

Source code excerpt:

	ENGINE_API bool MaterialUsesSceneDepthLookup_GameThread() const;

	/** The material usage mask of CustomDepth and CustomStencil*/
	ENGINE_API uint8 GetCustomDepthStencilUsageMask_GameThread() const;

	/** Note: This function is only intended for use in deciding whether or not shader permutations are required before material translation occurs. */
	ENGINE_API bool MaterialMayModifyMeshPosition() const;

	/** Get the runtime virtual texture output attribute mask for the material. */

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Public/SceneTexturesConfig.h:199

Scope: file

Source code excerpt:

	uint32 bPreciseDepthAux : 1;

	// (Mobile) True if CustomStencil are sampled in a shader
	uint32 bSamplesCustomStencil : 1;
	
	// (Mobile) True if MSAA targets can be memoryless
	uint32 bMemorylessMSAA : 1;

	// (XR) True if we can request an XR depth swapchain

#Loc: <Workspace>/Engine/Source/Runtime/Renderer/Private/CustomDepthRendering.cpp:394

Scope (from outer to inner):

file
function     bool FSceneRenderer::RenderCustomDepthPass

Source code excerpt:

	{
		const FSceneTexturesConfig& Config = FSceneTexturesConfig::Get();
		// TextureView is not supported in GLES, so we can't lookup CustomDepth and CustomStencil from a single texture
		// Do a copy of the CustomDepthStencil texture if CustomStencil is sampled in a shader.
		if (IsOpenGLPlatform(ShaderPlatform))
		{
			if (Config.bSamplesCustomStencil)
			{
				FRDGTextureRef CustomStencil = GraphBuilder.CreateTexture(CustomDepthTextures.Depth->Desc, TEXT("CustomStencil"));
				AddCopyTexturePass(GraphBuilder, CustomDepthTextures.Depth, CustomStencil);
				CustomDepthTextures.Stencil = GraphBuilder.CreateSRV(FRDGTextureSRVDesc::CreateWithPixelFormat(CustomStencil, PF_X24_G8));
			}
		}
		else
		{
			CustomDepthTextures.Stencil = GraphBuilder.CreateSRV(FRDGTextureSRVDesc::CreateWithPixelFormat(CustomDepthTextures.Depth, PF_X24_G8));
		}

#Loc: <Workspace>/Engine/Source/Runtime/Renderer/Private/MobileShadingRenderer.cpp:134

Scope: file

Source code excerpt:

{
	bool bUsesCustomDepthStencil = false;
	// whether CustomStencil is sampled as a textures
	bool bSamplesCustomStencil = false;
};

static FMobileCustomDepthStencilUsage GetCustomDepthStencilUsage(const FViewInfo& View)
{
	FMobileCustomDepthStencilUsage CustomDepthStencilUsage;

#Loc: <Workspace>/Engine/Source/Runtime/Renderer/Private/Nanite/NaniteComposition.cpp:677

Scope (from outer to inner):

file
function     void EmitCustomDepthStencilTargets

Source code excerpt:

		PassParameters->ClusterPageData				= Nanite::GStreamingManager.GetClusterPageDataSRV(GraphBuilder);
		PassParameters->CustomDepth					= CustomDepth;
		PassParameters->CustomStencil				= CustomStencilSRV;
		PassParameters->RenderTargets[0]			= OutCustomStencil ? FRenderTargetBinding(OutCustomStencil, ERenderTargetLoadAction::ENoAction) : FRenderTargetBinding();
		PassParameters->RenderTargets.DepthStencil	= FDepthStencilBinding(OutCustomDepth, ERenderTargetLoadAction::ENoAction, StencilLoadAction, FExclusiveDepthStencil::DepthWrite_StencilNop);

		FPixelShaderUtils::AddFullscreenPass(
			GraphBuilder,
			View.ShaderMap,