bAllowCompilingThroughWorkers

bAllowCompilingThroughWorkers

#Overview

name: bAllowCompilingThroughWorkers

The value of this variable can be defined or overridden in .ini config files. 1 .ini config file referencing this setting variable.

It is referenced in 10 C++ source files.

#Summary

#Usage in the C++ source code

The purpose of bAllowCompilingThroughWorkers is to control whether shader compilation can be performed through worker processes, allowing for multi-core utilization during shader compilation.

This setting variable is primarily used in the shader compilation system of Unreal Engine. It is referenced in the ShaderCompiler module, specifically in the FShaderCompilingManager and FShaderCompileThreadRunnable classes.

The value of this variable is typically set in the engine configuration file (GEngineIni) under the [DevOptions.Shaders] section. It can also be overridden through command-line parameters or console variables.

bAllowCompilingThroughWorkers interacts with other variables and systems, such as:

Developers should be aware of the following when using this variable:

  1. Setting it to false will force shader compilation to occur in-process and single-threaded.
  2. It can be automatically disabled if the platform doesn’t support multi-threading or if specific command-line arguments are used.
  3. It affects the number of shader compiling threads used, both during normal operation and game execution.

Best practices when using this variable include:

  1. Ensure it’s enabled for better performance on multi-core systems.
  2. Consider disabling it for debugging purposes or when investigating shader compilation issues.
  3. Be aware of its interaction with other shader compilation settings and adjust accordingly based on the specific needs of your project and development environment.
  4. Monitor shader compilation performance and adjust the setting if necessary, especially on machines with few cores where compile latency might be prioritized over editor performance.

#Setting Variables

#References In INI files

Location: <Workspace>/Engine/Config/BaseEngine.ini:1949, section: [DevOptions.Shaders]

#References in C++ code

#Callsites

This variable is referenced in the following C++ source code:

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:4320

Scope (from outer to inner):

file
function     int32 FShaderCompileThreadRunnable::CompilingLoop

Source code excerpt:

int32 FShaderCompileThreadRunnable::CompilingLoop()
{
	if (!Manager->bAllowCompilingThroughWorkers && CVarCompileParallelInProcess.GetValueOnAnyThread())
	{
		int32 NumJobs = Manager->GetNumPendingJobs();
		if (NumJobs == 0)
		{
			return 0;
		}

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:4384

Scope (from outer to inner):

file
function     int32 FShaderCompileThreadRunnable::CompilingLoop

Source code excerpt:

		return 0;
	}
	else // compile either through worker processes or single-threaded in-process (depending on Manager->bAllowCompilingThroughWorkers)
	{
		// push completed jobs to Manager->ShaderMapJobs before asking for new ones, so we can free the workers now and avoid them waiting a cycle
		PushCompletedJobsToManager();

		// Grab more shader compile jobs from the input queue
		const int32 NumActiveThreads = PullTasksFromQueue();

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:4400

Scope (from outer to inner):

file
function     int32 FShaderCompileThreadRunnable::CompilingLoop

Source code excerpt:

		}

		if (Manager->bAllowCompilingThroughWorkers)
		{
			// Write out the files which are input to the shader compile workers
			WriteNewTasks();

			// Launch shader compile workers if they are not already running
			// Workers can time out when idle so they may need to be relaunched

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:4413

Scope (from outer to inner):

file
function     int32 FShaderCompileThreadRunnable::CompilingLoop

Source code excerpt:

				// Fall back to local compiles if the SCW crashed.
				// This is nasty but needed to work around issues where message passing through files to SCW is unreliable on random PCs
				Manager->bAllowCompilingThroughWorkers = false;

				// Try to recover from abandoned workers after a certain amount of single-threaded compilations
				if (Manager->NumSingleThreadedRunsBeforeRetry == GSingleThreadedRunsIdle)
				{
					// First try to recover, only run single-threaded approach once
					Manager->NumSingleThreadedRunsBeforeRetry = 1;

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:4455

Scope (from outer to inner):

file
function     int32 FShaderCompileThreadRunnable::CompilingLoop

Source code excerpt:

				{
					UE_LOG(LogShaderCompilers, Display, TEXT("Retry shader compiling through workers."));
					Manager->bAllowCompilingThroughWorkers = true;
				}
			}
		}

		return NumActiveThreads;
	}

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:5610

Scope (from outer to inner):

file
function     FShaderCompilingManager::FShaderCompilingManager

Source code excerpt:


	// Read values from the engine ini
	verify(GConfig->GetBool( TEXT("DevOptions.Shaders"), TEXT("bAllowCompilingThroughWorkers"), bAllowCompilingThroughWorkers, GEngineIni ));
	verify(GConfig->GetBool( TEXT("DevOptions.Shaders"), TEXT("bAllowAsynchronousShaderCompiling"), bAllowAsynchronousShaderCompiling, GEngineIni ));

	// Explicitly load ShaderPreprocessor module so it will run its initialization step
	FModuleManager::LoadModuleChecked<IModuleInterface>(TEXT("ShaderPreprocessor"));
	
	// override the use of workers, can be helpful for debugging shader compiler code

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:5620

Scope (from outer to inner):

file
function     FShaderCompilingManager::FShaderCompilingManager

Source code excerpt:

	if (!FPlatformProcess::SupportsMultithreading() || FParse::Param(FCommandLine::Get(), TEXT("noshaderworker")) || (CVarAllowCompilingThroughWorkers && CVarAllowCompilingThroughWorkers->GetInt() == 0))
	{
		bAllowCompilingThroughWorkers = false;
	}

	if (!FPlatformProcess::SupportsMultithreading())
	{
		bAllowAsynchronousShaderCompiling = false;
	}

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:5689

Scope (from outer to inner):

file
function     FShaderCompilingManager::FShaderCompilingManager

Source code excerpt:


	TUniquePtr<FShaderCompileThreadRunnableBase> RemoteCompileThread;
	const bool bCanUseRemoteCompiling = bAllowCompilingThroughWorkers && ShaderCompiler::IsRemoteCompilingAllowed() && AllTargetPlatformSupportsRemoteShaderCompiling();
	BuildDistributionController = bCanUseRemoteCompiling ? FindRemoteCompilerController() : nullptr;
	
	if (BuildDistributionController)
	{
		UE_LOG(LogShaderCompilers, Display, TEXT("Using %s for Shader Compilation."), *BuildDistributionController->GetName());
		RemoteCompileThread = MakeUnique<FShaderCompileDistributedThreadRunnable_Interface>(this, *BuildDistributionController);

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Private/ShaderCompiler/ShaderCompiler.cpp:5894

Scope (from outer to inner):

file
function     void FShaderCompilingManager::CalculateNumberOfCompilingThreads

Source code excerpt:

	}

	NumShaderCompilingThreads = (bAllowCompilingThroughWorkers && NumVirtualCores > NumUnusedShaderCompilingThreads) ? (NumVirtualCores - NumUnusedShaderCompilingThreads) : 1;

	// Make sure there's at least one worker allowed to be active when compiling during the game
	NumShaderCompilingThreadsDuringGame = (bAllowCompilingThroughWorkers && NumVirtualCores > NumUnusedShaderCompilingThreadsDuringGame) ? (NumVirtualCores - NumUnusedShaderCompilingThreadsDuringGame) : 1;

	// On machines with few cores, each core will have a massive impact on compile time, so we prioritize compile latency over editor performance during the build
	if (NumVirtualCores <= 4)
	{
		NumShaderCompilingThreads = NumVirtualCores - 1;
		NumShaderCompilingThreadsDuringGame = NumVirtualCores - 1;

#Loc: <Workspace>/Engine/Source/Runtime/Engine/Public/ShaderCompiler.h:712

Scope (from outer to inner):

file
class        class FShaderCompilingManager : IAssetCompilingManager

Source code excerpt:

	uint32 ProcessId;
	/** Whether to allow compiling shaders through the worker application, which allows multiple cores to be used. */
	bool bAllowCompilingThroughWorkers;
	/** Whether to allow shaders to compile in the background or to block after each material. */
	bool bAllowAsynchronousShaderCompiling;
	/** Whether to ask to retry a failed shader compile error. */
	bool bPromptToRetryFailedShaderCompiles;
	/** If enabled when we enter the prompt to retry we will break in the debugger if one is attached rather than prompting. */
	bool bDebugBreakOnPromptToRetryShaderCompile = false;