AddWork

AddWork

#Overview

name: AddWork

This variable is created as a Console Variable (cvar).

It is referenced in 29 C++ source files.

#Summary

#Usage in the C++ source code

The purpose of AddWork is to add new tasks or work items to a work queue or job system within the Unreal Engine. This function is used in various subsystems to schedule work that needs to be executed, often in a parallel or asynchronous manner.

Based on the provided callsites, AddWork is utilized in several Unreal Engine subsystems, plugins, and modules:

  1. UnrealEd: Used for testing and debugging purposes in the editor.
  2. VirtualHeightfieldMesh: A plugin for rendering virtual heightfield meshes.
  3. UnrealBuildAccelerator: A tool for speeding up build processes.
  4. Networking: Used in the NetworkServer class for managing network-related tasks.
  5. Storage: Used in various storage-related operations, including file compression and decompression.
  6. Work Management: Part of a general work management system in the engine.

The value of this variable is typically set when calling the AddWork function, which takes a work item (usually a lambda function or a function object) and a count of how many times to add that work item.

AddWork often interacts with other variables and systems:

Developers should be aware of the following when using AddWork:

  1. Thread safety: Ensure proper synchronization when adding work from multiple threads.
  2. Work item lifecycle: Be cautious about capturing variables in lambdas, ensuring they remain valid for the duration of the work item’s execution.
  3. Performance impact: Adding too many small work items can lead to overhead; consider batching work when appropriate.

Best practices for using AddWork include:

  1. Use descriptive names for work items to aid in debugging and profiling.
  2. Balance work distribution to avoid overloading the work queue or creating too fine-grained tasks.
  3. Consider the priority and dependencies of work items when adding them.
  4. Use the appropriate WorkManager for the context (e.g., rendering, networking, etc.).
  5. Monitor and profile the work queue to ensure efficient execution and identify bottlenecks.

#References in C++ code

#Callsites

This variable is referenced in the following C++ source code:

#Loc: <Workspace>/Engine/Source/Editor/UnrealEd/Private/GlobalEditorNotification.cpp:165

Scope: file

Source code excerpt:

}
FAutoConsoleCommand StartWorkTestCommand(TEXT("StartWorkTest"), TEXT(""), FConsoleCommandDelegate::CreateStatic(&StartWorkTest));
FAutoConsoleCommand AddWorkTestCommand(TEXT("AddWork"), TEXT(""), FConsoleCommandDelegate::CreateStatic(&AddWork));
FAutoConsoleCommand StopWorkTestCommand(TEXT("StopWorkTest"), TEXT(""), FConsoleCommandDelegate::CreateStatic(&StopWorkTest));

#Loc: <Workspace>/Engine/Plugins/Experimental/VirtualHeightfieldMesh/Source/VirtualHeightfieldMesh/Private/VirtualHeightfieldMeshSceneProxy.cpp:184

Scope (from outer to inner):

file
class        class FVirtualHeightfieldMeshRendererExtension : public FRenderResource

Source code excerpt:


	/** Call once per frame for each mesh/view that has relevance. This allocates the buffers to use for the frame and adds the work to fill the buffers to the queue. */
	VirtualHeightfieldMesh::FDrawInstanceBuffers& AddWork(FRHICommandListBase& RHICmdList, FVirtualHeightfieldMeshSceneProxy const* InProxy, FSceneView const* InMainView, FSceneView const* InCullView);
	/** Submit all the work added by AddWork(). The work fills all of the buffers ready for use by the referencing mesh batches. */
	void SubmitWork(FRDGBuilder& GraphBuilder);

protected:
	//~ Begin FRenderResource Interface
	virtual void ReleaseRHI() override;
	//~ End FRenderResource Interface

#Loc: <Workspace>/Engine/Plugins/Experimental/VirtualHeightfieldMesh/Source/VirtualHeightfieldMesh/Private/VirtualHeightfieldMeshSceneProxy.cpp:265

Scope (from outer to inner):

file
function     VirtualHeightfieldMesh::FDrawInstanceBuffers& FVirtualHeightfieldMeshRendererExtension::AddWork

Source code excerpt:

}

VirtualHeightfieldMesh::FDrawInstanceBuffers& FVirtualHeightfieldMeshRendererExtension::AddWork(FRHICommandListBase& RHICmdList, FVirtualHeightfieldMeshSceneProxy const* InProxy, FSceneView const* InMainView, FSceneView const* InCullView)
{
	UE::TScopeLock Lock(Mutex);

	// If we hit this then BegineFrame()/EndFrame() logic needs fixing in the Scene Renderer.
	if (!ensure(!bInFrame))
	{

#Loc: <Workspace>/Engine/Plugins/Experimental/VirtualHeightfieldMesh/Source/VirtualHeightfieldMesh/Private/VirtualHeightfieldMeshSceneProxy.cpp:549

Scope (from outer to inner):

file
function     void FVirtualHeightfieldMeshSceneProxy::GetDynamicMeshElements

Source code excerpt:

	{
		// Can't add new work while bInFrame.
		// In UE5 we need to AddWork()/SubmitWork() in two phases: InitViews() and InitViewsAfterPrepass()
		// The main renderer hooks for that don't exist in UE5.0 and are only added in UE5.1
		// That means that for UE5.0 we always hit this for shadow drawing and shadows will not be rendered.
		// Not earlying out here can lead to crashes from buffers being released too soon.
		return;
	}

#Loc: <Workspace>/Engine/Plugins/Experimental/VirtualHeightfieldMesh/Source/VirtualHeightfieldMesh/Private/VirtualHeightfieldMeshSceneProxy.cpp:560

Scope (from outer to inner):

file
function     void FVirtualHeightfieldMeshSceneProxy::GetDynamicMeshElements

Source code excerpt:

		if (VisibilityMap & (1 << ViewIndex))
		{
			VirtualHeightfieldMesh::FDrawInstanceBuffers& Buffers = GVirtualHeightfieldMeshViewRendererExtension.AddWork(Collector.GetRHICommandList(), this, ViewFamily.Views[0], Views[ViewIndex]);

			FMeshBatch& Mesh = Collector.AllocateMesh();
			Mesh.bWireframe = AllowDebugViewmodes() && ViewFamily.EngineShowFlags.Wireframe;
			Mesh.bUseWireframeSelectionColoring = IsSelected();
			Mesh.VertexFactory = VertexFactory;
			Mesh.MaterialRenderProxy = Material;

#Loc: <Workspace>/Engine/Source/Editor/UnrealEd/Private/GlobalEditorNotification.cpp:155

Scope (from outer to inner):

file
function     static void AddWork

Source code excerpt:

}

static void AddWork()
{
	int32 TaskIndex = FMath::Rand() % TestProgressNotificationTasks.Num();
	TUniquePtr<FTestProgressNotification>& TestProgressNotificationTask = TestProgressNotificationTasks[TaskIndex];
	//if (TestProgressNotificationTask)
	{
		TestProgressNotificationTask->RemainingJobs += 100;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Cli/Private/UbaCli.cpp:404

Scope (from outer to inner):

file
namespace    uba
function     int WrappedMain
lambda-function

Source code excerpt:

			storageServer.TraverseAllCasFiles([&](const CasKey& casKey)
				{
					workManager.AddWork([&, casKey]()
						{
							Storage::RetrieveResult res;
							storageServer.EnsureCasFile(casKey, TC("Dummy"));
							CasKey casKey2 = AsCompressed(casKey, false);
							if (!storageClient.RetrieveCasFile(res, casKey2, TC("")))
								success = false;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaFileMapping.cpp:566

Scope (from outer to inner):

file
namespace    uba
function     void FileMappingBuffer::UnmapView

Source code excerpt:

			};
		if (m_workManager && newSize == InvalidValue)
			m_workManager->AddWork([=, h = TString(hint_)]() { unmap(h.c_str()); }, 1, TC("UnmapView"));
		else
			unmap(hint_);
	}

	void FileMappingBuffer::GetSizeAndCount(FileMappingType type, u64& outSize, u32& outCount)
	{

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaNetworkServer.cpp:536

Scope (from outer to inner):

file
namespace    uba
function     void NetworkServer::Worker::DoAdditionalWorkAndSignalAvailable

Source code excerpt:

			// Both locks needs to be taken to verify if additional work
			// is present before making ourself available to avoid
			// a race where AddWork would not see this thread in the
			// available list after adding some work.
			SCOPED_WRITE_LOCK(server.m_availableWorkersLock, lock1);
			SCOPED_READ_LOCK(server.m_additionalWorkLock, lock2);
			// Verify there is not additional work while we hold both lock
			// and only add ourself as available if no additional work is present.
			if (!server.m_additionalWork.empty())

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaNetworkServer.cpp:779

Scope (from outer to inner):

file
namespace    uba
function     void NetworkServer::AddWork

Source code excerpt:

	}

	void NetworkServer::AddWork(const Function<void()>& work, u32 count, const tchar* desc)
	{
		SCOPED_WRITE_LOCK(m_additionalWorkLock, lock);
		for (u32 i = 0; i != count; ++i)
		{
			m_additionalWork.push_back({ work });
			if (m_workTracker)

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaSession.cpp:2026

Scope (from outer to inner):

file
function     bool Session::WriteFilesToDisk
lambda-function

Source code excerpt:

			if (IsRarelyReadAfterWritten(process, file.name.c_str(), file.name.size()) || file.mappingWritten > m_keepOutputFileMemoryMapsThreshold)
			{
				m_workManager->AddWork([mh = file.mappingHandle]()
					{
						CloseFileMapping(mh);
					}, 1, TC("CFM"));
				//CloseFileMapping(file.mappingHandle);
			}
			else

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaSession.cpp:2072

Scope (from outer to inner):

file
function     bool Session::WriteFilesToDisk

Source code excerpt:

			events[i].Create(true);

			m_workManager->AddWork([&, ii = i]()
				{
					writeFile(*files[ii]);
					events[ii].Set();
				}, 1, TC(""));
		}
		for (u32 i=0; i!=fileCount; ++i)
			events[i].IsSet();
		*/
		return true;
	}

	bool Session::AllocFailed(Process& process, const tchar* allocType, u32 error)
	{
		m_logger.Warning(TC("Allocation failed in %s (%s).. process will sleep and try again"), allocType, LastErrorToText(error).data);
		return true;
	}

	bool Session::GetNextProcess(Process& process, bool& outNewProcess, NextProcessInfo& outNextProcess, u32 prevExitCode, BinaryReader& statsReader)
	{
		if (!m_getNextProcessFunction)
		{
			outNewProcess = false;
			return true;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaSessionClient.cpp:307

Scope (from outer to inner):

file
namespace    uba
function     bool SessionClient::EnsureApplicationEnvironment

Source code excerpt:

			for (auto& m : modules)
			{
				m_client.AddWork([&handledCount, &m, this, &success, &keyStr]()
					{
						++handledCount;
						auto g = MakeGuard([&]() { m.done.Set(); });
						CasKey newCasKey;
						bool storeUncompressed = true;
						u64 fileSize;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaSessionClient.cpp:1181

Scope (from outer to inner):

file
namespace    uba
function     bool SessionClient::SendProcessAvailable

Source code excerpt:

					knownInputKey = AsCompressed(knownInputKey, false);

				m_client.AddWork([knownInputKey, this]()
					{
						Storage::RetrieveResult result;
						bool allowProxy = true;
						bool res = m_storage.RetrieveCasFile(result, knownInputKey, TC("KnownInput"), nullptr, 1, allowProxy);
						(void)res;
					}, 1, TC("KnownInput"));

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:261

Scope (from outer to inner):

file
namespace    uba
function     bool StorageImpl::WriteCompressed
lambda-function

Source code excerpt:

					{
						#if UBA_EXPERIMENTAL
						m_workManager->AddWork([=, f = TString(from)]() { UnmapViewOfFile(uncompressedData, fileSize, f.c_str()); }, 1, TC("UnmapFile"));
						#else
						UnmapViewOfFile(uncompressedData, fileSize, from);
						#endif
					});

				if (!WriteMemToCompressedFile(destinationFile, workCount, uncompressedData, fileSize, maxUncompressedBlock, totalWritten))

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:450

Scope (from outer to inner):

file
namespace    uba
function     bool StorageImpl::WriteMemToCompressedFile

Source code excerpt:


		rec->refCount = workerCount + 1; // We need to keep refcount up 1 to make sure it is not deleted before we read rec->written
		m_workManager->AddWork(work, workerCount-1, TC("Compress")); // We are a worker ourselves
		work();
		rec->events[rec->workCount - 1].IsSet();

		totalWritten += rec->written;
		bool error = rec->error;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:636

Scope (from outer to inner):

file
namespace    uba
function     bool StorageImpl::DecompressMemoryToMemory

Source code excerpt:

				workerCount = Min(workerCount, MaxWorkItemsPerAction); // Cap this to not starve other things
				rec->refCount += workerCount;
				m_workManager->AddWork(work, workerCount, TC("DecompressMemToMem"));
			}

			TimerScope ts(stats.decompressToMem);
			work();
			rec->done.IsSet();
			bool success = !rec->error;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:1464

Scope (from outer to inner):

file
namespace    uba
function     bool StorageImpl::CheckCasContent
lambda-function

Source code excerpt:

			{
				++entryCount;
				workManager.AddWork([&, filePath = TString(fullPath.data), name = TString(e.name), lastWritten = e.lastWritten]()
					{
						StringBuffer<> timeStr;
						writeTimeAgo(timeStr, lastWritten);

						
						//m_logger.Info(TC("Validating %s (%s ago)"), name.c_str(), timeStr.data);

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:1572

Scope (from outer to inner):

file
namespace    uba
function     bool StorageImpl::DeleteAllCas
lambda-function

Source code excerpt:

					if (!IsDirectory(e.attributes))
						return;
					workManager.AddWork([&, name = TString(e.name)]()
						{
							StringBuffer<> fullPath;
							fullPath.Append(m_rootDir).Append(name);
							u32 deleteCountTemp = 0;
							DeleteAllFiles(m_logger, fullPath.data, true, &deleteCountTemp);
							atomicDeleteCount += deleteCountTemp;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:2266

Scope (from outer to inner):

file
namespace    uba
function     CasKey StorageImpl::CalculateCasKey

Source code excerpt:

				workerCount = Min(workerCount, MaxWorkItemsPerAction); // Cap this to not starve other things
				rec->refCount += workerCount;
				m_workManager->AddWork(work, workerCount, TC("CalculateKey"));
			}

			work();
			rec->done.IsSet();

			hasher.Update(rec->keys.data(), rec->keys.size()*sizeof(CasKey));

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:2319

Scope (from outer to inner):

file
namespace    uba
function     CasKey StorageImpl::CalculateCasKey
lambda-function

Source code excerpt:

			auto udg = MakeGuard([&]() { 
					#if UBA_EXPERIMENTAL
					m_workManager->AddWork([=, fn = TString(fileName)]() { UnmapViewOfFile(fileData, fileSize, fn.c_str()); }, 1, TC("UnmapFile"));
					#else
					UnmapViewOfFile(fileData, fileSize, fileName);
					#endif
				});

			struct WorkRec

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorage.cpp:2378

Scope (from outer to inner):

file
namespace    uba
function     CasKey StorageImpl::CalculateCasKey

Source code excerpt:


			rec->refCount = workerCount + 1; // We need to keep refcount up 1 to make sure it is not deleted before we read rec->written
			m_workManager->AddWork(work, workerCount-1, TC("CalculateKey")); // We are a worker ourselves
			work();
			rec->done.IsSet();

			hasher.Update(rec->keys.data(), rec->keys.size()*sizeof(CasKey));

			bool error = rec->error;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorageClient.cpp:1039

Scope (from outer to inner):

file
namespace    uba
function     bool StorageClient::PopulateCasFromDirsRecursive
lambda-function

Source code excerpt:

				}

				workManager.AddWork([&, filePath = TString(fullPath.data), name = TString(e.name)]()
					{
						FileInformation info;
						if (!GetFileInformation(info, m_logger, filePath.c_str()))
						{
							m_logger.Error(TC("Failed to get information for file %s"), filePath.c_str());
							return;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorageProxy.cpp:246

Scope (from outer to inner):

file
namespace    uba
function     bool StorageProxy::HandleMessage

Source code excerpt:


						// Move the additional messages to a job to be able to return this one quickly.
						m_server.AddWork([f = &file, segmentCount, this]()
							{
								SCOPED_WRITE_LOCK(m_largeFileLock, lock);
								//TrackWorkScope tws(m_client, TC("SEGMENTS"));
								auto& file = *f;
								for (u32 i=0; i!=segmentCount; ++i)
								{

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaStorageProxy.cpp:259

Scope (from outer to inner):

file
lambda-function
lambda-function

Source code excerpt:

											auto mif = (MessageInFlight*)userData;
											mif->error = error;
											mif->proxy.m_server.AddWork([mif]() { mif->proxy.HandleReceivedData(*mif); }, 1, TC(""));
										}, mif);
									if (!res)
									{
										// TODO: Don't leak mif
										mif->error = true;
									}

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Private/UbaWorkManager.cpp:99

Scope (from outer to inner):

file
namespace    uba
function     void WorkManagerImpl::AddWork

Source code excerpt:



	void WorkManagerImpl::AddWork(const Function<void()>& work, u32 count, const tchar* desc)
	{
		SCOPED_WRITE_LOCK(m_workLock, lock);
		for (u32 i = 0; i != count; ++i)
		{
			m_work.push_back({ work });
			if (m_workTracker.load())

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Public/UbaNetworkServer.h:72

Scope (from outer to inner):

file
namespace    uba
class        class NetworkServer : public WorkManager

Source code excerpt:

		//void UnregisterOnConnection(u8 id);

		virtual void AddWork(const Function<void()>& work, u32 count, const tchar* desc) override final;
		virtual u32 GetWorkerCount() override final;

		struct ClientStats
		{
			u64 send = 0;
			u64 recv = 0;

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Public/UbaWorkManager.h:16

Scope (from outer to inner):

file
namespace    uba
class        class WorkManager

Source code excerpt:

	{
	public:
		virtual void AddWork(const Function<void()>& work, u32 count, const tchar* desc) = 0;
		virtual u32 GetWorkerCount() = 0;

		u32 TrackWorkStart(const tchar* desc);
		void TrackWorkEnd(u32 id);

		void SetWorkTracker(WorkTracker* workTracker) { m_workTracker = workTracker; }

#Loc: <Workspace>/Engine/Source/Programs/UnrealBuildAccelerator/Common/Public/UbaWorkManager.h:35

Scope (from outer to inner):

file
namespace    uba
class        class WorkManagerImpl : public WorkManager

Source code excerpt:

		WorkManagerImpl(u32 workerCount);
		virtual ~WorkManagerImpl();
		virtual void AddWork(const Function<void()>& work, u32 count, const tchar* desc) override;
		virtual u32 GetWorkerCount() override;
		void DoWork(u32 count = 1);
		void FlushWork();

	private:
		struct Worker;