HangDuration

HangDuration

#Overview

name: HangDuration

The value of this variable can be defined or overridden in .ini config files. 1 .ini config file referencing this setting variable.

It is referenced in 23 C++ source files.

#Summary

#Usage in the C++ source code

The purpose of HangDuration is to define the maximum time allowed for a thread or operation to complete before it is considered “hung” or unresponsive. This variable is used in Unreal Engine’s thread monitoring and hang detection system.

The HangDuration variable is primarily used in the Core module of Unreal Engine, specifically within the thread heartbeat system. This system is responsible for detecting and reporting when threads or operations are taking longer than expected to complete.

The value of HangDuration is typically set in the engine configuration files (GEngineIni) under the [Core.System] section. It can be overridden via command-line arguments or programmatically during runtime.

HangDuration interacts with several other variables and systems:

  1. StuckDuration: A shorter duration used to detect when a thread is stuck but not yet considered hung.
  2. PresentHangDuration: A specific duration for detecting hangs in frame presentation.
  3. LastHeartBeatTime: Used to calculate the time elapsed since the last heartbeat.
  4. LastHangTime: Tracks when the last hang was detected.

Developers should be aware of the following when using HangDuration:

  1. Setting HangDuration too low may result in false positives, while setting it too high may delay the detection of actual hangs.
  2. Different threads or operations may require different hang durations based on their expected execution times.
  3. The hang detection system can be disabled by setting HangDuration to a negative value.

Best practices for using HangDuration include:

  1. Adjusting the value based on the specific needs of your project and the capabilities of the target hardware.
  2. Using different durations for different types of operations (e.g., frame presentation vs. general thread execution).
  3. Implementing proper error handling and logging when hangs are detected to aid in debugging.
  4. Regularly reviewing and updating hang detection settings as your project evolves.

#Setting Variables

#References In INI files

Location: <Workspace>/Engine/Config/BaseEngine.ini:1447, section: [Core.System]

#References in C++ code

#Callsites

This variable is referenced in the following C++ source code:

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:154

Scope (from outer to inner):

file
function     void FORCENOINLINE FThreadHeartBeat::OnPresentHang

Source code excerpt:

}

void FORCENOINLINE FThreadHeartBeat::OnPresentHang(double HangDuration)
{
#if MINIMAL_FATAL_HANG_DETECTION

	LastHungThreadId = FThreadHeartBeat::PresentThreadId;
	FGenericCrashContext::SetEngineData(TEXT("HungThread"), TEXT("Present"));

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:171

Scope (from outer to inner):

file
function     void FORCENOINLINE FThreadHeartBeat::OnPresentHang

Source code excerpt:


#elif UE_ASSERT_ON_HANG
	UE_LOG(LogCore, Fatal, TEXT("Frame present hang detected. A frame has not been presented for %.2f seconds."), HangDuration);
#else
	UE_LOG(LogCore, Error, TEXT("Frame present hang detected. A frame has not been presented for %.2f seconds."), HangDuration);
#endif
}

void FORCENOINLINE FThreadHeartBeat::OnHang(double HangDuration, uint32 ThreadThatHung)
{
#if MINIMAL_FATAL_HANG_DETECTION

	LastHungThreadId = ThreadThatHung;
	FGenericCrashContext::SetEngineData(TEXT("HungThread"), LexToString(ThreadThatHung));

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:226

Scope (from outer to inner):

file
function     void FORCENOINLINE FThreadHeartBeat::OnHang

Source code excerpt:

			ThreadName = FString::Printf(TEXT("unknown thread (%u)"), ThreadThatHung);
		}
		UE_LOG(LogCore, Error, TEXT("Hang detected on %s (thread hasn't sent a heartbeat for %.2f seconds):"), *ThreadName, HangDuration);
		for (int32 Idx = 0; Idx < StackLines.Num(); Idx++)
		{
			UE_LOG(LogCore, Error, TEXT("  %s"), *StackLines[Idx]);
		}

		// Assert (on the current thread unfortunately) with a trimmed stack.

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:290

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::Run

Source code excerpt:

	while (StopTaskCounter.GetValue() == 0 && !IsEngineExitRequested())
	{
		double HangDuration;
		uint32 ThreadThatHung = CheckHeartBeat(HangDuration);
		if (ThreadThatHung == FThreadHeartBeat::InvalidThreadId)
		{
			ThreadThatHung = CheckFunctionHeartBeat(HangDuration);
		}

		if (ThreadThatHung == FThreadHeartBeat::InvalidThreadId)
		{
			ThreadThatHung = CheckCheckpointHeartBeat(HangDuration);
		}

		if (ThreadThatHung == FThreadHeartBeat::InvalidThreadId)
		{
			InHungState = false;
		}

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:313

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::Run

Source code excerpt:

			if (ThreadThatHung == FThreadHeartBeat::PresentThreadId)
			{
				OnPresentHang(HangDuration);
			}
			else
			{
				OnHang(HangDuration, ThreadThatHung);
			}
		}

		if (StopTaskCounter.GetValue() == 0 && !IsEngineExitRequested())
		{
			FPlatformProcess::SleepNoStats(0.5f);

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:357

Scope (from outer to inner):

file
function     void FThreadHeartBeat::InitSettings

Source code excerpt:

	{
		GConfig->GetDouble(TEXT("Core.System"), TEXT("StuckDuration"), NewStuckDuration, GEngineIni);
		GConfig->GetDouble(TEXT("Core.System"), TEXT("HangDuration"), NewHangDuration, GEngineIni);
		GConfig->GetDouble(TEXT("Core.System"), TEXT("PresentHangDuration"), NewPresentDuration, GEngineIni);
		GConfig->GetBool(TEXT("Core.System"), TEXT("HangsAreFatal"), bNewHangsAreFatal, GEngineIni);

		const double MinStuckDuration = 1.0;
		if (NewStuckDuration > 0.0 && NewStuckDuration < MinStuckDuration)
		{

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:400

Scope (from outer to inner):

file
function     void FThreadHeartBeat::InitSettings

Source code excerpt:

	for (TPair<uint32, FHeartBeatInfo>& Pair : ThreadHeartBeat)
	{
		if (Pair.Value.HangDuration < CurrentHangDuration)
		{
			Pair.Value.HangDuration = CurrentHangDuration;
		}
	}
	
	if (PresentHeartBeat.HangDuration < CurrentPresentDuration)
	{
		PresentHeartBeat.HangDuration = CurrentPresentDuration;
	}
}

void FThreadHeartBeat::HeartBeat(bool bReadConfig)
{
#if USE_HANG_DETECTION

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:429

Scope (from outer to inner):

file
function     void FThreadHeartBeat::HeartBeat

Source code excerpt:

	FHeartBeatInfo& HeartBeatInfo = ThreadHeartBeat.FindOrAdd(ThreadId);
	HeartBeatInfo.LastHeartBeatTime = Clock.Seconds();
	HeartBeatInfo.HangDuration = CurrentHangDuration;
	HeartBeatInfo.StuckDuration = CurrentStuckDuration;
#endif
}

void FThreadHeartBeat::PresentFrame()
{

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:439

Scope (from outer to inner):

file
function     void FThreadHeartBeat::PresentFrame

Source code excerpt:

	FScopeLock HeartBeatLock(&HeartBeatCritical);
	PresentHeartBeat.LastHeartBeatTime = Clock.Seconds();
	PresentHeartBeat.HangDuration = CurrentPresentDuration;

	static bool bFirst = true;
	if (bFirst)
	{
		// Decrement the suspend count on the first call to PresentFrame.
		// This enables frame-present based hang detection on supported platforms.

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:466

Scope (from outer to inner):

file
function     void FThreadHeartBeat::MonitorFunctionStart

Source code excerpt:

	FHeartBeatInfo& HeartBeatInfo = FunctionHeartBeat.FindOrAdd(ThreadId);
	HeartBeatInfo.LastHeartBeatTime = Clock.Seconds();
	HeartBeatInfo.HangDuration = CurrentHangDuration;
	HeartBeatInfo.SuspendedCount = 0;
#endif
}

void FThreadHeartBeat::MonitorFunctionEnd()
{

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:539

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::CheckHeartBeat

Source code excerpt:

					double TimeSinceLastHeartbeat = (CurrentTime - HeartBeatInfo.LastHeartBeatTime);

					if (TimeSinceLastHeartbeat > HeartBeatInfo.HangDuration && HeartBeatInfo.LastHeartBeatTime >= HeartBeatInfo.LastHangTime)
					{
						HeartBeatInfo.LastHangTime = CurrentTime;
						OutHangDuration = HeartBeatInfo.HangDuration;
						return LastHeartBeat.Key;
					}
					else if (HeartBeatInfo.LastHeartBeatTime >= HeartBeatInfo.LastStuckTime)
					{
						// Are we considered stuck?
						if (TimeSinceLastHeartbeat > HeartBeatInfo.StuckDuration)

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:591

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::CheckHeartBeat

Source code excerpt:

		if (ConfigPresentDuration > 0.0)
		{
			if (PresentHeartBeat.SuspendedCount == 0 && (CurrentTime - PresentHeartBeat.LastHeartBeatTime) > PresentHeartBeat.HangDuration)
			{
				// Frames are no longer presenting.
				PresentHeartBeat.LastHeartBeatTime = CurrentTime;
				OutHangDuration = PresentHeartBeat.HangDuration;
				return PresentThreadId;
			}
		}
	}

#endif

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:623

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::CheckFunctionHeartBeat

Source code excerpt:

			{
				FHeartBeatInfo& HeartBeatInfo = LastHeartBeat.Value;
				if (HeartBeatInfo.SuspendedCount == 0 && (CurrentTime - HeartBeatInfo.LastHeartBeatTime) > HeartBeatInfo.HangDuration && HeartBeatInfo.LastHeartBeatTime >= HeartBeatInfo.LastHangTime)
				{
					HeartBeatInfo.LastHangTime = CurrentTime;
					OutHangDuration = HeartBeatInfo.HangDuration;
					return LastHeartBeat.Key;
				}
			}
		}
	}
#endif

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:651

Scope (from outer to inner):

file
function     void FThreadHeartBeat::MonitorCheckpointStart

Source code excerpt:

		FHeartBeatInfo& HeartBeatInfo = CheckpointHeartBeat.Add(EndCheckpoint);
		HeartBeatInfo.LastHeartBeatTime = Clock.Seconds();
		HeartBeatInfo.HangDuration = TimeToReachCheckpoint;
		HeartBeatInfo.HeartBeatName = EndCheckpoint;
		HeartBeatInfo.SuspendedCount = CheckpointSuspendCount;
	}
#endif
}

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:690

Scope (from outer to inner):

file
function     uint32 FThreadHeartBeat::CheckCheckpointHeartBeat

Source code excerpt:

			{
				FHeartBeatInfo& HeartBeatInfo = LastHeartBeat.Value;
				if (HeartBeatInfo.SuspendedCount == 0 && (CurrentTime - HeartBeatInfo.LastHeartBeatTime > HeartBeatInfo.HangDuration && HeartBeatInfo.LastHeartBeatTime >= HeartBeatInfo.LastHangTime))
				{
					if (HeartBeatInfo.HangDuration > 0.0)
					{
						UE_LOG(LogCore, Warning, TEXT("Failed to reach checkpoint within alotted time of %.2f. Triggering hang detector."), HeartBeatInfo.HangDuration);

						HeartBeatInfo.LastHangTime = CurrentTime;
						OutHangDuration = HeartBeatInfo.HangDuration;
						LastHungThreadId = FPlatformTLS::GetCurrentThreadId();

						*((uint32*)3) = 0xe0000001;

						return 0;
					}

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:937

Scope (from outer to inner):

file
function     void FGameThreadHitchHeartBeatThreaded::InitSettings

Source code excerpt:

	static bool CmdLine_StackWalk = false;

	float OldHangDuration = HangDuration;

	if (bFirst)
	{
		bHasCmdLine = FParse::Value(FCommandLine::Get(), TEXT("hitchdetection="), CmdLine_HangDuration);
		CmdLine_StackWalk = FParse::Param(FCommandLine::Get(), TEXT("hitchdetectionstackwalk"));

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:969

Scope (from outer to inner):

file
function     void FGameThreadHitchHeartBeatThreaded::InitSettings

Source code excerpt:

	{
		// Command line takes priority over config
		HangDuration = CmdLine_HangDuration;
		bWalkStackOnHitch = CmdLine_StackWalk;
	}
	else
	{
		float Config_Duration = -1.0f;
		bool Config_StackWalk = false;

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:987

Scope (from outer to inner):

file
function     void FGameThreadHitchHeartBeatThreaded::InitSettings

Source code excerpt:

		if (bReadFromConfig)
		{
			HangDuration = Config_Duration;
			bWalkStackOnHitch = Config_StackWalk;
		}
		else
		{
			// No config provided. Use defaults to disable.
			HangDuration = -1.0f;
			bWalkStackOnHitch = false;
		}
	}

	if (OldHangDuration != HangDuration)
	{
		UE_LOG(LogCore, Display, TEXT("Hitch detector threshold: %dms"), int32(HangDuration * 1000.0f));
	}

	// Start the heart beat thread if it hasn't already been started.
	if (Thread == nullptr && (FPlatformProcess::SupportsMultithreading() || FForkProcessHelper::SupportsMultithreadingPostFork()) && HangDuration > 0)
	{
		Thread = FForkProcessHelper::CreateForkableThread(this, TEXT("FGameThreadHitchHeartBeatThreaded"), 0, TPri_AboveNormal);
	}
#endif
}

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/HAL/ThreadHeartBeat.cpp:1036

Scope (from outer to inner):

file
function     uint32 FGameThreadHitchHeartBeatThreaded::Run

Source code excerpt:

				FScopeLock HeartBeatLock(&HeartBeatCritical);
				LocalFrameStartTime = FrameStartTime;
				LocalHangDuration = HangDuration;
			}
			if (LocalFrameStartTime > 0.0 && LocalHangDuration > 0.0f && SuspendedCount == 0)
			{
				const double CurrentTime = Clock.Seconds();
				if (float(CurrentTime - LocalFrameStartTime) > LocalHangDuration)
				{

#Loc: <Workspace>/Engine/Source/Runtime/Core/Public/HAL/ThreadHeartBeat.h:57

Scope (from outer to inner):

file
class        class FThreadHeartBeat : public FRunnable
function     FHeartBeatInfo

Source code excerpt:

			, LastHangTime(0.0)
			, SuspendedCount(0)
			, HangDuration(0)
			, LastStuckTime(0.0)
			, StuckDuration(0.0)
			, HeartBeatName()
		{}

		/** Time we last received a heartbeat for the current thread */

#Loc: <Workspace>/Engine/Source/Runtime/Core/Public/HAL/ThreadHeartBeat.h:70

Scope (from outer to inner):

file
class        class FThreadHeartBeat : public FRunnable

Source code excerpt:

		int32 SuspendedCount;
		/** The timeout for this thread */
		double HangDuration;

		/** Time we last detected thread stuck due to lack of heartbeats for the current thread */
		double LastStuckTime;
		/** How long it's benn stuck thread */
		double StuckDuration;
		/** An optional FName */

#Loc: <Workspace>/Engine/Source/Runtime/Core/Public/HAL/ThreadHeartBeat.h:150

Scope (from outer to inner):

file
class        class FThreadHeartBeat : public FRunnable

Source code excerpt:

	CORE_API void InitSettings();

	CORE_API void FORCENOINLINE OnHang(double HangDuration, uint32 ThreadThatHung);
	CORE_API void FORCENOINLINE OnPresentHang(double HangDuration);

	CORE_API bool IsEnabled();

public:

	enum EConstants

#Loc: <Workspace>/Engine/Source/Runtime/Core/Public/HAL/ThreadHeartBeat.h:335

Scope (from outer to inner):

file
class        class FGameThreadHitchHeartBeatThreaded : public FRunnable

Source code excerpt:


#if USE_HITCH_DETECTION
	float HangDuration = -1.0f;
	bool bWalkStackOnHitch;
	double FirstStartTime;
	double FrameStartTime;
	int32 SuspendedCount;
	bool bStartSuspended = false;