abtest.NumResamples

abtest.NumResamples

#Overview

name: abtest.NumResamples

This variable is created as a Console Variable (cvar).

It is referenced in 3 C++ source files.

#Summary

#Usage in the C++ source code

The purpose of abtest.NumResamples is to determine the number of resamples used in AB testing to establish confidence in the test results. This variable is primarily used in the profiling and debugging subsystem of Unreal Engine, specifically for AB testing functionality.

The Unreal Engine subsystem that relies on this setting variable is the Core module, particularly the AB testing component within the profiling and debugging system. This can be inferred from the file location: Engine/Source/Runtime/Core/Private/ProfilingDebugging/ABTesting.cpp.

The value of this variable is set through a console variable (CVar) system. It is initialized with a default value of 256, but can be modified at runtime through console commands or configuration files.

The associated variable that interacts with abtest.NumResamples is CVarABTestNumResamples. This is a TAutoConsoleVariable that wraps the console variable, providing a programmatic interface to access and modify its value.

Developers should be aware that:

  1. This variable directly affects the statistical reliability of AB test results.
  2. Increasing the number of resamples will improve confidence but may also increase computational overhead.
  3. The value is read and applied when starting an AB test, as seen in the FABTest::Start function.

Best practices when using this variable include:

  1. Carefully consider the trade-off between statistical confidence and performance impact when adjusting this value.
  2. Monitor the impact of different resample numbers on your specific AB tests to find an optimal balance.
  3. Document any changes to this value in project configurations to ensure consistency across the development team.

Regarding the associated variable CVarABTestNumResamples:

The purpose of CVarABTestNumResamples is to provide a programmatic interface to the abtest.NumResamples console variable. It allows C++ code to easily access and modify the number of resamples used in AB testing.

This variable is part of the Core module’s AB testing system and is used to retrieve the current value of the resample count when starting an AB test.

The value of CVarABTestNumResamples is set automatically based on the abtest.NumResamples console variable. It’s updated whenever the console variable changes.

CVarABTestNumResamples interacts directly with the abtest.NumResamples console variable, effectively serving as its C++ representation.

Developers should be aware that:

  1. Changes to CVarABTestNumResamples will affect all AB tests in the current session.
  2. The value is read at the start of each AB test, so changes during an active test won’t take effect until the next test starts.

Best practices for using CVarABTestNumResamples include:

  1. Use GetValueOnGameThread() to safely retrieve the current value, as seen in the FABTest::Start function.
  2. Consider caching the value if it’s needed frequently, to avoid repeated calls to GetValueOnGameThread().
  3. If modifying the value programmatically, ensure it’s done in a thread-safe manner and consider the impact on ongoing or future AB tests.

#References in C++ code

#Callsites

This variable is referenced in the following C++ source code:

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/ProfilingDebugging/ABTesting.cpp:41

Scope: file

Source code excerpt:


static TAutoConsoleVariable<int32> CVarABTestNumResamples(
	TEXT("abtest.NumResamples"),
	256,
	TEXT("The number of resamples to use to determine confidence."));



FABTest::FABTest()

#Associated Variable and Callsites

This variable is associated with another variable named CVarABTestNumResamples. They share the same value. See the following C++ source code.

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/ProfilingDebugging/ABTesting.cpp:40

Scope: file

Source code excerpt:

	TEXT("The number of frames to run a given command before switching; this is randomized."));

static TAutoConsoleVariable<int32> CVarABTestNumResamples(
	TEXT("abtest.NumResamples"),
	256,
	TEXT("The number of resamples to use to determine confidence."));

#Loc: <Workspace>/Engine/Source/Runtime/Core/Private/ProfilingDebugging/ABTesting.cpp:349

Scope (from outer to inner):

file
function     void FABTest::Start

Source code excerpt:

	CoolDown = CVarABTestCoolDown.GetValueOnGameThread();
	MinFramesPerTrial = CVarABTestMinFramesPerTrial.GetValueOnGameThread();
	NumResamples = CVarABTestNumResamples.GetValueOnGameThread();

	Samples.Empty(HistoryNum);
	ResampleAccumulators.Empty(NumResamples);
	ResampleAccumulators.AddZeroed(NumResamples);
	ResampleCount.Empty(NumResamples);
	ResampleCount.AddZeroed(NumResamples);