Testing
Why have one system when you can have four
Why have one system when you can have four
Unreal Engine provides tools and features to help test the game.
The session frontend is one way to run the tests. In the editor it is found in the menu Window/Developer Tools/Session Frontend. It can also be run standalone in Engine/Binaries/Win64/ UnrealFrontend.exe. The Automation tab list the tests that are available in the session you are connected to. Select the tests using the checkboxes and the press the start tests button to run them.
Run UE4Editor-Cmd.exe and use the command-line argument -ExecCmds to run the automation console command. Multiple commands can be separated with semicolon.
List - Show the available tests. It's the same ones shown in the session frontend.
RunFilter - Run tests that belong to one of Engine, Smoke, Stress, Perf or Product.
RunTests - Select test on Group:, Filter:XXX (only test that start with XXX) or substring match.
RunAll - Run all the tests. Takes time so better to select the specific ones to run.
Besides -ExecCmds there are a few UE4 commands line arguments that are useful for testing.
-NullRHI - Don't instantiate the whole editor.
-testexit="Automation Test Queue Empty" - When the tests are completed the editor will log 'Automation Test Queue Empty'. This option will shut down the editor when it is printed.
-unattended - Disable anything requiring feedback from user.
-nopause - Close the log window automatically on exit.
-ReportOutputPath - Generates a report of all the tests in JSON format in the destination folder. It can be converted to a jUnit XML without to much work.
-log="loadmaps.log" - Set the filename of the log file to use. If running multiple test runs in a row it can be good to give them separate names.
There are a few tests in the engine that are easy and useful to run on the project.
Project.Blueprints.Compile Anims - Compile animation blueprints.
Project.Blueprints.Compile Blueprints - Compile blueprints.
Project.Maps.Load All In Game - Load all levels.
Automation Testing are declared by macros, overriding virtual functions from the FAutomationTestBase class. Both macros ( IMPLEMENT_SIMPLE_AUTOMATION_TEST and IMPLEMENT_COMPLEX_AUTOMATION_TEST ) takes three parameters, TClass, PrettyName and TFlags. TClass is the C++ class to create, PrettyName is the hierarchical test name that will appear in the UI and TFlags tells unreal how the test will be used. Two functions can be implemented for the class. RunTest() performs the actual test, return true if the test passed or false if it failed. GetTests() is used in Complex Tests only, it creates a array of parameters that will be used to call RunTest() once for each parameter. It can be used to get a list of maps for example and then run the test on each map. Each map will be shown as a seperate test in the UI.
To run code over multiple frames Latent Commands (IAutomationLatentCommand) are used. Use it with the macro ADD_LATENT_AUTOMATION_COMMAND with the constructor of the Latent Command you wish to add. Examples of a command is FWaitLatentCommand to wait a number of seconds. Define your own with the DEFINE_LATENT_AUTOMATION_COMMAND macro. I takes a single parameter, the name of the C++ class to create. Then override it's update(), it will be called every frame until it return true. To define commands with a parameter use the DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER macro.
Put tests in the Private\Tests directory. Name the file [ClassFilename]Test.cpp.
include Misc/AutomationTest.h
Wrap the tests in #if WITH_DEV_AUTOMATION_TESTS
The Functional Test (AFunctionalTest) is used to setup test in levels. The test itself can be built into the functional test (with code or blueprint) or created directly in the level blueprint. Override the PrepareTest, IsReady, OnTestStart and OnTestFinished as needed. When the test is complete call FinishTest with the test result. The name of the actor will be used as the test name in the Sessions Frontend.
Gauntlet is used to test the cooked game. Collecting profiling data is easy and it can run on platforms of your choice. The whole thing is easy to manage by CI tools such as jenkins. RunUAT is used to start the game and let's you select the configuration to run and the platform to run on. It also takes the name of the test to run.
RunUAT RunUnreal -project -platform -configuration -build -execcmds -test -uploaddir
project = Full path of the .uproject file.
platform = Platform to run the test on.
configuration = Configuration to use to run the test.
build = Directory where the builds for all platforms are stored. Gauntlet do not create the builds, it only runs the test on them.
execcmds = Any commands to run in the game. Open a map can be a good choice.
test = Name of the gauntlet test to run.
uploaddir = Where to put the output of the test.
The Test
The test is a C# class based on the class DefaultTest. The source build of the engine is needed for this and create your tests in a Class Library (.Net Framework). Set project name to be one that ends in .Automation. The code needs to use the GetConfiguration function to retrun a UE4TestConfig. One important parameter is the name of the GauntletController to use in the test. Look in Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Game\Samples for examples of tests.
GauntletController
The GauntletController is a class based on UGauntletTestController you write to control the test when the game is running. It gets ticked and let you EndTest when it is completed. Look in Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private for example controllers.
FCsvProfiler
One way to capture performence while testing is with the FCsvProfiler class. Use FCsvProfiler::Get() to use it. It is like the csvprofile console commando. Csv files are stored in Saved/Profiling directory. Use Engine\Binaries\DotNET\CsvTools\PerfreportTool.exe to turn a csv file into a html one.
Gauntlet Automated Testing and Performance Metrics in UE4 - 2020
Harnessing the Unreal Engine Automation Framework for Performance Measurement - 2020
2020 - Harnessing the Unreal Engine Automation Framework for Performance Measurement
2019 - Automated Testing of Gameplay Features in 'Sea of Thieves'
2019 - Automated Testing at Scale in Sea of Thieves
2018 - UE4 Unit Tests in Jenkins
JUnit XML reporting file format for Jenkins