Placating The Natives

Introduction

Blueprint Nativization is a relatively new feature of UE4, claimed to improve performance of Blueprint-heavy projects significantly by converting those Blueprints to pure C++ code.

Working on Conan, we identified Blueprint processing as a significant bottleneck and looked to nativization as one way to help improve that.

Could we just turn it on and enjoy the improvement? Maybe fix a couple of little bugs? Not quite. It didn’t work. Not even close.

Enabling Nativization

Enabling Blueprint Nativization is remarkably straightforward. Within the editor, under “Project Settings” > “Packaging”, set “Blueprint Nativization Method” to either “Inclusive” or “Exclusive”:-

• “Inclusive” – nativizes all Blueprints by default but allowing you to remove from this by editing DefaultEditor.ini and adding the following to the [BlueprintNativizationSettings] block (note that there are examples of these in BaseEditor.ini):-
  • “+ExcludedAssets=/Game/Path/BPName” to exclude individual Blueprints;
  • “+ExcludedBlueprintTypes=..” to exclude entire types of Blueprint (eg. you might want to remove all UMG Widget Blueprints).
• “Exclusive” – defaults to no Blueprints and allows you to add them by checking “Nativize” in “Class Settings” of each Blueprint. This reduces the likelihood of encountering nativization errors – but also makes it much harder to see the full benefit of nativization.

Note that on the cook and UAT commandlines, “-ini:Game:[/Script/UnrealEd.ProjectPackagingSettings]:BlueprintNativizationMethod=” followed by “Disabled”, “Inclusive”, or “Exclusive” can be added if you want to override the INI setting.

With this, cooking and packaging will trigger nativization under the following conditions:-

• in the editor cooks, it will be triggered automatically;
• in UAT, “-build” should be passed on the commandline. For engine versions prior to 4.18, “-nativizeassets” is also needed;
• with “Launch on” from the editor, nativization won’t be triggered (since this option is intended for fast iteration rather than final builds and performance testing);
• with “Cook On The Fly” (COTF), nativization also won’t be triggered as, well, it’s just not compatible;

If you’re looking to run your project through Visual Studio, there a couple of things you should/could do:-

• to build a nativized executable from VS you need to add the following to the NMAKE Build, Rebuild and Clean commandlines within Project Properties:-
  -PLUGIN “$(SolutionDir)\Intermediate\Plugins\NativizedAssets\Windows\Game\NativizedAssets.uplugin”
  (replacing “Windows” as appropriate for whatever platform you’re working on)
• to help with debugging nativized code, in UE4 Editor, go to Project Settings and, in the Advanced section below “Blueprint Nativization Method”, set “Include Nativized Assets in Project Generation” – this allows VS to see the generated nativized code when debugging, allowing you to step through nativized Blueprints as if they were normal C++ code.

Improving Log Output

As it’s often difficult keeping track of whether the log output from multiple runs of the project comes from a nativized or non-nativized build, we advise adding the following line just to make sure that it’s clear – by prepending a simple “Nativized: Yes” or “Nativized: No” line near the top of the output:-

Thousands Of Compile Errors…

It’s extremely likely that the first things that you’re going to see once you’re compiling with Nativized Blueprints are thousands of errors … the main reason for these will be to do with the order that headers are being included. Some example errors that you’re likely to see:-

• error C3646: ‘TestProp’: unknown override specifier
• error C4430: missing type specifier – int assumed. Note: C++ does not support default-int
• error C2059: syntax error: ‘const
• error C2238: unexpected token(s) preceding ‘;’

These occur in files that you would’ve likely seen compiling fine before nativization was enabled.

For the majority of them the fix is simple – you just need to add either a forward declaration of the unknown type or to add a relevant #include to the top of the file. Note that, sometimes, the compiler can be so confused about what’s happening, though, that it can’t tell you what the unrecognised type is.

Problems With UDynamicClass

If you have any references to UBlueprintGeneratedClass (or its subtypes, e.g. UAnimBlueprintGeneratedClass or UWidgetBlueprintGeneratedClass) in your code, for whatever reason, these will need fixing. A Nativized Blueprint class is no longer a UBlueprintGeneratedClass, but a UDynamicClass. The only safe way to handle it is to use a UClass or TSubClassOf<> type instead. If you are trying to access any properties of the class, you will have to find a different way to do it.

Show Me The Bugs

We encountered quite a few more issues after enabling nativization – from compile errors due to badly generated code, to crashes at runtime, to subtle incorrect behaviour. We have put together this blog with examples, and have submitted many of our fixes to Epic in a pull request: https://github.com/EpicGames/UnrealEngine/pull/4202 along with a test project replicating many of the bugs. Our hope is that Epic will take our fixes or fix the bugs in their own way so that others don’t have to go through this again – until then, feel free to use the fixes from this blog post yourself if you encounter any of these issues.

1: IsValid()

We’ll start with a simple one – one of our classes contained an ‘IsValid()’ function which, when a Blueprint child of the class was nativized, caused a compile error. Nativized Blueprints attempt to call the global ‘IsValid(UObject*)’ function many times – but the member function was hiding it. Easily resolved by adding the scope resolution operator to the generated code which intends to call the global function.

In ‘FSafeContextScopedEmmitter::ValidationChain()’, replace:-

with:-

2: Plugin include not found

Nativization only knew how to generate include paths for game and engine classes, but not plugin classes. This resulted in generating #include statements referencing a header from a plugin without a path, and then the compiler not being able to find it. We rewrote the function that generates the include paths to take plugins into account:
In FIncludeHeaderHelper::EmitIncludeHeader(), replace the “else” block with:-

3: Problems with Soft Object References (formerly Asset References)

This was straightforward – Blueprints allowed soft object references to be “expose on spawn” but UHT rejected it in the nativized class. As it clearly worked, this was simply solved by adding CPT_SoftObjectReference to FExposeOnSpawnValidator::IsSupported():-

4: Make Array Nodes

We were getting compile errors from array initialization statements – which were tracked down to “Make Array” nodes not generating casts for their inputs. This is actually a symptom of a difference between Blueprint and C++. Blueprint allows implicit casts between some types that C++ does not. In this case, we found it with a “Make Array” node that was making a “byte” array (uint8 in C++) and enum values. Enum values can be use in byte pins without a cast in Blueprint (implicit cast), but in C++ it generates C++11-style “enum class” for the enum types – which don’t support implicit casts to their base type. The solution is to add a call to FEmitHelper::GenerateAutomaticCast() to FBlueprintCompilerCppBackend::EmitCreateArrayStatement().

Replace:-

with:-

5: Array Casts Of Enums

Another case of implicit casts being allowed in Blueprints but absolutely forbidden in C++ is array casts. In Blueprint land, array pins can be connected to each other in the same way as single values – if you can connect a MyBP_Actor to an Actor pin, you can connect an “array of MyBP_Actor” to an “array of Actor” pin. C++ does not allow any of this (TArrays don’t implicitly cast to other types), so the Blueprint Nativizer has to generate casts using the helper type TArrayCaster. Unfortunately we found a few unhandled cases: Enums (again) and Soft Object References.
In FEmitHelper::GenerateAutomaticCast(), add the following “else” clause to “if (!RType.IsContainer())”:-

6: Array Casts Of Soft Object References

If you connect an array of soft object or class references of a derived type, to a function taking an array of soft object or class references of a base type, the generated C++ doesn’t compile as C++ doesn’t allow this cast to be implicit.This one was a more involved fix. We can reuse some of the machinery supporting implicit casts on objects, but we need to adjust some code to allow it.

In FEmitHelper::GenerateAutomaticCast(), add:-

As written, unfortunately this results in generating UMyType* instead of TSoftObjectPtr<UMyType>. This is because GetInnerTypeString() tries to cast the inner property to a UObjectProperty – but USoftObjectProperty isn’t actually derived from UObjectProperty. Thankfully they do share a common base – UObjectPropertyBase.

So we should go ahead and change GetInnerTypeString() and GetTypeString() as follows:-

However … this doesn’t work 100% – for soft class references (soft references to Blueprint classes), this still generates UMyType* instead of TSoftClassPtr<UMyType>. Looking at GetTypeString(), it has special handling for UClassProperty inners – we just need to duplicate that for USoftClassProperty (unfortunately, UClassProperty and USoftClassProperty don’t share a common base above UObjectPropertyBase):-

And now the nativizer generates correct casts for arrays of soft object / class references.

7: Interface function calls

With the fixes above implemented, on Conan, we were finally able to actually get a nativized version to run… except… it crashed. Hard.

The culprit, as it turns out, was bad code generation surrounding calls to interface functions. For interfaces, UE4 requires calling Execute_FunctionName(Object, Params) rather than Object->FunctionName(Params). If the variable connected to the “Target” input to a “call interface function” node in Blueprint was either “self” or an object that implemented the interface, rather than an interface variable, it tricks the nativizer into generating a regular function call instead of an interface call. A regular call crashes with the message: “0 && Do not directly call Event functions in Interfaces. Call Execute_FunctionName instead.” Execute_FunctionName exists due to non-nativized Blueprints being able to implement the interface, and Blueprint functions have to be called via name lookup, not via C++ virtual dispatch.

Fixing this required adjusting the heuristic as to what was considered an interface call. In FBlueprintCompilerCppBackend::EmitCallStatmentInner(), replace:-

with:-

The original requires that the call be on an object other than self, and that the type connected to the target (the FunctionContext) is an interface. Both of these assumptions are invalid. The only accurate test is that the original function is in a UInterface. The test for !Statement.bIsParentContext is to prevent issues with calls to a parent implementation – these need to be generated as “Super::FunctionName_Implementation(params)” instead of Execute_FunctionName(). This was already handled correctly by the code handling regular function calls, so we just exclude them from being considered interface calls.

Unfortunately just altering this heuristic causes the nativizer to crash – the later code doesn’t handle bInterfaceCallExecute being true for “self” calls, which may have a null FunctionContext – so we need to fix that also:-

8: Transient Defaults

After fixing that, and a few other issues that have since been fixed by Epic (e.g. parent calls on replicated functions, issues nativizing use of set/map, async loading crashes with nativized Blueprints), we were up and running. But all was not well – several key game features didn’t appear to work. I traced one of them down to transient variables not having their default values emitted to the nativized constructor – so they’d just end up null or 0. Annoyingly this seemed to be intentional – but clearly wrong.

In FEmitDefaultValueHelper::OuterGenerate(), change:-

to:-

9: Component Overrides

This one’s a doozy. Due to some broken logic, only the first modified component in a non-nativized child Blueprint will have its changes applied. If an actor consists of a mesh and a collision cylinder, and in a child you change the mesh and modify the cylinder height to match, only the mesh will change in the nativized build, the cylinder will silently be the size from the parent Blueprint. The cause, when it was eventually found, was the break in UBlueprintGeneratedClass::CheckAndApplyComponentTemplateOverrides():-

Unfortunately this is not doing what it thinks – it ends up breaking out of the loop through the components of the object after applying the changes to the first component.

If you think this break should be in the outer loop, perhaps – but as written it doesn’t generate the expected behaviour. The outer loop is iterating through the ancestor classes, in farthest to closest order. If you break here, it will only apply the component changes from the first class in a hierarchy, and ignore all children.

The correct fix would appear to be to remove this break altogether.

10: Component BodyInstance

We originally found this on a much older version of UE4 where it was causing changes in a child Blueprint to the collision settings of a component from a native parent to not be applied after that Blueprint was nativized. In more recent versions of UE4 I believe it actually works – but the generated code is still horrific:-

Blueprint components, on the other hand, generate lovely code for this:-

A quick search through the nativization code for “SetCollisionProfileName” turned up some code in FNonativeComponentData::EmitProperties() which handled this for Blueprint components. Overridden native components got through FEmitDefaultValueHelper::HandleInstancedSubobject() instead, which doesn’t have that code. A simple copy-paste job fixed the issue:

In FEmitDefaultValueHelper::HandleInstancedSubobject(), replace:-

with:-

11: GetClass()

This is an odd one. For some reason, on nativizing, any literal referencing the current Blueprint class will be replaced with a call to GetClass() instead of the literal. I don’t know why it does this, but it does introduce a wonderful bug – if the Blueprint has a child class, any references to the nativized parent class will use the child class at runtime. The responsible code is shown here in FEmitterLocalContext::FindGloballyMappedObject():-

It looks like this is primarily here to fix something with “SubobjectsOfClass”, so lets just extract that case and remove the code that’s outputting GetClass() otherwise:-

The later code handles the case where Object is the current class perfectly fine, generating a call to ::StaticClass() as it should.

12: Enum Properties

We were seeing import errors on load when nativization was used, with messages saying that either it couldn’t find enum/byte properties or it had a type mismatch trying to load an enum/byte property. Our initial investigations led to FKismetCompilerUtilities::CreatePrimitiveProperty():-

“enum class” type enums get compiled as UEnumProperty but the other two types (regular and namespaced) get compiled as UByteProperty instead. If we had a mismatch between Blueprint and the nativizer’s property types, we could get these errors. A quick check of UHT (which is responsible for generating the properties for native classes) revealed very similar logic, so the problem wasn’t with the properties themselves. Next up, we took a look at Blueprint enums. In FEnumEditorUtils::CreateUserDefinedEnum() we saw the following:-

So Blueprint enums are in namespaced form and, as such, create byte properties when the Blueprint is compiled. What about when nativized?

In FBlueprintCompilerCppBackendBase::GenerateCodeFromEnum():-

Woah, wait a second, that’s not in namespaced form, it’s “enum class”, meaning that the nativized version got UEnumProperty when the Blueprint version got UByteProperty!

Namespaced form enums are actually considered outdated so using “enum class” for Blueprint enums as well is, perhaps, preferable. Let’s go ahead and make the changes:-

In FEnumEditorUtils::CreateUserDefinedEnum():-

In FEnumEditorUtils::UpdateAfterPathChanged():-

In UUserDefinedEnum::UUserDefinedEnum():-

In UUserDefinedEnum::Serialize():-

And in UUserDefinedEnum::GenerateFullEnumName():-

After that the import errors went away and everything seemed to work fine.

13: BONUS: Crash Loading Blueprint After Deleting An Enum

We didn’t actually encounter this one in Conan, but while trying to replicate some of the issues with enums and nativization for the pull request to Epic. Essentially, if you have a Blueprint that references a native “enum class” enum, or have taken fix #12 and reference a Blueprint enum, when that enum is deleted the Blueprint that formerly referenced it will crash on open here:

In EnumProperty::ExportTextItem():-

This is simply fixed by returning an invalid string in this case instead:-

Wrapping Up

That’s quite a lot of fixes that we’ve gone through above… however, it may just be the tip of the iceberg as Blueprint Nativization is extremely complicated – we fully expect that other teams’ projects will likely turn up other problems. If you do try it out, and have any problems (or not), feel free to get in touch with us.

Credit(s): Gareth Martin (Coconut Lizard)
Status: Currently partially fixed in UE4.18