Package Versioning… How It Works… And An Optimization

Unreal Engine includes some excellent functionality for ensuring that editor packages and cooked content can originate from various legacy engine versions… and to allow programmers to easily change data structures that are used in these packages. There are several mechanisms in place to allow this. Today, we’re going to look at the package versioning code and how that works through GetLinkerUE4Version() and related functions.

There are several engine versions that we need to know about:-

• UE4Version: this version number increases every time that Epic add something to the codebase that requires package data to change
  • It should only be changed by Epic
• UE4LicenseeVersion: works in the same way – but this is a “safe” version number that licensees of the engine (you!) can change without fear of clashing with Epic’s versioning
  • Licensees (most people reading this) should change this
• CustomVersion: these are used in several places to better allow different system/game engineers to make changes that won’t conflict… this is a new system that Epic are now using much more regularly than UE4Version

First off, though, let me point out something that should be adhered and that I’ve seen cause problems for several developers:-

Licensees should only change UE4LicenseeVersion – NOT UE4Version

How To Use Package Versioning

There are 2 functions that are regularly used for dealing with package versioning (thereby allowing support for legacy package versions), GetLinkerUE4Version() and GetLinkerLicenseeUE4Version().

Let me give you an example of one place where Epic use GetLinkerUE4Version to fix some content…

At some point in time, somebody added a Blueprint property for setting visibility. Except, when they did so, they made a small error, missing the third “i” of visibility (4 i’s in one word is, I agree, excessive – even for the English language..) and so adding it as visiblity.

Somebody down the line spotted this – so they added the following code, in UWidgetBlueprint::PostLoad(), to patch the mistake:-

They then added this define to the end of the list in ObjectVersion.h (nb. since then, many more versions have been added – so you’ll now find the define closer to the middle of the file):-

So here’s what this does:-

• When a package is loaded containing the spelling error, the package’s version number will be something less than VER_UE4_RENAME_WIDGET_VISIBILITY
• The above code will trigger, fixing the spelling mistake so that the user sees the correct spelling
• -If- the package is saved, it will be saved with the correct spelling and the version number will be set to be the latest (which should be equal to or higher than the new define)

There are many more examples dotted around the codebase… some example uses:-

• Adding new properties to classes – using versioning, we can default these properties to some sensible valuable and then allow the user to modify the value in the editor
• Deprecating properties from classes
• Patching content that we wish to change

And much more. Have a look around the codebase and you’ll see plenty of examples.

Package Version Defines

Here’s an excerpt from the start of ObjectVersion.h showing how Epic have defined some of the versioning – first EUnrealEngineObjectUE4Version:-

And EUnrealEngineObjectLicenseeUE4Version, in the same file:-

The comment on line 3 of the above, by the way, makes little sense – because there is no such comment below…

Some notes for adding things to these define lists:-

• As already mentioned, licensees should only add to EUnrealEngineObjectLicenseeUE4Version list;
• Only add new versions to the end of the list;
• When working with Perforce or other version control where two or more programmers may be adding new package versions, be very careful with any content attached to the code – if there’s a mismatch between the version number that the content was saved with and the version number being checked in (due to another programmer’s version being merged ahead) then the content may end up corrupt if we’re not careful. The simplest solution is to not allow content checkins alongside version number changes.

It’s for the above reasons that we should only modify UE4LicenseeVersion. Imagine the following situation:-

• A project is begun on UE4 4.11;
• We add some extra versioning to UE4Version for some cool new feature we’ve made;
• We merge UE4 4.12 .. and find that we’re getting a conflict with UE4Version – Epic have also changed it for 4.12… we need to either:-
  • Insert Epic’s new version numbers to the end of the list, leaving ours just before – this will work for our own content but will break any content pulled from Epic;
  • Move ours to the end – but this will break our own content.

These 2 issues mentioned above can be fatal. A programmer once made this mistake back when we were working on UE3…. it took us a while to figure out a way to re-patch our content … it wasn’t nice at all, I don’t wish that on anybody – so please, I say again, only use UE4LicenseeVersion.

When Package Versioning Goes Horribly Wrong…

Over the years, I’ve seen a few cases where problems have cropped up due to mistakes being made by programmers when changing the package defines…

To understand these problems, you need to think about what’s really going on under the hood, what the versioning code is really doing. So let’s say we have a package being loaded that was previously saved with, let’s say, version 256. The currently version is, let’s say, 3 ahead of that (259). As the package is loaded, it may need to apply patches 257, 258 and 259 – but it knows that patches 0-256 were previously applied so can be skipped.

Here’re some examples of how this might go wrong (through programmer error):-

• When merging changes before submitting to version control, another programmer’s version change is merged in -after- the version currently being added
  • Testing locally, everything will probably appear to work fine
  • Other users, though, start complaining about crashes or other problems
    • Those users already had the change from the other programmer that you merged above
    • On their version, that change would’ve had a different version number
    • Since you’ve moved that change to the end of the list, the system is now confused
    • As packages are loaded, the engine thinks that your own patch was applied (which it wasn’t)
    • And that the other programmers change wasn’t applied (it was)
• You’ve spot-integrated a change from Epic that added a new define to the versions list
  • However, in Epic’s ObjectVersion.h were some other changes, before and after the one you’ve integrated
  • You later do a full integration, adding the extra defines in the same order as Epic have
    • Crashes/bugs may now start to occur as several of the patches are now effectively out-of-order
  • The “correct” way to have done things would’ve been to -only- add the extra defines following any you’d already integrated
    • This could still cause problems if you were to integrate Epic content – if that content was saved somewhere between the first and last change that is being added
    • COMPLICATED – no easy fix for this problem…
• You’ve ignored all the advice I’ve given above and you’ve added a define into the non-licensee list
  • You’re trying to integrate Epic’s latest code and content
    • You’re going to find that the content is extremely likely to crash or have bugs…
    • You’re also going to find that there’s no easy fix…
    • If you find yourself in this mess, I hope to god you can get away without integrating Epic’s content – if you can, you’re ok

Our Optimization

When the full content for a game is cooked, the content saved will have its versions set to be the maximums. Herein comes the idea for the optimization…

Even if the game is loading cooked content, the engine doesn’t assume that it’s up to date. This is great because it means that you don’t need to recook your content every time that you change some package version code… however, constantly checking content to find out whether or not it needs to be patched is expensive… more often than not, particularly on a Shipping title, it isn’t necessary. So let’s make a few assumptions:-

• that Shipping titles will only work with cooked content
• that Shipping titles will only with with content fully cooked to the latest package versions

If we can make these assumptions – then we can optimise much of our serialisation code by making the compiler aware that we don’t need to run any of the patching code.

Essentially, we need to help the compiler out by making it aware that every check of the form “if ( GetLinkerUE4Version() < VER_…” will fail and every “if ( GetLinkerUE4Version() >= VER_” will pass. The simplest way is to have each of the Get—Version() functions return the max version number – we do this in the header file to ensure that the compiler is able to inline this appropriately (and thereby realise that each block of patching code will be unreachable).

So, here’s what we came up with…

We add the following code to the top of UObjectBaseUtility.h … note that here we’re assuming that a Shipping build is just for running cooked builds – change this to suit your own requirements (for our project, we actually set this for both Test and Shipping builds as we typically profile using Test).

Just before GetLinkerUE4Version() you should add this:-

And finish up after GetLinkerCustomVersion():-

As we’re now overriding the CPP version of the versioning functions, we also need to block-out that CPP… so we add this check to the very top of ObjectBaseUtility.cpp, right after the include:-

And we finish that at the very end of the file:-

That’s all that we need – the compiler should now be in a great position to totally strip out the patching code – leading to reduced and more optimal compiled code.

Additional Advice (to Epic)

I’ve seen it too often that licensees will go ahead and add new defines to EUnrealEngineObjectUE4Version instead of the licensee equivalent… To try to help prevent this, I’d maybe change the comments in the header file close to where they’d make the change… eg. change:-

to be:-

And, while we’re at it, the following should probably be changed too:-

probably to:-

It may also be worth moving the licensee defines up to the top of the file, possibly renaming EUnrealEngineObjectUE4Version to something like EUnrealEngineObjectPleasePleaseOnlyForEpicChangesUE4Version, and maybe adding a comment block to the top of the file explaining how the defines should be used. I’ve seen some very clever, very experienced developers that have made mistakes here…

And that’s it … I hope this article made some sense and is of some use to some of you … please feel free to comment below!

Credit(s): Robert Troughton (Coconut Lizard)
Status: Currently unimplemented in 4.12