The Cook, The Resave, His Garbage And Her Optimization

Summary: Speed up content cooking with the ResavePackages commandlet (51% faster cooking for us!) and speed up the ResavePackages commandlet itself with a small code change (nearly a 7x speedup in our tests!).

“Cooking” content is very, very important in Unreal Engine when working on run-time performance. Thankfully, UE4 offers several ways to do it – through the editor or commandlets, for example. More information here: docs.unrealengine.com/latest/INT/Engine/Basics/Projects/Packaging/index.html

I want to show you a lesser-known commandlet, ResavePackages… I couldn’t actually find much information about this in the UE4 docs – just the auto-generated page here: docs.unrealengine.com/latest/INT/API/Editor/UnrealEd/Commandlets/UResavePackagesCommandlet/index.html

Basically, when you’re cooking content, you might notice in the log several messages suggesting that you resave certain package files in order to fix a certain problem. You can either do that by just loading those packages in the editor, one by one, and saving them out again… or you can use ResavePackages which will update as many packages as you like in one go.

A long time ago, working on Wheelman (Unreal Engine 3), we used this commandlet a LOT. In fact, we had it set to run automatically every night at around midnight… it would grab all of the packages that it could from source control (we used Perforce), resave them and then, when it finished a couple of hours later, put them back into source control. This would update “most” packages to the latest version of the engine – which, for us, was changing daily (we worked hard!).

I decided that now was the right time to try the commandlet out again on our current project. To my horror, it took nearly 8 hours (!) to complete. It was going so slow that, at one point, I fired up Windows Task Manager just to see that the damn thing was still breathing – and that’s when I spotted something truly horrific. Since I’m constantly working on optimization, I have my Task Manager configured to show me several bits of information that it doesn’t by default: including file read/write statistics. The instance of UE4 that was running the commandlet had done 5.1 terabytes of file reads. Terabytes. That’s 5,100,000,000,000 bytes. A lot.

I took a step back to think about this… ResavePackages didn’t seem to have changed much from Unreal Engine 3 … so how had it gotten 4 times slower? Then it hit me: With Unreal Engine 4, we have assets. Small files – but a lot of them. Unreal Engine 3 tended to work with a much smaller number of larger files… so it struck me that the commandlet might be reloading dependent assets every time it started the resave process on a new file, without retaining previously used loads at all. I realized immediately the most-likely culprit: Garbage Collection.

Looking through the UResavePackagesCommandlet code, I very soon found this promising piece of code in ContentCommandlets.cpp:-

Note “GarbageCollectionFrequency”. It’s defined in ResavePackagesCommandlet.h as:-

The default value..? Zero.

• Find all “GarbageCollectionFrequency”, Subfolders, Find Results 2, “C:\Source\UE4”, “*.c;*.cpp;*.cxx;*.cc;*.tli;*.tlh;*.h;*.hh;*.hpp;*.hxx;*.hh;*.inl;*.rc;*.resx;*.idl;*.asm;*.inc;*.ini”
• C:\Source\UE4\Engine\Source\Editor\UnrealEd\Classes\Commandlets\ResavePackagesCommandlet.h(82): int32 GarbageCollectionFrequency;
• C:\Source\UE4\Engine\Source\Editor\UnrealEd\Private\Commandlets\ContentCommandlets.cpp(579): if (!GarbageCollectionFrequency || Counter++ % GarbageCollectionFrequency == 0)
• C:\Source\UE4\Engine\Source\Editor\UnrealEd\Private\Commandlets\ContentCommandlets.cpp(581): if (GarbageCollectionFrequency > 1)
• Matching lines: 3 Matching files: 2 Total files searched: 44378

So, as I’d thought, the garbage collector would run at the end of each and every asset being resaved. The garbage collector itself can be slow – but the bigger problem is that throwing away all that data means the next asset, which is very likely to be similar to the previous (since we tend to arrange assets into nice, orderly folders), will need to load from disk all of its dependent assets in order to do its own resave.

Here are some stats for you of different values that I tried:-

When I tried a frequency of 500, the commandlet spent much of its time using 24gb of RAM (out of 32gb total RAM on my system) – so the slowdown here is undoubtedly due to virtual memory paging. At a value of 100, it was only using around 4gb at peak. Here’s the code:-

Here’s the potential issue, then: everybody’s system will have a different amount of RAM.. and everyone’s content will be slightly different. So, while “100” worked well for us, it might not be as good for others… it’s worth playing around to find the right number. Also, perhaps a better solution would be to keep track of how much memory is in use as a percentage of system memory and to use that as an indicator of whether garbage collection is required – this would ultimately give the best solution as it would cope well dealing with both small and large asset files.

I may revisit this at a later date to implement such a system – for now, though, I’m going to stick with a GC frequency of 100 – 70 minutes isn’t bad at all (and is nearly 7 times faster than the default pass).

Following the resave, I tested out the content cooker again. Let me just jump right in with the stats:-

So, following a ResavePackages, content cooking is around 51% faster! This is 100% down to the content in the project, many files hadn’t been touched since we’d updated the engine, but it does highlight how useful the commandlet really can be.

That’s it for another time! As always, let me know if this article was any help.

Credit(s): Robert Troughton (Coconut Lizard)
Status: Currently unimplemented in 4.12