For Developers Working With Unreal Engine

The Cook, The Resave, His Garbage And Her Optimization

by

Summary: Speed up content cooking with the ResavePackages commandlet (51% faster cooking for us!) and speed up the ResavePackages commandlet itself with a small code change (nearly a 7x speedup in our tests!).

“Cooking” content is very, very important in Unreal Engine when working on run-time performance. Thankfully, UE4 offers several ways to do it – through the editor or commandlets, for example. More information here: docs.unrealengine.com/latest/INT/Engine/Basics/Projects/Packaging/index.html

I want to show you a lesser-known commandlet, ResavePackages… I couldn’t actually find much information about this in the UE4 docs – just the auto-generated page here: docs.unrealengine.com/latest/INT/API/Editor/UnrealEd/Commandlets/UResavePackagesCommandlet/index.html

Basically, when you’re cooking content, you might notice in the log several messages suggesting that you resave certain package files in order to fix a certain problem. You can either do that by just loading those packages in the editor, one by one, and saving them out again… or you can use ResavePackages which will update as many packages as you like in one go.

A long time ago, working on Wheelman (Unreal Engine 3), we used this commandlet a LOT. In fact, we had it set to run automatically every night at around midnight… it would grab all of the packages that it could from source control (we used Perforce), resave them and then, when it finished a couple of hours later, put them back into source control. This would update “most” packages to the latest version of the engine – which, for us, was changing daily (we worked hard!).


I decided that now was the right time to try the commandlet out again on our current project. To my horror, it took nearly 8 hours (!) to complete. It was going so slow that, at one point, I fired up Windows Task Manager just to see that the damn thing was still breathing – and that’s when I spotted something truly horrific. Since I’m constantly working on optimization, I have my Task Manager configured to show me several bits of information that it doesn’t by default: including file read/write statistics. The instance of UE4 that was running the commandlet had done 5.1 terabytes of file reads. Terabytes. That’s 5,100,000,000,000 bytes. A lot.

I took a step back to think about this… ResavePackages didn’t seem to have changed much from Unreal Engine 3 … so how had it gotten 4 times slower? Then it hit me: With Unreal Engine 4, we have assets. Small files – but a lot of them. Unreal Engine 3 tended to work with a much smaller number of larger files… so it struck me that the commandlet might be reloading dependent assets every time it started the resave process on a new file, without retaining previously used loads at all. I realized immediately the most-likely culprit: Garbage Collection.

Looking through the UResavePackagesCommandlet code, I very soon found this promising piece of code in ContentCommandlets.cpp:-

static int32 Counter = 0;
if (!GarbageCollectionFrequency || Counter++ % GarbageCollectionFrequency == 0)
{
  if (GarbageCollectionFrequency > 1)
  {
    UE_LOG(LogContentCommandlet, Display, TEXT("GC"));
  }
  VerboseMessage(TEXT("Pre CollectGarbage"));
  CollectGarbage(RF_NoFlags);
  VerboseMessage(TEXT("Post CollectGarbage"));
}

Note “GarbageCollectionFrequency”. It’s defined in ResavePackagesCommandlet.h as:-

/** Only collect garbage after N packages */
int32 GarbageCollectionFrequency;

The default value..? Zero.

Find all “GarbageCollectionFrequency”, Subfolders, Find Results 2, “C:\Source\UE4”, “*.c;*.cpp;*.cxx;*.cc;*.tli;*.tlh;*.h;*.hh;*.hpp;*.hxx;*.hh;*.inl;*.rc;*.resx;*.idl;*.asm;*.inc;*.ini”

  • C:\Source\UE4\Engine\Source\Editor\UnrealEd\Classes\Commandlets\ResavePackagesCommandlet.h(82): int32 GarbageCollectionFrequency;
  • C:\Source\UE4\Engine\Source\Editor\UnrealEd\Private\Commandlets\ContentCommandlets.cpp(579): if (!GarbageCollectionFrequency || Counter++ % GarbageCollectionFrequency == 0)
  • C:\Source\UE4\Engine\Source\Editor\UnrealEd\Private\Commandlets\ContentCommandlets.cpp(581): if (GarbageCollectionFrequency > 1)

Matching lines: 3 Matching files: 2 Total files searched: 44378

So, as I’d thought, the garbage collector would run at the end of each and every asset being resaved. The garbage collector itself can be slow – but the bigger problem is that throwing away all that data means the next asset, which is very likely to be similar to the previous (since we tend to arrange assets into nice, orderly folders), will need to load from disk all of its dependent assets in order to do its own resave.

Here are some stats for you of different values that I tried:-

Garbage Collection
Frequency
Resave Time
(mins)
0 475
100 70
500 160

When I tried a frequency of 500, the commandlet spent much of its time using 24gb of RAM (out of 32gb total RAM on my system) – so the slowdown here is undoubtedly due to virtual memory paging. At a value of 100, it was only using around 4gb at peak. Here’s the code:-

static int32 Counter = 0;
GarbageCollectionFrequency = 100;
if (!GarbageCollectionFrequency || Counter++ % GarbageCollectionFrequency == 0)

Here’s the potential issue, then: everybody’s system will have a different amount of RAM.. and everyone’s content will be slightly different. So, while “100” worked well for us, it might not be as good for others… it’s worth playing around to find the right number. Also, perhaps a better solution would be to keep track of how much memory is in use as a percentage of system memory and to use that as an indicator of whether garbage collection is required – this would ultimately give the best solution as it would cope well dealing with both small and large asset files.

I may revisit this at a later date to implement such a system – for now, though, I’m going to stick with a GC frequency of 100 – 70 minutes isn’t bad at all (and is nearly 7 times faster than the default pass).


Following the resave, I tested out the content cooker again. Let me just jump right in with the stats:-

Package State Cook Time
(mins)
Before ResavePackages 44.7
After ResavePackages 21.8

So, following a ResavePackages, content cooking is around 51% faster! This is 100% down to the content in the project, many files hadn’t been touched since we’d updated the engine, but it does highlight how useful the commandlet really can be.


That’s it for another time! As always, let me know if this article was any help.

 


Credit(s): Robert Troughton (Coconut Lizard)
Status: Currently unimplemented in 4.12


5 Comments

  1. Nice article and great sleuthing to find the garbage collection issue. This is definitely something I’ll be looking at scripting periodically. It would be great to be able to base the garbage collection frequency on memory usage with a LRU queue to really squeeze the most performance out of it 🙂

  2. Hi, any chance post “JUDICIOUS USE OF INLINE FUNCTIONS – PART 1” will become unprotected so we can read it? 🙂
    Cheers and thank you for writing this blog! 😉

  3. Hi mate,

    Great find with the resave packages commandlet. I have a task to convert the cooking to have a max memory as a percentage instead of a max memory limit.
    I’m sure you have seen the code it’s more of a soft limit anyway as garbage collection ends up frequently running due to uworld loading forcing garbage collection.

    Either way after the conversion to a percentage is done for the cooker would be super easy to duplicate that function (bleh) or preferably somehow share it.

    I’m really interested about the first line of the article. Says 51% faster cooking with resave packages commandlet? How is cooking been sped up?

    Thanks

    • Hey Daniel!,

      That all sounds great!

      Basically, for the current project, I found that a large number of asset files that were going through content cooking were around 2 engine versions out of date. During the cook process, there would be a ton of warnings thrown up recommending that various packages be resaved. Once I’d resaved packages locally, a full recook sped up by 51% as, I guess, the content no longer needed patching during the cook.

      Hope that makes sense..?

      Robert

  4. I changed GarbageCollectionFrequency and faced incomprehensible blueprint compile error.
    Resavepackages process may depend on clearing loaded objects by garbage collection on each load/resave cycle.
    Circular dependency in blueprints also may affect this issue.
    I concluded that I have to accept frequent GC and slow resave at least on current version(UE4.14).

Leave a Reply

Your email address will not be published.

*

Latest from ALL

Trash Compactor

We recently found a huge leak in the UE4 garbage collector, particularly

Placating The Natives

In this article we delve into Blueprint Nativization, a relatively new feature
Go to Top
%d bloggers like this: