Start-Games-Open-Source/SGDynamicTextAssets

https://github.com/Start-Games-Open-Source/SGDynamicTextAssets

The Start Games Dynamic Text Assets is an Unreal Engine plugin that provides a text-based (JSON, XML, YAML, can extend to others) data configuration system similar to Primary Data Assets but stored as external text files. These files serve as the source of truth for static game data configuration.

After 10+ years of building gameplay systems in Unreal Engine, one problem kept following me from project to project and every studio would try to implement their own solution around it: static game data locked inside binary .uasset files.

Overview

Primary Data Assets are the engine’s answer to data-driven design, but there is one major drawback they’re binary uassets. Don’t get me wrong, it’s great because when you cook out the data its part of the pak files and packaged build, so you know they’ll be picked up by Unreal’s asset manager framework. But then you run into collaboration problems and issues because they’re binary files:

• Merge conflicts cause somebody’s work to be lost.
• Only one person can modify it at a time.
• There is a large enough amount of effort with being able to update them at runtime in an automated way that makes it difficult to justify to teams.
• If you make a change and you want to test it on a live build, you have to wait for a full redeploy(meaning at least 45 minutes of waiting).

Every team I’ve been on has either worked on a workaround for it or implemented it and I got fed up with everybody basically rolling their own solution from square one. I wanted to build a plugin to at least get teams started and less time wasted trying to figure how to build this plugin as well as what they wanted from it. I built it to be extended on, overridden, even forked and rewritten entirely! Because its completely open source!

The short version for how its setup is;

1. You have your text files in the content folder while in the editor and normal development
2. When you use them at runtime there are wrapper objects used for gameplay usage.
3. When you package the game it turns them into binary files that are part of the pak file(s)
4. If you have a backend server for live ops you can layer that on top of your packaged binary file versions to override/replace/add/remove data
5. And it’s separately cached locally to speed up lookups from the server and such and it doesn’t touch your binary files to have the best of both worlds.

I built it to scale as best as I could, but I probably missed some parts of it which is why I want the community to give feedback on it!

High Level Outline

Some of you may say, “I have no problems with Primary Data Assets! I don’t need this plugin!” and that’s totally fine! You don’t HAVE to use this plugin, but there are developers(I’ve seen it many times in AAA) that do have problems that this plugin addresses(hopefully).

For reference, Dynamic Text Asset’s abbreviated is DTA.

Loading video…

Why making Data Tables isn’t the solution

In Unreal Data Tables are intended for just generic data that doesn’t reference other assets(which is a failure I’ve seen many times as well), and isn’t intended to be modified at runtime(which I’ve seen developers try to do as well with plugins) so you end up fighting the intended design of Data Tables.

The reason for why everybody tries to rearchitect them is because they want that fancy “Import from CSV” button. Which is great when you’re dealing with simple data schemas but when it gets more complicated or layered it starts to break apart and go past what CSV’s were intended for.

The great news about this plugin is that you can setup a simple pipeline of importing a JSON(or if you want to build out the CSV pipeline you can) and then boom problem solved! It’s instanced per text asset, easy to lookup, and easy to modify without opening the editor since its referring to the raw text files!

Design Goals

The main design goals were for designers and artists to be able to author content and data without stepping over each others work and make life easier to engineers that need to integrate it with live service types of features or at least be able to continually develop without many slowdowns.

High Level design

The plugin is structured into 3 layers:

1. On Disk text files
2. The Unreal Asset System
3. Gameplay Code

Asset Identity is GUID-based rather than path-based, so renaming or moving a file doesn’t break references elsewhere in the project. Configuration for the plugin itself lives in a Primary Data Asset rather than ini files to keep it within Unreal’s standard asset workflows and to avoid creating multiple Dynamic Text Asset types when we only need 1 settings asset file.

     Layer 1: On Disk Text Files

First, you have the external JSON, XML, or YAML files. These are the default file types provided by the plugin, but you can extend it with your own file types if you want, the plugin has documentation on how to do this(TLDR: it involves custom serializers).

These are the source of truth files that live alongside your project in the Content folder, they’re human readable, can be diffed and merged cleanly in any source control system.

The plugin does not care which format you use because they’re abstracted through a format handler to parse any of them.

     Layer 2: The Unreal Asset System

The middle layer is where most of the work is being done, each text file is given a corresponding UObject wrapper inside the project as a lightweight runtime asset that your gameplay code can reference and query like any normal Unreal object. The plugin provides a base wrapper type of USGDynamicTextAsset that you can inherit from, but you can implement your own base wrapper types as well since it just requires implementing an interface.

Currently I intended for the wrapper types to be declared in C++ only to keep the type's manageable, giving complete control to Blueprint users tends to create a runaway problem of too many type's that makes it unmanageable for the team.

These wrappers are what the engine actually will see and are created on demand thorugh async “loading” managed by a Game Instance Subsystem. Its async only to avoid blocking the game thread for parsing files on startup and such, the subsystem owns the lifecycle of load, cache, invalidate, and reload.

     Layer 3: Gameplay Code

The top layer is your gameplay code and Blueprints. They never touch the text files directly and only reference the UObject wrappers through FSGDynamicTextAssetRef structs which are lightweight ID holders to resolving to the appropriate wrapper object.

Semantic Versioning and Migration Hooks

The plugin supports versioning and migration overriding per DTA type, this is specified in the documentation for the plugin. This is so you can transfer data between versions without breaking previous DTA’s that have already been written and ensure development iteration continues.

Validation and Automated Testing

The plugin also has an Unreal Header Tool(UHT) validator that runs at compile time to catch any hard references inside the wrapper objects, this helps avoid direct dependencies and enforces good code practices within your project.

The plugin currently implements a bunch of unit automation tests to validate that functionality is setup correctly in case you fork the repo and make changes. This can help validate if another feature broke as a result of your changes.

You can access these automated tests using Unreal Frontend or running an automated tests script with the appropriate command line arguments.

There is also support for implementing your own custom validation per DTA type to ensure its been correctly configured as well as being able to validate the raw text files in case an outside change broke something.

Blueprint Support

There is a large amount of Blueprint supported functionality for both editor tooling and runtime gameplay code, a large amount of the usage really leverages casting to the wrapper types and then using that functionality.

Conclusion

I’m excited to see what people can do with this plugin and see the great games people make with it!

I would love to receive feedback for the plugin to make it better, and other developers are welcome to contribute to the plugin!

For any AAA developers that end up using the plugin, please give back to the community and make PR’s for this plugin!

We all end up making this sort of system, so why not help shortcut everybody’s work for the benefit of all!