[Spec] Savegame File

[davelsan]

Hey all,

I have glanced over the save file (global.dat) and noticed a few patterns that might be useful, in case someone has a good reason to modify some of the things in there. Although it is very unlikely, maybe a future me will want to come back to this post, to spend some time creating a save editor. In any case, I just wanted to share the bits of the spec that I have looked at (see what I did there?  ).

If you're in a hurry, just skip the motivation section. That's just there to prove that I'm a real human being... *ahem*.


MOTIVATION

I don't have a lot of time these days, but I really wanted to give a serious try to Underrail, being a long time fan of this kind of games. This is my very first post here, so I hope I'm not breaking any rules -- banned after the first post; now, wouldn't that be a grand entry to the community...

Anyhow, one of the things that kept me busy for a long while, trying stuff out with the online character build tool, is that there are no respecs in this game. So I just wanted to have a nice character that I'm happy role playing with, from beginning to end, without restarts. The other reason is that I very much enjoy building characters and imagining what they would be like, even for a game that I have barely played. But that's a story for another time.

(I might be a bit of min-maxer, too, but I'm not admitting to anything).

I think I got my nice character figured out. But in case I made a mistake, it would be nice to be able to tweak stats, skills, and so on. I'm not a fan of having memory hacks laying around my system, so I preferred to take a look at the actual file. As I understand, the save file is obfuscated, or that's what everybody says, but that doesn't mean there isn't a file spec that we can understand and deserialize.

Building on the work of others (link), we know that the good stuff is in a gzipped payload. I took a look at that and figured out the basics of how to edit a few of the things that are stored in global.dat. I'm going to start with the stats, later on I'll post about the skills. But there's a lot of potentially modifiable stuff in there (stats, skills, feats, inventory, quest global state, etc.).


CHARACTER STATS

Character stats seem to be distributed in blocks. I highlighted the strength block using HxD in the image link below.

https://i.imgur.com/5YkLVUL.png


In the next picture, I annotated a few interesting things.

- The first 32 bits (blue) are the block index.
- Stats are stored as a pair of 32-bit values.
- The first value (green) is the character unmodified stat.
- The second value (orange) has an unknown effect on the stat, if any.
- The block ends with the stat name (blue filled), preceded by its 8-bit length.

https://i.imgur.com/eXyDBD2.png

A few notes. I assumed a little-endian byte order, but the actual stat values could be perfectly ordered as big-endian, or just span a byte-length. Every block is populated with several key:value pairs, each preceded by its 8-bit length. I wonder what those are all about.

In the next post I'll describe skills. The skill names are not there, unlike stats, but their order is preserved and the block structure is the same.

[Azura_04]

Interesting.

A question, do you offer services to edit the global.dat s?

[davelsan]

I am testing this with the latest version of the game + expedition dlc. Assuming your save file has the same spec, with the stuff I posted you should be able to edit stats. Just need HxD and 7zip, really. I could guide you through the process, if you want.

Or is it something else you'd like to edit?

[davelsan]

@epeli that is amazing! I didn't know there was so much info already researched.

I'm actually in the midst of posting about the skills section. Also a humble apology, because I made a lot of wrong assumptions on that first post, as you surely noticed. I'm re-evaluating how I presented some things, so I'll post again shortly taking into account what you just shared here.

Again, amazing. Thanks!

[davelsan]

Well, I made quite a few wrong assumptions in that first post. The info that you posted has been really helpful, but I haven't figured out a basic struct for the base ability block yet. In fact, after looking at how skills and feats are stored in the file, at times it feels more like a big hash table than a collection of indexed blocks. Or maybe it is a bit of both, since I see evidence of some ordering and domain-based clustering.

If only for convenience, using the block structure I outlined in the first post, there are some interesting patterns. I'm going to use the Strength as an example:


- The Base Ability block seems to start with an index of sorts. I don't think it is always sequential for adjacent blocks, but for base abilities and skills it is.

Code: [Select]

Int32 index   // 0x5E7 (BigEndian?)
Byte  unknown // 0x0


- Then follows what appears to be a naming scheme of some kind, including a list of names that would make sense if they were linked to some values later.

Code: [Select]

Int8 + char[Int8] abilityType? // 0x4 + SBA2
Int32             nameCount    // 0x6
Byte[]            names        // nameCount * (Name) { S4:C, S4:V, S4:MV, S4:VC, S4:MVC, BA2:N }
Byte[]            unknown      // byte[nCount]
Int8 + char[Int8] entityClass? // 0x3 + BA3


- Each name in the list is just the string length + the string itself.

Code: [Select]

Int8 + char[Int8] name // 0x4 + S4:C


- Then comes a completely unknown chunk, where I would expect some sort of values or flags that correspond to the previous list of names.

Code: [Select]

Byte[] unknown    // byte[nameCount]
Int8 + char[int8] // 0x3 + ESI
Int32  esi1Val    // 0x2
Int8 + char[int8] // 0x3 + ESI
Int32  esi1Val    // 0x2
Byte[] unknown    // 0x2
Byte   unknown    // 0x9
Int32  unknown    // 0x2E


- Then come the actual ability values, as a pair of 32-bit integers.
- The first is the base ability value.
- The second could be the modified ability value, but it does not have any effect in either stats or skills when I changed it.
- This pattern is the same in the skill blocks.

Code: [Select]

Int32   baseAbilityValue      // 0x5
Int32   modifiedAbilityValue  // 0x5


- Then comes a strange byte array. This could actually be two flags and two 32-bit integers, but I have no idea what they could be used for. I'll explain in a moment, in the context of all blocks.

Code: [Select]

Byte[]  seqIndex  // 09 B7 03 00 00 0A 06 B8 03 00 00

- Finally comes the human-readable base ability or stat name.

Code: [Select]

Int8 + char[Int8] name // 0x8 + Strength


Now, about that strange byte array, what I called the seqIndex, when I looked what those bytes looked like in the different stat blocks, it appears they follow a sequential pattern:

Code: [Select]

STR: 09    B7 03 00 00
STR: 0A 06 B8 03 00 00

DEX: 09    BA 03 00 00
DEX: 0A 06 BB 03 00 00

AGI: 09    BD 03 00 00
AGI: 0A 06 BE 03 00 00

CON: 09    C0 03 00 00
CON: 0A 06 C1 03 00 00

PER: 09    C3 03 00 00
PER: 0A 06 C4 03 00 00

WIL: 09    C6 03 00 00
WIL: 0A 06 C7 03 00 00

INT: 09    C9 03 00 00
INT: 0A 06 CA 03 00 00


A part of this pattern even continues in the skill blocks. I wonder what this is.




Truly, I have no idea about which way to look at the file spec. But I think there are some obvious patters and it definitely can be edited, even if only for simple changes.

I would probably like to focus first on being able to read it from start to end. Then add a bit of editing functionality via CLI or a very basic UI. The problem I see is that I haven't found any block-sizes that I could use to speed-up the process of reading. It almost looks like it is all parsed in one pass.

I gotta say, though. It was never my intention to serialize the whole file. I'm just a curious fella who had a couple days free on the weekend. I just want to share what I can with the community, then try to play the actual game. But there is also the possibility that I enjoy mucking around game files more than playing the game. Oh well.

It'd be nice to have a community project where we can all contribute our little piece, I believe.


Yeah, I still have to write something about editing skills.

[davelsan]

Unfortunately, you would have to implement the entire spec to correctly parse whole Underrail files.

Yes, it seems like it. I really don't want to start writing code for a new project when I know I don't have the time to carry it through. But that would be the next step at this point, sure. I'll keep playing the game and see where that leads. Sometimes we just need to find that little something for everything to click into place. I'll write in this thread anything interesting I notice.

That includes a screenshot sharing where the skills are stored in the file and how to manually edit them, which is pretty much more of the same as above. In addition, I noticed feats can be stored in two different places: one is contiguous to skills and another close to the EOF. I think this has to do with the feat being a passive (e.g. Sixth Shell) or an active ability (e.g. Aimed Shot). I have read a forum post here where some fellas have swapped one feat for another. I'm actually surprised that even worked for any of them without at least also updating the size integer.

Like I said, I need to play more and keep looking at the file. Right now I'm just starting the game, so the save file is simple, but it does not show a complete picture yet.

[Azura_04]

I might have a challange for you, could you open a .char and add it's loadout to a player.char?

If you can, then you are truly a master. And I'll clean your shoes every afternoon.

[davelsan]

They are abbreviated like the data model element names. You can get those from Data\Knowledge\feats.k. For feat-granted abilities, check specialabilities.k and specialattacks.k. Knowledge files are the easiest to parse of all Underrail files.

My friend, you keep tempting me with all sorts of interesting stuff, and I'm trying to resist because I know where this road leads

I just glanced over feats.k and I think a few things are starting to make sense. Incidentally, I too believe this is a file we can easily de-serialize. Has anybody looked into it already?

Since knowledge files have strings longer than 127, this would be a good moment to tell you that LengthPrefixedString length isn't 1 byte uint8 like you assumed earlier, it's 1-5 bytes of uint7s.

I'm not sure I understand what you mean. I'm assuming a uint8 string length for base ability names, such as strength or intelligence, and for the abbreviated data model names. For something longer I would expect a uint32.

For example, in feats.k, the descriptions table has this structure (tentative structure, let's play it safe ):

uint16 index (Big Endian) - BLUE
uint32 string length (Big Endian) - GREEN
byte control character (e.g. SOH, but not always) - BROWN
string description (length = previous uint32) - RED

I have drawed it in the picture below. Notice how the element 0x606 has a SOH (0x1) control character not included in the length (0x93) of the description block.

https://i.imgur.com/HmWWf05.png

Or a variation of that. Maybe the index is a uint32 and the string description is a uint16? This is all conjectures, I just try to fit the data in the struct that would make most sense to me.

I haven't checked, but I'd imagine the hash table at the top of the file is mapping these indexes. What do you think?

[davelsan]

Yes, I need to spend some time reading up the links you posted. Thanks for all the info epeli, it's much appreciated

Talk to you again soon, I hope.

[davelsan]

Hey again, long time

I just wanted to drop by and say that I have not abandoned this. For once I wanted to play the game, instead of fiddling with the files during my leisure time, so I've been busy doing that.

But I have made progress nonetheless. I had to do a full respec (turns out I like energy pistols more than shotguns), including stats, skills, feats and special abilities; so that helped to figure out how those things are serialized in the file. Today I even temporarily changed the gender of my character from female to male, so that guards at a certain base don't go crazy, even though I wear their uniform (this is controlled by the byte at position 0x1A from the character name in the file, by the way).

I need to figure a lot more things before being able to fully parse the file, but a basic respec tool could be closer to reality than I first thought. This will take a while though, I don't have much time for this at the moment. But if anyone ever thinks about undertaking this one before I do, please do not hesitate to contact me and ask.

I think the best way to do this would be to take advantage of epeli's character builder. It already provides a nice interface to load and save builds, so we'd just need to decode/encode the base64URL the same way he does (it looks like an array of unicode character values, save for a couple of substitutions that would cause conflicts in a URL). It's all client side anyway, so the functions are all there, no need to even bother him.

Well that's all. I'm off to play a bit more now that I can have fun around that base.