Over the years of doing anything and everything Pip-Boy related, I’ve spent quite a lot of time in and out of the game files as part of recreating the interfaces in my own PIP-OS variants, it’s become a common part of the process but it’s not something I’ve written much about yet. Recently I did a much deeper dive into reverse engineering some of the in game implementations of the Pip-Boy and I’ve amassed a pile of notes, so I figured it was about time to “formalise” some of my notes in the interest of shared knowledge.
Fundamentals
Before we delve too deep let’s take a look at the high level of how the Pip-Boy is implemented in game, in particular I’ll be talking about Fallout 4, but the same is mostly true of Fallout 76 too. When you bring up the Pip-Boy in game, the interface you’re looking at is actually a Flash game (of sorts), for those that didn’t grow up on the internet in the early 00’s, we had a whole Flash era of gaming pre mobile era that was based entirely around Shockwave Flash (later Adobe Flash), a lot of my early internet memories were watching Flash animations and playing games on sites like Newgrounds, Joe Cartoon and StickDeath. Flash was discontinued in 2020 and remains a relic of a bygone era.
Inside of Fallout, Flash Player wasn’t actually used to run these interface, instead Fallout and many other game engines of the time used Scaleform, this allowed UI/UX designers to build the game UI’s in established tools for UI design such as Adobe Flash Professional, they could then export the projects as .swf files and then integrate them into the game through the Scaleform GFx middleware inside of the game engine. This was a fairly standard means of building in game UI’s and a ton of AAA game menu systems you’re familiar with have been built this way.
Companion App
Fallout 4 shipped alongside a companion app for your Android or iOS device that allowed you to run the same Pip-Boy interface from the game on your smartphone and it would interact with the game. The special edition of the game also came with a neat elaborate phone case shaped like a Pip-Boy so you really could live the dream; the very same Pip-Boy that started my whole foray into Pip-Boy replicas, so it holds a special place for me. My recent deep dive back into this topic came about after discussion on Discord about how this app communicated with the game, there was still some gaps in available knowledge on exactly how all elements of this worked so I went spelunking.
How does it work?
Note: I’m not going to write too much detail here about the exact methods I used to explore the app. I tried to find an EULA for the Pip-Boy app but I cannot find one anywhere, so I can’t be sure what is and is not permissible. The Fallout 4 EULA does have some things to say about de-compilation of binary assets, especially when it comes to sharing and re-use of those things, so I can only assume the mobile app is something along the same lines, but to err on the side of caution I won’t mention too many specifics.
The companion app actually runs the same Scaleform SWF files as the game uses, but rather than via the Creation Engine middleware, is actually using the Unity game engine with a Scaleform plugin, and then leveraging Unity’s features for exporting to Android/iOS to create the mobile apps. But are the SWF files actually the same between the ones inside the game and the ones inside the app? Almost! But not quite. There’s a collection of additional Actionscript classes added under bhvr, amongst other things we’ll go over later.
Interesting note here, while it’s not explicitly stated anywhere it appears that the companion app was actually mostly developed by Behaviour Interactive (hence the bhvr packages), the developers behind Fallout Shelter, Behaviour Interactive do have Fallout 4 listed as “Other Noteworthy Games” on their website portfolio so my assumption is that they were tasked with bringing in the companion app functionality.
Unity components
So what else does the Unity layer add into the mix? The Scaleform apps have a bunch of integrations that would usually be hooks into the Creation Engine, this includes things such as playing audio, passing data to be presented and various other hooks such that when a user performs an action in the UI, something then happens in game. A lot of those things are replicated here in Unity, there’s an AudioManager GameObject, a PipboyDataStructure and so on. Importantly for the app functionality there is also the NetworkManager responsible for getting data to and from the game.
The “ExternalInterface”
A lot of the calls from the SWF UI elements out and into the game are via a class called the BGSExternalInterface. Inside of Fallout 4 this interface calls methods that are attached to each main class, but in the versions present in the app this class is actually modified and instead the Flash ExternalInterface class is called. Functions such as playSound are registered external to the SWF middleware, then whenever the UI elements need to play a sound a call is made to BGSExternalInterface.call("playSound", ...) and the implementation knows how to handle that call. Inside the game this will then use the game engines audio capability, inside of the app the modified version of this class calls the Flash external call which then hands off to the C# handlers inside of Unity to eventually hand off to the AudioManager instance in the case of playSound.
Netcode
The element I set out to learn more about was the networking code so let’s dive deeper into that. I would like to preface this by noting that this isn’t new knowledge by any means, there’s been various others over the years that have explored this quite thoroughly (most notably https://github.com/NimVek/pipboy). A lot of previous explorations of this have been via packet capture and analysis, I wanted to take a look at it with fresh eyes from the implementation up and see if there’s anything interesting to be gleaned from it.
The data exchange between the game and the app is abstracted from the SWF’s themselves, so for example your inventory, player stats, quests status etc. are stored in data classes in the engine and the SWF UI elements are just responsible for presenting that data to the player. To make the app work in tandem with the game, there just needs to exist a network layer that handles the synchronization of that data between the game and the app, it also needs to handle various other events such as pressing to use a stimpak, or opting to fast travel etc.
The network protocol is proprietary but relatively straightforward. The first 4 bytes received are a 32 bit unsigned integer that denote the length of the whole data packet so we know how many subsequent bytes we should read. The following byte (in orange in the below diagram) is an enum type for the type of packet that is being sent. There are 6 types that a packet can be, these are used to separate which things are data updates, results of commands and so on, in the below example we’re looking at type 3 which is a DataUpdate used to signal a piece of data that should be updated as part of synchronising game and app data.
DataUpdate packets then have their own structure, the first byte of this structure will denote what primitive data type is being sent, this is again an enum, and in the above case the green 06 shows that this update is of a string type data. Following the data type are 4 bytes parsed as another uint32 which is the data ID, each piece of data in the database has a unique identifier, this allows pieces of data to be updated by referencing it’s ID. In the above example the ID 12345 is being set. Previously we’d determined that the data type will be a string, so for string data we parse for a null terminated string, so in this case we keep reading bytes into the buffer until we reach a byte that is 00 and that will conclude this data update.
This is just an example of one piece of data being updated, when the app first connects to the game though there will be thousands of these DataUpdate packets the sync every piece of data to fully populate the app UI. Once this data is synchronised however, these packets will be sent each time any of these values changes, for example when you pick up a new item in game an updated value for the number of that item you have will be sent and the app inventory will be in sync with what you have in the game. Below you’ll see a more complete example of a set of data update packets setting various strings and values.
Conclusion
I’m going to call that enough for this part lest I never get around to completing it, in the future I’ll delve some more into the ExternalInterface calls and show how we might be able to make use of the calls to run the interface components ourselves. I may also revisit some of these network concepts to demonstrate how it can be used to sync data to other things and perhaps use it in our own Pip-Boy replicas.