Difference between revisions of "User:Thingy Person"

Revision as of 15:35, 25 October 2023

A Technical History of Beyond Skyrim (WIP)

This article outlines some technical aspects of Beyond Skyrim in order to preserve a correct history and debunk some myths.

Heightmap

Beyond Skyrim grew out of a forum thread some weeks after Skyrim's release, urging beginning province mods to avoid some of the mistakes seen in the modding scene for Oblivion. One such alleged mistake was that province mod projects did not collaborate, using custom heightmaps that did not fit together, rendering the mods incompatible. When Tamriel Heightmaps, a common heightmap resource for Oblivion, eventually did appear, it was too late and too unattractive for existing mod teams to consider using it.

Although there is visible land beyond Skyrim's borders, it is only LOD, i.e. low-resolution versions of the land that is visible from a distance but cannot be interacted with; furthermore, it cuts out roughly half of the way into the neighboring provinces and is not accurate to the scale of Tamriel implied by Skyrim's landmass. And thus, the All-Tamriel Heightmap project was started to create lore-friendly and correctly scaled heightmaps for the other provinces that are compatible with Skyrim and with each other. The heightmaps were modeled in World Machine and imported into Skyrim with TESAnnwyn. Eventually these heightmaps would become the collective heightmap for the Beyond Skyrim heightmap, contained in BSTamriel.esm, a master which each project depended on alongside BSAssets.esm.

The original plan was that the heightmaps would be placed into the vanilla worldspace, meaning one could travel out of Skyrim with no load screen. Several issues changed this plan.

• A problem with LOD, caused by difficulties with creating a custom lodsettings/Tamriel.lod file and having it be loaded by the game, jeopardized the prospects of being able to display LOD beyond the limits specified for the vanilla worldspace. Since there were no such difficulties involved with a lodsettings file for a custom worldspace, it was eventually decided to put all the custom provinces into one custom worldspace, separate from the vanilla one.
• The first "worldspace bug", where collision issues occur for actors that are too far east and west of the world origin. Actors - such as NPCs, enemies, and a horse the player is riding - will appear to sink into and pop out of the ground repeatedly. The problem occurs in cells beyond 64 and -64 along the X axis; it does not affect cells far to the north or the south that are within this east-west limit. Because of this, one proposed solution was to divide Tamriel into three world spaces, one of which would have included Morrowind and Black Marsh.
  • Then an ini tweak was found that works around the issue, along with a conjectured technical explanation. With this tweak, the plan for a unified heightmap was back on the table.
• The second "worldspace bug". If the player is beyond 64 or -64 cells, on either axis this time, and is facing away from the world origin, then all gameplay interactions break. Harvestables cannot be picked, doors cannot be opened, attacks will not hit enemies, and so on. This bug was discovered late because it took until someone placed an interactible object outside the safe limit - specifically, a level designer from the Northpoint project placing a flower. There is no known workaround for this one, and as such this bug definitively killed the idea of a large Tamriel worldspace.

Master Limit

This section details how the merging process evolved, and its impact on which master files a project could have as its dependencies.

Before the Creation Kit's release, many small mods were already being made, mostly using a utility called SkyEdit, which was developed by reverse engineering part of the plugin data format used by the engine, and did not allow for any level design. (Contrary to popular belief, Moonpath to Elsweyr is not one of these mods, as it was made using the Creation Kit. See here from the author himself. The first and perhaps only mod that managed to implement new, albeit very crude, level design before the Creation Kit is called Wounded Knee Gulch.)

For the Beyond Skyrim heightmap's development, not even the Creation Kit's release was enough to satisfy all requirements; a worldspace plugin produced by TESAnnwyn cannot be converted to an .esm with the Creation Kit, so the esm flag would have to be set using unofficial tools. In addition, plugin cleaning and merging, essential functionalities in modding including Beyond Skyrim, are tricky to do with the Creation Kit, if not poorly understood at the time.

For these functionalities, another utility called TESVSnip was widely used. However, TESVSnip had a huge problem, in that it used an incorrect compression library and irreversibly damaged many plugins it edited (see e.g. this forum thread). When this came to light, an initiative was mounted to obtain the source code for TES4Edit and update it for Skyrim, resulting in TES5Edit, which finally allowed these operations to be done safely. Another utility called TESVGecko had been in development with many of the same goals, but it was put on hold.

For a long time, however, TES5Edit still did not support editing or merging one kind of data: the VMAD subrecord, which contains information on scripts attached to objects, and their properties. This meant that plugins implementing quest content could not be merged into the master file using TES5Edit, which is a limitation that the Cyrodiil project ran into. At the time, only the Creation Kit's own Version Control system (which had been sufficiently documented by this point) allowed for merging script content, so the Cyrodiil project switched to using this for merging.

However, using Version Control introduced two new issues:

• Version control has a hard-coded limit of two master files. This means that the Creation Kit will refuse to merge plugins into master files that have more than two .esm files as dependencies. This was likely implemented to avoid a situation where a Bethesda employee accidentally loads two DLCs at the same time, which would make one dependent on the other; the two allowed masters would thus be Skyrim.esm and Update.esm.
  • Since this issue more or less coincided with the discovery of the second worldspace bug discussed earlier, it was decided to cut up Tamriel and store the worldspaces into the separate project esm files. This meant that BSTamriel.esm could be left out, and if Update.esm is similarly left out, then the only remaining dependencies are Skyrim.esm and BSAssets.esm, thus staying within the master limit. It is not known what tools were used to do this cutting up, but the process has been redeveloped more recently and is documented here.
  • A knock-on effect was that projects occupying the same worldspace henceforth also needed to have the same master file, as a worldspace cannot be introduced from two masters at the same time. Thus, the BSCyrodiil.esm and BSElsweyr.esm master files, which both used the BSHeartland worldspace contained in BSTamriel.esm, merged into BSHeartland.esm. Similarly, the High Rock and Hammerfell projects merged their masters into BSIliacBay.esm.
  • Another knock-on effect relates to using DLC assets in Beyond Skyrim projects. Since a DLC master file could not be used directly, any desired object forms had to be re-implemented into BSAssets.esm. BSAssets would only include the forms and not the models and textures themselves, so the DLCs were still requirements for running the mod, just not by having the DLC master file as a dependency.
  • In order to avoid a DLC item and its re-implementation showing up as two different kinds of items in the player's inventory, Beyond Skyrim: Bruma ships with a DLC patch .esp that edits instances of re-implemented DLC objects and switches them back to being instances of the originals. Any future Beyond Skyrim release would have needed such a plugin that the end user would then have to add to their load order.
  • Many years later, SSE Creation Kit Fixes implemented a patch that removes the two master limit, and since April 2022, Beyond Skyrim has had DLCs as masters, removing the need for re-implementing DLC records and greatly simplifying this aspect of the project.
• The Creation Kit automatically renames fragments scripts (such as Topic Info Fragments) on merge. Fragment scripts are a special kind of script which is named automatically by the CK, incorporating the FormID of the object it is attached to. Since the FormID changes on merge, the CK will rename the fragment script accordingly. It will do this for the record data in the .esm file, but doesn't take care of renaming the loose .psc script source file, changing its header, and recompiling it. This means that the scripting is broken on merge, as the game will look for a renamed script file that doesn't exist, as the script file still has its old name.
  • The functionality for handling the external files can be restored by installing Perforce (now called Perforce Helix Core), a professional version control software, and hooking it up with the Creation Kit. During the game's development, Bethesda used Perforce for version control, including checking out forms so that only one employee could edit any given form at a time, avoiding conflicts. In Beyond Skyrim, perforce is only used for its specific functionality of renaming and recompiling fragment script files. The process of setting this up is complicated and is documented here. See also this forum thread from when this was discovered.
  • Later on, TES5Edit came to support the VMAD subrecord, making it a viable merging tool for script content. It does not rename the scripts in either the .esm or in the loose script files, meaning the scripts will function, but be "misnamed" with regards to the FormID. This is only a problem in the very unlikely but theoretically possible case where a Topic Info Fragment script from a newer plugin ends up being created with the same name as an existing one from a previous plugin, causing a nasty conflict.
  • Merge Plugins also supported automatic rename and recompile of fragment scripts and was used by one of the projects for merging (see Masterfile Merging with MergePlugins) but this functionality was removed in a later version.

Legendary Edition and Special Edition

When Skyrim Special Edition (abbreviated as SE) released in 2016, it was initially treated as another platform to release on, but development continued on the old version of Skyrim, known as Legendary Edition (LE). This is because converting a mod from LE to SE was a known process, converting from SE to LE was not, so developing on LE meant that releases could be done on both platforms.

• Note: Legendary Edition was actually a re-release of Skyrim from 2013 with all the DLCs included, but it is nowadays used to refer to that version of Skyrim as a whole. There is no technical difference between having bought Skyrim and its DLCs separately, or having bought Legendary Edition.

Over time, some projects came to have so much content that they ran into memory limitations inherent in LE. While LE's game executable is 32-bit, SE's executable is 64-bit, allowing it to use vastly more memory. Even Beyond Skyrim: Bruma requires the use of SKSE plugins on LE in order to expand the memory limit somewhat and allow the mod to run. The SE version does not require this. Eventually, these projects switched their development to SE, forgoing the possibility of making releases available for LE, while other projects continued on LE.

With some projects on LE and others on SE, this meant that two different versions of BSAssets had to be maintained in parallel. This was a source of much complexity and regularly led to problems where, for example, some assets were not in both repositories, or one of them contained only outdated versions; or a tree mesh was developed for SE, would be back-converted to LE (which is an imperfect process) and then forward-converted to SE again instead of just using the original version, while its tree animation data would also somehow not make it into the .esm.

Eventually, by January 2022, all projects switched their development to SE, even those that had no pressing need to do so. Reasons included a shrinking playerbase on LE; a dearth of new devs who own LE, which had become difficult and extremely expensive to purchase; being able to use SSE CK Fixes, which was developed to address the regressions found in SE's version of the creation kit, but had become significantly better than even the original LE creation kit; and the prospect of being able to deprecate the LE version of BSAssets, as well as developers not having to have LE and SE installed in parallel anymore if they worked on different projects. This step greatly simplified the maintenance of the project.

Stripped Master

Stripped

SKSE

SKSE as end user requirement: dependency of Bruma LE (and used SKSE script functions before removal in order to work on Xbox), former dependency of Atmora (DOBJ manipulation for combat music)

SKSE as developer tool: Fuz Ro D'oh, Beta Comments, Ingame Editor, etc.

Animations

Improvements to ck-cmd and Skyrim Behavior Tool finally allowing for reliably creating custom behavior graphs