Arcane University:World space synchronization

< Arcane University

In this article, world space synchronization refers to the process of copying a part of one world space into another, so that the borders match up in both. This is needed for Beyond Skyrim projects whose provinces border each other. Once a project has their border region more or less finalized, it needs to be copied over to the other project so that they can level design their border region while accounting for what's directly on the other side; and once that's done, this border region needs to be copied back the other way.

This article will demonstrate how to do one such copying operation using xEdit, without the larger-scale project management involved; it is also useful for other situations where this may have to be done, and not just in Beyond Skyrim (many scripts discussed here were originally created for Skywind).

Preparation

You need two world spaces, one containing the landscape to copy (henceforth origin world space) and one to copy to (destination world space). For clarity's sake, this article will assume that they are contained in masters (.esm) to distinguish them from plugins (.esp), but the process is the same if they are in plugins. The two world spaces may be part of the same master file, which makes the process a bit simpler, but often will be in separate masters.

The copy happens in two passes that are independent and can be done in any order: copying of landscape, and copying of references. Both methods work on rectangular regions in a world space, so you must figure out what those rectangular regions are (there can be multiple that combine into a more complex shape, even just one cell large if necessary). It helps to have something like a grid map on hand.

First, find out the difference between the coordinate systems of the two world spaces. Pick a location that can be pinpointed in both world spaces, then note its cell coordinates in the origin world space, and its cell coordinates in the destination world space. Subtract the X and Y coordinates in the origin world space from the X and Y coordinates in the destination world space, respectively. This will give you the offset, or the number of cells to move the copy by along the X and Y axes. Also calculate the X and Y offsets in units by multiplying them with 4096 (as that is the number of units comprising the edge of a Skyrim world space cell). If the offset is not a clean multiple of a cell, then the method for copying landscape will not work, as the method for that works cell by cell. The method for copying references is unit-based, but will be of little use if the landscape cannot be accurately copied. Lastly, if the sea level is not at the same height in the two world spaces, write down the difference between them in units (this does not have to be a multiple of 4096).

Write down the cell coordinates of the northeast and southwest corners of every rectangle according to the origin world space's coordinate system. Add the offsets onto this to get the coordinates in the destination world space.

Lastly, also calculate what those coordinates amount to in Units. You will need the southwest corner of the southwest cell, which you can find by multiplying the X and Y cell coordinates by 4096. You will also need the northeast corner of the northeast cell, for which you must multiply the X and Y cell coordinates by 4096 and add an extra 4096.

This spreadsheet template will help you organize the information, calculate most of it for you, and present it in the form that's required in the next steps. File:World space synchronization template.zip

If the two world spaces belong to different masters, and the destination world space's file is not a master of the origin world space's file, then the copied area must be free of land textures and references whose base objects come from the origin world space's file. This can be resolved in advance (such as by re-implementing any non-vanilla assets in a shared master, and switching all references to use those shared base objects; for Beyond Skyrim, this is BSAssets) but can also be arranged at the end of the process; this article will show how to contain the copy in a patch plugin which can be tweaked before being sent off to be merged into the destination world space's master. If it is too difficult to integrate land textures, these can optionally not be copied as detailed further below.

To redirect all references of a certain base object in the origin file to a base object that is compatible with the destination file, one can use the "Search and replace" function in the Creation Kit, or the following xEdit script by Zilav, which has the advantage of also covering land textures (just replace OldFormID and NewFormID as needed):

unit UserScript;

const
  OldFormID = $01012345;
  NewFormID = $01054321;

function Process(e: IInterface): integer;
begin
  CompareExchangeFormID(e, OldFormID, NewFormID);
end;

end.

Sometimes, there may already be references in destination cells where references are going to be copied to. This could be due to earlier synchronization efforts or due to level design that went across the border. It would be prudent to delete all references in these cells before everything is copied over, as it will be difficult to tell what's what.

Landscape

Here, landscape refers to the LAND record in each cell, containing land height (i.e. heightmap data) and the land textures used (which refer to LTEX base objects). In xEdit, you need to disable the "Simple records" option or you will not be able to copy landscape. Find the options menu by clicking on the three horizontal bars, then "Options"; then under the General tab, make sure "Simple records LAND, NAVI, NAVM, CELL, WRLD (requires restart)" is unticked.

If you wish to copy land textures, and some of them still come from the origin world space's file, then you must temporarily make the origin file a master of the destination file for the copying to work; the copying will also not happen in a patch plugin, but is applied directly on the destination file. However, a later step will extract the edits into a patch plugin after all.

Right click anywhere and choose "Apply script". Choose the built-in script "Worldspace copy landscape area to another worldspace". If you want to copy land textures, edit the script so that bCopyLayers = True. You can do this directly in xEdit in the "Apply Script" dialogue by simply typing into the script after you have selected it in the drop-down. When you see "The script has been modified. Do you want to save it?", answer Yes.

A pop-up window appears. Configure the two world spaces; enter the cell coordinates of the southwest corner of the current rectangle, and sizes of the rectangle along both axes, which is the difference between the two X and Y coordinates respectively. A rectangle comprised of one cell will have sizes 1 and 1; a square of 2x2 cells will be 2 and 2, and so on. Then enter the southwest cell coordinates in the destination world space by summing the cell offsets with the origin world space's coordinates. You must leave "Create patch plugin" unchecked unless you have no problematic land textures or forego copying them entirely, and if you check it, every rectangle will go in its own patch plugin which you will want to merge afterward.

Then allow the script to run. LAND records are heavy and so this will take some time. For a large rectangle, xEdit may run out of memory, so consider using the 64bit version. After a rectangle has been copied, you can save to free the memory back up. If you see an error like "Load order FileID [06] can not be mapped to file FileID for file", then that means some land textures were used that are not from a master of the destination esm, and you may have to apply the aforementioned workaround.

Once you are done with all rectangles, and you had "Create patch plugin" checked, then you have a bunch of different plugins that you will now have to merge together. If you didn't have "Create patch plugin" checked, then the destination .esm was edited directly and you will now probably want to extract those edits into a patch plugin after all. Save and close xEdit just in case so you can make a back-up of your destination esm. Put a fresh copy of your destination esm in your data folder, one free of any land copying. In xEdit, load only this fresh esm. Once loaded, right click it, select "Compare to..." and select your modified esm. This will make a new load order in xEdit where the modified esm appears to override the records of the fresh one. Right click anywhere and select "Apply filter"; in the pop-up, check "by Base record signature" and check "LAND". Also make sure "by conflict status overall" is still checked, and deselect the top three options. Also check "Flatten Blocks/Sub-Blocks" and "Flatten Cell Children" further below. Run the filter and wait a (long) while. After it's done, only the changed LAND records should still be visible in xEdit. In your modified esm, Alt+Click the expand icon next to the destination world space to instantly open all cells, select one LAND record, press Ctrl+A to select all other LAND records as well, right click and select "Copy as override into...". Make it a new plugin. This plugin now contains all the results of the process, and you can test it alongside the fresh esm.

N.B.: The "Worldspace copy landscape area to another worldspace" xEdit script only takes into account the plugin file (.esm or .esp) which the origin world space is from; it does not incorporate overrides to the landscape from subsequent plugins in the load order. If you have such edits, you will therefore have to merge them into master first. If you are copying from a vanilla world space (from Skyrim.esm) and need to incorporate edits, then lord help you, because xEdit will not let you modify Skyrim.esm. You will need to create a copy of Skyrim.esm and rename it to something else, for example Skyrim2.esm; launch xEdit with the launch parameters -IKnowWhatImDoing and -AllowMasterFilesEdit; change the file header of the editing plugin so that its master is Skyrim2.esm instead of Skyrim.esm; merge the edits (for instance, right click and "copy as override (with overwriting)", and if that causes issues, clicking and dragging from right to left, one record at a time); save and quit xEdit, and rename Skyrim2.esm back to Skyrim.esm, replacing the original. You can then run the "Worldspace copy landscape area to another worldspace" script. If you are copying from the Tamriel world space, then you have an additional problem in that its FOrmID is 0000003C and FormIDs lower than xx000800 are reserved by the game engine and thus also treated specially by xEdit, so you may have to "Change FormID" on the world space before merging in the edits, and change it back afterwards.

References

The process with references will be to first make a plugin that just contains overrides of all the references that need copying (without actually changing anything yet), then turning those into new references, and finally applying a script to move them to the destination world space.

Load the origin esm (you don't need to load the destination esm yet at this point). To filter for references within the rectangular area, apply the following custom script; it doesn't matter on what, as it will check all records across all currently loaded files (feel free to improve the script and make it faster). Before applying, edit the values of xmin, xmax, ymin and ymax inside the Initizalize function to match the unit coordinates of your rectangle (again, feel free to improve in order to make it interactive). The script is a crude mishmash of code by km816 and Zilav.

unit FilterByPosition;

var
    xmin, xmax, ymin, ymax: float;


function Filter(e: IInterface): Boolean;
var
  xp, yp: float;
begin
  
  if GetIsDeleted(e) then exit;
  
  if (Signature(e)<>'REFR') and (Signature(e)<>'ACHR') then exit;

  xp := StrToFloat(GetElementEditValues(e,'DATA\Position\X'));
  if (xp<xmin) or (xp>xmax) then exit;

  yp := StrToFloat(GetElementEditValues(e,'DATA\Position\Y'));
  if (yp<ymin) or (yp>ymax) then exit;
  
  Result := True;
end;

function Initialize: Integer;
begin
  
  xmin := 0;
  xmax := 4096;
  ymin := 0;
  ymax := 4096;
  
  
  FilterConflictAll := False;
  FilterConflictThis := False;
  FilterByInjectStatus := False;
  FilterInjectStatus := False;
  FilterByNotReachableStatus := False;
  FilterNotReachableStatus := False;
  FilterByReferencesInjectedStatus := False;
  FilterReferencesInjectedStatus := False;
  FilterByEditorID := False;
  FilterEditorID := '';
  FilterByName := False;
  FilterName := '';
  FilterByBaseEditorID := False;
  FilterBaseEditorID := '';
  FilterByBaseName := False;
  FilterBaseName := '';
  FilterScaledActors := False;
  FilterByPersistent := False;
  FilterPersistent := False;
  FilterUnnecessaryPersistent := False;
  FilterMasterIsTemporary := False;
  FilterIsMaster := False;
  FilterPersistentPosChanged := False;
  FilterDeleted := False;
  FilterByVWD := False;
  FilterVWD := False;
  FilterByHasVWDMesh := False;
  FilterHasVWDMesh := False;
  FilterBySignature := True;   //
  FilterSignatures := 'REFR';  // refs only
  FilterByBaseSignature := False;
  FilterBaseSignatures := '';
  FlattenBlocks := True;
  FlattenCellChilds := True;
  AssignPersWrldChild := False;
  InheritConflictByParent := False; // color conflicts
  FilterScripted := True; // use custom Filter() function

  ApplyFilter;
  Result := 1;
end;

end.

Alt+Click the expand icon next to the world space, select one reference, press Ctrl+Alt to select all references in the world space, and "Copy as override into" a new plugin. This may take a while. If you have more rectangles, then right click and Remove filter, and run the script again. This time, "copy as override" into the plugin you previously created for convenience's sake.

This paragraph pertains to copying references for LOD only Areas that are far enough away from the playable border to never be in the loaded area, may still be visible from a distance and thus will have to be copied over to be included in LOD that is generated for the destination world space. Here, you may like to exclude references that won't be visible in LOD. Once you have your .esp from the previous step, apply a Filter for Base record signature "Tree" (shrubs and other small flora will also be lumped in). Copy these as override from your previous .esp into a new one. Remove the filter. Then do another filter by running the below script by Zilav, which retains only references whose base object has LOD meshes configured (the built-in "visible when distant" filter options are not effective). Copy all remaining references from the same previous .esp into the same second .esp you created. You may now discard the first .esp and continue with the second one.

unit ApplyCustomScripted;

function Filter(e: IInterface): Boolean;
var
  stat: IInterface;
begin
  stat := WinningOverride(BaseRecord(e));
  if Signature(stat) = 'STAT' then
    Result := ElementExists(stat, 'MNAM');
end;

function Initialize: Integer;
begin
  FilterConflictAll := False;
  FilterConflictThis := False;
  FilterByInjectStatus := False;
  FilterInjectStatus := False;
  FilterByNotReachableStatus := False;
  FilterNotReachableStatus := False;
  FilterByReferencesInjectedStatus := False;
  FilterReferencesInjectedStatus := False;
  FilterByEditorID := False;
  FilterEditorID := '';
  FilterByName := False;
  FilterName := '';
  FilterByBaseEditorID := False;
  FilterBaseEditorID := '';
  FilterByBaseName := False;
  FilterBaseName := '';
  FilterScaledActors := False;
  FilterByPersistent := False;
  FilterPersistent := False;
  FilterUnnecessaryPersistent := False;
  FilterMasterIsTemporary := False;
  FilterIsMaster := False;
  FilterPersistentPosChanged := False;
  FilterDeleted := False;
  FilterByVWD := False;
  FilterVWD := False;
  FilterByHasVWDMesh := False;
  FilterHasVWDMesh := False;
  FilterBySignature := True;   //
  FilterSignatures := 'REFR';  // refs only
  FilterByBaseSignature := False;
  FilterBaseSignatures := '';
  FlattenBlocks := False;
  FlattenCellChilds := False;
  AssignPersWrldChild := False;
  InheritConflictByParent := False; // color conflicts
  FilterScripted := True; // use custom Filter() function

  ApplyFilter;
  Result := 1;
end;

end.

Filtering Tree References for LOD After copying all tree references for LOD into your esp, it is ideal to filter these references so that only those with LOD billboards are preserved. The easiest way to obtain a list of which tree records lack billboards is to run xLODGen with your LOD reference plugin loaded. Select the worldspace where the references are added, and only tick the box for "Tree LOD". Generate LOD. The program's log will proceed to list every tree record in the worldspace both with and without LOD. Copy the list of records that lack billboards and save it in a new file. You can now open your LOD reference plugin in xEdit and remove the record types you've listed from your esp. Depending on the area you're copying from, you can reduce the esp's references between 50% and 90%. As references without LOD billboards will not be seen in LOD only cells, these references do nothing but bloat the esp needlessly.

Once your patch .esp has all the needed references from all rectangles, save and close xEdit and make a backup. Load it again, apply a filter that only does "Flatten blocks/sub-blocks" and "flatten cell children", select all references in the plugin using the Alt+Click/Ctrl+Click method, right click, and "Change FormID". Select the current plugin. This will change all the overrides into new references, but it will take some time.

Save, close, backup, and reload, this time including the destination esm. Run the same filter to flatten blocks and cell children, select all references in the patch plugin, right click, and apply the built-in script "Worldspace move references into another worldspace". Indicate the destination world space; check "Move persistent references" and "only from current plugin". Then enter the offset values expressed in units, which you should've calculated earlier.

After the script is done, your plugin should be ready, unless there were references whose base object comes from the origin esm. In that case, your plugin will depend on both esms and must be tweaked and cleaned before it can be merged into the destination esm.

Final checks and extras

You independently created a patch plugin for landscape and another one for references; you may as well merge them together. If base objects from the origin esm are still in use, you can detect them by right clicking your plugin, and applying the built-in script "Report masters", then selecting your origin esm. This will list all such references and what base objects they use, and also LAND records where an incompatible land texture is used (look for mentions of "/ Layers" in the xEdit output); but frustratingly, it will not list the land texture itself.

To replace base objects, you can still use the script shown above in the Preparation section. After everything is replaced as needed, right click on your patch plugin and select "Clean masters". If nothing from the origin esm is used anymore, then it will now be removed as master and the plugin can be merged.

Among the references, remove any map markers, doors and other gameplay-related content that doesn't belong. Most of these can be easily found in the persistent reference cell.

Although copying landscape and copying references both require rectangular areas, they don't have to be the same rectangles. Landscape copying is destructive (it will wholesale replace any affected land) so precise rectangles are needed. Reference copying actually does not change the destination cells if the origin cells have no references, so those rectangles can be made cruder if it saves effort.

As a final note, if the two world spaces have a height difference, this may be due to an error, as Skyrim's water height of -14000 is typically standardized on. In this case, rather than continuing to enter Z offsets during copying, it may be better and safer to fix the height of the errant world space. The script below will do this for you on a world space you apply it to; edit the script to set the ShiftZ variable, as well as the MinX, MinY, MaxX and MaxY variables to cover the entire world space. It is based on the built-in "Worldspace change height" script, but expanded by Elscrux to also cover teleport markers, navmesh, CELL-based water levels, and water height properties in the WRLD record. If you have a lot of exterior navmesh, this may take a long time.

{
  Changes height of exterior worldspace cells in specified range: references, landscape, door teleport coordinates, navmeshes, CELL water level and WRLD properties (default land height, default water level, and LOD water level).
  Navmesh will have to be refinalized and the NAVI regenerated.
  Works for all games (?), apply to worldspace in xEdit.
  ShiftZ must be divisible by 8
  Min/Max constants set the range of cells to update
}
unit WorldspaceChangeHeightRedux;

const
  ShiftZ = 888;
  MinX = -10;
  MinY = -10;
  MaxX = 10;
  MaxY = 10;
  CellSize = 4096;
  sRefSignatures = 'REFR,ACHR,PGRE,PMIS,PHZD,PARW,PBAR,PBEA,PCON,PFLA';
 
//===========================================================================
function Initialize: integer;
begin
  if wbSimpleRecords then begin
    MessageDlg('Simple records must be unchecked in xEdit options', mtInformation, [mbOk], 0);
    Result := 1;
    Exit;
  end;
end;

//===========================================================================
function Process(e: IInterface): integer;
var
  Sig, baseValue, doorRef: string;
  i: integer;
  cell, doorCont, door: IInterface;
begin
  Sig := Signature(e);

  if Sig = 'LAND' then begin
    cell := LinksTo(ElementByName(e, 'Cell'));
    if not Assigned(cell) then
      Exit;
    if (GetElementNativeValues(cell, 'XCLC\X') >= MinX) and
       (GetElementNativeValues(cell, 'XCLC\X') <= MaxX) and
       (GetElementNativeValues(cell, 'XCLC\Y') >= MinY) and
       (GetElementNativeValues(cell, 'XCLC\Y') <= MaxY)
    then
      SetElementNativeValues(e, 'VHGT\Offset', GetElementNativeValues(e, 'VHGT\Offset') + ShiftZ/8);
  end
  else if Sig = 'WRLD' then begin
	SetElementNativeValues(e, 'NAM4', GetElementNativeValues(e, 'NAM4') + ShiftZ);
	SetElementNativeValues(e, 'DNAM\Default Land Height', GetElementNativeValues(e, 'DNAM\Default Land Height') + ShiftZ);
	SetElementNativeValues(e, 'DNAM\Default Water Height', GetElementNativeValues(e, 'DNAM\Default Water Height') + ShiftZ);
  end
  else if Sig = 'CELL' then begin
	if (GetElementEditValues(e, 'XCLW') <> 'Default') then
		SetElementNativeValues(e, 'XCLW', GetElementNativeValues(e, 'XCLW') + ShiftZ);
  end
  // update position of navmeshes
  else if Sig = 'NAVM' then begin
	SetElementEditValues(e, 'NVNM\Min Z', GetElementNativeValues(e, 'NVNM\Min Z') + ShiftZ);
	SetElementEditValues(e, 'NVNM\Max Z', GetElementNativeValues(e, 'NVNM\Min Z') + ShiftZ);
	for i := 0 to ElementCount(ElementByPath(e, 'NVNM\Vertices')) - 1 do begin
		SetElementEditValues(e, 'NVNM\Vertices\[' + IntToStr(i) + ']\Z', GetElementNativeValues(e, 'NVNM\Vertices\[' + IntToStr(i) + ']\Z') + ShiftZ);
	end
  end
  // update position of reference records
  else if Pos(Sig, sRefSignatures) <> 0 then begin
	baseValue := GetElementEditValues(e, 'NAME');
	if (Copy(baseValue, Pos('[', baseValue) + 1, 4) = 'DOOR') then begin
		doorCont := ElementByPath(e, 'XTEL\Door');
		if Assigned(doorCont) then begin
			door := LinksTo(doorCont);
			SetElementEditValues(door, 'XTEL\Position/Rotation\Position\Z', GetElementNativeValues(door, 'XTEL\Position/Rotation\Position\Z') + ShiftZ);
		end;
	end;
  
    if (GetElementNativeValues(e, 'DATA\Position\X')/CellSize >= MinX) and
       (GetElementNativeValues(e, 'DATA\Position\X')/CellSize <= MaxX) and
       (GetElementNativeValues(e, 'DATA\Position\Y')/CellSize >= MinY) and
       (GetElementNativeValues(e, 'DATA\Position\Y')/CellSize <= MaxY)
    then
      SetElementNativeValues(e, 'DATA\Position\Z', GetElementNativeValues(e, 'DATA\Position\Z') + ShiftZ);
  end;
end;

end.

If you need to batch mark as delete a selection of references (for instance, to clear the land and make room for a copy coming in), select the references in xEdit and apply this custom script:

{
  Undelete and disable references, almost the copy of internal
  xEdit code.
  Does not require applied filter.
}
unit UserScript;

  
function Process(e: IInterface): integer;
var
  Sig: string;
  xesp: IInterface;
begin
  Result := 0;
  
  if not (IsEditable(e)) then
    Exit;
  
  Sig := Signature(e);

  if 
    (Sig <> 'REFR') and
    (Sig <> 'PGRE') and
    (Sig <> 'PMIS') and
    (Sig <> 'ACHR') and
    (Sig <> 'ACRE') and
    (Sig <> 'NAVM') and
    (Sig <> 'PARW') and // Skyrim
    (Sig <> 'PBAR') and // Skyrim
    (Sig <> 'PBEA') and // Skyrim
    (Sig <> 'PCON') and // Skyrim
    (Sig <> 'PFLA') and // Skyrim
    (Sig <> 'PHZD')     // Skyrim
  then
    Exit;

  
  AddMessage('Deleting: ' + Name(e));
  
  // delete
  SetIsDeleted(e, True);
  
end;

end.