Various updates

GitHub has locked me out due to Microsoft’s forced 2FA policy which I don’t want to comply with. As a result, I can no longer log in to the website to make pull requests, participate in discussions etc. However, at the moment I can still push to repos (wow, so secure huh?), so I can still work on SA Tools, the Mod Loader and other projects. They may pull the plug in the future but at least for now I can contribute to projects that are still on GitHub.

After being locked out I was able to make the following updates:

SA Tools

I added more complete support for reading and writing NJ and GJ files from various games. The update needs some testing so it will be in a separate branch for now.

SA Tools / Texture Editor

1) Unmodified texture data is no longer re-encoded unless the archive format is changed. Adding (instead of importing) textures will now add them as-is if the texture format matches the archive format (e.g. you can add DDS to PAKs or GVRs to GVMs).

2) You can choose between DDS and PNG for textures in PAKs.

3) If you open a single PVR/GVR/XVR texture, it will auto-add all other textures located in the same folder.

4) You can open and edit non-indexed PAKs (SOC folder).

5) You can save individual textures in their original format using the “Save” button.

6) Support for YUV422, Bump, 4-bit Indexed + Mipmaps, Indexed 8-bit + Mipmaps and ARGB8888 indexed PVR pixel formats.

7) Code cleanup, various bugfixes with texture encoding and decoding.

SADX Mod Loader

The game can now load textures in Gamecube GVM files without conversion. This opens up new modding possibilities, particularly for a project I’ve been contemplating to do for a while. I’ll introduce it soon.

Huge thanks to Exant, who decompiled original GVM loading code in SA2 PC, and Kell, who fixed the issues with mipmaps and paletted textures.

Sonic Adventure VMS + Chao Editor first release

Today I’m releasing a new tool based on the original SA1 DLC editor. This tool has the DLC editor integrated, but its functionality expands to cover more VMS files in SA1. Eventually I would like it to support all of the formats described here, but the highlight for today’s release is the inclusion of a new Chao editor.

Like the original DLC tool, the updated editor will be included in SA Tools, which also have a big update underway.
UPDATE: It’s been released and is now included in SA Tools.
The Chao editor is experimental and will be updated as more information on SA1 Chao data becomes available. There is no documentation for it at the moment, although the most important data should be self explanatory.

The Chao system used in SA1 is the least researched of all Chao games. There are several Chao editors for SADX and SA2, but the only Chao editor for the Dreamcast version of SA1 was Tyro’s VMU editor that requires running on an actual Dreamcast VMU or a VMU emulator, which I found inconvenient. Besides, there have been some updates regarding SA1 Chao data (for example we now have the full SA1 Chao data struct from SADX X360 symbols), and I wanted to make use of that new knowledge to help advance SA1 Chao research. Although several aspects of it are now known better, Tyro’s notes on SA1 Chao data were one of the most important sources of information for developing the editor. I would also like to thank the Chao Island member UltimaNumber, who kindly shared the bits of information I was missing and explained how SA1 Chao data works.

The tool can import all Chao VMS data. Data produced by the Japanese version is also supported:

  • SA1 Chao Garden save file
  • Download Data (Black Market Chao)
  • Chao Adventure (created when you take a Chao out for a walk)
  • Upload Data (created when you hold Y+A as the Chao goes in the transporter)

At the moment saving is only supported for Chao Adventure and Download Data, but you can export selected Chao from the garden save file as Download Data or Chao Adventure data. There’s also an option to export files with headers used by the Japanese version of SA1.

As Chao data is rather complex and the editor is experimental, there may be some bugs, especially with saving data. Also, expect UI changes in the future.

I’d like to thank Speeps for providing a screenshot of Buddy the Chao that I used as an icon for this tool.

Sonic Adventure DLC tool: GUI version released

I’ve updated the SA Tools with a new version of the SA1 DLC tool that lets you create and edit VMS files for SA1 on the Dreamcast.

Some documentation is available here.

Apart from the GUI making it easier to edit things, this version also fixes several issues with the original tool that made it produce incorrectly aligned VMS files. Now it should be possible to modify and rebuild any of the official DLCs, or create completely new DLCs from scratch.

The tool ended up being a bit more complex than I originally anticipated, and testing it was a challenge. I’d appreciate it if you try it out and let me know on x-hax if you run into any issues. Have fun!

New SA1 Dreamcast split

Following the complete SADXPC split, here’s my attempt at a near-full level/model/animation rip of SA1 Dreamcast. It includes all levels, almost all object models, the majority of motions and some data from the older split such as level object lists for most levels. When the SA Tools update is complete, it should be possible to copy this over the SADXPC split (except the object lists) and be able to load SA1 levels and objects in SADXLVL2.

This split is less complete than the SADXPC one. Here’s a list of things missing from it:
-Individual landtable items are not ripped as separate files.
-Various miscellaneous data, such as texlists, dialogue text etc. are not ripped.
-Some levels are missing object lists.
-EV files are not ripped (maybe I’ll get to that someday).
-All shape motions are missing.
-Some other motions may be missing.
-Some object models may be missing (shouldn’t be a lot).
-We don’t have source filenames for SA1 Chao assets so I created the file structure from scratch.
-Some Chao animal parts may be missing or have wrong filenames.
-There is some minor inconsistency with character upgrades which have Adventure Field versions. Some of those are placed in the respective Adventure Field objects folder rather than the character’s folder.
-Some models and animations don’t have proper labels because I couldn’t match them with SADX assets.
-Some character animations may have wrong labels.

Other than the above, it’s a complete rip of levels, models and animations that follows the structure of the new SADXPC split. To try it, extract SA1 1.005 (US, animated title screen) ISO to SA Tools\SA1\data and run SA Tools\SA1\splitSA1.bat. The output will be in SA Tools\SA1\output.

A few days ago I also added support for SA1 Gamecube .REL files that use special compression (thanks Exant for the decompression code!). The tools should now be able to work with these files, and I will likely make a split configuration for SADX Gamecube as well. It will be less complete than SA1 or SADXPC but it will focus on the assets that are different between Gamecube and PC versions. A similar split configuration may also be made for SADX Preview eventually.

New SADXPC split – all models ripped

Last month I wrote about a new split configuration I made for SADXPC split that ripped models from sonic.exe with a file/folder structure resembling the original source files. Recently this new configuration was expanded to include models from all SADXPC binary files, including DLL and NB files. This means we now have a 100% complete rip of all models in the entire game. And not just models, but also motions and shape motions (most of which are properly mapped to their respective models), camera actions, fog data and texture lists.

I didn’t think it was possible but we were able to identify source filenames even for models in the DLLs, for which we don’t even have labels! It turns out that most models are arranged in an alphabetical order. There were some exceptions to that, which required manual editing with some guesswork, but overall it wasn’t as unrealistic as I originally thought. I’d like to thank ItsEasyActually for identifying character-related models and motions in CHRMODELS, as well as Past objects and animations in ADV03MODELS.DLL. Also, NB files are included in this. We don’t have source filenames for them, but based on the labels used in the X360 version I was able to a at least find or guess the names for all of the objects and animations.

ItsEasyActually has been working on migrating the currently known data from the old SADXPC split to this new system. If it works out, we will be able to rebase SA Tools and its project system around the new folder structure, and the new split configuration will become the main one. I’m still not sure if this means much for mods that don’t use code. At least I’d like to help make SADXLVL2 display more object models, and I also want to make some kind of wiki for the source filenames to help people find assets easier. A long-term goal is to make DC Conversion use this system, and create a more complete split for the Dreamcast version that uses a similar folder structure.

Some statistics: there are about 12500~13000 models in the entire game (approximate number because of duplicate models), about 3450 if you don’t count level pieces. The largest models (in terms of data size) are the Egg Carrier (Sky Chase and the transformation cutscene), Sonic and Tails on the Tornado, Perfect Chaos and Chaos 6. The high poly SA1 title screen model, which is one of the biggest models in the X360 version, is not present in the 2004 PC port. Some unreferenced models are also included in this split, such as an unused Adventure Field version of Amy’s Long Hammer, the SA1 DLC Christmas tree, the cowgirl, various unused Chao toys and the unused Chao flower I wrote about some time ago.

New sonic.exe split

For the last couple of months I’ve been working on a project related to SA Tools and the split tool, which I recently brought back in its original command line form, and today it’s finally ready for the public. This is a complete rip of levels, models and animations included in SADXPC’s sonic.exe that mimics the filenames and folder structure of the original source files.

This was done in multiple steps. First, Exant got me a list of source filenames from the PDB file that came with the leaked X360 prototype. The list was filtered to only include models (.NJA, .NAD and sometimes .DUP files) and animations (.NAM, DUM), and I also added levels (.c files).

Then I exported all SonicApp.exe labels using IDA and wrote a program that made lists of labels that were similar to source filenames. With some manual tweaking I was able to match all source models and animations with their labels in the database. I made the lists resemble the Dreamcast version’s file structure. For example, all Emerald Coast stuff would go in “STG01”, Station Square would be in “ADV00” etc. Both PC 2004 and the X360 prototype have most of the stuff stored in the main executable (for which we have symbols), but there’s a lot of models and animations in separate libraries too, so lists for characters and the Adventure Fields were pretty short.

Then I made it generate .INI files based on those lists and the addresses associated with the labels. Basically a bunch of split .INI files for SonicApp.exe that extracted the models and animations from it with a structure resembling the matched source assets. Now it was the time to get something similar out of the PC version.

This is the part where it got a bit more challenging. Since we didn’t have a complete split of sonic.exe, I decided to update my object scanner and make it more precise, as well as add support for motions. Motions were particularly challenging because there is no reliable way to detect the number of model parts they’re supposed to use if the model using them is unknown, which was the case with a lot of motions. But I was happy with the results since the scanner found almost all motions that I needed. Of course the models also had to be found, and to make it easier for myself I also made the scanner detect levels.

After that I had to come up with a solution to match those with the source filenames I got from the X360 version. My idea was to rely on the fact that the data should be identical between the two versions as long as it’s ripped correctly. I made some adjustments to SA Tools to make it possible to output 100% identical files from both versions as long as the model and animation data was the same, mostly by removing labels that used addresses, which were obviously different between the games. I added a bunch of undocumented command line switches to the split tool and programmed several new modes for it, including one that compared files byte by byte and made a list of file matches between two folders, and another mode that created split .INI files using lists of addresses and filename matches. This took a while but finally I was able to produce a big .INI file that contained a rough version of the file/folder structure I was looking for.

Then I split it into separate files again, and went through each file to confirm the contents. I checked for duplicates and false matches, added model part counts to all motions and assigned motions to models where possible. There were many animations that required manual editing because the models were named differently (so the tool couldn’t detect the model part count) or were missing altogether because they were supposed to be loaded from a DLL while the motion was in the EXE. For example, a lot of cutscene animations had situations like this with characters. Fixing this all was the most tedious part of the whole process, which I only finished today.

I think it was worth it. We now have a full split of all models and animations in sonic.exe (over 13000 items), including about 2720 non-level models. And, since most animations are now assigned to their models, you can preview them straight in SAMDL. Including shape motions (implemented by MainMemory earlier in the SA2 Event Viewer), which you can now also view in SAMDL.

You can try the new split configuration by downloading the latest version of SA Tools, putting sonic.exe in SATools\SADXPC\data and running splitnew.bat in the SADXPC folder.

image

You can now make your own SA1 Dreamcast DLC

I’ve added my DLC tool to the SA Tools suite, you can find it in the SA1 folder. It’s a simple command line tool that unpacks VMS files and rebuilds them from source assets and metadata. I want to make a GUI version of it someday, but for now I want to work on something else. The tool and the data should be self explanatory for the most part, and there’s also plenty of documentation in the readme file.

For those who want some juicy details, here’s the layout of SA1 DLC files:

 * VMU HEADER
 * OFFSET	SIZE		TYPE		DESCRIPTION
 * 0		16		string		DLC title
 * 10		32		string		DLC description
 * 30		16		string		Application title
 * 40		2		ushort		Number of icons
 * 42		2		ushort		Animation speed
 * 44		2		ushort		Eyecatch type (unused)
 * 46		2		ushort		CRC (unused)
 * 48		4		uint32		Size without the header
 * 4C		20		null		Reserved
 * 60		32		ushort		Icon palette, 16 colors
 * 80		512		byte		Icon graphics
 * 
 * SECTIONS HEADER (SIZE 64 BYTES) 
 * 280		4		uint32		Pointer to item layout table
 * 284		4		uint32		Item count
 * 288		4		uint32		Pointer to string table
 * 28C		4		uint32		String item count
 * 290		4		uint32		Pointer to PVM
 * 294		4		uint32		Number of PVMs (always 1)
 * 298		4		uint32		Number of textures in the PVM
 * 29C		4		uint32		Pointer to MLT
 * 2A0		4		uint32		Number of MLTs (either 0 or 1)
 * 2A4		4		uint32		Pointer to PRS
 * 2A8		4		uint32		Number of PRSes (always 1)
 * 2AC		4		uint32		Checksum
 * 2B0		16		null		Unused
 * 
 * ITEM LAYOUT TABLE HEADER (SIZE 12 BYTES)
 * 2C0		4		uint32		DLC ID (e.g. 504 in SONICADV_504)
 * 2C4		1		byte		Enable Sonic / Enable Tails
 * 2C5		1		byte		Enable Knuckles / Enable Gamma
 * 2C6		1		byte		Enable Amy / Enable Big
 * 2C7		1		byte		Unknown, probably unused
 * 2C8		4		uint32		Regional lock
 *
 * ITEM LAYOUT TABLE (ARRAY BEGINS AT 0x2CC, ITEM SIZE 30 BYTES)
 *  0		1		uint8		Level ID
 *  1		1		uint8		Act ID
 *  2		1		uint8		Scale X multiplied by 10
 *  3		1		uint8		Scale Y multiplied by 10
 *  4		1		uint8		Scale Z multiplied by 10
 *  5		1		uint8		Rotation speed X
 *  6		1		uint8		Rotation speed Y
 *  7		1		uint8		Rotation speed Z
 *  8		1		sint8		Item type (0: model, -128: sprite, -1: invisible)
 *  9		1		uint8		Texture ID
 *  A		2		ushort		Flags
 *  C		1		uint8		Object ID for collectibles or the number of objects to collect
 *  D		1		byte		Unknown
 *  E		1		uint8		Message ID to show when touching the object
 *  F		1		uint8		Trigger distance
 *  10		1		uint8		Level ID to warp or soundbank ID (8 for MLT, 15 for ADX music)
 *  11		1		uint8		Act ID to warp to or sound/music ID to play
 *  12		2		ushort		Rotation X
 *  14		2		ushort		Rotation Y
 *  16		2		ushort		Rotation Z
 *  18		2		short		Position X
 *  1A		2		short		Position Y
 *  1C		2		short		Position Z
 *
 * DLC OBJECT FLAGS
 *  BIT_0	Unknown
 *  BIT_4	Unknown
 *  BIT_8	Solid
 *  BIT_9	Play sound
 *  BIT_10	Show message
 *  BIT_11	Hide object and disable everything except collision
 *  BIT_12	Warp
 *  BIT_13	Collectible item
 *  BIT_14	Timer item
 *  BIT_15	Starts the challenge
 *
 * REGIONAL LOCK BITS
 *  -1	Disable regional lock
 *   1	Japan
 *   3	US
 *   4	Europe
 *   7	All regions