Blog

2021 plans

PkR

I wanted to post an update regarding my projects (and projects I’ve been involved in) and what will happen to them in 2021. Of course it’s hard to predict how the whole year is going to turn out, but at least I wanted to give a rough estimate of where thing are heading.

The overall direction is shifting from making mods to research and making tools, streamlining existing projects (can’t do so much these days!) as well as writing tutorials and updating/rewriting existing mods and tools to make them easier to understand. There are also projects I’m burned out on or don’t find much interest in anymore, so those will be in maintenance mode without active development.

Here are the projects I want to talk about in detail:

Dreamcast Conversion
There will likely be a Dreamcast Conversion update about as major as Update 11 was.
1) While the overall experience won’t be noticeably different, it will be reworked internally with better solutions for problems it’s trying to fix. For example, problems with transparency will be resolved in a more elegant way than they are now, and a lot of code will be rewritten for better readability for anyone interested in learning how to make mods. The new code will use labels from leaked X360 symbols wherever possible. This will be accompanied by Mod Loader updates as well, which will enable other people to use original structures and function names in addition to what’s available in the Mod Loader at the moment.
2) I’m also setting a new goal – I want to restore ALL of the object models. There are about 2720 non-level models in sonic.exe (plus several hundred in the DLLs), and DC Conversion restores about 800 of them at the moment. The goal is to get all of them.
3) There will also be some bugfixes and minor feature updates. If I figure out a good way to do it, I might add a separate config file “for nerds” that would allow to customize every single aspect of the mod.
4) Another big change will be related to file structure, as I will try to match the source file structure we managed to recreate for SADXPC this month.
5) There may be a big Chao-related update based on the work done by Exant earlier this year. It may be a separate mod, or a part of DC Conversion. We’ll see.

SA Tools
Since mid-2020 I’ve been updating various aspects of SA Tools to make them easier to use, and I want to do more of it later on. I’ve also made a few tools myself, such as the object scanner. There will probably be more tools made by me in the future. I want to make a Dreamcast Chao editor and recreate PL Tool in C# at some point, though these are only ideas at this point.
Lately I’ve been working a lot on researching and recreating the original SADX source file structure. With the help of other x-hax members, mainly ItsEasyActually, this will likely continue with relevant updates to the tools and possibly the Mod Loader. The main goal of this work is being able to extract most (ideally all) data from the game’s binaries to a set of files/folders with names resembling the original source assets. I must say though that I’m not interested in SA Tools’ project system (not in its current form at least) so this won’t be compatible with automatic/manual mod building, and figuring out how to put all this data back into the game as a non-DLL mod is a completely different story. But there are some starting points, and someone else will probably figure it out eventually.

SADX Mod Installer
I think 2020 was a very important year for SADX Mod Installer. It got a new website, a new configuration tool to replace the one that comes with SADX Steam, and a lot of language translations. I’d like to thank everyone who contributed a translation.
The installer is a project I’ve sunk a lot of time and effort into, and I think it’s solid enough now to feature freeze it. Sadly there’ve been some rare but persistent issues with Steam to 2004 conversion I’ve never been able to reproduce, and overall developing and supporting the installer has been more of a chore than an enjoyable experience. I don’t think it will have as many updates as it did in 2020. I’m still open to language contributions, but otherwise the installer will be in maintenance mode from now on.

Other projects
Unless significant progress is made in related areas in the Mod Loader, HD GUI and Sound Overhaul will be in maintenance mode. I want to continue my investigation into MLT soundbanks and sequenced sound – maybe someday we’ll be able to imitate it the way it was on the Dreamcast. Also, I wanted to make a new page on Dreamcastify that discusses some SA1- and SADX-related myths and also new information we got from the leaked X360 build, but it’s been increasingly difficult to find motivation to work on it. When I made the blog in 2017, I wanted to make a difference by letting people see how much of a letdown SADX was in comparison to the original game, and I think my attempt was fairly successful. Since then, my focus has long shifted towards research and tool development rather than trying to bring SADX issues to public attention. So, no guarantees but I do want to update the blog when I have the energy.

That’s it! 2021 will be an important year for me in real life so I don’t know how much I can commit to all these projects, but I will try to make or contribute to stuff when I can. Happy new year!

SADX Mod Installer update

PkR

I’ve updated the Steam conversion part of SADX Mod Installer with a new UI that tells you what it’s doing and reports all errors right away. If the program crashes while trying to do anything (for example, if there’s an access error because SADX is running or some files in its folder are open in another program), it will let you try again. It also saves logs for both the installer and the Steam conversion tool in SADX folder.

This update is mainly for those who have trouble getting the conversion to succeed and end up with the game missing videos and sounds. The tool should now be able to tell you what’s wrong, though you might still have to reinstall the game from scratch and run the installer again for it to work properly.

This is likely the last feature update for the mod installer. There may be language and maintenance updates in the future, but I won’t be adding any new features or mods to it.

New sonic.exe split

PkR

For the last couple of months I’ve been working on a project related to SA Tools and the split tool, which I recently brought back in its original command line form, and today it’s finally ready for the public. This is a complete rip of levels, models and animations included in SADXPC’s sonic.exe that mimics the filenames and folder structure of the original source files.

This was done in multiple steps. First, Exant got me a list of source filenames from the PDB file that came with the leaked X360 prototype. The list was filtered to only include models (.NJA, .NAD and sometimes .DUP files) and animations (.NAM, DUM), and I also added levels (.c files).

Then I exported all SonicApp.exe labels using IDA and wrote a program that made lists of labels that were similar to source filenames. With some manual tweaking I was able to match all source models and animations with their labels in the database. I made the lists resemble the Dreamcast version’s file structure. For example, all Emerald Coast stuff would go in “STG01”, Station Square would be in “ADV00” etc. Both PC 2004 and the X360 prototype have most of the stuff stored in the main executable (for which we have symbols), but there’s a lot of models and animations in separate libraries too, so lists for characters and the Adventure Fields were pretty short.

Then I made it generate .INI files based on those lists and the addresses associated with the labels. Basically a bunch of split .INI files for SonicApp.exe that extracted the models and animations from it with a structure resembling the matched source assets. Now it was the time to get something similar out of the PC version.

This is the part where it got a bit more challenging. Since we didn’t have a complete split of sonic.exe, I decided to update my object scanner and make it more precise, as well as add support for motions. Motions were particularly challenging because there is no reliable way to detect the number of model parts they’re supposed to use if the model using them is unknown, which was the case with a lot of motions. But I was happy with the results since the scanner found almost all motions that I needed. Of course the models also had to be found, and to make it easier for myself I also made the scanner detect levels.

After that I had to come up with a solution to match those with the source filenames I got from the X360 version. My idea was to rely on the fact that the data should be identical between the two versions as long as it’s ripped correctly. I made some adjustments to SA Tools to make it possible to output 100% identical files from both versions as long as the model and animation data was the same, mostly by removing labels that used addresses, which were obviously different between the games. I added a bunch of undocumented command line switches to the split tool and programmed several new modes for it, including one that compared files byte by byte and made a list of file matches between two folders, and another mode that created split .INI files using lists of addresses and filename matches. This took a while but finally I was able to produce a big .INI file that contained a rough version of the file/folder structure I was looking for.

Then I split it into separate files again, and went through each file to confirm the contents. I checked for duplicates and false matches, added model part counts to all motions and assigned motions to models where possible. There were many animations that required manual editing because the models were named differently (so the tool couldn’t detect the model part count) or were missing altogether because they were supposed to be loaded from a DLL while the motion was in the EXE. For example, a lot of cutscene animations had situations like this with characters. Fixing this all was the most tedious part of the whole process, which I only finished today.

I think it was worth it. We now have a full split of all models and animations in sonic.exe (over 13000 items), including about 2720 non-level models. And, since most animations are now assigned to their models, you can preview them straight in SAMDL. Including shape motions (implemented by MainMemory earlier in the SA2 Event Viewer), which you can now also view in SAMDL.

You can try the new split configuration by downloading the latest version of SA Tools, putting sonic.exe in SATools\SADXPC\data and running splitnew.bat in the SADXPC folder.

image

Dreamcast DLC mod release

PkR

The new version of the Dreamcast DLC mod is now out! You can use this mod to play SA1 DC downloadable content in SADX PC, as well as develop your own custom downloads that will work both in the mod and even in the Dreamcast version provided that you convert it using SA Tools first. Also, this version includes Tikal’s challenge, a custom DLC created previously for the Dreamcast version.

Get the mod here or simply update the previous version through the Mod Loader. If you get a “directory not empty” error while updating the mod, just hit cancel and it’ll work fine. I don’t know why it happens.

Here’s an excerpt from readme/release notes:
==CREDITS==
-SonicFreak94 for the code to play the SEGA/Sonic Team voice
-Darksecond for figuring out SA1 DLC and rank data encryption and posting it on ASSEMbler
-Sappharad for providing C# code to decrypt SA1 DLCs and for cracking the integrity check
-Daguar for giving me a hint on where to look for DLC item table
-Exant for making a lightweight “sitting Chao” model for the Tikal DLC
-Moopthehedgehog for the rediscovered Kadomatsu DLC
-TheLegendOfXela for high quality music and sound rips
-LastBreath for beta testing

==CHANGES FROM THE OLD DREAMCAST DLCS MOD==
-The mod has been rewritten from scratch to work more like the original game’s DLC system
-Text strings in all languages used in original DLC content have been preserved
-DLC content can now load in Adventure, Trial or Mission Mode
-Added Tikal’s challenge
-Lighting accuracy updates
-Custom codepage support: add system\dlc\codepage.ini with “FrenchCodepage=….” etc. to make French, Spanish or German strings use a codepage other than 1252
-Famitsu and QUO challenges no longer replace Station Square camera files with their counterparts from the original 1998 release of SA1
-Twinkle Circuit options for characters have been removed – you can now use the “Mercury GP” menu to go to any course with any character
-The “Alternative challenge timer GUI” option has been removed in favor of a more accurate SA1-like timer
-Seasonal DLC options have been rearranged
-Workarounds from the old mod to prevent DLC content from interfering with story progression have not been reimplemented

Alternative music in XMAS99 DLC

PkR

There are two NiGHTS-related music tracks on Sonic Adventure GD-ROM. One is called NIGHTS_A, which is the a capella cover of “Dreams Dreams”. The other is called NIGHTS_S, which is another mix of “Dreams Dreams” called “Sweet Mix in the Holy Night”. The A version is the one you hear with the Christmas 99 DLC when you touch any of the Christmas trees, the other one you don’t get to hear ingame normally but there’s a bit more to it than the song being unused.

A while back I found out that the S version is actually referenced in the Christmas 99 DLC, but the object that plays the song never loads because it has the “wrong” level ID, and the one that plays the A version gets loaded in its place because it has the “right” level ID. In the game’s internal level list Station Square has the ID of 26, while the objects playing the S version use 27 and 28. It’s pretty clear that at some point in development the game had the day, evening and night versions of Station Square use different internal IDs, even though the final game only uses 26. Even the “Easy Menu” in SADX Preview prototype refers to 27 and 28 as evening and night versions respectively.

The XMAS99 DLC places four Christmas trees around Station Square. If you look at its object placement (which you can do with the DLC Tool in SA Tools), you can see that for each of those trees there are “evening” and “night” versions that are supposed to load in levels 27 and 28. One of the parameters in those objects is the music track ID to play. The regular “day” trees use 68, which corresponds to NIGHTS_A, but some of the “evening” and “night” trees use 70, which is NIGHTS_S. For example, this one in the City Hall area:

While it is trivial to switch the IDs around and make these trees play the S version at all times, I don’t know of a way to restore the seemingly intended functionality with the trees playing different songs at different times of day. The old Dreamcast DLCs mod for SADX had this behavior hardcoded, but the new mod loads objects in a way a bit more similar to the Dreamcast version, so it doesn’t work for the same reason it doesn’t work in the original. I’ll probably just make this tree play the S version because why not?

There’s one more interesting thing about this DLC and unused NiGHTS songs in SA1. We’ve seen both music IDs 68 and 70 used, but what about 69? While it isn’t referenced in the DLC, we know from the PC version that there was supposed to be yet another NiGHTS song called NIGHTS_K, which would be using this ID. As no version of the game has this file, we can’t say for sure what was supposed to be. The Kids version of “Dreams, Dreams”? The Karaoke version? “Know Thyself!” maybe? To me, the Karaoke version makes the most sense, but we can only guess.

The updated DLCs mod for SADX PC is coming along well. I rewrote the entire mod from scratch and made it work in a way that resembles the functionality used in the original game. It’s also partially compatible with original data from SA1 DLC files provided that you convert it first using the DLC tool and replace some of the stuff that isn’t supported by the PC version, such as MLT soundbanks. Model and metadata files are fully compatible. So in the future if you decide to make a custom DLC for the Dreamcast version, you will be able to prototype it in SADX PC first using the new DLC mod, which makes object placement easier.

New Twinkle Circuit menu preview

PkR

I’m working on a complete rewrite of the Dreamcast DLC mod to practice making something that uses C++ classes. With the new version you will be able to modify original SA1 DLCs and create completely new DLC content that will work in both SADX PC and SA1 Dreamcast with a simple conversion (except custom sounds and music which aren’t compatible).

Some features of the old mod won’t make it to this version. For example, all Twinkle Circuit settings in the old mod will have to go. To compensate for it I made this little menu that will show up when you enter the special gate that appears when you enable the Samba GP DLC. The look may or may not be final, I’m still messing around with its design. I’m a bit limited in how much I can do in it.

Just… why

PkR

I’ve been wondering for a long time how SA1 draws its “Now Saving” and DLC timer strings. Apart from being italicized (and looking glitchy with letters overlapping each other) the font looks thicker than the font used in debug menus, and the 0 character looks different, and I started thinking it was a different font altogether.

Today Exant figured it out, and the answer to all this is just… Weird. It’s the same font, but there’s code that… replaces the zeroes in text with the letter O. The rest is a lot more believable: the font looks thicker because the letters are stretched horizontally, and the italics are done by offsetting the top right X coordinate of the letter’s rectangle.

I’ll be updating DC Conversion and the DLCs mod to improve the look of the font, though I also want to rework the DLCs mod from the ground up. I want to make it possible to load the data produced by the SA1 DLC tool I made for SA Tools. That way you’ll be able to convert DLCs between SA1 and SADX. There’s some fundamental incompatibility with sounds and music, which I won’t be able to resolve in a while, so it will probably be only partially compatible, but anyway I’ll save that for another post.

You can now make your own SA1 Dreamcast DLC

PkR

I’ve added my DLC tool to the SA Tools suite, you can find it in the SA1 folder. It’s a simple command line tool that unpacks VMS files and rebuilds them from source assets and metadata. I want to make a GUI version of it someday, but for now I want to work on something else. The tool and the data should be self explanatory for the most part, and there’s also plenty of documentation in the readme file.

For those who want some juicy details, here’s the layout of SA1 DLC files:

 * VMU HEADER
 * OFFSET	SIZE		TYPE		DESCRIPTION
 * 0		16		string		DLC title
 * 10		32		string		DLC description
 * 30		16		string		Application title
 * 40		2		ushort		Number of icons
 * 42		2		ushort		Animation speed
 * 44		2		ushort		Eyecatch type (unused)
 * 46		2		ushort		CRC (unused)
 * 48		4		uint32		Size without the header
 * 4C		20		null		Reserved
 * 60		32		ushort		Icon palette, 16 colors
 * 80		512		byte		Icon graphics
 * 
 * SECTIONS HEADER (SIZE 64 BYTES) 
 * 280		4		uint32		Pointer to item layout table
 * 284		4		uint32		Item count
 * 288		4		uint32		Pointer to string table
 * 28C		4		uint32		String item count
 * 290		4		uint32		Pointer to PVM
 * 294		4		uint32		Number of PVMs (always 1)
 * 298		4		uint32		Number of textures in the PVM
 * 29C		4		uint32		Pointer to MLT
 * 2A0		4		uint32		Number of MLTs (either 0 or 1)
 * 2A4		4		uint32		Pointer to PRS
 * 2A8		4		uint32		Number of PRSes (always 1)
 * 2AC		4		uint32		Checksum
 * 2B0		16		null		Unused
 * 
 * ITEM LAYOUT TABLE HEADER (SIZE 12 BYTES)
 * 2C0		4		uint32		DLC ID (e.g. 504 in SONICADV_504)
 * 2C4		1		byte		Enable Sonic / Enable Tails
 * 2C5		1		byte		Enable Knuckles / Enable Gamma
 * 2C6		1		byte		Enable Amy / Enable Big
 * 2C7		1		byte		Unknown, probably unused
 * 2C8		4		uint32		Regional lock
 *
 * ITEM LAYOUT TABLE (ARRAY BEGINS AT 0x2CC, ITEM SIZE 30 BYTES)
 *  0		1		uint8		Level ID
 *  1		1		uint8		Act ID
 *  2		1		uint8		Scale X multiplied by 10
 *  3		1		uint8		Scale Y multiplied by 10
 *  4		1		uint8		Scale Z multiplied by 10
 *  5		1		uint8		Rotation speed X
 *  6		1		uint8		Rotation speed Y
 *  7		1		uint8		Rotation speed Z
 *  8		1		sint8		Item type (0: model, -128: sprite, -1: invisible)
 *  9		1		uint8		Texture ID
 *  A		2		ushort		Flags
 *  C		1		uint8		Object ID for collectibles or the number of objects to collect
 *  D		1		byte		Unknown
 *  E		1		uint8		Message ID to show when touching the object
 *  F		1		uint8		Trigger distance
 *  10		1		uint8		Level ID to warp or soundbank ID (8 for MLT, 15 for ADX music)
 *  11		1		uint8		Act ID to warp to or sound/music ID to play
 *  12		2		ushort		Rotation X
 *  14		2		ushort		Rotation Y
 *  16		2		ushort		Rotation Z
 *  18		2		short		Position X
 *  1A		2		short		Position Y
 *  1C		2		short		Position Z
 *
 * DLC OBJECT FLAGS
 *  BIT_0	Unknown
 *  BIT_4	Unknown
 *  BIT_8	Solid
 *  BIT_9	Play sound
 *  BIT_10	Show message
 *  BIT_11	Hide object and disable everything except collision
 *  BIT_12	Warp
 *  BIT_13	Collectible item
 *  BIT_14	Timer item
 *  BIT_15	Starts the challenge
 *
 * REGIONAL LOCK BITS
 *  -1	Disable regional lock
 *   1	Japan
 *   3	US
 *   4	Europe
 *   7	All regions

My comments on Cybershell’s video on SADX downgrades

PkR

Youtuber and streamer Cybershell, who is known for his Sonic videos from a few years back, has made a comeback with a video on SADX downgrades, which you can watch here:

It’s a great video and I recommend it to anyone interested in the subject. At the same time I feel the need to highlight several very important points that weren’t mentioned in the video:

“Dreamcast Conversion” is the name I chose for my mod. It’s a total conversion (hence the name) for SADX PC that restores Dreamcast levels, special effects, object models, textures and a lot of other things, fixes several bugs in the PC version and so on, but it doesn’t include two important aspects of the original experience: Dreamcast lighting and Dreamcast characters.

I understand that “Dreamcast Conversion” sounds like an umbrella term for all projects restoring Dreamcast stuff in SADX, which can be confusing because there are Dreamcast projects for SADX made by other people. With this in mind, I feel it is incredibly unfair to the developers of those other projects, whose authorship doesn’t belong to me. So let me clear up the situation and give shout-outs to the people whose efforts should be acknowledged just as much as (if not more than!) mine:

SonicFreak94, the developer of the Lantern Engine (palette lighting) mod. I want to make it clear that, while I helped around with research and testing SA1 lighting, the actual implementation of authentic Dreamcast lighting in SADX PC is his work. It was his incredible insight and dedication that inspired me to work on Dreamcast Conversion in the first place. I’ve contributed to several of SonicFreak94’s projects, such as SADX: Fixed Edition and the Input Mod (and he did some work on Dreamcast Conversion as well), and I am very proud to have been involved in the development of his Lantern Engine to some degree, but overall my contribution to the actual implementation of it in Direct3D was minimal, and I wouldn’t have been able to restore Dreamcast lighting on my own. So the full credit for Dreamcast lighting on PC goes to SonicFreak94, not me.

ItsEasyActually, the developer of the Dreamcast Characters mod. It’s his mod that restores original character models, and it is not part of my Dreamcast Conversion mod. While I did help with the development of this mod here and there, as well as made some minor contributions to it, it is overall a completely independent work with a ton of research and effort put into it. So we need to thank ItsEasyActually for putting all that effort in restoring Dreamcast character models in SADX PC.

MainMemory and SonicFreak94, the main developers of SADX Mod Loader and SA Tools. The Mod Loader and its mod framework make it possible to develop mods for SADX PC with different levels of complexity, from simple file replacement to custom code that rewrites parts of the game’s programming. SA Tools let us work with data from both games and are extremely useful for making comparisons or converting assets between different versions of the game. Without the Mod Loader and SA Tools it would’ve been much more difficult to develop all these mods, to the point that I probably wouldn’t have started work on Dreamcast Conversion if those tools weren’t available.

So yes, while I did have some involvement in all of these projects to some extent, it is important to understand that the restoration of Dreamcast assets in SADX PC is a combined effort of many people, and much of that effort went towards reverse engineering the game’s internals and making tools long before I came to the scene to make Dreamcast Conversion.

To sum up:
1) My Dreamcast Conversion mod is one part of the Dreamcast restoration, which includes multiple mods and is a combined effort of many people, not just me;
2) SonicFreak94 and ItsEasyActually should be credited for Lantern Engine and Dreamcast Characters respectively;
3) The Dreamcast Conversion mod is built using MainMemory and SF94’s Mod Loader framework. “PkR’s Mod Loader” is actually my Mod Installer, which installs MainMemory and SF94’s Mod Loader and a few mods.

If you want to learn more about the mods and the people behind them, I recommend TheArcadeStriker’s interviews with me, SonicFreak94 and ItsEasyActually for further reading.

Sonic Adventure: “Tikal’s challenge” DLC contest!

PkR

This release is for the original Dreamcast game, not SADX PC. A PC version of the challenge will be available at a later date.

Today I’m releasing something a bit special. This is the first fully custom* Dreamcast Sonic Adventure DLC that adds a new challenge without hacking the base game. It’s built like the official SA1 DLCs, and it makes use of SA1’s internal system to add objects and challenges to the game.

Download for emulators or transfer from PC
VMI file download for Dreamcast browser

To stay true to the spirit of official SA1 DLCs I’d like to make this a contest with a “real” (though merely symbolic) prize. To participate, simply record your playthrough of this challenge and share the video. You can play it on an emulator or on a real Dreamcast. The fastest player who beats the challenge in the shortest amount of time will get a free Sonic Adventure DX Steam key from me.

The contest will run until September 23rd, 0:00 (UTC). The deadline may be extended depending on the number of entries.
Submit your results here

To get started, you need to import the file SONICADV_515.VMS or SONICADV_515.DCI to your memory card. On PC you can use a memory card editor (such as redream’s save manager). To play it on hardware you could burn the file to a CD together with Dream Explorer, or use an SD card adapter with DreamShell. If your Dreamcast is connected to the Internet, you can download it from the revived official webpage directly from Sonic Adventure!

This DLC was made using a tool that I’ve been developing during the last several days. Once the tool is released you will be able to make your own DLCs for Sonic Adventure from scratch (within the limits of the game’s DLC system of course)!

*Technically it’s not the first edit of a DLC, but it is the first DLC that adds custom objectives and makes extensive use of the DLC system rather than edits object properties. Huge thanks to Sappharad for figuring out the integrity check for the DLCs, which made this whole thing possible.

Credits:
-Darksecond for figuring out SA1 DLC and rank data encryption and posting it on ASSEMbler
-Sappharad for providing C# code to decrypt SA1 DLCs and for cracking the integrity check
-Daguar for giving me a hint on where to look for DLC item table
-Exant for making a lightweight “sitting Chao” model
-The graphics are based on Sonic Adventure (+DX) assets

Enjoy!