Sonic Adventure VMS + Chao Editor first release

Today I’m releasing a new tool based on the original SA1 DLC editor. This tool has the DLC editor integrated, but its functionality expands to cover more VMS files in SA1. Eventually I would like it to support all of the formats described here, but the highlight for today’s release is the inclusion of a new Chao editor.

Like the original DLC tool, the updated editor will be included in SA Tools, which also have a big update underway. While the SA Tools update isn’t ready for release yet, you can download the editor here:
Download (temporary link)
The Chao editor is experimental and will be updated as more information on SA1 Chao data becomes available. There is no documentation for it at the moment, although the most important data should be self explanatory.

The Chao system used in SA1 is the least researched of all Chao games. There are several Chao editors for SADX and SA2, but the only Chao editor for the Dreamcast version of SA1 was Tyro’s VMU editor that requires running on an actual Dreamcast VMU or a VMU emulator, which I found inconvenient. Besides, there have been some updates regarding SA1 Chao data (for example we now have the full SA1 Chao data struct from SADX X360 symbols), and I wanted to make use of that new knowledge to help advance SA1 Chao research. Although several aspects of it are now known better, Tyro’s notes on SA1 Chao data were one of the most important sources of information for developing the editor. I would also like to thank the Chao Island member UltimaNumber, who kindly shared the bits of information I was missing and explained how SA1 Chao data works.

The tool can import all Chao VMS data. Data produced by the Japanese version is also supported:

  • SA1 Chao Garden save file
  • Download Data (Black Market Chao)
  • Chao Adventure (created when you take a Chao out for a walk)
  • Upload Data (created when you hold Y+A as the Chao goes in the transporter)

At the moment saving is only supported for Chao Adventure and Download Data, but you can export selected Chao from the garden save file as Download Data or Chao Adventure data. There’s also an option to export files with headers used by the Japanese version of SA1.

As Chao data is rather complex and the editor is experimental, there may be some bugs, especially with saving data. Also, expect UI changes in the future.

I’d like to thank Speeps for providing a screenshot of Buddy the Chao that I used as an icon for this tool.

Sonic Adventure Black Market Chao

The first Sonic Adventure on the Dreamcast had a Black Market just like the sequel. The archived version of the original website still has the pages and CGI scripts for it, as well as VMU files, which we’re going to look at today.

I didn’t access the Black Market back when it was online, but I’ve heard that the US version never worked because of some bug. The messages from Chao Doctor all the way to January 2000 mention that the Black Market was about to launch, and the comments in the CGI script used on the website suggest it was indeed broken at the time of archival:

The Japanese version was functional though, and it was possible to download several jewel Chao. The Ruby and Emerald Chao didn’t have any emblem requirement. It looks like they were given out “for free” to celebrate the Dreamcast’s US and EU launch anniversaries. However, according to this page, the Emerald Chao was released during an event “Magical Garden: Yume no (‘Dream’) Dreamcast Festival” that took place on March 27, 1999. I can’t find any other information on the event so I’ll have to leave it there for now.

The Amethyst Chao was unlocked when you had 100 emblems. The Sapphire Chao isn’t referenced in the pages or scripts but its VMU data exists. Apparently the US version was going to add a new “NiGHTS” Chao that would also cost 100 emblems. Although the NiGHTS Chao is referred to as “very special Chao”, it’s actually a regular Fly type Chao with animal parts rather than a unique character Chao, which SA1 doesn’t support anyway.

Chao downloads are stored in 8-block VMS files, and Japanese files have a different header although the data is compatible between all versions. There’s a script on the website (likely used for Chao daycare) to generate Chao VMS files, and it alternates between Japanese and US headers, which are stored as binary data on the website. The header is set up like a VMU game header, similar to Chao Adventure. Here are all the files found in the site’s archive:

DescriptionVMI fileScreenshotEmblem requirement
Emerald ChaoCHAO_EME0
Ruby ChaoCHAO_RUB0
Amethyst ChaoCHAO_AME100
Sapphire ChaoCHAO_SUF?
NiGHTS Chao (US only?)KNIGHTS100
Download all of them here

I’ve also made Chao Download data for BUDDY, the developer Chao mentioned in my previous post. Download it here to get it in your copy of SA1 DC.

The SA1 website, including the Black Market, was restored by the DreamPipe team and is now accessible on hardware if you connect through DreamPi or with edited DNS settings. The site is located at http://sonic1.dreampipe.net/.

List of VMU filename references in Sonic Adventure

There are a lot of mysteries about Sonic Adventure’s files on the VMU, some of which we’re going to unravel today. I’d like to also use this post to list known VMU filenames in SA1 for future research.

Let’s start with the better known stuff. The main save file is called SONICADV_SYS in the original Japanese release, and SONICADV_INT in all other versions. The newer versions contain text and voice language settings which aren’t available in the original Japanese version. Normally the Japanese version doesn’t load US/EU/International save files, and vice versa. However, the game will recognize the other version’s save file if you rename it.

Now let’s talk about Chao files. The main Chao save is called SONICADV_ALF and is compatible between all regions. When you take a Chao out for a walk using the VMU-shaped transfer machine, the game creates a file called SONICADV__VM on the VMU. This file contains the VMU game Chao Adventure with the Chao’s data written directly into the game file.

The filename SONICADV__VM was also used for Chao downloads from the website. In the Japanese version of the game you could download rare Chao (Sapphire, Ruby, Emerald) from the website if your main save file had a certain number of emblems. These are discussed in a separate post.

Upload Data

There’s also a bunch of files that are created on the VMU at certain points in the game. These files were meant to be uploaded to the official website using the game’s built-in web browser. Among such files were Chao, level clear times and results of DLC challenges. They contain an HTML page with a hidden field that contains encrypted data encoded as a Base64 string. The encryption is the same between these files and DLCs.

The Japanese version had different strings for the webpage so the embedded HTML is different from later revisions, also the Japanese version uses different filenames. Here are all of them:

Filename (JP)Filename (US/EU/INT)DescriptionHow to create
SONICADV_H00SONICADV_H07Chao Upload dataHold Y+A when a Chao is going into the transporter until you hear the ring sound
SONICADV_H01SONICADV_H04World Ranking DataHold Y+A on the file select screen until you hear the ring sound
SONICADV_H02SONICADV_H05Twinkle Circuit DataCreated automatically when you finish the DLC Twinkle Circuit course
SONICADV_H03SONICADV_H06Event Result DataCreated automatically when you finish a DLC challenge

Hidden Chao on SA1 US GD-ROM

Interestingly there are files SONICADV_H00.BIN and SONICADV_H04.BIN in the root folder of SA1 US GD-ROM. These appear to be earlier versions of the data produced normally by newer revisions of the game. Apart from using the Japanese version’s naming scheme they have some differences in HTML compared to the regular files. While it isn’t compatible with either version of the game, the interesting part is the actual data stored in these files.

The Chao file contains a baby Chao called ♥BUDDY♥ that was submitted by segaeigh@concentric.net. Yep, there’s been a Chao hidden on the game’s disc all these years. Here’s what it looks like:

Stats: Swim: 19 Run: 17 Fly: 17 Power: 27 HP: 13 Max HP: 17

Eventually you’ll be able to edit Chao upload and download data, including this file, as well as the main Chao save. I just need to get to writing a Chao editor for SA Tools.

Character Voice Packs

When one of these files is placed on the VMU, the game changes the character voicing the menu screens. The files themselves don’t have any meaningful data other than an icon, a description and a single string Team Andromeda. The game seems to set the voice mode based on the filename. Again, the Japanese version looks for different filenames, so there were actually 18 unique files for voice packs! Do we even have their Japanese versions archived?

Filename (JP)Filename (US/EU/INT)Description
SONICADV_V01SONICADV_V11Sonic’s voice
SONICADV_V02SONICADV_V12Tails’ voice
SONICADV_V03SONICADV_V13Knuckles’ voice
SONICADV_V04SONICADV_V14Amy’s voice
SONICADV_V05SONICADV_V15Gamma’s voice
SONICADV_V06SONICADV_V16Big’s voice
SONICADV_V07SONICADV_V17Dr. Eggman’s voice
SONICADV_V08SONICADV_V18Tikal’s voice
SONICADV_V09SONICADV_V19Random voice

Events and Challenges

A lot has been written about SA1 DLCs, and they are now fairly well understood, up to the point that you can make your own DLCs that will work on an actual Dreamcast without hacking the base game. So here I’m going to talk about things that are a but more obscure.

One major limitation of the DLC system in SA1 is that you can only have one custom model. While I haven’t been able to get the game to load more models, I got it to load a model that isn’t stored in the DLC file by editing the pointer in the DLC file’s model binary to make it point to a different location in memory. It’s still just one model per DLC and you have to supply its textures separately but at least you can load stuff outside of the DLC file if you really want to. Maybe it’s possible to add more pointers and make the game load them somehow? I’ll look into that sometime soon.

https://cdn.discordapp.com/attachments/446023672147345420/753894751509413910/unknown.png

The filenames for the DLCs are SONICADV_aaa, where aaa is a number from 000 to 999. IDs from 000 to 200 are reserved for DLCs that only have text in Japanese. Early events released in Japan (Famitsu challenge, QUO challenge etc.) only contain Japanese message strings, and their string table is smaller compared to later events that had text in other languages. However, these old events still work with newer versions of SA1 with Japanese text displayed correctly regardless of the game’s language setting. The game doesn’t load strings for other languages if the DLC ID is lower than 200. This is why there is a gap between Japanese and international event filenames: the last Japan-exclusive event was called SONICADV_003 and the first international event was called SONICADV_501 (though SONICADV_200 would’ve worked too).

The Weird Stuff

The game’s main binary has several filename strings that suggests some interesting functionality. Earlier this year we learned from Exant’s fascinating discovery that we can still activate the game’s built-in lighting editor that looks like this:

Image

Apparently this editor can load lighting files from the VMU, and that functionality seems to be still in the game, we just need to learn how to activate it. The palette file is supposed to be called SONICADV_P00, and the SL file is SONICADV_S00 (in the Autodemo they’re just PL_ and SL_). It’s possible that numbers other than 00 were also used. There is a function loading these files at 0x8C011D46 (US 1.005). It also seems like LANTERN_PALETTE and LANTERN_SOURCE were VMS titles, while LANTERN/パレット情報 and LANTERN/環境情報 were VMS file descriptions. I’ll experiment with custom VMS files to see if the game picks them up.

There are also some strings suggesting there was VMU to PC transfer at some point. The filenames used in that transfer were SONICADV.VMS and SONICADV.DL0. Doesn’t look like it’s used, but who knows, maybe we’ll recover it eventually.

Sonic Adventure patches (Better 60 FPS + drift fix)

If you’re interested in hacking Sonic Adventure, you’ve probably seen 60 FPS and widescreen codes for the Dreamcast version floating around. The problem with 60 FPS codes for SA1 is that they make cutscenes run at double speed. Here’s my understanding of why it happens and a solution for version 1.005 (US, animated title screen).

The game uses two values to control framerate and speed: the frame increment (8C754EBC) and the multiplier (8C754EC4) used in physics and other things. The frame increment controls the actual framerate (1 for 60 FPS, 2 for 30 FPS), and the multiplier speeds up or slows down the physics to match it (1 is original speed). The Dreamcast version runs at 30 FPS everywhere except menus and Twinkle Circuit. In 30 FPS mode the frame increment is set to 2, and the multiplier is also set to 2 to speed up the physics to match the reduced framerate. In menus and Twinkle Circuit both values are set to 1, so the game runs at its original speed at full framerate.

This would be all fine, but the problem is the cutscenes. They are programmed in double speed, so the game sets the frame increment to 2 and leaves the multiplier at 1 to get them to work at half speed. This is also the real reason SADX cutscenes are still capped at 30 FPS – updating them would require fixing the timings in all functions used by cutscenes to work properly at 60 FPS, which is a lot more work than simply unlocking the 60 FPS mode in levels like the developers did.

The problem with existing codes is that they don’t account for the above and simply force both values to 1, which causes cutscenes to run at double speed. To fix this, instead of changing the values directly we can do the same thing SADX developers did, and change them depending on what happens in the game. Like in SADX, there is a function that is called to set the required frame increment and multiplier. All we have to do is patch the arguments that are passed for this function when the game loads levels, and leave cutscenes alone. Here are the addresses for version 1.005 (US, animated title screen). Just change the bytes at these addresses from 2 to 1, and the game will set framerate modes like SADX. If you use Demul with Cheat Engine, replace the starting 8C with 2C.

8C04956E
8C04E662
8C054736
8C082AAE
8C09A1D2

Here’s also a code that you can add as a custom cheat for redream. It’s a regular CodeBreaker code so you could use it on other emulators or on real hardware.

cheat=0204956E 420BE501 0204E662 4F22E501 02054736 430BE501 02082AAE 420BE501 0209A1D2 D252E501

There are still two problems remaining with the 60 FPS code:

  • The hardware cannot handle the game at 60 FPS most of the time (sometimes it struggles even at 30 FPS), so you will get a slow motion effect like in older SA1 preview videos. Unfortunately this also affects emulators – for example, at the moment the game does not reach full framerate in redream. There was a similar problem with Sonic Colors, which the Wii can’t handle at 60 FPS, but that was resolved with Dolphin’s overclock feature. Maybe we’ll get that in Dreamcast emulators someday?
    UPDATE: It seems to be perfect on flycast, no slowdown at all.
  • There are glitches that happen when the game runs at 60 FPS. It’s possible that some things in the game were added after the decision to switch to 30 FPS was made, and perhaps they were only tested at 30 FPS. A lot of such glitches appear in SADX. For example, the Leon badnik almost never attacks the player and disappears too fast. There are visual glitches too, for example Tails’ tails are too fast, and the background in the tornado section in Windy Valley loses the multi-layered effect for one layer. A lot of these glitches are fixed in the Dreamcast Conversion mod for the PC version, and it’s also possible to fix them in the Dreamcast version if we really want it. Interestingly some issues that affected DX at 60 FPS don’t affect SA1 at 60 FPS – for example, the Egg Hornet engine sound glitch.

There’s another issue with SA1 on emulators related to not being able to run in a straight line with the keyboard and some controllers. You can fix this by disabling analog interpolation by writing 00 00 00 00 to 8C051278 and 8C051430. CodeBreaker code:

02051278 00000000 02051430 00000000

It’s not an ideal solution but that will do for now. A better fix may show up very soon.

Sonic Adventure DLC tool: GUI version released

I’ve updated the SA Tools with a new version of the SA1 DLC tool that lets you create and edit VMS files for SA1 on the Dreamcast.

Some documentation is available here.

Apart from the GUI making it easier to edit things, this version also fixes several issues with the original tool that made it produce incorrectly aligned VMS files. Now it should be possible to modify and rebuild any of the official DLCs, or create completely new DLCs from scratch.

The tool ended up being a bit more complex than I originally anticipated, and testing it was a challenge. I’d appreciate it if you try it out and let me know on x-hax if you run into any issues. Have fun!

New SA1 Dreamcast split

Following the complete SADXPC split, here’s my attempt at a near-full level/model/animation rip of SA1 Dreamcast. It includes all levels, almost all object models, the majority of motions and some data from the older split such as level object lists for most levels. When the SA Tools update is complete, it should be possible to copy this over the SADXPC split (except the object lists) and be able to load SA1 levels and objects in SADXLVL2.

This split is less complete than the SADXPC one. Here’s a list of things missing from it:
-Individual landtable items are not ripped as separate files.
-Various miscellaneous data, such as texlists, dialogue text etc. are not ripped.
-Some levels are missing object lists.
-EV files are not ripped (maybe I’ll get to that someday).
-All shape motions are missing.
-Some other motions may be missing.
-Some object models may be missing (shouldn’t be a lot).
-We don’t have source filenames for SA1 Chao assets so I created the file structure from scratch.
-Some Chao animal parts may be missing or have wrong filenames.
-There is some minor inconsistency with character upgrades which have Adventure Field versions. Some of those are placed in the respective Adventure Field objects folder rather than the character’s folder.
-Some models and animations don’t have proper labels because I couldn’t match them with SADX assets.
-Some character animations may have wrong labels.

Other than the above, it’s a complete rip of levels, models and animations that follows the structure of the new SADXPC split. To try it, extract SA1 1.005 (US, animated title screen) ISO to SA Tools\SA1\data and run SA Tools\SA1\splitSA1.bat. The output will be in SA Tools\SA1\output.

A few days ago I also added support for SA1 Gamecube .REL files that use special compression (thanks Exant for the decompression code!). The tools should now be able to work with these files, and I will likely make a split configuration for SADX Gamecube as well. It will be less complete than SA1 or SADXPC but it will focus on the assets that are different between Gamecube and PC versions. A similar split configuration may also be made for SADX Preview eventually.

New SADXPC split – all models ripped

Last month I wrote about a new split configuration I made for SADXPC split that ripped models from sonic.exe with a file/folder structure resembling the original source files. Recently this new configuration was expanded to include models from all SADXPC binary files, including DLL and NB files. This means we now have a 100% complete rip of all models in the entire game. And not just models, but also motions and shape motions (most of which are properly mapped to their respective models), camera actions, fog data and texture lists.

I didn’t think it was possible but we were able to identify source filenames even for models in the DLLs, for which we don’t even have labels! It turns out that most models are arranged in an alphabetical order. There were some exceptions to that, which required manual editing with some guesswork, but overall it wasn’t as unrealistic as I originally thought. I’d like to thank ItsEasyActually for identifying character-related models and motions in CHRMODELS, as well as Past objects and animations in ADV03MODELS.DLL. Also, NB files are included in this. We don’t have source filenames for them, but based on the labels used in the X360 version I was able to a at least find or guess the names for all of the objects and animations.

ItsEasyActually has been working on migrating the currently known data from the old SADXPC split to this new system. If it works out, we will be able to rebase SA Tools and its project system around the new folder structure, and the new split configuration will become the main one. I’m still not sure if this means much for mods that don’t use code. At least I’d like to help make SADXLVL2 display more object models, and I also want to make some kind of wiki for the source filenames to help people find assets easier. A long-term goal is to make DC Conversion use this system, and create a more complete split for the Dreamcast version that uses a similar folder structure.

Some statistics: there are about 12500~13000 models in the entire game (approximate number because of duplicate models), about 3450 if you don’t count level pieces. The largest models (in terms of data size) are the Egg Carrier (Sky Chase and the transformation cutscene), Sonic and Tails on the Tornado, Perfect Chaos and Chaos 6. The high poly SA1 title screen model, which is one of the biggest models in the X360 version, is not present in the 2004 PC port. Some unreferenced models are also included in this split, such as an unused Adventure Field version of Amy’s Long Hammer, the SA1 DLC Christmas tree, the cowgirl, various unused Chao toys and the unused Chao flower I wrote about some time ago.

2021 plans

I wanted to post an update regarding my projects (and projects I’ve been involved in) and what will happen to them in 2021. Of course it’s hard to predict how the whole year is going to turn out, but at least I wanted to give a rough estimate of where thing are heading.

The overall direction is shifting from making mods to research and making tools, streamlining existing projects (can’t do so much these days!) as well as writing tutorials and updating/rewriting existing mods and tools to make them easier to understand. There are also projects I’m burned out on or don’t find much interest in anymore, so those will be in maintenance mode without active development.

Here are the projects I want to talk about in detail:

Dreamcast Conversion
There will likely be a Dreamcast Conversion update about as major as Update 11 was.
1) While the overall experience won’t be noticeably different, it will be reworked internally with better solutions for problems it’s trying to fix. For example, problems with transparency will be resolved in a more elegant way than they are now, and a lot of code will be rewritten for better readability for anyone interested in learning how to make mods. The new code will use labels from leaked X360 symbols wherever possible. This will be accompanied by Mod Loader updates as well, which will enable other people to use original structures and function names in addition to what’s available in the Mod Loader at the moment.
2) I’m also setting a new goal – I want to restore ALL of the object models. There are about 2720 non-level models in sonic.exe (plus several hundred in the DLLs), and DC Conversion restores about 800 of them at the moment. The goal is to get all of them.
3) There will also be some bugfixes and minor feature updates. If I figure out a good way to do it, I might add a separate config file “for nerds” that would allow to customize every single aspect of the mod.
4) Another big change will be related to file structure, as I will try to match the source file structure we managed to recreate for SADXPC this month.
5) There may be a big Chao-related update based on the work done by Exant earlier this year. It may be a separate mod, or a part of DC Conversion. We’ll see.

SA Tools
Since mid-2020 I’ve been updating various aspects of SA Tools to make them easier to use, and I want to do more of it later on. I’ve also made a few tools myself, such as the object scanner. There will probably be more tools made by me in the future. I want to make a Dreamcast Chao editor and recreate PL Tool in C# at some point, though these are only ideas at this point.
Lately I’ve been working a lot on researching and recreating the original SADX source file structure. With the help of other x-hax members, mainly ItsEasyActually, this will likely continue with relevant updates to the tools and possibly the Mod Loader. The main goal of this work is being able to extract most (ideally all) data from the game’s binaries to a set of files/folders with names resembling the original source assets. I must say though that I’m not interested in SA Tools’ project system (not in its current form at least) so this won’t be compatible with automatic/manual mod building, and figuring out how to put all this data back into the game as a non-DLL mod is a completely different story. But there are some starting points, and someone else will probably figure it out eventually.

SADX Mod Installer
I think 2020 was a very important year for SADX Mod Installer. It got a new website, a new configuration tool to replace the one that comes with SADX Steam, and a lot of language translations. I’d like to thank everyone who contributed a translation.
The installer is a project I’ve sunk a lot of time and effort into, and I think it’s solid enough now to feature freeze it. Sadly there’ve been some rare but persistent issues with Steam to 2004 conversion I’ve never been able to reproduce, and overall developing and supporting the installer has been more of a chore than an enjoyable experience. I don’t think it will have as many updates as it did in 2020. I’m still open to language contributions, but otherwise the installer will be in maintenance mode from now on.

Other projects
Unless significant progress is made in related areas in the Mod Loader, HD GUI and Sound Overhaul will be in maintenance mode. I want to continue my investigation into MLT soundbanks and sequenced sound – maybe someday we’ll be able to imitate it the way it was on the Dreamcast. Also, I wanted to make a new page on Dreamcastify that discusses some SA1- and SADX-related myths and also new information we got from the leaked X360 build, but it’s been increasingly difficult to find motivation to work on it. When I made the blog in 2017, I wanted to make a difference by letting people see how much of a letdown SADX was in comparison to the original game, and I think my attempt was fairly successful. Since then, my focus has long shifted towards research and tool development rather than trying to bring SADX issues to public attention. So, no guarantees but I do want to update the blog when I have the energy.

That’s it! 2021 will be an important year for me in real life so I don’t know how much I can commit to all these projects, but I will try to make or contribute to stuff when I can. Happy new year!

SADX Mod Installer update

I’ve updated the Steam conversion part of SADX Mod Installer with a new UI that tells you what it’s doing and reports all errors right away. If the program crashes while trying to do anything (for example, if there’s an access error because SADX is running or some files in its folder are open in another program), it will let you try again. It also saves logs for both the installer and the Steam conversion tool in SADX folder.

This update is mainly for those who have trouble getting the conversion to succeed and end up with the game missing videos and sounds. The tool should now be able to tell you what’s wrong, though you might still have to reinstall the game from scratch and run the installer again for it to work properly.

This is likely the last feature update for the mod installer. There may be language and maintenance updates in the future, but I won’t be adding any new features or mods to it.