DreamShell and Region change

[sundance]

I just acquired an SD Card reader on Serial port for my Dreamcast.
This is an unmodified PAL version of the Dreamcast, which works very well.
The external SD Card reader was selected because it doesn't require any internal modification.
That being said, I know that it comes with limitations and performance compared to more invasive gd-ide and similar mods.

The SD-Card can be used with DreamShell.
I actually use RetroDream, which embeds DreamShell, so I can easily switch between the 2.

This is my first experience with the software. While selecting and playing a game is very simple, setting up the system to get there is much less straightforward, if not poorly documented. I could quickly make it run Star Wars Racers, which was a nice beginning, but afterwards, making other titles work proved more challenging.
Fortunately, I then found an excel sheet, likely maintained by @DragonCity, which is a gold mine of information on how to setup games with DreamShell, and it helped.

But it doesn't solve everything.
Case in point : Ikaruga, on Dreamcast PAL.
Following the recommendations of the excel document is not enough : the screen remains black, the SD Card reader is inactive.

I then realized that Ikaruga is Japan only, while my Dreamcast is PAL, hence a different region.
Fortunately, DreamShell is said to provide a region changer.
I made a quick attempt, but it did not work, but maybe it's just me which doesn't know how to make it work.

It's unclear to me how it's supposed to work, on a pristine unmodified Dreamcast.
I mean, does this functionality requires to modify the BIOS to pretend being a Japan console ? And aren't original BIOS non-programmable ?

Any place where there is some information on the DreamShell region changer ?
More generally, anyone having successfully run Ikaruga on a PAL Dreamcast ? (in case there are other ways than DreamShell region changer, such as, for example, patching the game ?)

[pcwzrd13]

When booting games from DreamShell, region is ignored, so that's not your issue. There are many games that don't work with the serial SD reader. Ikaruga might be one of them (it's on an old compatibility list saying it doesn't work) but I haven't personally tested it. That being said, make sure you optimize your GDI files for the best compatibility using opimize_gdi.bat which is included in the DreamShell ISO Make Pack. Just copy that file along with the "sys" folder, and the GDI to another folder on your PC and run the .bat file.

And to answer your question about the Region Changer (even though it's not relevant), yes, you need a modded Dreamcast to use it.

[sundance]

Thanks @pcwzrd13, that's very helpful.

> When booting games from DreamShell, region is ignored

Very interesting.
I wonder what's the point of the region changer then ?
Load native GDRom from other regions ?


> it's on an old compatibility list saying it doesn't work

Any way to reach this compatibility list ?
The lists I found are more gdide/gdemu related,
it tends to be more difficult to find information targeting the serial sdcard reader.


> make sure you optimize your GDI files for the best compatibility using opimize_gdi.bat

My current copy of Ikagura is indeed a gdi.
It references 2 *.bin files and 1 *.raw file.

Not sure what the optimizer can do about that, but let's try.
I run Linux, so this *.bat utility is not very convenient. I hope it can be emulated from Linux.

[sundance]

Running the gdi optimizer under Linux wasn't successful.
Fortunately, for such case, I may access a derelic Windows system and do there what I can't on Linux. Not my preference, but it seemed worth it.

Regarding the gdi optimizer :
the only thing I noticed is that it translated the *.bin files into *.iso files. And that's basically it.
I can see 2 benefits to this operation :
1) .iso files are smaller. The saving is > 10% for .bin files, so it's not negligible and welcomed.
I presume the savings come from the .iso files only storing content, while .bin likely store both content and error correction codes, which are indeed not necessary on an SDcard or flash.
2.a) because files are smaller, I would expect this translate into an equivalent bandwidth boost, due to no longer needing to transport the redundant checksum
2.b) there might be a cpu saving associated too. I'm not sure how it works exactly, but it's not clear if it's enough to just discard the checksum. Maybe it's actually required to run the error correction algorithm to regenerate data, in which case, the cpu is busy doing this work at each load. *.iso files save that load.

I applied the optimizer to Ikaruga, but it did not help make it run.

Then, I noticed that my copy of Star Wars Racer also uses *.bin files, and could benefit from the same optimization. At a minimum, it helps save > 100 MB.
But while in game, I noticed other small improvements too :
- The video where _almost_ good before, but there was easily some stuttering. Now, it's much smoother. It's not perfect, there are still stuttering here and there, especially during large movements, but it's perceptibly better, cutting the amount of stuttering by more than half.
- In game, there are a few loads here and there during the race. I would notice it because the game would drop a few frames when that happens (quite often, every few seconds). Now, it's no longer noticeable, the race remains completely smooth from start to finish.

This is a very nice improvement. It's true that, with limited bandwidth of the serial port, every little thing help, so this optimizer made a welcome job.

By the way, I noticed that my copy of Propeller Arena uses the *.cdi format.
It works, but similarly, would it benefit from getting "optimized" into a different format, such as .gdi ?

As for Ikaruga, well, it's disappointing, but indeed, this trick is not enough.
I don't think that the issue is performance oriented.
There must be something else, more important, that prevent the game from running at all.

I noticed that the game is mentioned as "supporting DMA", which I read as letting the HDD load data into memory without passing through the cpu, which is helpful I suppose. But I also suppose this is not possible when using the serial port. Now, there is a difference between being helpful, and being compulsory. I wonder if it could have a role in this outcome.

Anyway, any kind of documentation about the different configuration options available on DreamShell / RetroDream would be helpful. I haven't found any so far, and this part of the system remains mysterious.

[megavolt85]

1) in new versions of the loader SD support is discontinued
2) dragoncity compatibility list is only valid for IDE bus devices
3) optimizer for linux can be assembled from source code

[sundance]

> optimizer for linux can be assembled from source code

Thanks for the link.
I could compile and run the optimizer on Linux.
And it seems to work : it was tested on a copy of Coaster Works, converting the *.bin into *.iso files, and the resulting gdi works great on redream.

Just for my understanding, what is the optimizer doing ?
Is it limited to transform *.bin files into *.iso (and I guess editing the corresponding *.gdi file), or is there more to it ?

The repository also contains another source, called "afs tools".
Unfortunately, there is no doc, so it's unclear to me what it's supposed to be useful for.

> in new versions of the loader SD support is discontinued

I must be lucky, as my own version of retrodream/dreamshell seems to be working fine with a SDcard reader on the serial port. So far, I successfully tested Propeller Arena, GigaWings 2 and Star Wars Racer.
Of course, as you already know, the compatibility is more limited, and I could not get Ikaruga running, for example.
But all in all, it's not bad.

_edit_ : I noticed in the log that the loader is said to be v0.6.0.
I think I read somewhere that latest version of the loader is v0.7, though looking at the source code, I rather find v0.6.7.
So I presume my copy of RetroDream is intentionally packaged with an old version of isoldr to preserve SD Card support.

[megavolt85]

Just for my understanding, what is the optimizer doing ?

remove don't used ECC/EDC
loader needs to read less data, increasing download speed

The repository also contains another source, called "afs tools".

is software for repack AFS game archive
need if you translate game

HDD load game very fast and have better compatibility
all WINCE games don't work from SD

[sundance]

Are there ways to contact the DreamShell developers ?

I noticed that the latest version of isoldr seems to support compressed iso (.cso), and that would be a nice way to improve the speed of the serial interface.
For example, Gigawing 2 is compressed to 1/3 of its original size, so that could be exploited to generate 3x faster virtual speed. As far as I know, 3x would be enough to make the serial interface competitive with the GDrom reader.

However, if only the latest version of isoldr supports compressed iso, but at the same time it doesn't support serial interface, then both features don't combine.

Of course, it's not the end of the story, it's also necessary to decompress data afterwards, which costs cpu. But some algorithms can be surprisingly light. LZ4 for example enjoys a reputation of being almost as fast as memcpy(). I presume it's not supported, but I noticed LZO in the source code, so that would be the next best thing. zlib would compress even more, but I would then start to worry about decompression time. Anyway, worth testing.

I'm a C developer, so I could help for the code.
But the problem is not the code, it's first and foremost the environment. Just setting up a tool chain able to compile objects in the appropriate format is a serious undertaking, and scarcity of documentation make this exercise very difficult. I guess some ready-to-use VM or container could help, but do they even exist ...
And then, getting patch accepted, proper testing methodology, etc. All this requires good social interactions. So that's where it should begin.

_edit_ : seeing that megavolt85 has accepted patches from cpasjuste, I presume you are part of the developers team ?

_edit 2_ : found sdk/toolchain/ubuntu_build.sh, that might help to setup environment

_edit 3_ : found compress_iso.bat within iso_make, which converts *.iso images into compress *.cso ones, using LZO algorithm, confirming that this is a supported approach. Although it is still not clear to me what are the limitations (is it compatible with .gdi header files ? is it compatible with serial port ? Does it require a minimum version of isoldr ? )

_edit 4_ : made a few tests, unfortunately I couldn't make lzo-compressed .cso images load from my SDcard. Maybe it's unsupported by my isoldr version (v0.6.0), or maybe it's unsupported for SDcard, no way to know...

[megavolt85]

isoloader v0.6.0 have support CSO for SD
for create cso image use ISO Make pack

In order to inflate CSO it is necessary to spend processor time, not in all games CSO increases speed, sometimes the opposite reduces

CSO source code

[sundance]

Thanks @megavolt85,
this answer seems to confirm that *.cso is a supported DreamShell feature, which is great !

Indeed, there is a trade-off to select in term of processing power, and that's the reason why LZO was suggested, as it's supposed to be much lighter on cpu resources.

Anyway, the issue currently is unfortunately much more blunt : after trying both the cso image compressor from ISO Make pack, on Windows, and compiling the ciso source code to run it on Linux,
the generated *.cso file don't load at all, nothing starts, in either cases. Depending on the exact scenarios (I made a ton of tests with minor variants), either the ISO-Loader screen freezes and the console must be manually rebooted, or the loader tries something and fails, showing a lot of error messages, or the loader doesn't even attempt to load the file (no LED activity on the serial port).
I even updated the isoldr driver to v0.6.7, to no effect.

The corresponding non-compressed *.iso file was checked first, and it works great (for this test, an "easy" ISO was selected).

So, since *.cso files don't work at all, either they are incorrectly generated, or there is something in my copy of DreamShell which is buggy with regards to this format. Also, is it confirmed that the .cso compressed format is supported on SDCard over Serial Port, not just on IDE/ATA port ?