hatariB new Libretro core

[rainwarrior]

I had contributed some maintenance to the older libretro/hatari core in the past, but for various reasons I thought we should have a fresh start. I've been adapting Hatari 2.4.1 as a Libretro core for RetroArch etc. and at this point it does everything I wanted it to do. This is brand new so it's definitely in need of some testing. If anyone is interested, please take a look:

https://github.com/bbbradsmith/hatariB/

The short description is that it's Hatari with all of its user interface removed and reconnected to the Libretro interface instead. Some important features:

• Fully configurable gamepad, every button can be remapped to any key/jostick/etc.
• On-screen keyboard, so you can do everything with a gamepad
• Default free TOS (EmuTOS), so you can still run software if you don't have TOS ROMs set up in your system folder.
• Saves go to a copy of the disk image in the saves folder, no overwriting your originals
• Savestates are a seamless, frame-perfect restore so run-ahead and netplay are functional, almost*.
• Hard disk support.
• MIDI. Libretro has an experimental MIDI feature now, so I hooked it up. Tried out some MT32 games with it.
• IPF. The nightly build has this feature now, if you can provide the capsimg.dll.

This is not an official RetroArch core, and I don't have the resources to build on all the targets that Libretro has, but maybe if it goes well it can become that in the future. For now there's builds for:

• Windows 64-bit
• Windows 32-bit
• Ubuntu Linux
• MacOS

* The netplay and run-ahead features do work. However, the savestate restore has really high CPU load, so in practice my PC can't quite keep up with it. There is a slight stutter when the remote player moves, for me, but at least no frames are lost and there is no divergence. I'd like to try to optimize the savestate restore to be more lightweight in the future.

[troed]

Oh wow this is awesome! I'll check it on Retropie. Thanks for your work!

[rainwarrior]

Ah, if that's Retropie on an Ubuntu PC the provided build should hopefully work. If for Raspberry Pi, though, I don't have a build for this yet. I'm sure the code will eventually be viable for ARM, but I have not attempted it.

[troed]

Well I have now (RPi4 Retropie). Compilation worked out of the box. Installed the existing Hatari optional package, rewrote the cfg to the .so format and to use the one I built and ... it looks to work just fine. Will playtest more tomorrow, just brought up EmuTOS, onscreen keyboard and joy-mouse movement etc now.

Huge thanks - this is well on its way to a proper libretro core

[rainwarrior]

Ah, that's great to hear! I tried to keep the make/cmake as plain and generic as possible, but I never know what I'll discover when trying a new target. (MacOS especially had a lot of special pains.)

[troed]

Yeah there's something about the library include path in RetroPie I'm struggling with. I've failed getting hatariB to find capsimg.so regardless of if I add it to ldconfig or if I put it in the RetroArch system path (which is ~/RetroPie/BIOS/). So far the only way that works is to add something like LD_LIBRARY_PATH="/usr/local/lib/capsimage" to the launch command. I'm sure it's just something I'm missing but I don't know what atm.

Just posting it here in case someone else tries to get it going.

[npomarede]

I had contributed some maintenance to the older libretro/hatari core in the past, but for various reasons I thought we should have a fresh start. I've been adapting Hatari 2.4.1 as a Libretro core for RetroArch etc. and at this point it does everything I wanted it to do. This is brand new so it's definitely in need of some testing. If anyone is interested, please take a look:

https://github.com/bbbradsmith/hatariB/

Hi
thanks for your work ; some retroarch users are sometimes complaining that Hatari version was too old and not updated, so it's a good thing if you can improve the situation.
If all goes well when you complete more of your work, maybe we can merge some/all parts into Hatari later to ease maintenance or to make patching easier ?

Nicolas

[troed]

Nicolas - just to highlight that this is much more than just a version refresh A libretro core is usable without needing to connect mouse and keyboard, and is fully integrated in the rest of RetroArch with regards to options etc. Current Hatari for RetroArch is an optional package that needs such things connected while this version is "plug'n'play" and anyone who already has a RetroArch installation can just start using it with a joypad etc.

There's quite a few ifdefs and changes (dynamic loading of capsimage instead of compile time for example) but I've looked through them and I think it would be great if this could be upstreamed without adding too much ifdef-hell.

[npomarede]

Nicolas - just to highlight that this is much more than just a version refresh A libretro core is usable without needing to connect mouse and keyboard, and is fully integrated in the rest of RetroArch with regards to options etc. Current Hatari for RetroArch is an optional package that needs such things connected while this version is "plug'n'play" and anyone who already has a RetroArch installation can just start using it with a joypad etc.

I understand this, don't want to lower the work made here
And as many users are just interested in playing STF/STE games, retroarch have some common features to all cores (gfx filters, UI, network play,save state...) that are quite convenient.

There's quite a few ifdefs and changes (dynamic loading of capsimage instead of compile time for example) but I've looked through them and I think it would be great if this could be upstreamed without adding too much ifdef-hell.

That would be the idea, adding ifdef or merging some concept / refactoring some code if this can be made more generic for both "normal" Hatari and retroarch version

Nicolas

[rainwarrior]

I'm not sure that I'd suggest making upstream changes based on what was needed for a core. The vast majority of what changes is just disabling something that is never used (e.g. dialogs), or replacing some point of user interaction (e.g. replacing SDL_PollEvent with a provided event queue). If I found a bug or some general improvement, I've already submitted those as PR via Hatari's github.

I made every change within an #ifdef so that I can compare against the code was replaced, which I would hope facilitates future merges for me. The overall changes look something like this:

• Cut the main execution loop in to beginning/middle/end, so that I can run the "middle" part on demand to produce the next frame.
• Disable dialogs, shortcuts, etc.
• Remove or replace almost all use of SDL. Basically all that remains, besides the type definitions, are SDL_MapRGB pixel colour conversions, and the font and box drawing used by sdl-gui for the statusbar, which I reused for the core's on-screen keyboard.
• Replace all direct use of filesystem (fopen, fseek, fseeko, ftell, stat, etc.) with an equivalent "virtual" version.
• Some things that load directly from a file should instead load from a memory buffer (e.g. zip file).

So, where there might be future updates, it should mostly just re-applying replacements, I would think. It would have been "easier" if all these points of interface like fopen etc. were going through a central abstraction within Hatari, but it was generally simple to just replace them one by one.

The ability to dynamically load capsimg was mentioned... it doesn't really change anything else about how its used, there's just a block of added code to fetch the function pointers from the DLL/SO when needed. I don't think the implicit linking the stand-alone version does is a problem for it, but for a Libretro core it appeared to be necessary to do it optionally at runtime.

I did notice that the CPU system's tracing stuff seemed to have its own separate trace output instead of using the LOG_TRACE_PRINT macro. I replaced a couple of those just so I could get a CPU trace into my log redirection, but that seemed like something that should be used universally. Maybe that's something I'd suggest doing upstream, but I didn't actually do the complete version of that which I could submit.

As for the old libretro/hatari core, this is not a continuation of that, and there's no way it could be merged with it. I think it would be fine for that to remain as a separate alternative if others need it. (I don't know how many are using it, and if there's reasons like increased CPU demands that would make them not want to switch).

[Eero Tamminen]

I did notice that the CPU system's tracing stuff seemed to have its own separate trace output instead of using the LOG_TRACE_PRINT macro. I replaced a couple of those just so I could get a CPU trace into my log redirection, but that seemed like something that should be used universally. Maybe that's something I'd suggest doing upstream, but I didn't actually do the complete version of that which I could submit.

CPU core tracing code was moved to a single place in Hatari Git version:
https://git.tuxfamily.org/hatari/hatari ... a68556c643

Reason why code changed in above commit looks different from v2.4.1 code, is this earlier commit:
https://git.tuxfamily.org/hatari/hatari ... 92200ed60d

[rainwarrior]

CPU core tracing code was moved to a single place in Hatari Git version

Ah, well it's good to see it was already done.

Well I have now (RPi4 Retropie). Compilation worked out of the box. Installed the existing Hatari optional package, rewrote the cfg to the .so format and to use the one I built and ... it looks to work just fine. Will playtest more tomorrow, just brought up EmuTOS, onscreen keyboard and joy-mouse movement etc now.

I figured out how to set up a Raspberry Pi build cross-build through the Github Actions. I only have an RPi2 to test with, and while it does run, it's unfortunately too low-powered to keep up with this core. So... partial success. Presumably the same build should work on later RPi where it might run at OK speed, as long as they have 32-bit support available. I set up an aarch64 build too... though I have no way to test that currently.

[rainwarrior]

Since I had to go over them all anyway, would you be interested in a PR that replaces all direct filesystem access (fopen, stat, etc.) with an abstraction interface? (e.g ftell and ftello become "File_ftell"). Could probably just go in file.h/file.c.

I don't know if that would benefit the stand-alone version, which can probably use fopen forever, but for my core, that would basically eliminate 100 or so points of replacement across the code, for whenever the next release is ready to merge.

[Eero Tamminen]

Since I had to go over them all anyway, would you be interested in a PR that replaces all direct filesystem access (fopen, stat, etc.) with an abstraction interface? (e.g ftell and ftello become "File_ftell"). Could probably just go in file.h/file.c.

I don't know if that would benefit the stand-alone version, which can probably use fopen forever, but for my core, that would basically eliminate 100 or so points of replacement across the code, for whenever the next release is ready to merge.

I think that at least your own patch would look better with such a wrapper, than adding ifdefs to all those places.

For Hatari upstreaming, tthose wrappers could be just in "file.h" header:

Code: Select all

/* wrappers which usage helps things that embed Hatari, e.g. libretro */
#define File_ftell ftell
#define File_fopen fopen
...

And your further libretro specific commit would then just enclose them with "#ifndef LIBRETRO" etc.

I think you'd want to upstream also commit adding test to Hatari, which validates that ftell/fopen/etc are used directly only in "file.h". Something like this:

Code: Select all

for fn in fopen ftell ...; do
    git grep -e "\b${fn}\b" | grep -v -e "File_$fn\b" -e "/\*.*$fn" -e "//.*${fn}" -e "file\.h:";
done

Which will fail if there an any of matches to given function names elsewhere than in comments or "file.h".

As to getting that to Hatari itself... I'm bit on the fence on it. May be better to ask on hatari-devel mailing list as some Hatari developers read Atari forum only infrequently. Just send a link to your repo where you have that commit.

[troed]

I figured out how to set up a Raspberry Pi build cross-build through the Github Actions. I only have an RPi2 to test with, and while it does run, it's unfortunately too low-powered to keep up with this core. So... partial success. Presumably the same build should work on later RPi where it might run at OK speed, as long as they have 32-bit support available. I set up an aarch64 build too... though I have no way to test that currently.

I've tested your build (32-bit since that's what my RetroPie installation is currently) but unfortunately they don't work (silently failed launch through runcommand, no log). Difference compared to my build:

mine:

Code: Select all

$ file hatariB/build/hatarib.so 
hatariB/build/hatarib.so: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=873be6240e4e79ba76abb3287bd3e673275dc72d, not stripped

$ ld hatariB/build/hatarib.so 
ld: warning: cannot find entry symbol _start; defaulting to 0001015c
ld: hatariB/build/hatarib.so: undefined reference to `udev_monitor_enable_receiving'
ld: hatariB/build/hatarib.so: undefined reference to `udev_monitor_new_from_netlink'
ld: hatariB/build/hatarib.so: undefined reference to `udev_monitor_filter_add_match_subsystem_devtype'
ld: hatariB/build/hatarib.so: undefined reference to `udev_monitor_receive_device'
ld: hatariB/build/hatarib.so: undefined reference to `udev_monitor_get_fd'
ld: hatariB/build/hatarib.so: undefined reference to `udev_device_get_action'
ld: hatariB/build/hatarib.so: undefined reference to `udev_device_get_devtype'
ld: hatariB/build/hatarib.so: undefined reference to `udev_new

yours:

Code: Select all

$ file cores/hatarib.so 
cores/hatarib.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=cac77452b512db38bda10302587bfe38925d7060, not stripped

$ ld cores/hatarib.so 
ld: warning: cannot find entry symbol _start; defaulting to 00010154
ld: cores/hatarib.so: undefined reference to `log@GLIBC_2.29'
ld: cores/hatarib.so: undefined reference to `exp@GLIBC_2.29'
ld: cores/hatarib.so: undefined reference to `pow@GLIBC_2.29'

[rainwarrior]

In my testing it ran with current Raspberry Pi OS. However, the build won't allow static linking libgcc, and it seems every RPi OS distribution has a different GLIBC, so it's really hard to make something widely compatible. Unfortunately if a dependent library is missing, the SO won't load at all, and RetroArch for some reason doesn't even try to pop up a notification if a core fails to load, so logs are the only option. It was hard to even try to select a GLIBC version for the cross compiler, and the "ubuntu-latest" Github Action runner had one that was too new for every RPi OS. I only got it running because there was an older ubuntu option.

I did try to get it to run with Retro Pie, and I think it did run with the latest OS but at this point it's hard to remember (I was trying a lot of situations). The other OS I tried was Lakka, which is the one linked from the RetroArch website, but it's a very "locked down" installation, so it seemed like most users wouldn't be able to get their own cores onto it anyway.

The solution of building it on your own machine is maybe the best way to go overall. I added some suggestions about how to do that to the README. Maybe I'll also label the build specifically "Raspberry Pi OS". (It's hard to find capsimg libraries that match, too, for the same reason.)


I also got around to adding keyboard language variations this weekend, so now the on-screen keyboard should be usable with almost any TOS region, I hope.

I'm not sure if I should provide anything for host keyboard remapping. RetroArch actually ignores the OS keyboard layout and pretends it's always US layout, which as a result I think is mostly reasonable mappings if the user picks a TOS that matches their own modern keyboard region (e.g. the host's QWANTZ key Z should actually give Z on German TOS, even though RetroArch reports it as Y). RetroArch does let you remap individual keys at least in its input menu, so maybe that's enough for users to take care of the edge cases.

[troed]

Is it GitHub Actions that won't allow you to statically link libs?

Another thing: It took me _way_ too long to realise that "System/tos.img" means that tos.img needs to be in the same folder as the disk images on RetroPie Usually the BIOS folder is the one meant by "System" and that's where ... well, BIOS-files etc reside for other emulators. It would be the natural place for TOS-images too.

Works fine to change to another TOS though as soon as I figured it out

(This is surely RetroPie specifics different from vanilla RetroArch, but anyway)

[rainwarrior]

No it's the ARM compiler itself. I have the -static-libgcc flag but it is apparently ignored in this situation, and I couldn't figure out any way to force it.

I wrote "system/" in the descriptions, but it should mean RetroArch's system folder, wherever that is configured, so whatever the "System/BIOS" directory setting is on RetroArch? Normally the folder has that name, relative to some installation directory (maybe in home .config on linux, maybe elsewhere). The code doesn't actually ask for a path of "system/", it requests the path of the folder from RetroArch, but I thought in the description it was less confusing to just say "system/". Maybe that was a mistake...

So... it should be in the same place as other BIOS files. If it's looking in the same folder as disk images, I guess check the RetroArch directory settings while it's running to see if you're getting a weird configuration there?

When I was looking at Retro Pie it seemed like every platform was given a completely separate configuration by the Emulation Station system, so there weren't any global settings to work with, and I mostly found my way around by checking the directories settings after starting RetroArch.

[troed]

No weird configuration - other emulators are indeed reading their bios files from BIOS, but tos.img wasn't detected there. In the end I solved it by enabling debug logging and checking where hatariB tried reading it from.

Code: Select all

[libretro INFO] core_read_file('/home/pi/RetroPie/roms/atarist//tos.img')
[libretro DEBUG] core_read_file (VFS) not found: /home/pi/RetroPie/roms/atarist//tos.img
[libretro ERROR] Could not load TOS.

I'll play around with trying to build something statically. Don't think I've done that for ARM before.

[rainwarrior]

What does it say in the directories setting for system/BIOS after starting up the core? That suggests that the path RetroArch is providing for it is "/home/pi/RetroPie/roms/atarist//", which is a little odd with the extra / too.

[troed]

ahhh. Could this be something that's set because of how I add your core. I'm lazy and install the existing Hatari first, then modify it's launch cfg. When I check the directories now you're right, System/BIOS is set to "<Content Directory>" which of course would be in roms. I checked another emulator and there the same setting is "RetroPie/BIOS".

My bad. I didn't even know that could be changed by an emulator when installed. I'll create some installation files around your core instead of doing it this way from now on

[telengard]

Awesome! And perfect timing, I switched my settop box from using Steem to Retroarch a few days ago. I'll try and provide some feedback soon (I'm running windows x64).

EDIT:
[I'm testing with Defender of the Crown from automation 82]

Love the OSK one shot!

- I'm noticing that I sometimes have to press the OSK 'press' button multiple times. Was trying to press '1' and it takes 3 or 4 tries to register
- I have disabled the status bar at the bottom, but I sometimes see flickering in that same area

This is so great, so many options and it fixed the video geometry issues I was having with stock hatari_libretro.

thank you for all this work!

[rainwarrior]

I think I noticed flickering at the bottom of the border with Defender of the Crown once too, but I was unable to reproduce it. Do you have any steps to recreate it reliably? (...and does it happen in Hatari stand-alone or only in this core?)

Could you also give steps about having to press a key multiple times? I'm unsure of how to duplicate this.

[telengard]

I think I noticed flickering at the bottom of the border with Defender of the Crown once too, but I was unable to reproduce it. Do you have any steps to recreate it reliably? (...and does it happen in Hatari stand-alone or only in this core?)

The space taken up by the flickering /seemed/ to be the exact size of the status bar (which I had disabled). I will test with stock Hatari, etc and see if it looks the same. I only mentioned it because it seems to be the same area as the status bar, not a big deal at all, but as a programmer myself I always like to know of any potential bugs, etc.

Could you also give steps about having to press a key multiple times? I'm unsure of how to duplicate this.

Sure thing, I have the OSK one shot and "press" mapped to a gamepad, but hopefully it would happen just by pressing keys. I'm using this disk image for DotC:

https://exxosforum.co.uk/atari/games/au ... /A_082.zip

There is a intro screen that comes up, here I was trying to type '1' to choose DotC and it would take 3 or more tries to register the selection:

[rainwarrior]

Could you tell me, are you using the current nightly build, or the 0.1 beta? Also, what TOS are you using, and are there any non-default settings?

For both of these cases is it something that happens every time for you? Could you possible describe the buttons pressed and released after load content?