Bad Mood : Falcon030 'Doom'

[dhedberg]

I haven't posted here in ages but things are still crawling forward.

While I haven't had a lot of time for coding, I do now have two (!) of the original MIDI tracks replaying 90% correctly via Codec, with nearly all of the SFX replay hacks removed. So the other tracks should follow quite quickly. I started with those tracks having the greatest number of channels and instruments active so it would be all downhill after the first few

I am still optimizing this as I find the time, as it has collected a lot of garbage and mess over many sessions doing a little at a time, and having fixed a lot of annoying bugs. There does seem to be one bug left which is puzzling me - but hopefully I'll get enough time in one block to identify and fix it soon.

The Codec and HW MIDI replay systems are on different code branches and these will need to be brought together before long.

Not much else to report, that's all for now.

I'm glad to hear that things are still moving forward. Take your time, work on it when you feel like it... there's no hurry, the machine is well past its due date!

[dml]

Hi Daniel,

I'm glad to hear that things are still moving forward. Take your time, work on it when you feel like it... there's no hurry, the machine is well past its due date!

I'm definitely trying to get it finished, but recently much has been happening to keep me away from the keyboard so things are slow.


It has been some time since I posted any 'results' so here is an early demo of native MIDI replay via Codec, for the ingame music.

https://www.youtube.com/watch?v=slc24GsQU0A

I put some details in the video description. There are still some things quite wrong with it, but getting quite close. I suppose it sounds a bit like a MODfile, but it is essentially coming from MIDI data and while the MIDI events are simple enough to interpret, the consequences for overlapping sources can be a challenge to handle without dropping channels and destroying the music. Notes must sustain while held, and 'ring' or decay after being released, and the same instrument playing different notes results in polyphony - and many other minor difficulties all layered together. It has taken quite a lot of work to get this far with it.


Apart from fixing problems and trying to make it sound decent, I'm also trying to optimize the replay engine and work on the game side to integrate music with the game logic (for both MIDI HW and Codec replay).

The codec replay engine currently steals about 4% for the MIDI event processing and instrument calculations, and a further 8% for mixing, plus 1% for resampling - all stolen from the CPU. In total this is a bit too much for ingame use and it can be felt while playing, so I'm trying to cut that by 30% or more. This will take a bit more time.

I can't mix to Codec using the DSP without removing the DSP texturing feature, since the texturing trick relies heavily on being uninterrupted. The trade doesn't seem worthwhile here.


Anyway I'll try to record another vid soon with a track actually playing ingame. I can do this already but recording output from Hatari is awesomely slow and must be done via a recorded demo to have any hope of doing it properly - a demo which I haven't recorded yet

Bye for now.

[EvilFranky]

New title screen!!

Sent from my Nexus 4 using Tapatalk

[dml]

New title screen!!

Indeed, there will be quite a lot of new stuff in the next version including art, music, sfx and so on. The vid isn't representative of the final look since the menus get in the way etc. but will do for now.

[dml]

Here is a second video, proving the original ingame music + sfx combined. This is the oh-so-familiar e1m1 track, and the Codec manages to get it pretty close to the reference GM Soundcard version. In fact, maybe this one turns out a bit better I translated this MIDI file today in one go, without edits. The replay init was moved to the level start routine. It all worked first time.

You may also notice some new ricochet sounds.

https://dl.dropboxusercontent.com/u/12947585/ingame.avi

This is a 100mb file, so expect a wait.


I won't put this vid on YT because its really jerky. For some reason, recording AVIs here from Hatari results in accurate audio, but 'jerky' framerate. If I play the game normally in Hatari, I get the opposite - consistent framerate but some occasional glitches in the music. This is why I recorded the longer BM gameplay videos direct from the Falcon's composite video signal since results are always correct.

However its good enough to demo the ingame music feature.

I don't have time to figure out such mysteries with my PC/VM, drivers, Hatari etc... will focus on making sure its ok on Atari HW

BFN

[dma]

Wow! That's indeed sounding pretty good!

Congrats!

[DarkLord]

I don't have time to figure out such mysteries with my PC/VM, drivers, Hatari etc... will focus on making sure its ok on Atari HW

BFN

Amen!

[dml]

Thanks. I'll hopefully have most of the better tracks integrated quite soon.

Some translate easily because there are only a few channels (e.g. many of the Doom I tracks) and not much complexity. A few of the Doom II tracks hide a lot of problems though and some have required careful editing by hand to get them sensible. They do play, but cause dropouts in places.

For example - one track was triggering 6 different drums simultaneously within the same channel, to produce a specific sound - and there were 8 channels That's a bit too much for my player...

[dml]

So with a bit of optimizing last night (both data and code) I got the following results:

Code: Select all

MIDI event handling / replay:         ~3.0% (8 scanlines)
Mixing (6 channel, 16bit, stereo):    ~8.2% (22 scanlines)
Resampling:                           ~0.2%

There's still plenty of room to optimize the MIDI replay part but its close to usable at 3%. Mixing however is difficult to optimize further while keeping the stereo panning feature.

Given that stereo is mostly useful for identifying the direction of SFX during gameplay, and that music interferes with this a lot anyway, it might be a good plan to offer a mono mix option via the config file or menus. Or drop to mono by default when music is enabled, unless the config says otherwise. This would help cut mixing time on a base 16MHz machine and keep the framerate as high as possible. The user can then decide if they want to afford a few CPU % for stereo sound. It's not a huge difference in cost but its one of those small things that adds to the other small things.

I found another Doom II track which looked fine on first inspection but turned out to be a near-worst-case, with many of those tracks having multiple notes held at once More editing...

[Eero Tamminen]

Can you have effects as stereo and music as mono and/or switch to stereo only when there are effects playing? I'm assuming effects don't happen constantly, although with things like chaingun that might not be a correct assumption...

[dml]

Hi Eero,

Can you have effects as stereo and music as mono and/or switch to stereo only when there are effects playing? I'm assuming effects don't happen constantly, although with things like chaingun that might not be a correct assumption...

The SFX and music are served by the same mixer, which must modulate and write 32bit samples, for 16bit stereo (if we assume panning stereo - which is not the same as putting separate channels to left/right, sometimes used in modplayers).

Switching to mono would cut reads/writes in half. Even this will save probably 3% and not more. It's probably not worth doing anything severe with audio which saves less than that.

The biggest single saving could be made by eliminating volume modulation, since it needs done for every sample. This can be done by increasing storage cost by about 4x or more - otherwise I'd be doing that now. It might be considered as a final thing but not before all other costs are known, since it is very greedy.

By switching to mono and absorbing volume modulation costs, the speedup would be quite significant. If to do just one of these, mono is more practical IMO because storage remains flat.

[dml]

It seems I managed to get myself into a bit of a mess while optimizing the code last night. I was surprised to find that writing faster code resulted in slower performance. In the end I figured out that my version of Hatari was broken - the CPU emulation tables seemed to be wrong. Having updated Hatari the problem went away and the profiling figures I gathered are all now invalid, in some areas I'll have to start measuring again

MIDI processing time dropped quite a lot, and mixing time went up. In total it's around the same - perhaps a bit worse, but the total FPS is higher and gameplay is less jerky. It's different enough that I'll need to go through some steps again to get new figures and retest some ideas I already tried.

Unfortunately I can't easily test on real HW just now because I moved my development environment into a VM so I can use it during journeys etc. and the VM doesn't work with the transfer system I had before. I'll just retest occasionally from the desktop machine when I can.

[dhedberg]

It seems I managed to get myself into a bit of a mess while optimizing the code last night. I was surprised to find that writing faster code resulted in slower performance. In the end I figured out that my version of Hatari was broken - the CPU emulation tables seemed to be wrong. Having updated Hatari the problem went away and the profiling figures I gathered are all now invalid, in some areas I'll have to start measuring again

O'boy, that's kind of what happened to me a while back as well. I did lots of optimizations in Hatari that actually proved to run slower on real hardware.

MIDI processing time dropped quite a lot, and mixing time went up. In total it's around the same - perhaps a bit worse, but the total FPS is higher and gameplay is less jerky. It's different enough that I'll need to go through some steps again to get new figures and retest some ideas I already tried.

Great! I'm glad to hear that your efforts weren't all a waste of time!

Unfortunately I can't easily test on real HW just now because I moved my development environment into a VM so I can use it during journeys etc. and the VM doesn't work with the transfer system I had before. I'll just retest occasionally from the desktop machine when I can.

It's always a good idea (read critical) to try things out on real hardware now and then.
Out of curiosity, how was your old transfer system set up? Did you transfer stuff over the network?

[dml]

It's always a good idea (read critical) to try things out on real hardware now and then.
Out of curiosity, how was your old transfer system set up? Did you transfer stuff over the network?

I have been using a HxC USB floppy emulator fitted to a Falcon. It works pretty well for files up to about 1.7mb - then it becomes a problem

I know some people are using serial links, ethernet adapters etc. I'm interested to know if the CosmosEx device will be a better option but haven't looked at it closely yet to see.

[viking272]

Unfortunately I can't easily test on real HW just now because I moved my development environment into a VM so I can use it during journeys etc. and the VM doesn't work with the transfer system I had before.

On this point Doug, what VM are you using. Did you need USB working?

[dml]

On this point Doug, what VM are you using. Did you need USB working?

VMWare + win7 under OSX.

However it's not so much 'can't work' as 'currently doesn't work' - I'll probably have to find a good couple of hours to start mucking about with that. I had just planned on finishing the main code changes and then doing tests & performance tuning with the Falcon from the desktop PC, since they are both connected up anyway.

[Oldskool]

Hi dml,

For large file transfer I use a Gigafile device on my Falcon.
It's a small external sd card reader which connects via the scsi port.
It works great and is fast and ideal for transferring large files between my falcon and the PC.
Fat32 is also no problem when using mint and this device (I use vanilla mint distro on my falcy http://atari.nvg.org/VanillaMiNT/)

The device is however not cheap but I personally think it is worth the money.

Some guy in Germany makes them.

If you're interested in one please contact him first for the right scsi connector ( not every connector works)
( you also need hddriver)

You can buy these things here:

http://shop.inventronik.de/store

Maybe this helps you a bit.

Regards,

Ofcourse you can also use the internal ide with the cf device. However opening the Falcon every time was a bit frustrating at the end. And the scsi is faster.

[Eero Tamminen]

It seems I managed to get myself into a bit of a mess while optimizing the code last night. I was surprised to find that writing faster code resulted in slower performance. In the end I figured out that my version of Hatari was broken - the CPU emulation tables seemed to be wrong. Having updated Hatari the problem went away and the profiling figures I gathered are all now invalid, in some areas I'll have to start measuring again

O'boy, that's kind of what happened to me a while back as well. I did lots of optimizations in Hatari that actually proved to run slower on real hardware.

If it's something unrelated to data cache, it would be good to drop a note about that to Hatari devel mailing list or to the Hatari forum area.

Main thing Hatari provides is good overview of where the performance goes, so that you know on which area to concentrate. While ST emulation is pretty accurate, Falcon is less so. DSP side should be accurate and so should instruction counts. However, non-existing data cache emulation and some video emulation deficiences can distort cycle accuracy though, depending on what your code does.

Btw. I find profiler to be good for debugging too, as profiler shows all the addresses where code has been executed and how many times each instruction was executed. Just do a profile with symbols and then "profile save" to save information on all executed instructions.

[dml]

For large file transfer I use a Gigafile device on my Falcon.
It's a small external sd card reader which connects via the scsi port.

Seems like quite a good idea. It's not much fun swapping CF or SD cards for floppy images etc. but it would be worth the trouble for very big files. I might look into that option, thanks.

[dml]

If it's something unrelated to data cache, it would be good to drop a note about that to Hatari devel mailing list or to the Hatari forum area.

In fact this one was identified and resolved on the mailing list relatively recently. However I didn't think I had a build of Hatari affected by that bug until I started getting funny results Just getting a new build fixed it.

[dml]

As a small bonus I managed to get a few Quake II tracks integrated with the MIDI player. They do need more work in places but some parts sound pretty good ingame.

At this rate there should be plenty of tracks ready for the next release.

[dml]

I've got the following tracks (listed below) converted for Codec replay with most of them working well enough to be considered 'final', give or take a few tweaks or instrument changes. Some I haven't tested but they don't appear to be problematic - the simpler ones were skipped over to focus on the harder material.

Of all the tracks I processed, only two didn't work well. They might still be saved but probably not worth the trouble. One of them depends heavily on pitchbending and other controller effects and while this could be added, I'd need to remake my note tables at higher resolution and make code changes in several places to deal with it, and that is too much effort for the limited use it has among all the tracks.

So nearly all Doom2 tracks ready. Some required a lot of editing, some a bit of editing, some translated unedited. I barely started on Doom1 but looking at some of them indicates most should translate without editing. Should manage a few per evening.

It seems to cope with some of the Quake2 tracks but not all - will pick the best/most appropriate and leave the rest. 'Big Gun' worked surprisingly well

Music/SFX volume now both tied up to the menus. Music set at 50% by default. As the music volume is reduced, channel events get dropped below a threshold, making more room for SFX - with music volume at zero, only SFX events will compete for channels.

That's probably all for now, until I get the Doom1 tracks converted...

Doom 1:

e1m1
(remainder not yet processed)

Doom 2:

map01
map02 (**untested)
map03 (**untested)
map04 (**untested)
map05
map06 (**untested)
map07
map08
map09
map10
map18
map20
map23
map25
map28
map31
map30 - this one is troublesome, uses lots of weird effects, may drop it
map32
read_me

Quake 2:

Big Gun
Kill Ratio - this one did not translate well, very soundfont-specific, may drop it

[dma]

That's excellent.

Could you, by any chance, produce some kind of a music-disk with those tracks and your player, once the converting is done?
It would be super cool for anyone to listen those tracks renditions on any Falcon.

[dml]

Could you, by any chance, produce some kind of a music-disk with those tracks and your player, once the converting is done?
It would be super cool for anyone to listen those tracks renditions on any Falcon.

That's a nice idea. Yes I'll do that.

There is a separate convertor which turns .MID files into .BMZ (BadMooD) format which can be played by the engine but there are still some manual steps involved, and not all instruments or effects are mapped - so there will be a big YMMV for premade MIDI. For specially made music I think those steps could be ignored. It would help to have a separate player if some Atari fan wants to make new tracks for it and test them.

The manual steps are primarily for remapping missing instruments to available ones, setting octave thresholds for multisampled instruments and adjusting master channel volumes to suit the instruments - which are normalized and don't match the natural GM volume levels. This is done per .MID conversion but I suppose those things could be put in ASCII .CSV files and applied automatically during conversion.

Only the SW/Codec music player needs the offline conversion process. Hardware MIDI replay won't require any conversion steps - Saulot integrated conversion to his own .NKT format within the HW MIDI library so that gets done mostly invisibly. Will be possible to just play MID or MUS tracks from the originals.

[Eero Tamminen]

Will be possible to just play MID or MUS tracks from the originals.

That sounds really great! Running both BM in Hatari with WinAUE CPU core & DSP emulation, and a good softsynth in the same computer needs beefier machine than mine, to do all that without getting frame skips in Hatari though...