Bad Mood : Falcon030 'Doom'

[dml]

Ok, I will look into fixing the script to setup worst frame breakpoints better with the latest BM code. Might not happen this week though, maybe next week.

No hurry I'm also very busy.

The problem seems most obvious in Doom II, map15 or map13. At least its possible to record a demo now to repeat the same profile several times. I might do that next.

[dml]

Interesting that Romero in the GDC video claims John Carmack got the BSP sorting concept from Bruce Naylor. I don't know if that's true but I also learned a lot from Bruce and had ongoing correspondence with him about finding optimal BSPs in the late 90's. Tiny world

[dml]

Hmm. Romero also states they had the graphics engine locked at 35Hz, but the AI stuff was running at 10Hz?

Haven't seen any evidence of that in the source - everything is locked at 35. It would also be difficult to have the graphics running at a different speed from the game without camera interpolation (or at least tick the player at a nice multiple of the enemy AI) - interpolation didn't appear until the Quake series.

Modding it to run the AI at 10Hz just results in slow-motion behaviour and requires the AI stuff to be modified in lots of places (which I did have to do).

Still its a strange thing to say if not true, and would explain some problems

[calimero]

Interesting that Romero in the GDC video claims John Carmack got the BSP sorting concept from Bruce Naylor. I don't know if that's true but I also learned a lot from Bruce and had ongoing correspondence with him about finding optimal BSPs in the late 90's. Tiny world

it is not "tiny world" but it IS *inaccurate* world in which you *can not* get back in time and check facts!

[troed]

It's been mentioned here before, but replay can also be done by recording all YM activity at a sufficiently high rate and blasting the registers with raw data, using rudimentary compression. This can be done as a last resort but it's more or less wasteful in some direction, so it's probably better to get the native replay code working if possible - maybe also fewer glitches in the replay

For a regular 50Hz tune I recall it would be around ~30kB of RAM for a regular length YM song. The playback code would then need ~1 scanline of (ST) CPU time per VBL. Of course, that limits the composition to non-SID which I guess a lot of current musicians wouldn't approve of

Patching the playback code is surely much easier - if it's really about unnecessary initialisation. I did so for the Sid Sound Designer replay rout used in LoSTE last year (the source was available though) since it was timer wasteful.

/Troed

[dml]

For a regular 50Hz tune I recall it would be around ~30kB of RAM for a regular length YM song. The playback code would then need ~1 scanline of (ST) CPU time per VBL. Of course, that limits the composition to non-SID which I guess a lot of current musicians wouldn't approve of

Yeah I figured it might reduce the chances of new contributions even lower if the nice stuff was unavailable

Patching the playback code is surely much easier - if it's really about unnecessary initialisation. I did so for the Sid Sound Designer replay rout used in LoSTE last year (the source was available though) since it was timer wasteful.
/Troed

I will return to it, however it probably helps to get a good understanding of what the export options actually do to the embedded replay code, to limit the amount of patching needed. My last attempt involved tracing all of the init code and identfying/nopping out the stuff I thought would be a problem (and there was a lot of that!) however the more extensive changes just crashed the replayer, and the more conservative ones still stopped the DMA timera events - so I got fed up and stuck it on the shelf for later!

[Eero Tamminen]

I will return to it, however it probably helps to get a good understanding of what the export options actually do to the embedded replay code, to limit the amount of patching needed. My last attempt involved tracing all of the init code and identfying/nopping out the stuff I thought would be a problem (and there was a lot of that!) however the more extensive changes just crashed the replayer, and the more conservative ones still stopped the DMA timera events - so I got fed up and stuck it on the shelf for later!

If you add symbols to the YM code, Hatari's profiler will tell you how many times they get called and from where. This can be used e.g. to check which code gets interrupted by interrupt handlers (for which you've added symbols).

[Yglika]

@DML: Hi. I just upgraded my Falcon to 14MB and wanted to try this badly, unfortunately I'm one of those guys who have "out of sync" problem with VGA monitors.

I tried two monitors:
- Dell 3008WFP (which is actually relatively new LCD panel with VGA port), that one reports "out of sync".
Then I tried old, actually high-end at its time CRT monitor (Belinea 108035 21") and it reports the same, complaining about horizontal frequency being 15.6K

Is here any way to force 60Hz via config file?

[Yglika]

Hmm, I actually isolated the problem. It happens with full version wad but not shareware wad.
Interesting. Either way it works with shareware wad what is good enough for me.
Awesome engine btw, I <3 it.

[dml]

Hi,

Hi. I just upgraded my Falcon to 14MB and wanted to try this badly, unfortunately I'm one of those guys who have "out of sync" problem with VGA monitors.

Ok, well chances are since you reported it here it can now be fixed I did try to find a cause and made some speculative changes but so far haven't found anything specific which looks wrong.

I have some doubts about the refresh rate of 54Hz but its hard to see how that can produce 15KHz horizontal. At least now we can test it to find out.

Then I tried old, actually high-end at its time CRT monitor (Belinea 108035 21") and it reports the same, complaining about horizontal frequency being 15.6K

Before we do anything else - lets check some basic things.

1) Are you booting it clean before running BadMood? Any interesting auto folder stuff?
2) Have your Falcon's clocks been modded? Sometimes if a bus accelerator has been installed one or more of the video clocks gets modded at the same time and can mess up custom video modes (I don't think it can be this but it could confuse matters when trying things).

Next step should be for me to send you a .SCP config file for Screens Pain, which encodes the same video settings that I use in BM. If you get the same problem with that, it must be the cause. If not, then it has to be something in the engine itself and I'll need to dig deeper.

If its just a problem with the SCP then it can be quickly adjusted and a fixed/updated version pushed out quickly.

[Yglika]

Hi Dml.

I know now what's the problem, I was using doom.wad version 1.1 from some ancient backup I had. So pretty much my fault, but the error is a bit misleading.
It works fine with shareware 1.9 and probably with full 1.9 as well.
Awesome work, I have only one plea to it - please add music support too.

El.

[dml]

Maybe you can give this one a try - exactly the same build but with 60Hz VGA modes.

https://dl.dropboxusercontent.com/u/129 ... lph02b.zip

If it fixes the problem let me know so I can work it into the next version. If not, let me know if there is *any* change in your monitor's behaviour. I'll probably then follow up with a debug build which prints the type of monitor it thinks it has detected...

[dml]

Tonight I did a first experiment with the game/AI code.

A new linedef special and AI thinker has been added to implement proper elevators (so the ceiling and floor move together). The elevator can be sent up or down using the new linedef tag.

I noticed ZDoom can support this if you write a script for it but I figured it would be nice to have something like this built in, and its good practise for adding new game code

[dml]

Hi Dml.

I know now what's the problem, I was using doom.wad version 1.1 from some ancient backup I had. So pretty much my fault, but the error is a bit misleading.
It works fine with shareware 1.9 and probably with full 1.9 as well.
Awesome work, I have only one plea to it - please add music support too.

El.

Ah - ok. Well I haven't kept track very well of the 'effects' from running older WADs. I should add a version check for them - a CRC or something and a big banner

Glad it works!

Beta version is well underway. Haven't got around to native music support yet - MIDI might not be far off though!

[Yglika]

That 60Hz version is actually sweet.

I had a minor problem on my Dell LCD with the original version that it fails to autocenter VGA signal correctly and shifts the screen about 20 pixels to the left, clipping it. Not a big issue and it is actually stupidity of the monitor so I didn't even mention it but with 60Hz it is now flawless. So I suggest adding 60Hz option in config file for future releases even though the original problem had actually nothing to do with it.

[dml]

Here are some grabs from the 'elevator' test. I had to grab these from BM because normal versions of Doom don't have the elevator logic built-in.

(P.S. that's not a BM sky my cache is still dirty from testing PWADs!)

Layout in the editor:

elev1.png

Viewing in the game, from outside:

elev2.png

Going in:

elev3.png

elev4.png

Going down:

elev5.png

At the bottom, new room:

elev6.png

[Eero Tamminen]

The problem seems most obvious in Doom II, map15 or map13. At least its possible to record a demo now to repeat the same profile several times. I might do that next.

Demo would be nice, that way it's also possible to verify the effect of optimization on whole demo and find difference to next worst frame.

[Eero Tamminen]

Regarding worst frame profiling, it still seems to start from correct place (first A_Chase() call i.e. when first monster notices player), but it doesn't end at level end. Currently it ends at first call to either P_DeathThink() or D_AdvanceDemo(). What would be good symbol to check that player reached level end?

When going over multiple levels, worst frame render is 8s of mostly:

Code: Select all

Time spent in profile = 8.31849s.
...
Used cycles:
  39.63%  39.77%  39.77%    52882332  53076532  53076532   _D_QuantCubeFillLookupTable
  22.80%  22.88%  22.88%    30425108  30534148  30534148   _D_QuantCubeSortColorListRegion
  11.37%  11.41%  77.42%    15168532  15220804 103312268   _D_ImageQuantizerCreateOptimalPa
   8.59%   8.62%   8.62%    11467184  11508512  11508512   correct_element
   7.27%   7.30%   7.30%     9704660   9740568   9740568   D_TextureMipGen_16_16
   2.03%   2.04%   2.04%     2715452   2725720   2725720   _D_QuantCubeCalcColor
   1.68%   1.68%  10.27%     2239212   2247772  13711556   create_local_palette_64levels
   1.20%   1.20%   1.20%     1595096   1600412   1600412   _D_ImageQuantizerAddPixelsToHist
   0.88%   0.89%   0.89%     1179736   1183636   1183636   D_PatchTargetGetApproxRGB
   0.88%                     1175324                       set256_2
   0.85%   0.86%   0.86%     1140556   1144508   1144508   _D_QuantCubeScanColorUsage
   0.56%                      752440                       R_Column_NMip_2x1

Instruction cache misses:
  19.49%  19.68%  19.68%       27661     27932     27932   _D_QuantCubeCalcColor
  12.72%  12.80%  12.80%       18049     18163     18163   _D_QuantCubeScanColorUsage
  11.82%  12.39%  91.11%       16781     17592    129334   _D_ImageQuantizerCreateOptimalPa
  11.52%  12.91%  12.91%       16350     18325     18325   _D_QuantCubeSortColorListRegion
  11.51%  14.76%  14.76%       16337     20955     20955   _D_QuantCubeFillLookupTable

And DSP side:

Code: Select all

Used cycles:
  99.90%  99.91%  99.91%   266634304 266664868 266664868   R_DoColumnPerspCorrect

EDIT: files mentioned before this frame were:

Code: Select all

BMC\TEX\SW2COMP.BMT
hd\patch\COMP03_4.bmp
hd\patch\COMP04_5.bmp
d\patch\SW2S1.bmp
BMC\TEX\SW2COMP.BMT

First 2 weren't found, 3rd was found, 4th wasn't found, last wasn't found, but created. This frame wasn't only one where "render" part took 8s, there were also few others.

[dml]

Regarding worst frame profiling, it still seems to start from correct place (first A_Chase() call i.e. when first monster notices player), but it doesn't end at level end. Currently it ends at first call to either P_DeathThink() or D_AdvanceDemo(). What would be good symbol to check that player reached level end?
[/code]

Ok I'll look again to see if an obvious place exists for ending the profiling. It's tricky because there isn't a direct transition - the main tick 'thread' can have any number of processes going at once and the end of the game process overlaps with the start of some other frontend stuff.

It might be useful to insert a dummy call to a special 'profiler_stop' function when the player's health depletes to zero, and leads to the other processes. This will make sure the profiling stops as soon as possible. If there is no other obvious place, I'll just add that.

And DSP side:

Code: Select all

Used cycles:
  99.90%  99.91%  99.91%   266634304 266664868 266664868   R_DoColumnPerspCorrect

That's probably because the DSP has been told to begin shading but is waiting for the CPU to finish messing with texture preprocessing from disk. i.e. it's not a typical measurement during gameplay.

[dml]

Have added a call to BM_E_PlayerDeath() as soon as the console player dies. This should work as a profiling event to stop recording etc.

[dml]

EDIT: files mentioned before this frame were:

Code: Select all

BMC\TEX\SW2COMP.BMT
hd\patch\COMP03_4.bmp
hd\patch\COMP04_5.bmp
d\patch\SW2S1.bmp
BMC\TEX\SW2COMP.BMT

First 2 weren't found, 3rd was found, 4th wasn't found, last wasn't found, but created. This frame wasn't only one where "render" part took 8s, there were also few others.

Do you mean this happens every time you run the same profile?

That stuff should happen the first time it sees a level, and shouldn't repeat unless BMC/ is deleted. If you find it is happening more than once for any file then there is a problem.

Basically I don't much care about the time taken to convert textures - but I do care if it tries to do that more than once


I'll try to record a demo tonight and will run 2 profiles in sequence - the first to populate the cache, and the second to actually do the profiling. If something is wrong I should probably see it on the 2nd run.

[Eero Tamminen]

Ok I'll look again to see if an obvious place exists for ending the profiling. It's tricky because there isn't a direct transition - the main tick 'thread' can have any number of processes going at once and the end of the game process overlaps with the start of some other frontend stuff.

If level end detection goes slightly after level actually ends, that's not a problem as long as time from last render/think start isn't longer than what the worst frame so far took.

Have added a call to BM_E_PlayerDeath() as soon as the console player dies. This should work as a profiling event to stop recording etc.

I think profiling should already catch player death OK, the problem is catching level end.

Do you mean this happens every time you run the same profile?

That stuff should happen the first time it sees a level, and shouldn't repeat unless BMC/ is deleted. If you find it is happening more than once for any file then there is a problem.

I don't thing so, but it's hard to say, as every play is different (they last long so I don't do them that often ). That's why having 100% reproducible test-case is important.

[dml]

I recorded a demo this evening, which is pretty bad, especially given that I loaded the wrong mouse/keys config So I'll probably scrap it and try again.

I did have to deal with a bug first in demo record/replay - caused by adding freelook - so at least that's fixed now.

[dml]

Hi Eero,

I finally recorded a demo against a sensible revision of the game code, using map15 from Doom2.

You can find the demo file here:

BadMood/autoprofile/wads/d2m15.lmp

To run the demo just use:

-timedemo d2m15


It's not great - I didn't finish the level. Partly because you can't cheat when recording demos and partly because I forgot which key I bound for 'run' so I couldn't make a critical jump. Gave up after trying a couple of times heh!

I made some changes to the game code before making the recording. The original code stores 8-bit angles for demo replay, and is careful to record them then immediately replay them - while you play the game. This is because it quantizes the player angle quite a lot, and causes the view to snap quite badly. I didn't really notice this before with Doom but I think all old versions do it. IMO it looks awful and makes gameplay more difficult so I fixed it to record the full 16bits so its nice and smooth.

I also restored 8-bit bools, vs 32-bit. Don't know if it helps performance but it's unlikely to hurt.

I mention both of these things, because demo compatibility depends on them. Change them and the demos will stop working. I don't plan to change these again so hopefully demos recorded from now onwards will be ok.

[EDIT]

BTW for whatever reason, I saw *no* evidence of the slowdown thing happening while recording the demo. I did recently delete the BMC cache and start from scratch - not sure if that has something to do with it. Or it could be specific to the super-shotgun, which I tend to use when I cheat (and didn't find while recording).

I might have to find another level where the SS is easy to collect in case it is related.

The SS fires lots of pellets and makes lots of puff sprites on the target. Maybe these are involved - I don't know.

The demo recorded pretty smoothly and replays the same. :-z

[dml]

I have modded the autoprofile script profile.conf/profile.sh to use the new recorded demo with doom_t4.ttp (TIMEBASE_CONTROL=4, full optimization).

It takes ages to run, given that it's capturing profile info from both CPU and DSP, and it needs 2 runs to fully populate the disk cache and profile the game properly, so i will leave it running overnight. Will look in the morning to see what I find there.