[A]tari [G]ame [T]ools - 2D prototyping engine for STE

[Anima]

Thanks Doug for your ongoing work on this.

Looking forward to any new posts here.

[dml]

Many thanks Sascha,

I am still working on a few things but the last changes have been difficult and progress is bit slow...

So there will be a pause before the next updates, but more is on the way.

[dml]

Not quite an update yet...

...but I have replaced the sprite generator with a new version. Currently testing it and trying to measure & settle all of the optimizations involved.

The speed gain varies from small - on narrow square sprites - to significant, on big irregular sprites. I'm seeing a 20-40% speedup on some of the bigger arcade sprites I'm working with, when compared with the non-code-generated EMS sprite system still in AGT. (Comparing with the previous code-generated sprite system, the savings are smaller but still very significant, around 15-30% but it is very shape-dependent).

This speedup mainly comes from changing the way the sprite generator breaks down the sprite frame into blocks of Blit ops. Previously it was cutting out sections of 32xH and converting them into 48xH chunks, taking fully-dead lines into account within blocks (treated as skips). As a bonus, it could also coalesce adjacent 32xH blocks to make them effectively 48-wide sources without overlaps using hardware FISR, which did provide some good savings for wide sprites. This was the main 'gain' over just using 32-wide sprites in a grid. However the Blitter spans were all effectively 36 (or less) bus cycles wide due to the 3-word-wide enforcement and tended to transfer a lot of empty words/holes in the resulting rectangles, wasting time.


The new generator doesn't have to deal with rectangles anymore, and does not need to work with 32- or 48-pixel wide Blits. I have come up with a hybrid mixture between AGT's old polygon-like, shifted-span SLAB renderer and Anima's 32-wide EM sprite renderer, taking the best things from each.

Namely:
- hardware EM, code-generation, const-prop capabilities of the sprite system
- arbitrary-width spans & ad-hoc layout capability from the SLAB system

The new system can generate Blitter spans up to any bus-cycle cap it is configured for. This is important because the cost of a Blit varies according to the masks applied. To the Atari Blitter, a hardware mask of all-1's (FFFF, or 'solid fill') avoids a redundant memory fetch from the destination/screen, which means lines containing (or starting/ending with) this mask are cheaper, and their size can be increased accordingly. The important point here is that *wide* sprites tend to have FFFF in the middle of many spans, allowing them to decompose into fewer, longer Blits.

A hardware mask of all-0's (empty holes) are not however optimized by the Atari Blitter to avoid redundant reads & writes, meaning wasted cycles. The new system tries to trim these out of the layout, at the edges and also the interior, again increasing the size of Blits and reducing their number.

Since Blits can now be wider (and no longer confined to rectangles), fewer horizontal components are needed to make up the sprite. Shapes which previously took 2-4 rendering components can now be rendered just in 1 or 2. The 'components' are no longer iterated sprite pieces, just one unbroken sequences of spans which hop around the shape to maximise coherence, also saving time.


The Blit bus cycle cap can be configured anywhere between around 36 bus cycles (equivalent to the old system, if rectangular Blits were enforced) and 64 bus cycles (equivalent to Blitter co-op mode). The ideal cap is somewhere in-between, to play nicely with MFP & raster interrupts.

There are several other optimizations in the system, including pre-calculation of NFSR + FISR redundant reads within the sprite, based on mask and shift knowledge. Line reordering to reduce state changes, constant-propagation, peephole substitutions and so on.


To summarize.... AGT just got measurably faster across the board, but especially for large sprites.


At this point, there are shapes which can Blit more cheaply on STE than the equivalent on Amiga OCS (assuming rectangular Blits in 4 planes). Perhaps that is being a bit unfair since we're dealing less and less with rectangles now on Atari - but in the end we're interested in the sprite active source area vs cycles spent, and that ratio keeps improving.

The storage cost for the new sprites is also a bit lower than the previous code-generated (EMX) sprites, mainly because the total Blit count is reduced.


This is not finished, but it is working - I will be pushing a version very soon. I will continue to tweak the optimizations after the first version is in.


Again, a big thanks to Anima who got this piece of R&D off the ground in the first place. While this effort has been focused on maximising AGT's sprite throughput by combining various methods, most of the EM sprite techniques are still fundamental for this revised system to perform well!

[TheNameOfTheGame]

That sounds fantastic! AGT keeps getting better and better!

[Cyprian]

great news DML

[calimero]

...at the end we will see Cho Ren Sha on STe...

[Anima]

Great progress. Thanks for the detailed overview of the improvements.

...at the end we will see Cho Ren Sha on STe...

Well, I think it would be kind of doable today. At least Galaga 88 is a good candidate to have a look at.

[calimero]

Galaga 88 is pretty massive!
...playing Galaga 88 on Falcon it is already impressive (I just also played Galaga 88 on MAME/CoinOps/Xbox... - it is prefect on Falcon)

if STe could pool it...

btw
can you restore animated background in Cho Ren Sha? Or it is too much work/complicated...?

[mrbombermillzy]

I think the DML/Anima combo is really pushing the envolope here.

This must be Atari 16/32 bits finest hour for game creation, surely?

To have this level of control end user (or coder!) facing is such an achievement too (and lots of extra work, I'll bet).

I can only bow and take my hat off to you guys!

[masteries]

...at the end we will see Cho Ren Sha on STe...

hehehe Cho Ren Sha makes Falcon indistinguishable from a NeoGeo

[masteries]

Great progress. Thanks for the detailed overview of the improvements.

Well, I think it would be kind of doable today. At least Galaga 88 is a good candidate to have a look at.

A dualfield coloured version of Ghouls and Ghost arcade is a better candidate... xD

its not a joke, I see it completely possible with current AGT features!

[dml]

Anima has already produced a version of Ghouls and Ghost with his super fast sprites and I think it is running a version of the real game code (?) as well, which probably has a strong influence on how the sprites/objects have to be drawn. From what I've seen though its pretty fast! and it is full game logic.

There are probably a lot of other good candidates which haven't been visited yet - I have a short list myself

[masteries]

Anima has already produced a version of Ghouls and Ghost with his super fast sprites and I think it is running a version of the real game code (?) as well, which probably has a strong influence on how the sprites/objects have to be drawn. From what I've seen though its pretty fast! and it is full game logic.

There are probably a lot of other good candidates which haven't been visited yet - I have a short list myself

Yes, Anima executes the native arcade code... with a replacement of the video related code, amazing effort!

Today I discovered that the arcade game is based on two rounds... that is not the best case, players want more levels, not a second round

[TotOOntHeMooN]

Today I discovered that the arcade game is based on two rounds... that is not the best case, players want more levels, not a second round

I disagree with that. All the Makaimura games work around two rounds to be able to got a secret weapon to beat the (real) final boss. If peoples have the time and skill to produce new levels from it, better to do a new sequel to the game.

[calimero]

Today I discovered that the arcade game is based on two rounds... that is not the best case, players want more levels, not a second round

I disagree with that. All the Makaimura games work around two rounds to be able to got a secret weapon to beat the (real) final boss. If peoples have the time and skill to produce new levels from it, better to do a new sequel to the game.

Here is great review

Ghosts N' Goblins (NES) - Angry Video Game Nerd (AVGN)

https://www.youtube.com/watch?v=94Y6y1MOoEo

(search AVGN for "true ending" )

btw
what is correct syntax for including YT video on page?

[masteries]

Today I discovered that the arcade game is based on two rounds... that is not the best case, players want more levels, not a second round

I disagree with that. All the Makaimura games work around two rounds to be able to got a secret weapon to beat the (real) final boss. If peoples have the time and skill to produce new levels from it, better to do a new sequel to the game.

Here is great review

Ghosts N' Goblins (NES) - Angry Video Game Nerd (AVGN)

https://www.youtube.com/watch?v=94Y6y1MOoEo

(search AVGN for "true ending" )

btw
what is correct syntax for including YT video on page?

hehehe... The same feeling when I discovered... the same...

I remember that the ZX Spectrum version is more affordable, some graphics are hard to notice due to colour clash, but easier to play.

[Anima]

Well, I think I need to get back on track again with my ideas and projects. So many things which are not even finished and I was pushed back by the Blitter hardware problems I was facing some weeks ago.

However, this thread should return to the AGT discussion again.

I would like to add the AGT build tools in my Raspberry Pi "development device" since I have different platforms here from where I'd like to develop and it seems to be a real nightmare to get it running everywhere.

I'll check how far I'll get with the instructions from Dougs website.

[dml]

I would like to add the AGT build tools in my Raspberry Pi "development device" since I have different platforms here from where I'd like to develop and it seems to be a real nightmare to get it running everywhere.

I'll check how far I'll get with the instructions from Dougs website.

I think the main problem will be the fact that the tools are x86 binaries. Although I recently had the main tool (agtcut) building on Linux and it should probably also compile for ARM (probably!). The source is in the repo.

The pcs palette gen tool is gcc-based source so it probably can also build for ARM, although I have never tried it. The source for this is not in the repo currently and is in fact in a bit of a state right now - not quite a new release yet but a large number of files checked out & edited :-/ I should really deal with this soon as I had new features waiting to be released there.

RMAC assembler might already be available for ARM (?) as ggn uses RPI for a bunch of stuff too. I bet he has compiled it for RPI already (not wishing to land him in hot water here

The remaining tools aren't all that important IMO. agtcut + RMAC are the important ones to build any of the AGT projects. The rest (packers etc.) can be worked around or ignored.

[Anima]

I think the main problem will be the fact that the tools are x86 binaries.

Thanks for the info. I've got into some troubles as well with the x86 based DSP assembler I am using for my Falcon projects. Will check out tomorrow how far I'll get.

[masteries]

Are there current state of the art commercial compilers for 68k?

Usually I use commercial ARM compiler instead of GNU due to the code optimization and overall quality are far beyond free compilers.

Obviously, commercial ones can be expensive or hard to acquire... but I am curious about 68k

[dml]

I'm still gradually making my way through the sprite upgrades. Clipping this stuff is a nightmare!

However the progress has been good.

Here's a before/after using a worst-case sprite. Well not quite worst-case but a poor case for sure.

before.png

The sprite is a single 256x128 cut, with independent regions and an internal line gap. The old system could cope with internal line gaps but only where a component (column) has completely empty lines.

Previously a component was a 48-wide rectangle of any height (32-wide for the first/leftmost component). In the example sprite, 5 components were needed to cope with this shape (corresponding to the 5 green timing bands). The 48-wide optimization really helped here - it would have been 8 components for 32-wide columns!

The components/columns could be different heights, so time is saved per component. But still, fitting these circles into adjacent rectangles is not efficient.

Now look at the results for the new version...

after.png

A single component, rendered in less than half the time. This is the benefit of using arbitrary-length, offset spans instead of the fixed-width columns. Far less data being moved.


Some notes...

The improvement on small sprites is neglegable or zero - if the sprite is no wider than 32 pixels, there is no guarantee the offset spans are going to provide useful savings, except in a few cases. So for most of the AGT samples (which mainly use small sprites), there won't be much, if any speed gain. For sprites in this domain, Anima's method remains the optimial one.

The primary benefit with this upgrade is for big sprites. The bigger the sprite, the more time saved.

The clearing time here is exaggerated because it uses the default (rectangle) restore. In the second case, the clearing time is actually larger than the drawing time (!). There are smarter clearing methods in the engine - so for bigger sprites, this isn't necessarily the case. I just didn't want to complicate the test.

While the sprite uses only 1 colour, its still a 16-colour/4-plane sprite. No, I don't cheat for this stuff

I'm also working on a hybrid method which is similar to this, not quite as fast but requires less RAM and still way faster than the old method for big sprites. This will take a bit longer though, it is more complicated to implement.

[AdamK]

Are there current state of the art commercial compilers for 68k?

Usually I use commercial ARM compiler instead of GNU due to the code optimization and overall quality are far beyond free compilers.

Obviously, commercial ones can be expensive or hard to acquire... but I am curious about 68k

There are (see https://www.ghs.com/products/68k_development.html).

[masteries]

I'm still gradually making my way through the sprite upgrades. Clipping this stuff is a nightmare!

However the progress has been good.

Here's a before/after using a worst-case sprite. Well not quite worst-case but a poor case for sure.


before.png


The sprite is a single 256x128 cut, with independent regions and an internal line gap. The old system could cope with internal line gaps but only where a component (column) has completely empty lines.

Previously a component was a 48-wide rectangle of any height (32-wide for the first/leftmost component). In the example sprite, 5 components were needed to cope with this shape (corresponding to the 5 green timing bands). The 48-wide optimization really helped here - it would have been 8 components for 32-wide columns!

The components/columns could be different heights, so time is saved per component. But still, fitting these circles into adjacent rectangles is not efficient.

Now look at the results for the new version...


after.png


A single component, rendered in less than half the time. This is the benefit of using arbitrary-length, offset spans instead of the fixed-width columns. Far less data being moved.


Some notes...

The improvement on small sprites is neglegable or zero - if the sprite is no wider than 32 pixels, there is no guarantee the offset spans are going to provide useful savings, except in a few cases. So for most of the AGT samples (which mainly use small sprites), there won't be much, if any speed gain. For sprites in this domain, Anima's method remains the optimial one.

The primary benefit with this upgrade is for big sprites. The bigger the sprite, the more time saved.

The clearing time here is exaggerated because it uses the default (rectangle) restore. In the second case, the clearing time is actually larger than the drawing time (!). There are smarter clearing methods in the engine - so for bigger sprites, this isn't necessarily the case. I just didn't want to complicate the test.

While the sprite uses only 1 colour, its still a 16-colour/4-plane sprite. No, I don't cheat for this stuff

I'm also working on a hybrid method which is similar to this, not quite as fast but requires less RAM and still way faster than the old method for big sprites. This will take a bit longer though, it is more complicated to implement.

Its more than excellent progress!

Clearly, clearing process needs improvement; also the Occlusion method can help here; but obviously clear a giant rectangle shall be no efficient.
IIRC you suggested a slab restore based method for these.

[dml]

Clearly, clearing process needs improvement; also the Occlusion method can help here; but obviously clear a giant rectangle shall be no efficient.
IIRC you suggested a slab restore based method for these.

Yes, and this part is now implemented I'll forward some details very soon.

[masteries]

Some of the work in progress performed today...

The Morden´s army is taking the STE territory, also on foot:






On this last image, the nearest soldier is using the back part of the rifle against the player,