Test Input lag: Megadrive vs MiSTer FPGA and also Megadrive vs RGB-Pi... Which one?

[Newsdee]

Everyone that plays retro games knows that low input lag is needed.

I like healthy discussion but i REALLY hate this crappy lag topic which is based only on imagination of hypothetical situation.
Users with lag obsession may go to look for another system. No one keeps you using the MiSTer which has so bad lags making games unplayable for you (and only for you).

Some people will always complain. There were complaints on the MiST that it had no HDMI, then on MiSTer that it had no VGA out of the box, etc. etc.

There is however one factor that can be a genuine source of lag: crappy/cheap USB adapters with a low polling frequency. This guy makes Neogeo/Atari adapters and advertises they run at 1 Mhz: http://www.2600-daptor.com/index.htm . I can confirm from personal experience with a bad adaptor that this can be a problem.

I wonder if anybody would be interested in using the FPGA and serial IO abilities of MiSTer to make an adapter test suite? For example, the core could use serial to send a signal that is connected to a gamepad button on the gamepad PCB, and measure time difference from getting the input command via USB. I'm not sure if serial has the right voltage for that, though.

[SaschaFFM]

I do not notice any lag on games that I did not (regularly) play on the real hardware. When there is no comparison from my muscle memory.
But then there is Kick Off 2 on the Amiga. Must have played it thousands of hours (since the early 90s up to today). I would say I play at a decent level and the extremely finicky movement of the players has gone to my subconscious.

And playing via HDMI (that is on a "lagless" gaming-monitor or any other HDMI-Monitor) the controls behave "muddy". It is not that it adds to my reaction, but more that when I press a direction there is some delay until the players sprite follows my command. With all the micro-management that is happening and with the muscle memory trained over the years - you just feel that something is off. Feels "muddy" as I said above.

This may not be experienced by everybody. And probably not with every game. But there are some that do notice it. And I would say it is not fair to say it would only be only our imagination.

That is one of the two reasons that I am happy that there is a VGA output. I do not see any difference via VGA to the real hardware. If there is an USB-Delay then it is to small for me to notice. Perhaps there are people that notice even that. I play guitar - not as good as I play Kick Off 2 - and there are far better musicians than me who have an incredible feeling for timing. But just because I lack these abilities it is still true that others have them. Not everybody is equal.

[cacophony]

The brain only tricks you into perceiving less input lag then there really is. The real reaction time is still longer with higher input lag and games which needs short reaction time like Punch Out will be much more difficult.

it looks like you didn't read what you've quoted.
In regular predictable reaction brain compensates the WHOLE chain of lags.
In unpredictable reactions adding 2-3 frames of lag on top of ~250ms of human lag makes no difference.

The brain compensates in many situations (eg. a far off pit you're approaching to jump over), and in that case the game will just feel tiny bit less natural/responsive when the lag gets large enough.
But you're incorrect in situations with an unpredictable event. If your typical human lag is 250ms then every bit of added input/scaler/display lag will only make that reaction progressively more difficult until it gets to the point where it's actually impossible. In Mike Tyson's Punchout you have only 13 frames to hit dodge after indication to avoid being knocked down. That's only 216ms. So if your human reaction time is 250ms you will never beat Tyson, even on original hardware connected to a CRT. Somebody with quicker reactions might be able to react in 200ms, and in that case adding just a single frame of lag will make beating Tyson impossible. And for somebody with a faster reaction time, they'll just see their rate of success decrease, because reaction times do vary. So it will absolutely have a noticeable affect on gameplay.

I understand that you don't want to pursue lowering lag yourself, but perhaps we could make a thread where options for reducing scaler lag can openly discussed with Grabulosaure?

[Newsdee]

And for somebody with a faster reaction time, they'll just see their rate of success decrease, because reaction times do vary. So it will absolutely have a noticeable affect on gameplay.

I understand that you don't want to pursue lowering lag yourself, but perhaps we could make a thread where options for reducing scaler lag can openly discussed with Grabulosaure?

I think the real problem is lack of objective measures and the implications of that.

Scaler lag is one thing, the XRGB adds one to two frames of lag and that's pretty good, many people were happy with it. Then you got the OSSC which is lagless, but it also can't convert 50hz to 60hz or vice-versa. That is not good for me, but its fine for some people. Then some people dislike LCD TVs altogether (and some TVs are truly terrible to be fair) so a CRT is the only acceptable option.

So you already have three camps in this area, and whoever deals with this "lag" issue will never see the end of it because of people complaining for different things (from different goals and assumptions).

[cacophony]

So you already have three camps in this area, and whoever deals with this "lag" issue will never see the end of it because of people complaining for different things (from different goals and assumptions).

My hope is just to allow open and constructive discussions about possible approaches.

[Newsdee]

My hope is just to allow open and constructive discussions about possible approaches.

I'd love a way to test the various USB adapters and controllers I have. Some may be worse than others. Maybe we need a controller test core, haha.

For video there is the 240p test suite:
https://sourceforge.net/projects/testsu ... rce=navbar
One can attach a display to HDMI and another to a CRT via RGB, then film both with a high speed camera like on an iPhone (which I think can record at 240fps).

[bluescrn]

Everyone that plays retro games knows that low input lag is needed.

Yes. And i've played a lot of games on MiSTer using USB wireless pad with HDMI output and don't notice the lag at all.
So, i know what i'm talking about.

I like healthy discussion but i REALLY hate this crappy lag topic which is based only on imagination of hypothetical situation.
Users with lag obsession may go to look for another system. No one keeps you using the MiSTer which has so bad lags making games unplayable for you (and only for you).

Latency isn't hypothetical at all. It's a very real issue. I got interested in FPGA-based retro gaming after very poor experiences with emulation on a Raspberry Pi (combined with a laggier-than-average TV). The setup must have had around 100ms total latency between input and screen response, and you can *really* feel that when playing fast-paced shmups or platformers - you just can't dodge bullets or jump precisely.

It's not all about human reaction times. While it may well take something like 250ms for a human to *react* to something, we can execute a pre-planned move (such timing a jump) with much more precision - perhaps within a 2-3 frame window.

But if you're noticing lag, the first thing to look at is your TV/monitor. Some LCD screens can add as much as 60ms even in game mode (If you really care about the issue, you can buy a gadget to measure display lag: http://www.leobodnar.com/shop/index.php ... cts_id=212)

[cacophony]

My hope is just to allow open and constructive discussions about possible approaches.

I'd love a way to test the various USB adapters and controllers I have. Some may be worse than others. Maybe we need a controller test core, haha.

For video there is the 240p test suite:
https://sourceforge.net/projects/testsu ... rce=navbar
One can attach a display to HDMI and another to a CRT via RGB, then film both with a high speed camera like on an iPhone (which I think can record at 240fps).

To be clear, I meant discussion of possible approaches for lowering lag, not measuring it. I know Grabulosaure has some ideas for approaches that he's interested in pursuing, but there are still some open questions.

[Newsdee]

To be clear, I meant discussion of possible approaches for lowering lag, not measuring it. I know Grabulosaure has some ideas for approaches that he's interested in pursuing, but there are still some open questions.

To me it is part of recognizing and solving the problem. How do we know the approaches are working? How do people know on their own if they have an optimal setup with their gear?

But more importantly, how bad is the lag that it needs to be such an important issue? If it is one to two frames (which puts the MiSTer scaler in the same category as the XRGB) I would say that it is already amazingly good.

I am all for open and objective discussion, and I am sure the upscaler will see more improvements, but it annoys me when I see people (not you) in some forums blaming the MiSTer for being "laggy" without ever having seen it or without being willing to make objective comparisons.

[Sorgelig]

I am all for open and objective discussion, and I am sure the upscaler will see more improvements, but it annoys me when I see people (not you) in some forums blaming the MiSTer for being "laggy" without ever having seen it or without being willing to make objective comparisons.

That especially flooded with Genesis perfectioning and SNES core release. Looks like commercial competitors are unhappy with open source MiSTer. So, time for trolls

I've played many games recently on PCE, Genesis, SNES with full blown so-called lag producing chain: USB wireless gamepad with HDMI output and triple buffer scaler. And i had no problems with reactions.

Complainers going to complain anyway.
Complains are targeted to lag in general but not about MiSTer specifically. This is what is called non-constructive discussion.

P.S.: "Punch out" obsessed users may go to find other solutions I never played this game and have no interest in it.

[SaschaFFM]

I have been trying to measure the lag with a more "scientific" approach.

Test-Setup:

Latest NES-Cores on MiST and on MiSTer (as of December 23rd of 2018). Game used for testing was 60hz Super Mario Bros (MD5-Checksum: 811b027eaf99c2def7b933c5208636de). No scanlines or other non-standard features where activated.

Many people think that older console/computer games were next-frame. That is (mostly) not true. Every game had some inherent delay. This can be tested on Retroarch by pausing the game (default: "p"), pressing a button and then advancing frame-by-frame with the "k"-button and count the frames until Mario jumps. I tried this approach with Super Mario Bros. on NES (above version) on World 1-1 on the initial screen with the jump-button. It was *always* two frames. No matter if it was a lower or a higher jump.

Two frames on 60hz translates to approx. 33-50ms of delay. Depending on when exactly the button was pressed during building of the screen, it might even be a bit faster.

Now I used the slow-motion camera of my iPhone XS and took a video that captured my iBuffalo-Classic-USB-Controller and the Mario Sprite. I pressed the jump button several times in a row. Downloaded the video to Final Cut and counted the frames (240 fps on iPhone Slow-Mo) between the button pressed and Mario jumping. There is probably a slight issue in telling when exactly the button was pressed. But this should not be more than one frame (approx. 4ms).

First test was with MiST connected via VGA to BNC-Cable to my Sony CRT.

# Frames between button pressed and Mario jumped (240fps-Frames, so 4,16ms per frame)
8
9
6
11
6
11
8
9
10
7
9
9
10
10

Numbers were from 6 to 11 frames (that is 240fps-Frames!). If you devide by 4 you will get 60hz-Frames.
This is exactly in the range that you would expect from a 2-frame-delay.

Then I did the same test with MiSTer:

# Frames between button pressed and Mario jumped (240fps-Frames, so 4,16ms per frame)
10
9
9
8
8
9
12
7
7
11
7
6
10

Those are just the same numbers as on MiST. The MiSTer itself does not introduce any lag. Neither to the original console, nor to MiST. Of course you would want to test with other cores. I picked Mario and the NES because they are easy to test and the core exists on both platforms.

Next up will be a test with HDMI. This will be harder to measure as I obviously cannot take the input-lag of my HDMI-Monitor out of the test. It will always add to the delay that comes with MiSTer. I expect the numbers to be higher than on VGA>CRT setup.

[Locutus73]

Many people think that older console/computer games were next-frame.

Absolutely not, AFAIK most NES games and Super Mario Bros in particular had 2 frames (60FPS) of input lag (real hardware on CRT), i.e. watch this https://www.youtube.com/watch?v=_qys9sdzJKI (please ignore the RetroArch final part of the video)
and most SNES games, and Super Mario World in particular had 3 frames.
Approaching real hardware results, while not being obsessed and considering pro and cons of each option (i.e. buffering for converting frame rate without tearing) should be a reasonable goal. In addition an open project like this can offer many different options with different trade offs (some option with more lag, but more compatibility and other no compromise options with less compatibility).

Next up will be a test with HDMI. This will be harder to measure as I obviously cannot take the input-lag of my HDMI-Monitor out of the test.

Maybe you could using vga_scaler=1 and CRT, so you force MiSTer to use the same scaler as HDMI in its VGA out and you can rule out your HDTV lag using your Sony CRT.

P.S.: "Punch out" obsessed users may go to find other solutions I never played this game and have no interest in it.

Punch out isn't an obsession, it is just an example and a test bed for providing some "scientific" approach to a discussion that easily slips into fanboyism (i.e. this is better than that) and fierce debate, while it should be remain quiet, objective and peaceful.

Regards.
Locutus73

[SaschaFFM]

Second part of my tests. This time with HDMI. Again used the same 60hz version of Mario Bros as above on the NES. Hooked up the MiSTer to my iiyama ProLite B2780HSU. This is advertised with 2ms grey-to-grey. No idea about the input lag.

Did the same series of tests. Video-Out was set to 1920x1080p at 60hz. Used the internal filter as HDMI-Scaler.

# Frames between button pressed and Mario jumped (240fps-Frames, so 4,16ms per frame)
14
18
17
14
17
15
14
17
18
17
15
17
18

The numbers are higher, but honestly I expected them to be even further away from CRT. They translate to approx. 2 frames of added lag compared to VGA>CRT or the original console.

Again - I cannot tell how much my monitor contributes to these numbers. If I should try any other filters, please tell me. I do not follow this topic, as I am more of a CRT-guy.

I might give the idea of Locutus73 a try and will activate the scaler for VGA.

[Sorgelig]

here is my draft article about the subject: https://github.com/MiSTer-devel/Main_Mi ... oth-output

[ijor]

Many people think that older console/computer games were next-frame.

Absolutely not, AFAIK most NES games and Super Mario Bros in particular had 2 frames (60FPS) of input lag (real hardware on CRT), i.e. watch this https://www.youtube.com/watch?v=_qys9sdzJKI (please ignore the RetroArch final part of the video)
and most SNES games, and Super Mario World in particular had 3 frames.

I don't know much about the NES or the SNES. But many older 8-bit systems do have a "next frame" reaction. That is, the output of a specific frame is already affected by the input from the previous frame. In some cases the software even reacts immediately right after the joystick switch is detected by the hardware. Not sure how common or rare this is.

Second part of my tests. This time with HDMI.
...
Did the same series of tests. Video-Out was set to 1920x1080p at 60hz. Used the internal filter as HDMI-Scaler.
...
The numbers are higher, but honestly I expected them to be even further away from CRT. They translate to approx. 2 frames of added lag compared to VGA>CRT or the original console.

There is not much need to need the measure the lag produced by the HDMI output. As already said, it will oscillate between one and two extra frames. This is a direct consequence of the triple buffer mechanism. The frequency of the oscillation in turn depends on the relation between both the input and output refresh rates. If vsync is enabled it would be very low.

There might be an extra latency added by the pipeline logic. But this should be minimal and on the order of a couple of scan lines in the worst case.

This is, of course, just the HDMI output without considering delays by the monitor itself.

[paulbnl]

P.S.: "Punch out" obsessed users may go to find other solutions I never played this game and have no interest in it.

You will try to combine the low latency scaler with vsync_adjust right? When that is working people will be happy to play Punch out on MiSTer through hdmi.

[kitrinx]

I've played many games recently on PCE, Genesis, SNES with full blown so-called lag producing chain: USB wireless gamepad with HDMI output and triple buffer scaler. And i had no problems with reactions.

The lag as it stands is very good and the overwhelming majority of things are playable without issue. The MiSTer does an impressive job at creating compatibility for these systems with such a huge range of timings and resolutions. Although people rarely say it, I'm sure many appreciate this fact.

That said, the difference between 16.7ms of latency and 50ms of latency is not a lot, but it absolutely IS perceptible. Activities like wall jumping in Super Metroid or rotating pieces in Dr. Mario have timings that we're used to and are trained into us. I think there is a happy medium here where we can optionally enable single buffering or Grabulosaure's low latency mode, and perhaps optimize or thread the main loop to deliver lower latency input without having to sacrifice anything at all other than the time to implement. Lower is better if you don't have to give up anything for it, right?

On the other side of the aisle, there's also caring too much. The ~4ms of latency from USB polling is not going to be perceptible, and what you gain from this sacrifice is enormous. The ability to remap buttons, use any gamepad with any system, create virtual keys on the buttons, etc. Other FPGA systems only take one kind of controller input, typically input that is native to the single core the systems represents, so it's very easy for them to have no ADDED latency. It's important to remember that MiSTer has to be a LOT more adaptable than just about anything else in circulation, so constantly saying "xxx does it this way, can we do it that way?" is not really constructive, especially if you don't understand the technical details of how these things work. The people involved with working on the scaler and input are aware of what's out there and how it operates for the most part.

As a last note, there's no need to measure the video latency of the MiSTer. The code is open and you can look at what it does. It's going to be two frames in triple buffering mode, and this can be easily confirmed using two crt's, an hdmi to vga converter, and the 240p test suite's latency test (the one that shows the dots). Spoiler: Bob did this and found two frames. The ADDED input latency is usb_delay + main_loop_delay. The usb_delay, as far as I can tell, should be an average of around 4ms. The main loop delay is less clear to me, but it can't be more than 16.7ms as far as I can tell, and might be much less.

[ijor]

You will try to combine the low latency scaler with vsync_adjust right? When that is working people will be happy to play Punch out on MiSTer through hdmi.

I doubt he will. And honestly it is not fair to blame him. It is not easy at all to implement this universally for all cores. This probably will need to be left as an option to individual core developers.

I think there is a happy medium here where we can optionally enable single buffering or Grabulosaure's low latency mode, and perhaps optimize or thread the main loop to deliver lower latency input without having to sacrifice anything at all other than the time to implement. Lower is better if you don't have to give up anything for it, right?

If you disable triple buffering (the only way to avoid extra latency) and you don't match both refresh rates exactly, the one that outputs the emulated systems and the one used by the scaler, then you would get tear. So it is certainly not "free". I'm not sure how many people would like to sacrifice tearless video for low latency, but it might be left as an advanced option perhaps.

[kitrinx]

If you disable triple buffering (the only way to avoid extra latency) and you don't match both refresh rates exactly, the one that outputs the emulated systems and the one used by the scaler, then you would get tear. So it is certainly not "free". I'm not sure how many people would like to sacrifice tearless video for low latency, but it might be left as an advanced option perhaps.

In the sense that the option already exists in the scaler and has tangible, positive effects when enable and paired with vsync_adjust on, it's pretty low-cost. Single buffering at least. Of course disabling compatibility options is going to make the signal less compatible, that's obvious, but the choice is not hard to give people who don't have the experience to edit and compile cores. Give some of Grabulosaure's test builds a try and see how you feel. I think a lot of people would be made happy to simply have the choice.

[Sorgelig]

vsync_adjust has nothing to do with low latency.
It's FPS lock, but it's not 100% match. So HDMI refresh will slowly drift forward or backward. You will see one frame drop or repeat once per several hours. So you cannot fully lock the refresh.
So doing this with single buffer, and you will see tearing slowly drifting up or down.

[vanfanel]

I would like to point to the lagless solution the SuperNT uses: it moves the SNES refresh rate to an HDMI-compatible mode. The implemented system, when this option is activated, is a bit slower than the original NTSC system, but the difference is very small so it can't be noticed while playing, yet we get perfectly smooth scrolls and no desyncs ever.
What I am trying to say that an option like this would be a good compromise for original system lag while retaining smooth movement as in the original system.

[Locutus73]

I would like to point to the lagless solution the SuperNT uses: it moves the SNES refresh rate to an HDMI-compatible mode. The implemented system, when this option is activated, is a bit slower than the original NTSC system, but the difference is very small so it can't be noticed while playing, yet we get perfectly smooth scrolls and no desyncs ever.
What I am trying to say that an option like this would be a good compromise for original system lag while retaining smooth movement as in the original system.

For completeness sake: SNT zero lag mode doesn’t slow down the whole system, it underclocks by a 60/60.09 factor just the CPU and the PPU (for having a 60Hz refresh rate), while mantaining the SPC at the original clock (for running the music program without pitch alterations). It may appear a weird solution but it seems to work: the only known issue is with a homebrew demo, Bad Apple Demo, which apparentely plays a digitized music through a ring buffer filled by the CPU and read by the SPC through DMA (or HDMA or whatever is called); at the end of the demo the music has some loops, probably because the SPC surpasses the underclocked CPU in the buffer.
Anyway, solutions like these may be cool, leading to a few lines (sub 1 frame) buffering with zero tearing, but obviously they are not system wide (I mean global MiSTer solutions), they can be very specific options for some cores if not totally different compilations of these cores.

Regards.
Locutus73

[Locutus73]

The lag as it stands is very good [...]The main loop delay is less clear to me, but it can't be more than 16.7ms as far as I can tell, and might be much less.

Amen sister!

Locutus73

[kitrinx]

I would like to point to the lagless solution the SuperNT uses: it moves the SNES refresh rate to an HDMI-compatible mode. The implemented system, when this option is activated, is a bit slower than the original NTSC system, but the difference is very small so it can't be noticed while playing, yet we get perfectly smooth scrolls and no desyncs ever.
What I am trying to say that an option like this would be a good compromise for original system lag while retaining smooth movement as in the original system.

This is exactly what I was talking about two posts up.

[ijor]

I would like to point to the lagless solution the SuperNT uses: it moves the SNES refresh rate to an HDMI-compatible mode. The implemented system, when this option is activated, is a bit slower than the original NTSC system, but the difference is very small so it can't be noticed while playing, yet we get perfectly smooth scrolls and no desyncs ever.

We talked about that method already. It is not an universal solution that can be implemented globally to every core. So it's up to individual developers to implement this for specific cores.

Note that the MiSTer allows the user complete freedom to define custom video resolutions, refresh rates and pixel clock. This makes more difficult to match the refresh rates. And it is important to be able to output the optimal native parameters supported by the monitor. Otherwise some monitors might perform a scale internally, and that might add even more latency.

Btw, this method was implement by Mark (Foft) on his Atari 8-bit system long before the SuperNT. But again, it was a specific core running at a fixed resolution.