Scaler

[ghogan42]

The menu option for changing the coefficients is actually a pretty nice user experience feature. The coeff.txt method provides the maximum flexibility and the ability to experiment and grow, but having that menu option with those presets makes life a lot easier, especially for less experienced users. Maybe some middle ground could be found?

I assume we could have several built in options and then a "custom" option that loads the coefficients from the coeff.txt. That way, most people could just pick their favorite off of the list and advanced users could still have use whatever they want to make.

But that's getting ahead of ourselves because right now, all of the filters are implemented in a different way then the "16 phase 4 tap polyphase" way. Right now, Grabulosaure has coded up each algorithm from scratch from the mathematical description of each algorithm. So we'll see what work can get done. I'll at least try to add such a filter, but I don't know how long that would take me. I coded up the sharper bilinear algorithm I wanted, but in that same time, Grabulosaure already did a ton of work on the scaler. So other people will probably be faster than me if a generic polyphase filter too.

[Sorgelig]

I've started to integrate ascaler to LITE revision.
Request for width/height of active area is removed - scaler already supports it. So, i've implemented aspect ratio/integer scaling already.

Currently i see only one flaw in this scaler: corruptions on edges. I don't know if i will be able to fix it. I will try. But would be good if Grabulosaure would fix it as he is more familiar with the code.

I didn't look into possibility of coefficients replacement, but i see tables in the code, so it should be doable.

[Sorgelig]

Another problem is inability to downscale. Some cores with enabled HQ2x may produce the video bigger than scaled one.
Is it hard to add downscaling?

[Sorgelig]

Resources for current implementation of ascaler against VIP:

ascaler/vip:
ALM: ~1.4K/14K
M10K: 28/136

So, it's a big save in resources!

[Sorgelig]

I've pushed changes with scaler to Genesis repository.

[Sorgelig]

Trying to understand the scaler code. If i understood correct, the scaling process doesn't care of edges, so pixels supposed to be outside the visible area on edges are taken from wrong place. Probably it should be taken from the same pixel or black - i don't know which is better for scaling algo.
I had similar problem with HQ2x scaler. It uses 2x2 pixels. So, for first line i set 2 buffers to the same locations. Pixel count is delayed by 1 and get the first pixel from the blanking.

[ghogan42]

Trying to understand the scaler code. If i understood correct, the scaling process doesn't care of edges, so pixels supposed to be outside the visible area on edges are taken from wrong place. Probably it should be taken from the same pixel or black - i don't know which is better for scaling algo.
I had similar problem with HQ2x scaler. It uses 2x2 pixels. So, for first line i set 2 buffers to the same locations. Pixel count is delayed by 1 and get the first pixel from the blanking.

Trying to understand the scaler code. If i understood correct, the scaling process doesn't care of edges, so pixels supposed to be outside the visible area on edges are taken from wrong place. Probably it should be taken from the same pixel or black - i don't know which is better for scaling algo.
I had similar problem with HQ2x scaler. It uses 2x2 pixels. So, for first line i set 2 buffers to the same locations. Pixel count is delayed by 1 and get the first pixel from the blanking.

Since all of the filters are weighted averages of source pixels, if you extend past the edge with black pixels then the areas near the edges will be too dark compared to the center. Usually people will either duplicate the last pixel or mirror the values:

Code: Select all

Original:   318 319 320
Black:      318 319 320 | Black Black Black
Duplicate:  318 319 320 | 320 320 320
Mirror:     318 319 320 | 320 319 318

Probably duplicating pixels is fine.

BTW just to try to become familiar with VHDL and the ASCAL code (is it called Temlib Avalon sccaler?), I made a test where I implemented a few of scalers that I previously produced coeff.txt files for. Just the variations of bilinear with and without scanlines. I uploaded my test to my google drive here: https://drive.google.com/drive/folders/ ... tfQLi2xvVQ. There is a zipped Genesis source folder with the modified "ascal.vhd" with different filters than the default. One of the scanline versions is broken in that file I think,but the others work. You can take a look at those and see which scaling filters you would be interested among all of the variations: bilinear, Quilez bilienar, sharp bilinear, bicubic, lanczos, etc. Also the scanline versions in that file just loads the coefficients and then multiply them. So that's pretty close to the polyphase filter you want -- without the hard part of loading coefficients from somewhere else.

I haven't learned enough yet to do what you talked about regarding loading coefficients from somewhere else so I can't do that yet. But I can write filter functions. Even that is slow because I don't know hardware/digital logic. To understand the biliniear implementation in ascal I had to get out a piece of paper are write out binary numbers and trace through it line by line!

I might be able to make a pull request, but I don't know which scaler types you would want to include in the OSD menu. I don't want to make you useless pull requests. Could you try out my old compiled Genesis Core from the google drive and let me know which ones are worthwhile to include? This is all from before you added ascal to the Genesis core yourself (if you wonder why aspect ratio is not working and so on in my compile). I would redo it based on current source.

[Sorgelig]

I'm not sure what exactly i need to test?
Altera scaler with your coefficients or compile the sources from your archive and test?

[Sorgelig]

I'm more toward coeff files than hardcode it to core.
it's not that coeff file can be the single file. It's possible to have several one and let the user choose from OSD menu from system settings.
This way scaling will be consistent through all cores. And no need to add scaler option in OSD of every core.

[breiztiger]

good idea

one directory like scaler at home and several files for coefficients params

and with that anybody can create his prefs and test

[Sorgelig]

Sharp scanlines looks very good! Just like real scanlines. Sharp bilinear looks ok, although i'm not fun of bilinear filtering. I prefer either NN or Polyphase with Lanczos 2 from last ascal version.
But as i've said it's most likely coefficients will be uploaded from MiSTer for flexibility.
Current task is to fix garbage on the edges. Current implementation use full stop and full run at the edges while originally rendering should be started 1 or 2 pixels in advance and stop 1 or 2 pixels after active part and then clipper will cut it at the proper resolution leaving the garbage outside.

[ghogan42]

I'm more toward coeff files than hardcode it to core.
it's not that coeff file can be the single file. It's possible to have several one and let the user choose from OSD menu from system settings.
This way scaling will be consistent through all cores. And no need to add scaler option in OSD of every core.

What I was meant to ask is if a pull request with some of the variations of filters would be welcome. So if you tried on the compiled version on my drive, then you could tell me which ones to include in the pull request.

But you're correct that probably we should hold off until a method that takes arbirary coefficients is produced, so that the code does not have to contain separately implemented filters that only vary in the coefficients used. Then the OSD option would just choose which coefficients to load. And we could even keep an option to load them from disk.

If I do this, it may take a week (or more!) because right now, I don't even know how to do things like load a set of coefficients into a block of memory and then read that memory from a function in VHDL. So it will be slow going while I learn.

In the mean time, maybe you could temporarily fix the "ghogan" (bad name for that btw) function in ascal.vhd? It's produces incorrect output right now.

Code: Select all

  -----------------------------------------------------------------------------
  -- Ghogan
  -- Shift Bilinear weights away from the center (towards 0 or 1) to
  -- make scaling sharper
  FUNCTION gho_calc(frac : unsigned(11 DOWNTO 0)) RETURN unsigned IS
  BEGIN
  	case frac(11 DOWNTO 8) is  --We don't need so many bits
		when "0000" => Return "00000000";
		when "0001" => Return "00000001";
		when "0010" => Return "00000010";
		when "0011" => Return "00000110";
		when "0100" => Return "00010000";
		when "0101" => Return "00100000";
		when "0110" => Return "00110110";
		when "0111" => Return "01010101";
		when "1000" => Return "10000000";
		when "1001" => Return "10101010";
		when "1010" => Return "11001010";
		when "1011" => Return "11100000";
		when "1100" => Return "11110000";
		when "1101" => Return "11111010" ;
		when "1110" => Return "11111110";
		when "1111" => Return "11111111";
    end case;
  END FUNCTION;
  
  SIGNAL o_h_gho_pix,o_v_gho_pix : type_pix;
  SIGNAL o_h_gho_frac,o_v_gho_frac : unsigned(7 DOWNTO 0);

EDIT: The reason sharp bilinear is necessary is because it add just enough blur to nearest neighbor to hide non-integer nearest neighbor artifacts.
Like when you scale [P ,Q, R, S] with nearest neighbor by 3.5 factor you get:

[PPP QQQQ RRR SSSS]

which looks terrible in motion

And with linear you get a blurry mess.

Sharp bilinear hides the uneven scaling without being blurry like bilinear/bicubic. This (or maybe a good lanczos) is imo what should be the default output because Nearest Neighbor has artifacts and Bicubic/Bilinear are pretty ugly. But I guess defaults don't matter too much.

BTW, what is called "ghogan" in ascal (fixed function above) is "Sharp Bilinear"

[Sorgelig]

Ok, i will fix this filter and will rename it.

[ghogan42]

Ok. I'm starting to understand the code. I originally ignored all of the bicubic and and lanczos implementations assuming they would be too hard for me to understand. but I see now that lanczos is already implemented as a polyphase filter.

I'm going to make an attempt at implementing all of the available filters through the polyphase part of the filter code. With all of the filters implemented like this, then you could either keep the hardcoded filters (everything but lanczos and an currently unused set of bilinear coefficients) or throw them away to save FPGA space. Right now, I guess that switching scaling methods is swapping between several filtering modes that are always running. If you only keep the polyphase filter, then changing filters will only swap in a different set of coefficients. That seems more efficient.

[Grabulosaure]

I have uploaded a new version.

- Fixes to edges, bad synchronisation (but not quite perfect yet)
- Downloadable polyphase coefficients
- Many details, such as subpixel resolution.

Be patient!

Another problem is inability to downscale. Some cores with enabled HQ2x may produce the video bigger than scaled one.
Is it hard to add downscaling?

Downscaling will be fixed. At least for nearest neighbour and linear.
This scaler stores the source image in RAM (whereas the VIP scaler stores the scaled image in RAM).

With upscaling, adjacent source image pixels are reused, so for each horizontal and vertical line, between 0 and 1 horizontal pixel and vertical line is needed. with downscaling, non-adjacent pixels must be fetched, several input pixels must be processed for each output pixel, which is a memory bandwidth problem.
I will probably add hardware in the input part to do the linear interpolation, and store in RAM the shrinked image to support arbitrary downscaling.
Is bicubic/poly really needed for downscaling ? Maybe instead averaging over the colour of two or more pixels?

Besides fixing all edges and downscaling, which other important part is missing ? Maybe a short latency mode, with a fixed vertical refresh period locked to the input signal ?
(I also need to look at frequency limits, maybe some parts will need deeper pipelining or replicated hardware. Anyone tried with high resolution outputs ?)

I'm going to make an attempt at implementing all of the available filters through the polyphase part of the filter code.

If the polyphase mode can emulate well enough all other filters, it can be used exclusively, it would save resources in the FPGA. I coded BiCubic first, it is the most complex, full of fixed point arithmetics, then polyphase was coded in less than an hour, and I couldn't find any difference with default coefficients with BiCubic on my small screen.

[ghogan42]

If the polyphase mode can emulate well enough all other filters, it can be used exclusively, it would save resources in the FPGA. I coded BiCubic first, it is the most complex, full of fixed point arithmetics, then polyphase was coded in less than an hour, and I couldn't find any difference with default coefficients with BiCubic on my small screen.

I've coded a number of these filters as Pixel Shaders for use in opengl and direct3d to scale pixel graphics with various effects. My only exposure to a polyphase filter was the VIP filter in MiSTer. But I couldn't really find a difference either between the polyphase implementation and the pixel shader versions that use normal floating point math to compute the exact coefficients. I've seen low precision in low-end 3d hardware (like the Nintendo Classic Editions) cause problems and it pretty much always shows up the scanline versions of the shaders. But the polyphase versions I tested didn't have these problems. Not in the VIP scaler or in your scaler either. So for myself I'd only want/need the polyphase filter.

If it were up to me, and I was writing the scaler for use in many projects, I might keep something simple like NN and/or bilinear separate and do everything else with the polyphase filter so that there would be something to fall back on if for some reason the polyphase filter didn't work for something. But I don't know if makes sense. After all, all of the currently implemented filters can just be a set of coefficients for the polyphase filter.

I don't know if you saw the versions I made scanlines for the VIP shader or the test compile of the Genesis core I made of your scaler on my google drive: https://drive.google.com/drive/folders/ ... tfQLi2xvVQ but maybe it would be a good idea to allow selection of the scaling method on the x-axis and y-axis with separate OSD options. That way, options like your "Linear Horizontal + NN Vertical" or "Bicubic Horz + 20% Scanlines Vertical" would be easier without having a million different options in a single menu item.

Anyway, I just want to say: Great Work! Your scaler started out good and keeps improving so quickly! And I'm sure other future fpga projects will probably use your scaler too because it will be the highest quality open source option. So you're making a huge contribution to the community with this.

[Sorgelig]

Just woke up and didn't check the new version of scaler yet - will give feedback later.

Just want to write my opinion about filters.
I think, it should be enough for scaler to use just polyphase loadable coefficients. NN coefficients can be default if none is sent from MiSTer. I don't know if Sharp Bilinear can be implemented in coefficients as well or not. If not, then it worth to include as a separate filter. It looks good alternative for those who don't like to see ring effects on Lanczos filter.

For downscaling - polyphase is used as a low-pass filter. Probably simpler method will be OK as well. Currently i have no much experience in downscaling artifacts. So, need to see implementation and then it will be more clear what filters can be used. Usually downscaling is rare. If 1920x1080 output resolution is used, then downscaling should not be happened. But since sometimes downscaling happens, it's better to implement it at some acceptable level to avoid bad output.

[Sorgelig]

I have uploaded a new version.

this version is much better!
Still have edge problems:

1) upper line looks ok.
2) bottom line has been cut.
3) left column displays wrapped line from the right.
4) right column either cut or displays the garbage depending on aspect ratio (and probably resolution).

There is also dot on left top corner. Probably it belongs to left wrapped column.

Please use my latest commit of Genesis. Your scaler is already there with aspect ratio implemented. Switching aspect ratio back/forth shows the left/right wrappings and garbage.

For testing the edges I use 2 main apps:
240p Suite (v1.17):
https://sourceforge.net/projects/testsu ... CD_MegaCD/
You can check test patterns there (red-white cells grid)

Sonic 1 game:
It has moving background, so easy to check the wrappings not visible on test pattern.

Please also check the lines on edges to make sure they have the same width as other lines.

[ghogan42]

I made a compile with Bicubic, Sharp Bilinear and a couple of Scanline options implemented through the polyphase filter. I think they look good. Now it's already out of date
That build is here: https://drive.google.com/open?id=1YyNr3 ... BBfjZqEC1V

Since then, there are already updates with coefficient loading?! Wow, things move fast. Probably I'll have to read the new code and I'll learn how redo things better than I did this time after I see how things work.

[Sorgelig]

I see scaler calculates ratio input_res/output_res and then add this ratio at every step. I think this is the source of error at the end of line/frame. Probably calculation of input pixel with every output pixel will remove this error as the calculation at the end will give correct end on input_res. Not sure if multiply/divide can keep be done at output pixel clock, but may be it can be simplified somehow?

[Sorgelig]

fixed top,bottom and left. Right column is hardest part.. Hope to fix it.

[cacophony]

Besides fixing all edges and downscaling, which other important part is missing ? Maybe a short latency mode...

Yes, this. After the bugs are fixed, having a version with no frame buffer would be the next important milestone IMO.

[Sorgelig]

Yes, this. After the bugs are fixed, having a version with no frame buffer would be the next important milestone IMO.

May be you want to be able to levitate as well? Don't be shy
No frame buffer means output frame rate should match input frame rate. HDMI standard has strict definition of parameters. It's more strict than analog TV to which all retro systems were developed.
If you lock the output framerate to input then you have to change one of HDMI parameters to alter frame rate. Not all monitors and TVs support this feature.
MiSTer already has this feature called vsync_adjust in ini file.

With altera scaler it's impossible to control the latency. With this scaler it will be possible to make output frame start like several lines later at the same frame as input to minimize the latency. It won't require any change in scaler as it's external sync operation. Probably just one signal holding the scaler sync till some point.

It doesn't matter having frame buffer or not. It's only matter when you will start to read from buffer.

P.S.: even with pixel clock adjust to tweak output frame rate, it's impossible to make it exactly match as output pixel clock won't be multiple of input pixel clock. It means output will slowly run away or late relative to input frame. With close clocks match and additional frame between, the tearing can be eliminated. You will just see one frame drop or repeat like once per hour which you won't notice.
It's possible to force wait input vsync before start output sync - but it will introduce one more out of HDMI standard quirk and reduce amount of compatible TVs/Monitors even more.

[cacophony]

Yes, this. After the bugs are fixed, having a version with no frame buffer would be the next important milestone IMO.

May be you want to be able to levitate as well? Don't be shy
...

It doesn't matter having frame buffer or not. It's only matter when you will start to read from buffer.

...

yeah, obviously I meant a mode that doesn't depend on having a full frame buffer, or depends on it as little as possible.

Here are the three modes that the Super Nt offers, from the excellent unofficial SNT guide by Great Hierophant:

"The buffering modes on the Super Nt address the issue that the SNES’s native refresh rate is
60.0988fps. Modern displays and HDMI interfaces expect either 60fps or 59.94fps and are
generally not tolerant of refresh rates that vary from that specification. The [Super Nt] has three
modes that deal with the frame rate differential in different ways.

1. The Fully Buffered option buffers full frames to achieve a 60.0988 frame rate using only 60fps.
This is mode avoids tearing at the cost of latency. The Super Nt must render at least 1 frame
ahead of the game’s internal rendering to stay ahead.

2. The Zero Delay slows the SNES down to achieve a true 60fps frame rate, a speed difference
of 0.16% There is no latency penalty with this method but this method causes the Super Nt to
fall 1 second every 10 minutes behind an original SNES running the same software. This
time delay makes it non-ideal for speedrunning.

3. Finally, the Single Buffer option is something of a compromise between the two methods
described above. Like the Full Buffer option, the correct frame rate is being generated within
the Super Nt. Unlike the Full Buffer, only a portion of the next frame is being pre-rendered,
giving latency of no more than 1 frame depending on when the player activates an input. The
drawback is a recurring retrace line that is visible once per every several seconds."

Almost everyone wants the zero delay mode because zero lag is more important then running at precisely the correct speed.

[Sorgelig]

You can immediately delete option 2. No one will slowdown or speedup the systems in favor of matched frame rate. And it's not always can be achieved. So, simply forget about this option as universal one. MiSTer is not a single core device.