Scaler

[ghogan42]

I think LCD and scanlines filters should be updated to have darker parts on the edges instead of middle. This will outline original pixels and will have better outlook.

This is exactly what it's supposed to do! Both scanlines and the LCD effect are correct with my original coefficients for both VIP and ASCAL.

I think we must not be getting the same results on our screens!

For me, I always get the correct results with coefficients that I produce. But somehow you must get different results. And that's why we disagree about the correct coefficients.

Please tell me if you get these results. The top row is VIP and ASCAL from your release today with full/vip and lite/ascal with MY lcd coefficients. The bottom is with your modified coefficients that you posted Nov 15 above. Both are at 1920x1080 on a 1920x1080 TV. Sorry for some blur, but you can still see that I get correct results only with my original coefficient set. This is just the opening screen from 240P test suite.

VIP-ASCAL-LCD-Effect-Differences-Nov18.jpg

I used your release from today for the full version and I compiled the lite version myself. I erased Genesis stuff from my config folder just in case that made a difference.

To be perfectly clear. The top two pictures used this coefficient set:

Code: Select all

# range -128..128
# sum of line must not exceed the range!

# Even Sharper Bilinear with pixel outline
# effect (for LCD simulation)

# horizontal coefficients
   0, 128,   0,   0
   0, 128,   0,   0
   0, 127,   0,   0
   0, 125,   0,   0
   0, 120,   2,   0
   0, 110,   6,   0
   0,  95,  13,   0
   0,  76,  26,   0
   0,  50,  50,   0
   0,  26,  76,   0
   0,  13,  95,   0
   0,   6, 110,   0
   0,   2, 120,   0
   0,   0, 125,   0
   0,   0, 127,   0
   0,   0, 128,   0

# vertical coefficients
   0, 128,   0,   0
   0, 128,   0,   0
   0, 127,   0,   0
   0, 125,   0,   0
   0, 120,   2,   0
   0, 110,   6,   0
   0,  95,  13,   0
   0,  76,  26,   0
   0,  50,  50,   0
   0,  26,  76,   0
   0,  13,  95,   0
   0,   6, 110,   0
   0,   2, 120,   0
   0,   0, 125,   0
   0,   0, 127,   0
   0,   0, 128,   0

And the bottom two pictures use the coefficients below:

Code: Select all

# range -128..128
# sum of line must not exceed the range!

# Sorgelig Modified LCD

# horizontal coefficients
   0, 128,   0,   0
   0, 128,   0,   0
   0, 127,   0,   0
   0, 125,   0,   0
   0, 122,   0,   0
   0, 116,   0,   0
   0, 108,   0,   0
   0, 102,   0,   0
   0, 100,   0,   0
   0, 102,   0,   0
   0, 108,   0,   0
   0, 116,   0,   0
   0, 122,   0,   0
   0, 125,   0,   0
   0, 127,   0,   0
   0, 128,   0,   0

# vertical coefficients
   0, 128,   0,   0
   0, 128,   0,   0
   0, 127,   0,   0
   0, 125,   0,   0
   0, 122,   0,   0
   0, 116,   0,   0
   0, 108,   0,   0
   0, 102,   0,   0
   0, 100,   0,   0
   0, 102,   0,   0
   0, 108,   0,   0
   0, 116,   0,   0
   0, 122,   0,   0
   0, 125,   0,   0
   0, 127,   0,   0
   0, 128,   0,   0

[ghogan42]

After my post I think I figured out what you wanted.

I think that you want, for example,a set of coefficients that have all of the non-zero constants in column 2 like your nearest neighbor coefficients but with the darkened part changed to be at the edge of the pixel. You don't think that your previously posted LCD coefficients are correct. But you don't like mine either because you believe still that my coefficients have a shift built into them. Is this right?

I could make a set of coefficients like that, but I'm 99% sure that they would be incorrect. Which we've both posted about before...

But I'll try to explain how I understand it one more time. Here is what the VIP scaler doc says:

As can be seen in the figure, phase 0 centers the function over tap 1 on the x-axis. By
the equation above, this is the central tap of the filter.
• Further phases move the mid-point of the function in 1/Pv increments towards tap
2.
• The filtering coefficients applied in a 4-tap scaler for a particular phase are
samples of where the function with that phase crosses 0, 1, 2, 3 on the x-axis.

Here is a graph I made to help show the coefficient calculation is done by me. It's Lanczos2 but they all work the same.

Phase_Coeff.PNG

This is exactly like the one in the VIP scaler PDF docs but I've labeled many points. (well I have T1 centered at x=0 so numbering on the x-axis is off by 1)

Ph0 C0 is the C0 coefficient at phase 0 (Note that it is 0)
Ph0 C1 is the C1 coefficient at phase 0 (Note that it is 1)
Ph0 C2 is the C2 coefficient at phase 0 (Note that it is 0)
Ph0 C3 is the C3 coefficient at phase 0 (Note that it is 0)

Ph8 C0 is the C0 coefficient at phase 8 (Note that it is a small negative)
Ph8 C1 is the C1 coefficient at phase 8 (Note that it is 0.572)
Ph8 C2 is the C2 coefficient at phase 8 (Note that it is 0.572)
Ph8 C3 is the C3 coefficient at phase 8 (Note that it is a small negative)

Ph15 C0 is the C0 coefficient at phase 15 (Note that it is 0)
Ph15 C1 is the C1 coefficient at phase 15 (Note that it is a small positive)
Ph15 C2 is the C2 coefficient at phase 15 (Note that it is ~0.992)
Ph15 C3 is the C3 coefficient at phase 15 (Note that it is a small negative)


Compare this with MY calculated coefficients in the zip file I posted:
Phase 0: 0, 128, 0, 0
Phase 8: -8, 72, 72, -8
Phase15 0, 5, 127, -4

And they match perfectly if you scale by 128.

My point is the following:

(1) My picture matches the picture in the official scaler docs below.
(2) My picture also matches my coefficients
(3) Phase 0 is CENTERED on tap 1. That is why the center of the peak of the sinc function is at phase 0. Phase 8 is NOT the center of the pixel. Phase 8 is the edge between Tap1 and Tap2.
(4) All properly calculated coefficients that blend pixels have 2nd column start at 128 and go towards 0 by phase 15. And also have column 3 start at 0 and towards 128 by phase 15.
(5) Your coeff_nn and coeff_pp on github (not anything in the forum) match what I say here perfectly. But the last few days you have been saying something different (all coefficients for NN in column 2). And this new idea is incorrect IMO.
(6) If you put the dark pixels for scanlines at the beginning (Phase 0) you'll be putting them at the location of the PEAK in the picture. Which, again, is the center of Tap 1. So that's wrong.
(7) If you put the dark pixels for scanlines at the end (Phase 15) you'll put dark pixels at the third peak in my/official picture. That's near the center of Tap 2. And that's wrong.
(8) The dark part should be centered at phase 8. Which is the directly between Tap1 and Tap2. That's why your modified coefficients don't work. But moving the dark part to pahse 0 or 15 is not the correct way to fix that.

VIP_Lanczos.PNG

I really don't know how to say it more clearly so I hope you understand what I'm saying here.

[Sorgelig]

I'm appreciating your detailed explanation!
Although i know the theory, your posts are very good for everyone to understand it.
Really impressive!

I've told in my previous posts where i've posted new coefficients that somehow VIP scaler shifts the picture and cuts the halves of each top and right pixels if NN coeffs switch the tap on the phase 8.
You can check yourself in 240p suite on red+white grid.

[ghogan42]

I'm appreciating your detailed explanation!
Although i know the theory, your posts are very good for everyone to understand it.
Really impressive!

I've told in my previous posts where i've posted new coefficients that somehow VIP scaler shifts the picture and cuts the halves of each top and right pixels if NN coeffs switch the tap on the phase 8.
You can check yourself in 240p suite on red+white grid.

Yes I know that you must know how this works pretty well. I don't mean to lecture you. I just couldn't understand why we didn't think the same thing.

I think I actually understand what you're saying now. I'm just slow

Here's what I see:

(1) You are correct. With the coefficients the way I made them the VIP scaler shifts exactly the way that you said it does.
(2) You are also correct. If you switch the NN coefficients to all 128 in Column 2 then the VIP scaler produces correct output.
(3) Therefore, you want any other coefficients (scanlines/LCD effect) to be shifted the same way. But if you do the obvious thing (like your LCD coefficients you showed me) then you get a set of coefficients with the scanline in the middle of the pixels.
(4) This means that you must shift the scanlines/LCD border back to the edge of the pixels or you get what I showed in my pictures a couple of posts ago.

But here's the thing: the coefficients the way I have them are the correct coefficients. You can even see it in an official Altera PDF https://reds.heig-vd.ch/share/cours/vtf ... essing.pdf. Phase 0 is centered at Tap1 and phase 15 is centered almost at Tap2. And the scanline should be centered at phase 8 (obviously Lanczos2 doesn't have a scanline but phase 8 is 0.5 between Tap1 and Tap2).

Altera_Coefficients.PNG

So if I modify my Matlab code to produce coefficients with the scanline centered at phase 0 we will have incorrect coefficients. But they will produce the correct results: No cutoff pixels and scanlines at the edge between taps. Which is of course what you want: perfect output.

So the coefficients that will produce correct results with the current VIP scaler implementation are incorrectly calculated coefficients.

I wonder if somehow in MiSTer's hdmi code you need to something different? When you are drawing at the left edge of the screen (for the first half of source pixel 1), T2 and T3 would be the first two pixels of the source image and taps T0 and T1 would be sampled from outside the source image to the left. Does MiSTer handle this correctly?

Or would you have to know to repeat the first pixel 3 times for T0, T1, T2 and then have the second pixel be T3?

Because that's the only explanation I can think of for this behavior. Either VIP scaler isn't doing the right thing or MiSTer isn't doing the right thing for pixels along the top and left sides.

But at least I finally understand what you're telling me!

I can make new coefficents that will do what you want now. But I worry that if I do this then we will later find out that there is a small bug in the MiSTer scaling code and then we'll throw out all of the coefficients and recalculate them all again.

[Sorgelig]

What if shift the darker parts to phase 0 and phase 15? Not in phase 8 like now.

[ghogan42]

What if shift the darker parts to phase 0 and phase 15? Not in phase 8 like now.

That's what I'm saying now. This will produce the correct output. EDIT: I'll make a couple of coefficent sets and post them here.

But the coefficients will be wrong from a theoretical standpoint.

So I don't know whether to make these coefficients or to try to find a bug in MiSTer's code. I notice that with the polyphase VIP scaler with lanczos that you get pixels cut from the left edge but repeated on the right edge.

[Sorgelig]

i've no idea where bug can be. I have some ideas to figure out where is the problem. I will try.

[Sorgelig]

well, i've checked in old Genesis release which doesn't support coeff loading (20181015) and it has the same issue of cutting top/left pixels.
So, if it's the bug then it's inside the VIP scaler, not in MiSTer.
This gives me idea that quality of Lanczos PP filter can be improved over standard one if corrected shift will be taken in account.
What do you think?

[ghogan42]

I'm not sure how to improve quality with Lanczos over what we have now. Except for the edges of the screen I think the results are correct.

Normal Lanczos2 is pretty blurry. The custom coefficients you use (for horizontal only?) are much sharper at the cost of halos around edges. You can get sharp pixels with no halos from the "sharp binlinear" and "even sharper bilinear" coefficients I made. It's just preference I guess, but I don't know anything better than what we have.

Right now, I don't know how to get correct output from Lanczos AND fix the shift so as to not cut off the edges. We can do one or the other but not both I think because when I account for the shift and make coefficients for Lanczos2 I get these:

Code: Select all

    72    72    -8     0
    62    84   -10     0
    51    95   -11     0
    40   104   -11     0
    30   112   -11     0
    21   119   -10     0
    13   124    -8     0
     6   127    -4     0
     0   128     0     0
    -4   127     6    -0
    -8   124    13    -1
   -10   119    21    -1
   -11   112    30    -2
   -11   104    40    -4
   -11    95    51    -5
   -10    84    62    -7
  

Just look at the top row. Rows no longer sum to 128 because there should be another -8 to the left of the 72.

So then you have to normalize each row to get something you can use with the scaler. But it won't be correctly computed Lanczos anymore.

Anyway, I attached a couple of shifted coefficient sets. I don't think these fix the edges 100% though.

Shifted_Filter_Test.zip

But I just had a thought. You say that half a pixel was cut from the left. Maybe the pixel data starts going to the the scaler too early? I don't know hardware enough to be helpful

[Sorgelig]

But I just had a thought. You say that half a pixel was cut from the left. Maybe the pixel data starts going to the the scaler too early? I don't know hardware enough to be helpful

there is nothing i can do there. Scaler gets exactly the same video data as VGA output. You can see that VGA has perfect edges.

Normal Lanczos2 is pretty blurry. The custom coefficients you use (for horizontal only?) are much sharper at the cost of halos around edges. You can get sharp pixels with no halos from the "sharp binlinear" and "even sharper bilinear" coefficients I made. It's just preference I guess, but I don't know anything better than what we have.

Yeah, sharp bilinear is good alternative.
Did you try Lanczos 3? It should be more sharper.

Anyway, I attached a couple of shifted coefficient sets. I don't think these fix the edges 100% though.

These coefficients produce correct centering in VIP! And LCD/Scanlines effects are aligned to pixels correctly. So you can use it as a base for other filters for VIP.
Cores with borders won't get any differences. But cores without border will benefit from perfect edges.

[ghogan42]

there is nothing i can do there. Scaler gets exactly the same video data as VGA output. You can see that VGA has perfect edges.

Ahh. You just set the config and then throw your video output at it. They do the rest. Is there nowhere where the input size could have an off by 1 error? Or do you even specify that?

.
Did you try Lanczos 3? It should be more sharper.

I haven't because going from Lanczos2 (defined from -2 to +2) is a 4 tap filter and lanczos3 (define from -3 to 3) would be a 6 tap filter. I could still compute the coefficients for a 4 tap filter with lanczos3 but I think that's not how you're supposed to do it.

These coefficients produce correct centering in VIP! And LCD/Scanlines effects are aligned to pixels correctly. So you can use it as a base for other filters for VIP.
Cores with borders won't get any differences. But cores without border will benefit from perfect edges.

I can do that. But when I was looking at the test charts and I don't think this is perfect still. Look at the Overscan test pattern on 240P test suite and add 1 pixel of overscan to the top and sides. It looks like I shifted it slightly too much. So maybe I should shift it 6 or 7 phases instead of 8 phases like I posted for my test.

Keep in mind though that even if we make these shifted coefficients that
(1) Ascal still produces slightly different results. So Lite/Full can't both be perfect at the same time.
(2) Any filter that actually requires 4-taps will not be correct in the shifted versions. We're actually missing 8 coefficients because you would need Tap(-1) coefficients for phases 0 to 7 for it to be mathematically the same as the correct version. But it will still look ok I guess if you scale up the other coefficients.
(3) 2-Tap filters will be fine. So the bilinear variations and scanline/LCD effects will be fine still.

[PhantombrainM]

Are these filters lag free? Or does it depend?

[ghogan42]

Are these filters lag free? Or does it depend?

The filters don't care. They are a set of numbers that give the weights for combining the input pixels into the output pixels. These are what makes the scaler use the different scaling methods: Nearest Neighbor, Bicubic, Lanczos, Scanlines, etc. The lag/delay in the filter (VIP or ASCAL) doesn't come into it.

When the Temlib Ascal scaler can do it's low/no lag mode reliably with MiSTer, then you will get any interpolation method you want with no lag. But we're not there yet.

Right now, Sorgelig has discovered that the VIP scaler is cutting off a small amount of the left/top of the screen. We're trying to see if we can just adjust the filter coefficients to fix it. We sort of can, but it's kind of a kludge.

[hyperterminal]

Are these filters lag free? Or does it depend?

Today I did my own measurements with the 240p test suite on the Genesis core.

This was my setup:

1. run
MiSTer RGB > gscartsw switch > BNC to CRT Sony BVM 14G5E
MiSTer HDMI > HDMI to LCD Samsung LS22C350HS (5ms GTG response time)

2. run
MiSTer RGB > gscartsw switch > BNC to CRT Sony BVM 14G5E
MiSTer RGB > gscartsw switch > DE-15 > OSSC > HDMI to LCD Samsung LS22C350HS (5ms GTG response time)

On the second run, the interval between the image showed on the CRT and the LCD was one frame closer.

This leads me to the conclusion that by further tweaking the scaler of the MiSTer, approximately one frame of lag can be saved.

[Sorgelig]

This leads me to the conclusion that by further tweaking the scaler of the MiSTer, approximately one frame of lag can be saved.

Good! We will wait lag-free Scaler from you.

[Grabulosaure]

I've posted a new version of ASCAL, with scaler demo versions of AscalTest, Megadrive, C2650 (and recompiled Gameboy and SNES)

I cannot believe the number of times I rewrote this scaler. Pathetic.

Interface changes :
- Removed the option of loading several sets of poly. coefficients
- Added "freeze" signal

(1) Ascal still produces slightly different results. So Lite/Full can't both be perfect at the same time.

I don't know, maybe some rounding differences.
I have tried rounding and using a reduced accumulator instead of fractional deltas. Differences are only visible with extreme scaling.

With AscalTest, it is possible to zoom patterns and see how interpolation works.

Type : (without newlines)
LAF
B9999999999C0000000000
B9999999999C0000000000
B9999999999C0000000000B99
D9999999999E0000000000E0000

Then KKKK... to change patterns


ASCAL is not quite finished yet, but its closer.
(And it is free and much smaller than Altera's VIP...)

[ghogan42]

I've posted a new version of ASCAL, with scaler demo versions of AscalTest, Megadrive, C2650 (and recompiled Gameboy and SNES)

I cannot believe the number of times I rewrote this scaler. Pathetic.

Interface changes :
- Removed the option of loading several sets of poly. coefficients
- Added "freeze" signal

(1) Ascal still produces slightly different results. So Lite/Full can't both be perfect at the same time.

I don't know, maybe some rounding differences.
I have tried rounding and using a reduced accumulator instead of fractional deltas. Differences are only visible with extreme scaling.

With AscalTest, it is possible to zoom patterns and see how interpolation works.

Type : (without newlines)
LAF
B9999999999C0000000000
B9999999999C0000000000
B9999999999C0000000000B99
D9999999999E0000000000E0000

Then KKKK... to change patterns


ASCAL is not quite finished yet, but its closer.
(And it is free and much smaller than Altera's VIP...)

Hey, even if you rewrite it multiple times, it's already been very useful! There are several discord channels (I think you joined one of them) where people discuss MiSTer and work on small fixes to things. And the addition of a scaler that can be compiled for the lite version has been very helpful for those that want to get into the MiSTer code because they can compile their own cores now and get good hdmi output.

One thing I will say regarding the differences between lite/full builds: The ~0.5 pixel offset with the VIP build is either a MiSTer bug or a VIP scaler issue. Because after thinking about it a little, I realize that if I modify my matlab code to produce filter coefficients with the ~8 phase offset that Sorgelig wants, effectively you have only three taps for the first 8 phases. So only 2 tap filters can be adjusted the way Sorgelig has asked for before.

It make me wonder if matching the VIP scaler is actually what we want since according to Sorgelig it produces this offset even with it's internal scaler (not custom polyphase coefficients).

Either way, excellent work!

[Sorgelig]

Actually Ascal doesn't need to be matched to VIP in coefficients. When Ascal will replace the VIP, we will make new sets of coefficients. It's not that hard to do.
So no need to worry about it.

[Sorgelig]

I've posted a new version of ASCAL

this is error:

Code: Select all

poly_a2<=poly_a(FRAC+2 DOWNTO FRAC+3) & poly_a(FRAC+1 DOWNTO 2);

what should be instead of (FRAC+2 DOWNTO FRAC+3)?

[ghogan42]

Actually Ascal doesn't need to be matched to VIP in coefficients. When Ascal will replace the VIP, we will make new sets of coefficients. It's not that hard to do.
So no need to worry about it.

Sounds good. I put my matlab code and calculated filters up because people ask me about them on discord: https://github.com/ghogan42/Filter-Coef ... For-MiSTer

And I just made a page on the MiSTer wiki so there would be instructions for people.

I know you usually compile your own builds when the scaler is updated but if you haven't yet you should check out the his newly compiled genesis core with my original "LCD effect" coefficients. In the 240P Test Suite Grid, it is very close to being perfectly sized and centered. Better than either scaler in older versions. You set the scaler to "Poly" and then choose coefficients with your normal OSD option on page 2.

[Sorgelig]

And I just made a page on the MiSTer wiki so there would be instructions for people.

i've made invitation on github for you to join MiSTer-devel group.
If you accept it, then you will be able to create your repository with coeffs inside the MiSTer-devel. That would be better.

[Sorgelig]

I cannot believe the number of times I rewrote this scaler. Pathetic.

i see you pain
New version is very good!
And interlace mode is working now. Is it hard to add some de-interlacer to scaler? Currently it's perfect for static pictures, but has jagged edges on moving screens.
I didn't try to compile Minimig yet. Using high-res in workbench on MiSTer is a usual part of using Minimig. And as i've said earlier all high-res on Minimig are interlaced modes.

Edges almost perfect but i still see irregularity of left/right edges. Use 240p Suite in Genesis. Grid 320x224 and 256x224. In 320 mode edges look almost perfect. In 256 mode left edge has doubling pixel in 16:9 ratio. 4:3 looks normal.

And downscaling works now - perfect!
So, this version is nearly perfect and definitely the best among all previous.

[ghogan42]

And interlace mode is working now. Is it hard to add some de-interlacer to scaler? Currently it's perfect for static pictures, but has jagged edges on moving screens.
I didn't try to compile Minimig yet. Using high-res in workbench on MiSTer is a usual part of using Minimig. And as i've said earlier all high-res on Minimig are interlaced modes.

As I'm sure you know. The two most basic de-interlacers that can be implemented are Bob and Weave.

In Bob de-interlacing you make a progressive frame by simply doubling fields. So for 480i with Bob de-interlacing:

Frame 1: Input: Odd lines, Output: Odd lines only but each line repeated to get 480 lines
Frame 2: Input Even lines, Output: Even lines only but each line repeated to get 480 lines
Frame 3: Input: Odd lines, Output: Odd lines only but each line repeated to get 480 lines

In Weave de-interlacing you make a interleaving the fields in every frame. So for 480i with weave de-interlacing:

Frame 1: Input: Odd lines, Output: I'm not sure. Either the same as Bob or the Odd lines weaved with black lines
Frame 2: Input: Even lines, Output: Odd lines from frame 1 combined with even lines from frame 2 to get 480 lines
Frame 3: Input: Odd lines, Output: Odd lines from frame 3 combined with even lines from frame 2 to get 480 lines

So you wouldn't want bob de-interlacing on a computer core like Minimig because you throw away half of the spacial resolution.

But weave de-interlacing has the weave artifacts that you don't like when there is motion.

The thing is, all other de-interlacing methods are pretty complicated compared to those two and have their own artifacts such as ghosting instead of the weave artifacts. And to implement better methods, you would have to combine pixels from different fields in some kind of smart way. There are many methods used in video. VIP scaler implements some and probably that's why you haven't had to worry about it before now.

So if you want to minimize weave artifacts for now without changes to the scaler you can do what old video equipment before computers were fast: lowpass the video vertically. This can be done by choosing a blurry filter for vertical interpolation. But weave de-interlacing is very common so it's ok for now isn't it?

EDIT: I'll make a git repository for the Filter Coefficients tomorrow and then edit the wiki to point to the right place. I want to fix a few things in the my matlab code and add/remove some of the filter sets to minimize redundancy.

[Sorgelig]

Bob is not simple field to frame conversion! If you will do this, you will have flickering on static images. So even with bob you have to detect the motion, so static won't flicker and each frame for static must include both fields in Bob mode.

[Sorgelig]

@Grabulosaure,
it seems Ascal doesn't like frames with different vertical sizes - this is usual for Interlace modes. That's why SNES in interlace makes Ascal crazy.
Note: the visible lines in both fields are the same but total lines in one of fields is one less. This is how interlace mode works. Genesis has the same total lines for both fields in interlace mode - this is just Genesis specific, not normal.