Have you tried adjusting drive strength settings on the SDRAM signals?
(I once had trouble on a Xilinx-based board where I just couldn't get the SDRAM to work reliably, reducing the drive strength is what finally fixed it.)
How is drive strength reduced? sorry but I don't know all the quartus options
It's the CURRENT_STRENGTH_NEW in the qsf, set it to 4MA.
It affects other parameters, it is not just the model and speedgrade of the sdram.
For example, I changed the clk_sdram pin in SiDi for another. Now the SNES core doesn't work for me at any phase. The only way to make it work has been in phase 0 and with the option in PLL Zero delay mode. Now it is very stable.
Just check if the pin used is a dedicated output of a PLL. Using the dedicated output greatly reduces clock delay. However if the SDRAM doesn't require any phase shift @126 MHz, then it's really fast and connected with 0 clock skew.
If you make the scheme and tell me the part numbers of what you want, I make the circuit, perhaps a DAC audio would also be interesting.
On the sdram, my intention was 32mb as standard and a Mister-compatible expansion connector to be able to use the same memory modules and perhaps having two memory buses is good.
I'm not really an expert designing such hardware, but I think about a 10CL055 FPGA (the cheapest variant is 41 EUR @Mouser) - higher speed grade would be better of course, if it fits the budget.
I don't think it wouldn't make sense to have a 32MB SDRAM now - the controller code for 32MB could be used unchanged, maybe if it does full page burst, then it has to be adjusted, but I'm not aware of any core doing that. External SDRAM would be cool, as two independent module will give tremendous amount of bandwidth. However the external connector should be planned very cleverly to avoid problems with long traces and noise.
The ARM probably doesn't have to be changed - SD card in SPI mode couldn't speeded up more, only 4 wire mode could have higher speeds. But that requires many code changes and re-thinking of the SD-connection.
Of course the usual JTAG, SW12, SAM-BA pins should be there.
And 3x8 bits digital output pins, pixel clock, etc.. - maybe somebody will design a fully digital HDMI scaler for it.
About a better DAC - I'm not an audiophile, and these systems also not high-end audio equipments, thus I don't care.
Well, this would be my next dream single board FPGA, however I don't know if it would attract some more developers, since without them, it's just a paperweight.