SCP image file format for emulators

Moderators: DrCoolZic, Moderator Team

JimDrew
Atari Super Hero
Atari Super Hero
Posts: 758
Joined: Mon Nov 04, 2013 5:23 pm

SCP image file format for emulators

Postby JimDrew » Sat Oct 27, 2018 6:15 pm

I wanted to open a new thread to discuss a change coming to the .scp image file format. There are many emulators now supporting the .scp image file format. Ijor has released a FPGA based Atari ST emulator for the MiSTer project that can also support .scp image files. However, there is a limited amount of memory available in the FPGA setup (32MB total), and so it would be advantageous to have a means of reducing the file size of the .scp image files. We know that .7zip is the best compression to use for flux image files. We also know that we don't really need "preservation" quality for images that are intended to be run on emulators. There is an option in the SCP imaging software called "Preservation". If you un-tick that option, the image file will be reduced from 25ns captures to 100ns captures. The end result is that the image file compresses to a dramatically smaller size (like from 11MB to under 1MB). STeeM, Win-UAE, etc. all support using .scp image files that are compression with .zip and .7zip.

There are some emulators (like Ijor's FPGA based Atari ST emulator) that do not have the ability to handle compressed files, or can't afford the time it takes to decompress on the fiy and still maintain cycle accurate emulation. I have proposed a small addition to the .scp image file format that will let you cut the .scp image file size basically in half. Currently, all image files generated by the SCP software use 16 bits for storing the bitcell time. There is already an option in the .scp image format spec to change the number of bits used for the storage. Since all of the 720K-800K disk formats require at least 11 bits of storage space for the maximum bitcell time, 16 bits (as opposed to bit packed) has always been used because it made it easy for extracting the bitcell times. I have now assigned the reserved byte at offset 0x0B in the image file format to be the capture resolution. Note the new change:

BYTE 0x0B is resolution of the capture. The base resolution is always 25ns. So a value of 0
is the standard 25ns capture time. If the value is non-zero, then it is a multiplier value
of 25ns. So, a value of 1 would be 25ns (base resolution) + 1 * 25ns = 50ns. 2=75ns, 3=100ns, etc.

I will be making a change to the SCP software to allow saving with the new resolution along with storing the data as 8 bits instead of 16 bits. I will also make an option for doing conversions from existing .scp image files to a reduced size version.
I am the flux ninja

JimDrew
Atari Super Hero
Atari Super Hero
Posts: 758
Joined: Mon Nov 04, 2013 5:23 pm

Re: SCP image file format for emulators

Postby JimDrew » Sat Oct 27, 2018 6:36 pm

As a side note. When selecting the new storage mode option, offset 0x09 (encoding bits) will contain 8 instead of 0 to indicate that bitcells are stored as 8 bit values instead of the normal 16 bits.
I am the flux ninja

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3590
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: SCP image file format for emulators

Postby Sorgelig » Sat Oct 27, 2018 8:33 pm

Disk images on MiSTer not required to be loaded fully at once. It's possible to mount image and read it by sectors (512b).
That's how i use 7GB of HDD image in Minimig.
Also, there are 512Mb of DDR3 memory dedicated to FPGA use (ok, minus ~20mb for VIP), so it's more than enough for .scp. DDR3 latency is not an issue for disk storage.

JimDrew
Atari Super Hero
Atari Super Hero
Posts: 758
Joined: Mon Nov 04, 2013 5:23 pm

Re: SCP image file format for emulators

Postby JimDrew » Sat Oct 27, 2018 8:38 pm

Ijor told me that his emulation requires loading the entire disk image into memory in order to have a cycle exact emulation. .scp image files are flux level, not sector level. So, they are typically megabytes in size.
I am the flux ninja

Sorgelig
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3590
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: SCP image file format for emulators

Postby Sorgelig » Sun Oct 28, 2018 1:59 am

then DDR3 can be used.

JimDrew
Atari Super Hero
Atari Super Hero
Posts: 758
Joined: Mon Nov 04, 2013 5:23 pm

Re: SCP image file format for emulators

Postby JimDrew » Mon Oct 29, 2018 3:48 am

The DDR memory access is apparently too slow. The image file has to be loaded into the main 32MB core memory block that is connected directly to the FPGA.
I am the flux ninja

ijor
Hardware Guru
Hardware Guru
Posts: 3702
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: SCP image file format for emulators

Postby ijor » Tue Oct 30, 2018 2:49 am

Even for sector level images, cycle accuracy might need the whole image to be loaded into RAM.

In my case I'm using SDRAM, but using DDR is probably good enough. DDR at the MiSTer is not slower, it just has a more variable and less predictable latency.

But even when it is indeed possible to load huge images into DDR RAM, it takes some time and it is a waste of resources. Neither an emulator, neither a FPGA core needs all the details that you do normally want for preservation purposes.

If the values you propose are the better or not, Jim, that probably deserves some second thought. I'm currently out of home. Give me a few days and I'll comment a few ideas.
Fx Cast: Atari St cycle accurate fpga core


Social Media

     

Return to “SuperCard Pro Disk Copier”

Who is online

Users browsing this forum: No registered users and 0 guests