PASTI/STX File Format

In this forum you'll find more information about the Pasti & VAPI Tools and the Preservation Project built around these tools. Come on in to find out more about it and discuss these projects.

Moderators: Mug UK, ijor, Moderator Team

sarnau
Atari User
Atari User
Posts: 36
Joined: Tue Sep 07, 2010 4:22 am

PASTI/STX File Format

Postby sarnau » Tue Sep 21, 2010 12:24 am

Hi,

This should help the guys working on e.g. Hatari to deal with Pasti files.

http://www.sarnau.info/atari:pasti_file_format


Markus

SofiST
Atari Super Hero
Atari Super Hero
Posts: 591
Joined: Tue Apr 28, 2009 12:25 pm

Re: PASTI/STX File Format

Postby SofiST » Tue Sep 21, 2010 11:39 am

Great ! Maybe it will move things in right direction now :D

User avatar
simonsunnyboy
Moderator
Moderator
Posts: 4923
Joined: Wed Oct 23, 2002 4:36 pm
Location: Friedrichshafen, Germany
Contact:

Re: PASTI/STX File Format

Postby simonsunnyboy » Tue Sep 21, 2010 4:50 pm

That's interesting stuff. I forwarded the URL to the Hatari development mailing list.
Simon Sunnyboy/Paradize - http://paradize.atari.org/

Stay cool, stay Atari!

1x2600jr, 1x1040STFm, 1x1040STE 4MB+TOS2.06+SatanDisk, 1xF030 14MB+FPU+NetUS-Bee

sarnau
Atari User
Atari User
Posts: 36
Joined: Tue Sep 07, 2010 4:22 am

Re: PASTI/STX File Format

Postby sarnau » Tue Sep 21, 2010 7:13 pm

I'll see if I have time to add this to Hatari myself.

User avatar
snoopy
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Wed Mar 29, 2006 12:15 pm

Re: PASTI/STX File Format

Postby snoopy » Sun Oct 10, 2010 12:30 pm

Congrats. Maybe this will spark more interest in preservation using pasti in the Linux community (as I understand, Hatari is the only/best Atari St emulator on Linux which is still in development and which is also occasionally included in distros, right?). Anyways, keep up the good work.
Snoopy aka Roloway

Dal
Administrator
Administrator
Posts: 4123
Joined: Tue Jan 18, 2011 12:31 am
Location: Cheltenham, UK
Contact:

Re: PASTI/STX File Format

Postby Dal » Tue Oct 12, 2010 9:52 pm

Yes fingers crossed Hatari will soon support the pasti format as it is rather fab!
Mega"SST" 12, MegaSTE, STE: Desktopper case, IDE interface, UltraSatan (8GB + 512Mb) + HXC floppy emulator. Plus some STE's/STFM's

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Wed Oct 13, 2010 7:59 pm

I am not sure that for an emulator reading the pasti format is the best choice. These is a dll to link with that take care of reading the file plus offering more feature. I think that Ijor should be involved in this discussion

Dal
Administrator
Administrator
Posts: 4123
Joined: Tue Jan 18, 2011 12:31 am
Location: Cheltenham, UK
Contact:

Re: PASTI/STX File Format

Postby Dal » Wed Oct 13, 2010 8:33 pm

The issue is that a dll is useless for a multi platform application. Needs to be rewritten as a .h library really.

With an ever increasing number of STX images to preserve copy protected titles, it is imperative that emulators (especially a multi platform one with active development) embrace the format IMHO.
Mega"SST" 12, MegaSTE, STE: Desktopper case, IDE interface, UltraSatan (8GB + 512Mb) + HXC floppy emulator. Plus some STE's/STFM's

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Wed Oct 13, 2010 9:47 pm

Yes you are right for multi platforms this is a problem :oops:
If you need to write parser information from Sarnau is pretty good.

For info I have also looked at the problem of reading (in fact I am also interesting in writing) stx file for usage with the DC cartridge and for an existing parser you should look at the stx parser in the HxC 2001 code :wink:
Code is already pretty good but does not seems to interpret all info found by Sarnau.

ijor
Hardware Guru
Hardware Guru
Posts: 3261
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: PASTI/STX File Format

Postby ijor » Tue Nov 09, 2010 9:20 pm

sarnau wrote:This should help the guys working on e.g. Hatari to deal with Pasti files...


Nice work Markus (not angry at all, just in case you wonder) :)

Please keep in mind that the public imaging tool wasn't the only one used for producing the images. Many images out there were produced, or sometimes post-produced, by tools not widely/publicly available.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Tue Nov 09, 2010 9:47 pm

ijor wrote:Nice work Markus (not angry at all, just in case you wonder) :)

Please keep in mind that the public imaging tool wasn't the only one used for producing the images. Many images out there were produced, or sometimes post-produced, by tools not widely/publicly available.

Hi Ijor,
Nice to hear from you :wink:
Any comment on the work from Markus?
Can we assume the description is accurate and complete or not publicly available tools can produce different format?
Cheers
Jean

ijor
Hardware Guru
Hardware Guru
Posts: 3261
Joined: Sat May 29, 2004 7:52 pm
Contact:

Re: PASTI/STX File Format

Postby ijor » Wed Nov 10, 2010 12:57 am

DrCoolZic wrote:Can we assume the description is accurate and complete ...


Hi Jean, nice to talk with you again.

Well, I really don't know. I just made a quick browse to his web site. I'm sure you realize it would require a detailed study to verify if it is indeeed accurate and complete.

...or not publicly available tools can produce different format?


The format is basically the same, but some things could be different. Again, I don't remember the details. I would need to check it.

There were also a few bugs in the imaging tool(s). The DLL has code to overcome and workaround them, in some cases depending on which version of the imaging tool produced the image.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Sat Jan 07, 2012 11:33 am

This a question for Ijor and/or Sarnau

Sarnau I have looked again at the stx format description that you provide here http://www.sarnau.info/atari:pasti_file_format
I have started to analyzed how to use the stx format, that you describe, for all possible protections I know.
From your description I do not see how it is possible to store the information concerning the Macrodos protection (see viewtopic.php?f=47&t=19948&start=25#p194260)
This protection uses different intra-sector bit with variation. In other words some sections of a sector have bytes written with a slower clock and some bytes with a faster clock and the resulting sector has a normal timing (because the bytes in sections compensate). In the format you describe there is only a provision to store the sector timing (0 = normal, value = actual value). In case of macrodos the value would be 0 (overall normal timing). So the question is how do you store this kind of information in a stx file?

Ijor/Sarnau: Is it possible to store macrodos protection in stx? If yes how?
In my experimental program I can measure timing for chunks of 16 bytes (corresponding to DMA fifo size) and actually I do not see any way of measuring timing at the byte level (plus it would be useless). So question is: is there a way to store intra-sector bytes timing information in stx file

By the way I really like the way the fuzzy bits are stored ;) This is extremely extremely smart as it can even store the "special case" of Dungeon Master !!!

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Tue Jan 10, 2012 9:46 am

Other remarks of the pasti format:

FDC status it looks like the FDC status is not an exact copy of the status returned by the FDC.
As you mention it seems that bit 7 = motor on is used for fuzzy bytes and I have also noticed that bit 0 (busy) seems to be used for ??? something

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Tue Jan 10, 2012 11:05 pm

DrCoolZic wrote:Ijor/Sarnau: Is it possible to store macrodos protection in stx? If yes how?

Seems like answer is yes ;)
I have found where the information is stored (just following the sector data) but I have not yet found how the clock variation is stored :(
Ijor any help on these 512 bytes of clock (clock variation?) data
Also not quite sure about how to detect this fact (apart from extra 512 bytes) I suspect a bit in FDC status but not sure (the sector flags are always 00 :( )

User avatar
npomarede
Atari God
Atari God
Posts: 1191
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: PASTI/STX File Format

Postby npomarede » Fri Dec 20, 2013 9:37 am

Hello

bringing this old subject to life, I'm actually looking at the pasti format description made by Sarnau, and I'm also wondering how clock variation are stored for read track or read sector ?

DrCoolzic, did you find more information about this (maybe you wrote your own parser in the meantime) ?
Maybe by comparing macrodos' disks with other disks we could see a bit in a flag that indicates clock variation infos are present ?

Nicolas

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Fri Dec 20, 2013 10:25 am

npomarede wrote:Hello

bringing this old subject to life, I'm actually looking at the pasti format description made by Sarnau, and I'm also wondering how clock variation are stored for read track or read sector ?

DrCoolzic, did you find more information about this (maybe you wrote your own parser in the meantime) ?
Maybe by comparing macrodos' disks with other disks we could see a bit in a flag that indicates clock variation infos are present ?

Nicolas

Yes indeed I have the full STX description :) and where the timing info are located. I have discussed the format with Ijor and now I believe I got it right.
there is only one thing that is missing this is the the timing unit used in the "timing block"
and yes I have a parser that read STX file.
Can you give me few days because I need to find where all this info is located ...
I am currently spending 200% of my time on my new program that analyzes the Kryoflux stream output. You can look at some of the outputs in http://forum.kryoflux.com/viewtopic.php?f=3&t=749
The goal of this program is to analyze the input (currently stream file) to detect protections and to produce .st or .stx files (so yes I will soon need to look at stx format information).

I hope you will support stx format in Hatari because the IPF road seems to be blocked for quite some time as SPS people seems to have other priorities :(
FYI I have just learned about a new product called SuperCard Pro that is better than the Kryoflux board. This guy knows exactly what user wants: be able to either copy or generate emulation files from their FD without worrying about nonsense technical details. I am in contact with the person responsible of the product and I intent to support the format as input ... more on this latter :)

User avatar
npomarede
Atari God
Atari God
Posts: 1191
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: PASTI/STX File Format

Postby npomarede » Fri Dec 20, 2013 10:38 am

DrCoolZic wrote:Yes indeed I have the full STX description :) and where the timing info are located. I have discussed the format with Ijor and now I believe I got it right.
there is only one thing that is missing this is the the timing unit used in the "timing block"
and yes I have a parser that read STX file.
Can you give me few days because I need to find where all this info is located ...
I am currently spending 200% of my time on my new program that analyzes the Kryoflux stream output. You can look at some of the outputs in http://forum.kryoflux.com/viewtopic.php?f=3&t=749
The goal of this program is to analyze the input (currently stream file) to detect protections and to produce .st or .stx files (so yes I will soon need to look at stx format information).

I hope you will support stx format in Hatari because the IPF road seems to be blocked for quite some time as SPS people seems to have other priorities :(

Yes, I'd like to add Pasti support myself to Hatari, as I think we won't see any multiplatform "official" library.
With all the work / accuracy improvements I added to recent Hatari devel version, I splitted "mechanical" part (type I commands, index and rotation speed handling, ...) from the byte/DRQ part.
So, now how I did all this abstraction layering, I'm reaching a point where I can basically plug any decoder as long as it returns byte value + byte duration for all the bytes that would be returned by read sector, read address and read track.
Of course, the decoder part specific to each format (pasti in this case) is still a lot of work, but I think it should be doable.
So, I'm eagerly waiting on your docs :)
FYI I have just learned about a new product called SuperCard Pro that is better than the Kryoflux board. This guy knows exactly what user wants: be able to either copy or generate emulation files from their FD without worrying about nonsense technical details. I am in contact with the person responsible of the product and I intent to support the format as input ... more on this latter :)

Yes, I read about it too ; I'm less interested in the duplication/preservation method than in the resulting file and its use in emulator. From what I read, their better sampling frequency is not necessary required to correctly preserve disks. In the meantime, SPS also told they would release a next version that allow ssome kind of user scripting to turn your own kryoflux dump into an IPF compatible file (as it already exists for the Amiga). But that's another debate not related to Pasti, we'll see what we get.

Nicolas

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Fri Dec 20, 2013 4:42 pm

After all it did not took me too much time to check my Past documentation.
So here you are attached to this post.
The things that you will have to watch is the section on timing variation specially speedlock/macrodos.
This is hopefully clearly explained but let me know if something is not clear.
For the code it is embedded with other part of another experiment so I will need few days to publish.
There is a .h file that define the format and a reader library + a test program that check and display the content of any Pasti file.
I will probably post it soon but may be not before next week or year :)
I will make all this public domain so that it can be used as a reference for further development probably on Github.

Stay tune :mrgreen:
Pasti documentation-v02.rar
You do not have the required permissions to view the files attached to this post.

User avatar
npomarede
Atari God
Atari God
Posts: 1191
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: PASTI/STX File Format

Postby npomarede » Fri Dec 20, 2013 5:15 pm

DrCoolZic wrote:After all it did not took me too much time to check my Past documentation.
So here you are attached to this post.
The things that you will have to watch is the section on timing variation specially speedlock/macrodos.
This is hopefully clearly explained but let me know if something is not clear.
For the code it is embedded with other part of another experiment so I will need few days to publish.
There is a .h file that define the format and a reader library + a test program that check and display the content of any Pasti file.
I will probably post it soon but may be not before next week or year :)
I will make all this public domain so that it can be used as a reference for further development probably on Github.


Thanks for the doc, although I was only missing the infos on the timing, the gfx are very clear and this gives a very good reference doc.
From previous discussions here, I thought some games could also used inter gap bitcell variations, which would require a per-track timing table instead of a per-sector one. But maybe this was just theoretical discussion and no game use such protection ?
From your experiments, are speedlock/macrodos games imaged with embedded timing table (version 2), or are they using the one contained in the DLL (version 0) ? Do you have the table used for version 0 ? (if not, I could build it, if I recall correctly, macrodos are made of only 4 or 8 blocks with different timings per sector).

Nicolas

User avatar
npomarede
Atari God
Atari God
Posts: 1191
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: PASTI/STX File Format

Postby npomarede » Sat Dec 21, 2013 12:30 am

Just a question about your description of the sector header : you say the default read time is 16480 or 515 bytes for a sector, which seems strange to me, because as the DMA read bytes by blocks of 16, there's no way to know the timing for 512+3=515 bytes which is not a multiple a 16.
Also, what would be those 3 extra bytes ? Sector AM + 2 CRC bytes ?

In its docs Sarnau gives a default value of 16384, which gives 512 bytes at 32 microsec, which seems more coherent to me.

Maybe you made a typo in your doc ? (note I didn't have a look to a real STX file to check its value, so maybe I'm wrong, it's just that 515 bytes seems odd)

Nicolas

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Sat Dec 21, 2013 10:00 am

npomarede wrote:Thanks for the doc, although I was only missing the infos on the timing, the gfx are very clear and this gives a very good reference doc.
From previous discussions here, I thought some games could also used inter gap bitcell variations, which would require a per-track timing table instead of a per-sector one. But maybe this was just theoretical discussion and no game use such protection ?
From your experiments, are speedlock/macrodos games imaged with embedded timing table (version 2), or are they using the one contained in the DLL (version 0) ? Do you have the table used for version 0 ? (if not, I could build it, if I recall correctly, macrodos are made of only 4 or 8 blocks with different timings per sector).

Nicolas

I am not aware of inter-gap bit cell variation but I suspect (also never seen) possible inter-gap layout variation. By that I mean the clock rate is not changed during inter-gap, but the number of inter-gap bytes may vary (also never seen).
Note that also I never heard of bit cell variation in inter (or for that matter intra) gap it is possible for the inter/intra gaps to be above below normal as part of an overall above/bellow for the complete track.

The only game that I have with speedlock/macrodos is colorado track 00.1 sector 1
Below is the images from my analyzer: You can see that the data segment of the sector is roughly divided in 4 parts so each part is 128 bytes: first part is at nominal clock 2µs so we have 128*32= 4096 µs at 2000
followed by 128 at roughly 2100 (5% increase) followed by 128 at 1900 (5% decrease) followed by 128 at nominal speed. So what is interesting is that the overall length is the same as a normal sector but parts are different.
colorado-track.01.0.JPG
colorado zoomed.JPG

Below is an image of the data read in sector 1. First column is byte position second column is the value of the clock as detected by the DPLL third column is the position of the firt byte in µs from strat of the data segment
colorado buffer.JPG

so you can see that the values are not exactly what I said but if I remeber from discussion with Ijor you can use the values that I gave because their is obviously some margin ...
You do not have the required permissions to view the files attached to this post.

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Sat Dec 21, 2013 10:05 am

npomarede wrote:Just a question about your description of the sector header : you say the default read time is 16480 or 515 bytes for a sector, which seems strange to me, because as the DMA read bytes by blocks of 16, there's no way to know the timing for 512+3=515 bytes which is not a multiple a 16.
Also, what would be those 3 extra bytes ? Sector AM + 2 CRC bytes ?

In its docs Sarnau gives a default value of 16384, which gives 512 bytes at 32 microsec, which seems more coherent to me.

Maybe you made a typo in your doc ? (note I didn't have a look to a real STX file to check its value, so maybe I'm wrong, it's just that 515 bytes seems odd)

Nicolas

Yes of course you are right the timing returned when reading the sector with a read sector command is 512 * 32 (the FDC eat the AM and the CRC and set status accordingly)
The timing of 515 * 32 is the timing returned when doing a read track where you have the AM and the 2 CRC bytes.
The confusion comes also from the fact that in my analyzer these three bytes are measured with the sector

Thanks I will correct the doc.
FYI: I plan also to add a section on how/when to write the different parts of the STX file based on detected protections. This will be only useful for people that want to write STX
By the way I have also found my STX dump program if interested I will publish the source?

User avatar
npomarede
Atari God
Atari God
Posts: 1191
Joined: Sat Dec 01, 2007 7:38 pm
Location: France

Re: PASTI/STX File Format

Postby npomarede » Sat Dec 21, 2013 10:27 am

DrCoolZic wrote:FYI: I plan also to add a section on how/when to write the different parts of the STX file based on detected protections. This will be only useful for people that want to write STX

This could be useful for later, but as far as I know, write to STX file with pasti.dll are lost when exiting the emulator, so this is less frequent case (even if some games like 'Sundog' require a write unprotected disk)
By the way I have also found my STX dump program if interested I will publish the source?

Sources are always nice, in my case I prefer programs that can run on linux too, but it could help other people (Dio was planning to add it to his 'Emu' emulator IIRC).
Apart from that, I really like the circular view of the disk layout you posted in another thread, very handy to spot specific areas of a protected disk.
Regarding your doc, maybe you could also add a few names of games which exist in stx format, to give examples of various encoding used in pasti (RNF sectors, fuzzy bits, address fields / sector with bad CRC, timing variation, ...).

Nicolas

User avatar
DrCoolZic
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2144
Joined: Mon Oct 03, 2005 7:03 pm
Location: France
Contact:

Re: PASTI/STX File Format

Postby DrCoolZic » Sat Dec 21, 2013 10:54 am

This source is in plain C/C++ so it should compile under Linux without problem.
Adding example sounds like a good idea but dont know when I will have time ...

Yes the FD layout view is cool as it allow to see where protections are located. On the prototype only a limited number of protections are tested but just looking at the layout is already very instructive.
For example I have discovered some game where the ID segment is located over the index. This is very nasty to detect on the Atari because, as you know, the last bytes from a read track usually get "stuck" in the DMA and therefore you do not see them (I still need to modify my FD library to fix this problem)...

Here is the dump of Computer Hits volume 2 disk 2
You can see that in the first tracks the ID is placed few bytes before the index and on later tracks like 78 the ID is indeed just over the index
Notice also that track 79 is an "overclocked" track (in dark blue) with 9 segments
computer-hits-vol2-disk-layout.JPG
You do not have the required permissions to view the files attached to this post.


Social Media

     

Return to “Pasti & VAPI”

Who is online

Users browsing this forum: No registered users and 2 guests