Question regarding STT images

[rb]

Hi there,

it's probably a thick question

but the CRC bytes of the sector structure.. are they simply the result of adding all bytes or do you
use a special algorithm?

i don't know whether steem actually verifies the checksum but since
the disk explorer should be able to convert other formats to stt images
it would be quite useful to have the correct CRCs saved

also are these two bytes actually a word in little endian?

cheers
rb

[Steem Authors]

Time for me to show my total ignorance of everything.

but the CRC bytes of the sector structure.. are they simply the result of adding all bytes or do you
use a special algorithm?

Actually we don't check the CRCs at all, I haven't worked out how to generate them. In the WD1772 docs it says the polynomial is x^16+x^12+x^5+1. We haven't figured out exactly what it means, maybe you can solve the mystery for us.

i don't know whether steem actually verifies the checksum but since
the disk explorer should be able to convert other formats to stt images
it would be quite useful to have the correct CRCs saved

Yes it could be, if we can work out how to generate the CRCs (we'll have to do that at some point).

also are these two bytes actually a word in little endian?

Sorry, I'm no help again, I don't know, it doesn't seem to say in any of the docs I have. It probably is a word but it could be in either order. Probably best to find out how to calculate the CRC first, it will be easy to see once we can do that.

Russ

[rb]

thanks a lot russ..

i should have done my homework i thought this crc is created by steem in order to verify the actual sector data...

i will try to find out more about this formula and let you know when successful..

anyway, for the time being i can skip this value..

cheers
rb

[rb]

alright.. have done some research...

basically, the crc algorithm is a ccitt-16 crc calculation..
the crc is calculated bitwise...
btw the resulting crc is big endian!

the following code seems to be the most straightforward...

/*
* this is the CCITT CRC 16 polynomial X^16 + X^12 + X5 + 1.
* This is 0x1021 when x is 2, but the way the algorithm works
* we use 0x8408 (the reverse of the bit pattern). The high
* bit is always assumed to be set, thus we only use 16 bits to
* represent the 17 bit value.
*/

#define POLY 0x8408 /* 1021H bit reversed */

unsigned short crc16(char *data_p, unsigned short length)
{
unsigned char i;
unsigned int data;
unsigned int crc = 0xffff;

if (length == 0)
return (~crc);
do
{
for (i=0, data=(unsigned int)0xff & *data_p++;
i < 8;
i++, data >>= 1)
{
if ((crc & 0x0001) ^ (data & 0x0001))
crc = (crc >> 1) ^ POLY;
else crc >>= 1;
}
} while (--length);

crc = ~crc;
data = crc;
crc = (crc << | (data >> 8 & 0xff);

return (crc);
}

hope that helps

cheers
rb

btw... each sector seems to have 2 different crc's.. one based on data from the address mark to the next crc mark, and the other one, which you probably use, is the one from the data address mark to the end of the sector data..

[Steem Authors]

alright.. have done some research...

Great, I love it when somebody else does all the work!

basically, the crc algorithm is a ccitt-16 crc calculation..
the crc is calculated bitwise...
btw the resulting crc is big endian!

the following code seems to be the most straightforward...

That's great, thanks for finding that. I had assumed that the CRC was WD177x specific, I suppose I really should have spent 10 minutes trying it in a search engine at some point. I blame this damn heat wave (even though I had the docs with the polynomial in them for 2 years).

btw... each sector seems to have 2 different crc's.. one based on data from the address mark to the next crc mark, and the other one, which you probably use, is the one from the data address mark to the end of the sector data..

Oh yes, never noticed that before, strange to make a CRC of only 4 bytes.

Typically I can't get the routine to work yet, it is probably something I am doing wrong. From what I can tell a 512 byte blank sector should generate a CRC of 0xda6e, I can't get that result. I haven't tried very much, I'm probably just making some stupid mistake.

Thanks again,
Russ

[rb]

hi there

well i suffer myself from heat strokes the fan beside my desk is barely able to blow hot air around...

this routine i found might be somewhat dodgy.. i haven't tested it myself yet... i am still far from the need of using such a function

i contacted the guys from caps and asked whether they would be willing to share the file format of the ipf format but the administrator seems to be on holiday for a couple of weeks... so in order to have one unified internal image format to run the emulation with i have to wait for the guys to respond.. i hate having various formats since it's quite error prone..

i slightly remember that you can verify the resulting CRC by running the result against this function.. this should return a 0 if my cooked brain cells don't deceive me...

the second CRC (or first one) is probably used to verify the sector header itself before any sector data is read.. i imagine this opens doors for copy protection and such

cheers
rb

[zorg]

Hello RB,

Here is the source of a little Delphi program i found on the net (http://www.efg2.com\lab) which creates 32bit and 16 bit CRCs.

I use 32 bit version in MSA Converter to find duplicate files and it gives the same results as Winzip CRC

It 's in Delphi and easy to use.

I hope it will help you.

Zorg

[rb]

thanks zorg

i just had a quick look at the delphi code... at first glance the algorithms seem to be quite different...

the code i have provided calculates with only one constant 0x1021.. i found that code on a site entirely dedicated to CRC

well maybe russ can try that one right quick to verify the outcome since he is ready to go anyway

@russ.. in case you are unfamiliar with delphi/pascal.. just let me know

cheers
rb

[Steem Authors]

well i suffer myself from heat strokes the fan beside my desk is barely able to blow hot air around...

That's rough, I don't find one good fan enough. I shouldn't be using the heat as an exuse, up here in the midlands it is about 5 degrees cooler than London and 10 cooler than Paris, but it's the only decent excuse for my laziness I have!

this routine i found might be somewhat dodgy.. i haven't tested it myself yet... i am still far from the need of using such a function

I think the routine is doing what it was designed for, but I'm not sure what the FDC actually uses to calc the CRC. Here's what it says in the docs:

Assuming that the CPU has heeded the 177x's data request, the controller writes 12 bytes of zeroes. The 177x then writes a normal or deleted Data Address Mark according to the a0 flag of the command. Next, the 177x writes the byte which the CPU placed in the Data Register, and continues to request and write data bytes until the end of the sector. After the 177x writes the last byte, it calculates and writes the 16-bit CRC.

So no help there, it could include the data mark or the zeros or neither. Another possibility is it works out the CRC from the bit stream it actually writes, in which case we would have to MFM encode the bytes first. And there is always the chance that the FDC CRC routine is slightly different even though it uses the same polynomial. Yet another problem is that the number I am expecting (0xda6e) could be wrong, I got it through read track and although the docs say the CRCs are always preserved, I don't trust them at all.

i slightly remember that you can verify the resulting CRC by running the result against this function.. this should return a 0 if my cooked brain cells don't deceive me...

Hmm, I can't get this to work either, maybe the routine is a dud after all.

I have tried zorg's CRC-16 generator (thanks for uploading that), unfortunately I can't get the right result out of it.

I suppose I will have to put my thinking cap on... Damn lost it... Well I'll play Captive instead.

@russ.. in case you are unfamiliar with delphi/pascal.. just let me know

Well I've written one extremely short program in it, so I consider myself an expert. I haven't attempted to convert the routine, I used the test program to try out 512 0 bytes, if the FDC is using the bytes passed to it during write sector to calc the CRC then surely that must be the way to do it.

Russ

[rb]

well i haven't done much the last few days... the heat literally sucked all motivation and energy out of me .. but it seems to get cooler again

well to be honest i had to read about and brush up various technologies because i had to be prepared for a couple of job interviews.. had one today (over 2 hours.. pheew) but quite promising, though... well this is certainly my highest priority..

i had a look at other FDCs and they all use the same polynomial... so when i format a 720kb floppy under windows and save some files on it then the atari fdc obviously does seem to be quite happy with the CRCs.. so there must be a uniform method for all 'standard' FDCs...

i will dig further.. i already found some old scanned documents but hardly readable which nonetheless seem to have quite some information..

cheers
rb

[obo]

Have you sorted this now? If not, I can post more details and working code...

[Steem Authors]

I don't know about rb but I certainly haven't made any progress. Any help you can provide would be greatly appreciated!

Cheers,
Russell Hayward

[obo]

The CRC does use the normal CCITT polynomial of 0x1021 and initialisation of 0xFFFF, but the bit order is reversed (perhaps due to the order bits arrive in the decoded MFM stream?).

As documented, the CRC includes the 0xA1 bytes from gap 2 (originally written as 0xF5 during formatting), the id (0xFE) or data (0xFD) address mark, as well as the id header or data itself. The CRC is also stored in MSB then LSB order when written to disk.


Starting with a more traditional CRC update routine, using a bit-reversed polynomial we can use:

Code: Select all

WORD crc;
for (int i = 0 ; i < 8 ; i++)
  crc = (crc >> 1) ^ ((((b >> (7-i)) ^ crc) & 0x0001) ? 0x8408 : 0);

the resulting CRC will also need bit-reversing, which isn't very convenient. Fortunately, we can update the bits in the normal order, and use the normal 0x1021 polynomial:

Code: Select all

for (int i = 0 ; i < 8 ; i++)
  crc = (crc << 1) ^ ((((crc >> 8) ^ (b << i)) & 0x0080) ? 0x1021 : 0);

Which gives the correct bit order in the final CRC. This can then be extended to use a normal CRC-CCITT look-up table for speed, giving a much faster:

Code: Select all

crc = (crc << 8) ^ crc_ccitt[((crc >> 8) & 0xff)] ^ b;

The look-up table is the normal one generated from the 0x1021 polynomial using something like:

Code: Select all

for (int i = 0 ; i < 256 ; i++)
{
  WORD w = i << 8;

  for (int j = 0 ; j < 8 ; j++)
    w = (w << 1) ^ ((w & 0x8000) ? 0x1021 : 0);

  crc_ccitt[i] = w;
}

Using this on the id field for the first normal sector on the disk, we'd CRC the data: A1 A1 A1 FE 00 00 01 02

That would give a CRC of 0xCA6F, which is stored in big-endian order on disk: CA 6F


Hopefully that should be all you need to add real CRC checking/generating to Steem!

[Steem Authors]

Thanks a lot, all works perfectly, that's a great help. Can I add you to the thanks list in Steem? If so, do you want me to call you obo or something else?

I'm sure I would never have figured that out by myself, hopefully it will have fixed a some programs. A few programs I have noticed go wrong on "read address" commands, that could be down to the old bad CRCs.

Thanks again,
Russ

[obo]

Thanks a lot, all works perfectly, that's a great help. Can I add you to the thanks list in Steem? If so, do you want me to call you obo or something else?

"Simon Owen" would be best - cheers

A few programs I have noticed go wrong on "read address" commands, that could be down to the old bad CRCs.

That reminds me... does the ST suffer the WD1772 READ_TRACK problems that the SAM Coupé has? Some byte sequences cause the decoder to trip up, so the rest of the track is returned as junk. If that doesn't sound familiar, I can dig up more details to help check it...

Si

[Gunstick]

do you actually know the book "scheibenkleister"?
A german book of over 800 pages just about storage on the Atari. For the floppy there are 300 pages!

Everything is explained. It's that book who brought us hyperformat and self-made copy protection.

So if you have any questions on the FDC I can answer them (or rather the book can)...

Georges

[Steem Authors]

Hi Simon,

Sorry this reply is so late, thanks for your name and all the help.

The ST does suffer the read track problems you mentioned, it certainly is a bit of a pain (I was hoping to just store the result of the read track command in the STT files, but that plan was soon scupperred). Do you think it would be possible to emulate the track corruption? Currently we just return the data without any corruption, but I know of at least one program that relies on some corruption being present.

Thanks again,
Russ

[Steem Authors]

Hi Gunstick,

I haven't seen that book, sounds very useful. If I come across anything I need to know I'll contact you.

There is one thing that I keep meaning to test, does it have details of how exactly the write protect bit in the status register works? TOS relies on it being up to date to detect media changes but lots of programs expect it to always be 0 even if the disk is write protected.

Cheers,
Russ

[Gunstick]

There is one thing that I keep meaning to test, does it have details of how exactly the write protect bit in the status register works? TOS relies on it being up to date to detect media changes but lots of programs expect it to always be 0 even if the disk is write protected.

Ah the write protect bit... one of the big silly things in the ST

A floppy drive has the signals "disk change" and "write protect"
The ST only has "write protect"
yeah, so to detect disk change, a change on write protect signal is used.
when no disk is in the drive it signals "disk is write protected"
If the floppy disk is protected, no change occurs when swapping disks

WP status flopy status
HIGH non protected disk inside
LOW non protected disk is removed
LOW no disk in drive
LOW protected disk is inserted
LOW protected disk is in drive

Is that what you needed?

Georges

[obo]

Hi Russ,

The ST does suffer the read track problems you mentioned, it certainly is a bit of a pain (I was hoping to just store the result of the read track command in the STT files, but that plan was soon scupperred).

I'm having to optionally do that for my SAM Coupé disk images, as there is one title that stores protection data in the inter-sector gaps! What stopped you storing it in the STT files?

Do you think it would be possible to emulate the track corruption? Currently we just return the data without any corruption, but I know of at least one program that relies on some corruption being present.

I'm 95% sure it's possible to emulate, but it does rely on having the correct raw track information to feed through the algorithm, so you'd probably have to read it with an Amiga/CatWeasel. For my own emulation I return stored track data where available, and am planning to fake the corruption in other cases (assuming normal gap information). I've not coded it yet, as I need to check on some of the notes I made last year.

I believe the corruption is down to the address mark detector being left on during READ_TRACK. It trips over a pattern in the bit-stream, and ends up returning the MFM sync bits rather than the data bits (that's why blank tracks appear mainly as 0xff). From what I remember, I'd pinned down the bit pattern causing the problem, but I'd not finished looking at the switch point to see if any bits were dropped.

Si

[Gunstick]

Hey, some more quotes from the Scheibenkleiste book about read track and copy protection

first you have to know that reading a track gives always different results as long as there are no sync bytes.
You would need to emulate the physical magnetization of the media and a floppy head reading that and then emulate the MFM decoding.

Now a copy protection would be written into the gap like this
00 00 00 00 F5 F5 F5 10 20 30 40
the protection code is the 10 20 30 40
the 00 00 00 00 part will be read as garbage differently at each read
the F5 bytes generate sync bytes on the disk

Now reading this gives you A1 bye instead of the F5, so reading A1 but writing F5
?? ?? ?? ?? 14 A1 A1 10 20 30 40

Better copy protection
writing 00 00 00 29 10 20 30 40
and then reading gives you crap, but it's always the same
the $29 is the so called killerbit pattern

now to complicate things we write
00 00 00 29 F5 10 20 30 40
And reading that gives
?? ?? ?? 14 0C 10 20 30 40
HUH? no A1 or C2 showing there is a sync byte, that info is masked by the preceding $29


More? Yes there's more
let's write F5 F5 F5 F7 F7 (the F7 should generate checksums)
and reading does not give 2 checksums:
14 A1 A1 CD B4 F7
the first F7 made the CDB4 checksum and the second F7 was written as is!
Writing that back would result in something like
14 A1 A1 CD B4 A1 F2 (the second F7 is now the first F7 and creates a checksum)

combining all this is the following pattern to write track
00 29 F5 F7 F7

the 29 will mask the sync byte and the second F7 gets hidden by a checksum


Having fun with adress fields
write F5 F5 F5 FE F7 F7 02 F7
note the double F7
reading gives
14 A1 A1 FE B2 30 F7 02 AA 14
ok, track $B2, side 48 and sector number $F7... huh, sector $F7 how to write that, because F7 generates a checksum. So writing it and reading back gives:
14 A1 A1 FE B2 30 00 00 ....


Georges

[obo]

first you have to know that reading a track gives always different results as long as there are no sync bytes.

I suppose that could be faked by randomly changing the start position in the bit-stream. For normal use the only unsynced bit is between the index pulse and the first sync mark, which could be fiddled if it was really needed. Something in the ST world might be sneaky enough to rely on it, but I'm certain no SAM disks do.

You would need to emulate the physical magnetization of the media and a floppy head reading that and then emulate the MFM decoding.

The FDI format (http://www.oldskool.org/disk2fdi) aims to read everything at MFM level, storing as much detail as is needed to represent the disk at the lowest level. It also has short-hand way of representing standard disk structure, so as not to make it too bloated for most disks.

I was going to use FDI for protected disks, but it's 2 years since the original beta, and it's still not finished. Even worse, it's no longer free so I'm giving up on it!

Better copy protection
writing 00 00 00 29 10 20 30 40
and then reading gives you crap, but it's always the same
the $29 is the so called killerbit pattern

From the notes I made last year, I have the bit pattern of "000101001" written down, which is an extra leading zero bit ahead of the 0x29 pattern you mention. I'll have to check it, but I have a feeling it didn't work if the leading bit was a 1 - I'll try out 01 29 when I get a chance. It's also worth noting that this killerbit pattern works anywhere in the bit-stream - it doesn't have to be aligned on a byte boundary to be triggered.

Does the book have any more details on what causes it? Is it just the address mark decoder being enabled?

<snip>
All interesting stuff - thanks for posting it Georges!

[Gunstick]

From the notes I made last year, I have the bit pattern of "000101001" written down, which is an extra leading zero bit ahead of the 0x29 pattern you mention. I'll have to check it, but I have a feeling it didn't work if the leading bit was a 1 - I'll try out 01 29 when I get a chance. It's also worth noting that this killerbit pattern works anywhere in the bit-stream - it doesn't have to be aligned on a byte boundary to be triggered.

Does the book have any more details on what causes it? Is it just the address mark decoder being enabled?

yes indeed it has. The book is VERY detailed

in MFM code:
00-F4 is written as is
F5 is written as sync byte A1 without any clock info, AND the checksum register is initialized to CDB4
F6 is written as sync byte C2, no CRC modif, and C2 is not detected by READ ADR or READ SECT
F7 writes the checksum, next byte is written as is
F8-FF adress marks

btw, short MFM explanation:
0 bit: write as x0
1 bit: write as 01
(x means complement of previous bit) this introduces inside the bytes some regular clock
so writing gives

Code: Select all

1 0 1 0 0 0 0 1  = the written byte (an $A1)
0100010010101001   = the MFM coding

NOTE: somehow the BBS is not formatting code as fixed width, at least not in preview, or not in Mozilla.

ok, now what does "writing A1 without clock" mean
well it has no clock between bit 4 and 5 (with $C2 it's bit 3 and 4)
so it looks like this

Code: Select all

0100010010101001 = normal A1
0100010010001001 = sync A1

the controller has an 'adress mark detector' triggering on missing clock and signals the FDC when it sees the bit pattern

during READ TRACK the adress mark detector is NOT switched off
after the index pluse the controller reads the bits (or the clocks) until a syncbyte
the first sync byte gets read as C2 or 14 and sometimes as 0A
Unfortunately the resync is done on the 000101001 pattern of 9 bits, this happens with
$29 and previous lower bit at 0
$52, $53 and 2 previous bits 0
$A4,$A5,$A6,$A7 and 3 previous bits 0
$14 and following byte's high bit set (bit 7)
$0A, $8A and following byte having bit6 set and bit 7 zero (e.g. $43)
$05, $45, $85, $C5 and following byte with bit 6 & 7 zero and bit 5 set (e.g. $21)
now that sync creates a half bit shift in the following data, and the controller now reads
clock bits as data bits.
This is great on track 41 ($29)
the adress mark looks like this
A1 A1 A1 FE 29 00 01 02 ....
and read track reads
14 A1 A1 FE 14 7F FE 7C ...

but why does it read $29 with preceding 0 bit as sync?
well it's a BUG !
in MFM it looks like this

Code: Select all

1010010001001001
 0 0 1 0 1 0 0 1    = $29
1 1 0 0 0 0 1 0     = $C2

so the controller confuses data and clock bytes, sees a $C2 and syncs on that
result, it shifts everything by a half bit


Georges

[Steem Authors]

Hi Gunstick,

Thanks a lot for the info about write protect and read track corruption, hopefully I'll be able to knock up some sort of fake corruption for the next version of Steem.

As far as write protect goes, I'm not 100% sure but I think that there is always some sort of change in the write protect bit when a disk is inserted, TOS really relys on it in order to detect media change. It might be that when a disk is half-inserted in the drive it reads as read-write, but this is something I must check.

Anyway, media change seems to work so I'm not overly concerned with it. What is confusing me is that after the FDC performs a read sector command the write protect bit is always clear, even on a write protected disk. I really need to know how long it stays clear and what can return it to its correct state, is there anything in the book about that?

Cheers,
Russ

[Steem Authors]

Hi Simon,

I'm having to optionally do that for my SAM Coupé disk images, as there is one title that stores protection data in the inter-sector gaps! What stopped you storing it in the STT files?

Actually all the code is in Steem's disk imager to put the result of read track in the STT files, and the code is in Steem to read it. Unfortunately I couldn't find any programs that were fixed by including the data, lots of them did read track, got what they would on the ST, and then crashed anyway. I thought it wasn't really worth adding an option to treble the size of the images when it didn't improve anything!

I'm 95% sure it's possible to emulate, but it does rely on having the correct raw track information to feed through the algorithm, so you'd probably have to read it with an Amiga/CatWeasel.

Ouch, that is a bit tricky, I don't have any way of making those images. It might be best for Steem to leave this until we can include CAPS images, unfortunately that requires a complete rewrite of the FDC, so it might be a while.

For my own emulation I return stored track data where available, and am planning to fake the corruption in other cases (assuming normal gap information). I've not coded it yet, as I need to check on some of the notes I made last year.

Yes I think that is the best option for Steem too, it doesn't even have to be that accurate really. Any ST programs that take any more than a glance at the results of read track on a normal disk image will figure out it is fake, fortunately most only glance.

Thanks again,
Russ