Trouble using FloImg or FdRawCmd

[ppera]

...Btw, How can I know which read-sector fail? When I execute "read data" for 10 sectors, if it returns true then I know it's ok, and if it returns false maybe some sectors are ok. How can I know what sectors are ok ?

Therefore I said that first read sectorwise, and after it read trackwise what was RNF - in case of FC Pro it is first sector on track, # of sec. depends from actual skew phase.

Btw. I played little with track lead GAP. When I set count about 6-7 PC can see first sector on track with scan, but not when reads sectors. But floppies made with FC Pro have even smaller GAP - by them first sector is not visible with scan... Don't see how so small GAP can increase any speed more than 0.1 %...

[ijor]

...why can't I find "A1A1A1" pattern for some headers? Suppose index mark is not active, how the A1A1A1 could be read? Still don't enderstand why finding that pattern. (maybe shifted, but consistant).

I have zero experience doing this with the PC FDC, but something similar might happen in the ST, and this is why:

When you are reading the disk pulses out of sync, you might get the data pattern, or you might get the clock pattern. If you get the clock pattern, then you can't retrieve the data. Usually you retry several times until eventually you read on the data pattern. Then you try all bit-shifting combinations scanning for the sync bytes. And finally you can retrieve the data.

However, in some tracks (or sections of certain tracks) you never get the data pattern, you always get the clock pattern no matter how many times you try. This depends on the exact content on the disk surface before the sync marks.

The PC is a bit different. With the mark detector being disabled, the behavior won't be the same as with the ST. But it is still possible that sometimes you will get this same effect (always in the clock pattern).

It is easy to see if this is what is happening. You can't get the data from the clock pattern, but you do can the other way around. So if you have already the data (in this case you do have), then you can easily check if you are constantly reading on the clock pattern.

[ijor]

The controller writes from index to index in a single pass, using a single data rate of 250Kbps. The MFM encoding uses 1 clock bit per data bit, so the raw bitrate is 500Kbps on ST double-density disks.

MFM does NOT use one clock transition per data bit. That's the whole point of MFM vs. FM. And therefore the flux transition rate on the disk surface is still 250Ktps, not 500Ktps.

500 Khz would be the precision you would need to decode the raw pulse pattern. It would also be the rate you would read binary data after the PLL phase (as is the case of the Amiga or the PC CP Option Board).

[leonard]

I wonder if I could write a "universal" disk reader on PC. Some tracks nerver get data (as ijor say, it may returns clock sync). sometimes I get valid data blocks, but I miss sector header, so I don't know what sector is. I'll try to do a "guess sector" function, supposing sectors are coming in order (1,2,3,4,etc...).

obo, I still have a question about your driver: when reading 10 sectors, how can I know which sector get RNF error? (para is right, I can suppose it's the first sector on fcopy-pro disk but I would like to get a universal function to get exactly the RNF sector list).

Maybe one day I would "image" every disk I have

[obo]

I enderstand with your explain that maybe the fact the index mark detector is not active could explain that. so suppose index mark is not active, why can't I find "A1A1A1" pattern for some headers?

You're relying on the controller returning the data stream, which is only half the track bitstream. If it falls out of sync it can return the clock bits instead, which doesn't contain enough to recover the original data bits. In these patches you'll fail to see anything useful, and you'll need to hope the sync gets back in step with the data later on the track.

Suppose index mark is not active, how the A1A1A1 could be read? Still don't enderstand why finding that pattern. (maybe shifted, but consistant).

Using the two-drive Disk2FDI method with a 500Kbps data rate you can get both the clock and data bits, which is what adfread does for its decoding. There are still issues in doing that on disks that aren't written in a single pass (like the Amiga does).

Si

[obo]

Btw, How can I know which read-sector fail? When I execute "read data" for 10 sectors, if it returns true then I know it's ok, and if it returns false maybe some sectors are ok. How can I know what sectors are ok ?

Use IOCTL_FD_GET_RESULT to read the controller result bytes after failure. The 'sector' member of that structure will show the sector that caused the failure. You can then either retry it or continue the rest of the multi-sector read from the sector following it.

Si

[obo]

MFM does NOT use one clock transition per data bit. That's the whole point of MFM vs. FM. And therefore the flux transition rate on the disk surface is still 250Ktps, not 500Ktps.

The effective bitrate is still 500Kbps on a 250Kbps track, when you consider both the data and clock bits that can be returned: http://en.wikipedia.org/wiki/Modified_F ... Modulation Reading at 250Kbps data rate will only return either clock or data bits. Are you just arguing semantics here?

500 Khz would be the precision you would need to decode the raw pulse pattern.

And that's exactly what is done to read Amiga disks using the 2-drive Disk2FDI method, which returns both clock and data bits.

Si

[obo]

I wonder if I could write a "universal" disk reader on PC.

That's exactly what Vincent had in mind for Disk2FDI, and why his image file format is very flexible. I believe he already has decoders for FM, MFM and GCR, as well as the slightly different format needed for the Amiga, and maybe more.

Though I think he might have given up on the 2-drive method now, preferring to use the floppy->parallel cable for more reliable sampling of the drive data line. That avoids all the sync issues with the controller, as you watch for the pulses by polling the parallel port (not something suitable for a multi-tasking OS!)

Si

[MiggyMog]

Has anyone ever written a program to try reading foreign format disks using an external USB 3.5 Floppy drive ?

Surely these have a customised controller interface built in which would avoid some of these issues.

The ones I have seen advertised seem to be able to read Mac & Pc disks when connected to the corresponding computer which gives me some hope that they could be used for reading other foreign formats?

Does anyone know much about how these interfaces work?
worse case scenario is that they are hardwired & exactly mimic internal hardware of a PC disk controller.

[ijor]

Sorry Simon, didn't mean to be agressive. May be it is just a semantic issue.

The effective bitrate is still 500Kbps on a 250Kbps track, when you consider both the data and clock bits that can be returned: http://en.wikipedia.org/wiki/Modified_F ... Modulation Reading at 250Kbps data rate will only return either clock or data bits.

As it happens many times, Wiki is wrong and/or misleading. That article was likely written by an Amiga user, and is looking at this from the output of Paula's point of view.

One thing is what you would use with such a device to make a binay representation of both the clock and data pattern. Another thing is the actual rate.

The pattern as it is written to, and read from the drive is not exactly binary. The intervals between flux transitions aren't always a multiple of the bit-rate. In particular, MFM stream might have transitions that are one and a half bit-period apart.

This can't be represented directly in a binary bytewise form. To make that conversion you need some way to express that "one a half" in binary. So you need to double the frequency/precision. And then the bit-rate as transmitted with something like Paula, would be at 500Kbps. Which I suppose is what you are calling "effective" bit-rate.

But it is wrong to say that MFM encoded at DD is 500Kbps as Wiki claims. In first place because once data is encoded, the concept of bits doesn't exist anymore. In second place because MFM will never produce transitions separated at two microseconds (as it would be the case it the bit-rate would indeed be 500Kbps). They will be at least 4 microseconds apart before pre-compensation. So the flux transition rate is still 250Kftps. And this is why DD disks and DD drives use the 250K figure as the formal specification.

Paula or Disk2FDI would work at 500Kbps, and then they both could use illegal MFM encoding. If you were to translate RLL to binary then you would need to, at least, triple the frequency. So you can say if you want that the "effective bit-rate" is three times as normal, but this is a wrong concept.

[obo]

But it is wrong to say that MFM encoded at DD is 500Kbps as Wiki claims. In first place because once data is encoded, the concept of bits doesn't exist anymore.

Maybe we're just thinking in terms of the different sides of the controller? I usually think of what is visible from the host side, which comes as a stream of data and/or clock bits. Things are very different on the disk side, but unless you're sampling the drive data line directly you'll never see it.

In second place because MFM will never produce transitions separated at two microseconds (as it would be the case it the bit-rate would indeed be 500Kbps).

Doesn't it still require resolution of 2us, even if it's not stored at that rate? I've got a memory of 4/6/8us times for different flux transitions in MFM, but I'm a bit fuzzy on the details.

I suppose all of this is straying a bit off-topic anyway! Leonard is still stuck by the lack of resync during the extra long read, because the address mark detector isn't enabled to pull it back on track. Any disks that fail to expose the data stream for the header+data that is missing will be unreadable with any single-drive method (without a special cable). Things are back to looking a bit bleak again.

I might still experiment with flicking the drive select and/or motor during the index period, in the hope that it allows the controller to see sectors a little earlier. I managed to create a problem disk on the PC by overformatting a double-density track with 21x512-byte sectors and a gap3 of 37 bytes (exact value depends on drive speed). After the final sector is laid (on the 2nd revolution) it continues writing up to the next index. Timings suggested the first sector was about 15 bytes from the start of the index hole.

Si

[obo]

Has anyone ever written a program to try reading foreign format disks using an external USB 3.5 Floppy drive ?

Surely these have a customised controller interface built in which would avoid some of these issues.

The drive units do contain their own floppy controller chip, but it's not accessible through the USB interface. They're visible to the OS as dumb block devices, which are usually limited to 720K and 1.44M formats only. Without access to the controller it's not possible to reprogram them to access anything different - shame!

The ones I have seen advertised seem to be able to read Mac & Pc disks when connected to the corresponding computer which gives me some hope that they could be used for reading other foreign formats?

Old-style Mac floppy drives used GCR, but I've got a feeling it might only refer to PC-style disks accessible to modern Macs.

Does anyone know much about how these interfaces work?
worse case scenario is that they are hardwired & exactly mimic internal hardware of a PC disk controller.

Under Windows they're handled by sfloppy.sys, which also deals with SCSI floppy drives. It queries the block size and total blocks on the disk, which is reported by the USB drive unit, and matches it up against known formats to determine what geometry to report. So the OS has no control over the format, only a method to query it.

Si

[MiggyMog]

Thanks for the answer obo.

Maybe we could persuade somone to make a USB to floppy iterface which bypass the crappy controller...

[Lautreamont]

I wonder if I could write a "universal" disk reader on PC.

I tried to write one (on linux) when I started to image my disks.
I gave up after a couple of weeks because there was always something which didn't work with every new disk tried.
I found things like tracks whith 12 headers and only 9 data blocks, the famous header 66, service tracks 80 and 81 that the ST was unaware of, single sided disks with two sides ...

All the disks I tried were copies of games cracks done with various copiers .

My best program does 25 READID on each track to pick up the sectors headers, and uses the READ command to read the sectors found. It can get most of a disk but not the whole most of the time. One day I ran 10 000 READID on t00h0 of a disk and never found sector 1. A blind READ on this sector succeded first time though.

Now I image my disk either directly on my ST, or I do several steps with several tools on my PC to get all the sectors.

I fear it would take hours and hours to become an ST->PC floppy disk guru.

[ijor]

I usually think of what is visible from the host side, which comes as a stream of data and/or clock bits.

And I'm probably too used to see the disk side

Doesn't it still require resolution of 2us, even if it's not stored at that rate?

Yes, that's what I said. But this doesn't make the bit-rate to be 500Kbps. When writing the FDC actually uses a resolution of 125ns for the write-precompensation adjustment. But you wouldn't say then that the "effective" bit-rate is 8Mhz, would you? Also when reading, and for the digital PLL purpose, you can say that the actual resolution used is much higher than the bit-rate.

I suppose all of this is straying a bit off-topic anyway! Leonard is still stuck by the lack of resync during the extra long read...
I might still experiment with flicking the drive select and/or motor during the index period, in the hope that it allows the controller to see sectors a little earlier.

I'm afraid that Leonard goal is hopeless

Drive select tricks might help. But again, other ways of reading might help as well. I tend to believe that the problem is the index, and not the short sync bytes as Pera suggested (which btw, that's quite easy to test by creating a similar situation in the middle of the track).

But more than likely that the problem is the proximity to the index falling edge, and not because the mark detector is off during the whole index active period. Because otherwise you would have problems with almost every 10 sectors per track disk.

I was thinking that the reason that I can read those disk (slowly, but I can) might be a drive issue. Depending on the drive aligment the culprit header might appear further, closer, or even at the other side of the index pulse. I can easily measure the drive index aligment. But unfortunately not on those PCs that I used. I would need some drive swapping to do it.

[ijor]

The drive units do contain their own floppy controller chip, but it's not accessible through the USB interface.

That is not exact, it depends on the specific brand and model.

But that is the standard. So no way to implement this universally for every USB floppy drive.

[leonard]

I gave up after a couple of weeks because there was always something which didn't work with every new disk tried.
I found things like tracks whith 12 headers and only 9 data blocks, the famous header 66, service tracks 80 and 81 that the ST was unaware of, single sided disks with two sides ...

I'm exactly in the same position I tryed several trick today in he few spare time minutes I get Using the read-track command, I *very* often get header without data or data wihtout headers. Btw, these "nasty" tracks often works pretty well with "read sector" command. so basiccaly my program does:

a) try to read using "read sector"
b) if some sectors are missing, do a read-track and parse the bitstream, looking for the missing sectors. Retry step b) N times.

what I didn't test yet, and maybe you could tell me what is your experience about it:

1) retrying to "read sector" N times too in step a)
2) your "read id" step. What does it mean exactly? The controller goives you a list of position of secteor data, sector header? If "read sector" or "read track" fail, what's the advantage of that method?

[obo]

2) your "read id" step. What does it mean exactly? The controller goives you a list of position of secteor data, sector header? If "read sector" or "read track" fail, what's the advantage of that method?

The read-id command just returns the header details for the next one that passes under the head. Calling it repeatedly gives a list of sectors in the order they're found on the track. There's nothing directly to give positional information, but you can track that yourself. My IOCTL_FD_TIMED_SCAN_TRACK function packs all of that into a single request: synchronising with the start of track, reading all headers up to the end of the track, with time stamps showing when each was found.

Unfortunately it doesn't really help with your problem, as the unreadable first sector will usually be hidden from the scan. Even if you can see the header, it still won't help read the data from it more than you've already tried. In cases where headers are close together (with no data field), the scanning might still miss the second header. Though for true IBM-compatible disks it does work very well.

Si

[obo]

Drive select tricks might help. But again, other ways of reading might help as well.

Anything seems worth a go! I'm particularly interested in a solution using standard hardware only, as it makes it easy for anyone to set up. Even the requirement of having 2 floppy drives is a problem, as more motherboards are limiting to A: only (if B: isn't in the BIOS, it usually isn't physically accessible).

I tend to believe that the problem is the index, and not the short sync bytes as Pera suggested (which btw, that's quite easy to test by creating a similar situation in the middle of the track).

The TRS-80 issues matches the ST problem, and shows that disabling the index is a way around it. The index halving cable (http://home.planet.nl/~srahman/trs80hw1.htm) makes the sector visible every other pass. There are also reports that completely disconnecting it works, though the controller appears to need to see at one index pulse to initialise before it can be completely removed.

But more than likely that the problem is the proximity to the index falling edge, and not because the mark detector is off during the whole index active period. Because otherwise you would have problems with almost every 10 sectors per track disk.

The thought behind the drive select change was to remove the index early, though switching back would just bring it back again. I had hoped to try this tonight but I didn't get around to it. Another one was blipping the motor briefly, which might delay by fractionally enough to make a difference. I'm not sure if that's the same idea behind twaddle in the Linux floppy driver, or a completely different fiddle.

Depending on the drive aligment the culprit header might appear further, closer, or even at the other side of the index pulse.

I do remember seeing headers that wouldn't always show up as the first header when scanning a track (starting from the index pulse), but would show at the end of the track but with a time from the index that was greater than the disk rotation time. They might also fall into the borderline case, where they are only sometimes visible, so a repeated read would eventually get them.

Si

[Lautreamont]

what I didn't test yet, and maybe you could tell me what is your experience about it:

1) retrying to "read sector" N times too in step a)
2) your "read id" step. What does it mean exactly? The controller goives you a list of position of secteor data, sector header? If "read sector" or "read track" fail, what's the advantage of that method?

1) About retries, I've posted that some time ago:
http://www.atari-forum.com/viewtopic.php?t=8555
It's worth to try the same read command many times.
It also tells that sectors of a copy of a disk can be much easier or harder to read than on the original.

2) I did that to salvage as much as possible of the disks I couldn't read fully. The numerous readids give an overview of which sector headers the controller could see on each track. Then I read them one by one and I try to use other tools to get the missing ones.

[ppera]

I wrote a quick version at lunchtime too, which you can download from: http://obo.homeip.net/DiskTest01.zip

I started to implement it in FloImg. First tries are promising. It decodes first sect. in 90% cases (without retries).

Currently I bother to format at least one floppy with FC Pro to end - but it likes me not at all

[ppera]

I'm afraid that Leonard goal is hopeless
... I tend to believe that the problem is the index, and not the short sync bytes as Pera suggested (which btw, that's quite easy to test by creating a similar situation in the middle of the track).

Me too. But still, I will play with READ_TRACK>Decode method. If nothing else, we will learn something.

I will make test floppy on Atari with short intersector GAPS to test it.
But it will not help in reading... OK, we will know it too

[obo]

I will make test floppy on Atari with short intersector GAPS to test it. But it will not help in reading... OK, we will know it too

Don't forget to write something to the disk after/during formatting, to break up the single formatting stream. Freshly formatted disks will be much easier to scan than ones that have been written.

I tried a few tests with the select and motor changes I mentioned yesterday. It didn't seem to help at all, despite varying the gap from a few tens of microseconds to a few milliseconds. I even tried resetting the controller at the start of the index area, but an immediate read-id would still only catch sector 2.

Si

[ppera]

Formatting program will write FAT and ROOT dir on begin of disk ('zeroing' them )
Anyway, I will copy something on...

I expecting Sam floppy with test files...

[leonard]

I started to implement it in FloImg. First tries are promising. It decodes first sect. in 90% cases (without retries).

I get the same result when writing my routine. but, once testing it on complete disk, I get tons of other errors. Very often , almost all disk is redable but some 2 or 3 tracks (anywhere in the disk, say track 32,70, 72) which are unredeable (even with 100 retry).

Are my disks posseded by the evil?