KryoFlux Programming

[DrCoolZic]

EDIT: This thread has been created from the thread "Create new and fresh original game disks" viewtopic.php?f=28&t=21316
I would like to change the title to "KryoFlux Programming" which is the main focus from this message

It is already two weeks that I have received the device and it works as expected

So far I have used it to experiment on getting good reliable flux reversal information from original floppy disks and for that it is perfect.

First goal was to understand the Stream File output. Many thanks to IFW that provided me with a lot of very useful information and to MrVince that provided me with some source code.
The result is a preliminary documentation (attached) on the Stream File format. I believe it is more explanatory than the original documentation, but it may just be because I wrote it?

Second goal was to be able to parse the file. For that I have finished a "test" program (attached) to test the Stream File parser. Not very useful but ...
This program parses a Stream File (including stream files produced by version 2.0 of the firmaware) and check content for consistency.
It will therefore detect if something went wrong during the transfer of the data from the KryoFlux device to the connected PC.
This is a Windows program that should run on any version of Windows (tested on Windows 7 X64). At the end of the excution the program provides:

the average transfer rate oveb the USB link in Bytes Per Second
The number of complete revolutions used for Imaging the track
Rotation Per Minute (RPM): the minimum, average, and maximum values
Number of flux transitions per track: the minimum, average and maximum values (usually the same)

From what I understand the parser will be published by SPS (they need to find the right copyright declaration) and therefore the source is not included.
However the .h files that I have been using are "embedded" in the project documentation generated with doxygen (KFCheck.chm). I do not think there is anything "secret" here but MrVince or IFW let me know if I need to remove this information.

Third goal was to "decode" the flux information from the stream file to display the bytes as a WD1772 would see them. I just made it to work last few days but it is not in a state that I can publish. So this is an ongoing task that will probably require few weeks ...

I want also to test a "more standard" usage of the KryoFlux device. That is to produce images of FD and send them to SPS to get IPF files. For that I will use my collection of original(?) FD. I have sent to SPS pictures of about 300 FD of non boxed games that I can image. Of course they are interested in getting these images but I must apologize because I have not sent any so far but promise I will start soon

here is a picture of the system connected to an old system running Windows XP

I am also updating my web site with the information on KryoFlux device and will keep you posted.

To get back to what has been discussed in this post. I believe that what is best for you really depends on what you want to do...
I have attached a small word doc that compare several preservation/backup techniques. I have tried to be as neutral as possible.

[mr.vince]

Kudos DrCoolZic!

Three things I'd like to address:

a) In the Amiga "scene", IPFs are very well known as correct images of originals and usually people recommend getting the IPF, if someone has trouble with any other source, e.g. cracked ADF. The reason that it's currently not supported by Atari emulators might be lack of knowledge (no offence, but reason for many other emulators where the floppy part is a big hack; can't judge because never looked at any Atari ST emulator source) or objections because of the licence (our lib can't be sold; it's completely free for anyone in private). We do have two common ST emulators that run IPF files, but these are internal versions for testing only.

b) It is of course our intention to also support writing of STREAM / DRAFT, but due to restrictions explained earlier, writing raw data won't work in all cases due to ambiguity present in some protection schemes (e.g. weak bits). This might as well happen with HxC, we would have to see what Jeff wants to make happen now that the docs and specs are out.

c) We have analysed dumps made with the DC and found most of the data present being processed; it looks like that due to memory constraints, unprotected data was saved as processed MFM, only some parts look like they are genuine raw. This for sure was a great a achivement back in the day, but since it's 2011 now, we'd like to store the complete disk as unmodified flux timing found on the original disk.


May I ask you to put your source / docs on our forums as well (Developers Area). Might do so as well. People are likely to be looking there first...

[DrCoolZic]

b) It is of course our intention to also support writing of STREAM / DRAFT, but due to restrictions explained earlier, writing raw data won't work in all cases due to ambiguity present in some protection schemes (e.g. weak bits). This might as well happen with HxC, we would have to see what Jeff wants to make happen now that the docs and specs are out.

Yes directly writing raw data is not a good idea! Actually I did tried that with DC. You could specify that you wanted to save everything as raw data, but when writing back it would not work for reason that you probably understand and that would be too long to explain here.

c) We have analysed dumps made with the DC and found most of the data present being processed; it looks like that due to memory constraints, unprotected data was saved as processed MFM, only some parts look like they are genuine raw. This for sure was a great a achivement back in the day, but since it's 2011 now, we'd like to store the complete disk as unmodified flux timing found on the original disk.

Actually the reason of MFM block was not really to save memory (also it was partially) but also for the reason exposed just before. Writing pure raw data does not work in general . MFM blocks are easy to control with CRC etc and easy to write back. Therefore only specific pieces of information was in raw format but with specific indications (for example data over index pulse). And this was the difficult part of using the DC you had to provide control files that would indicate how to do the imaging ...

I know that probably nobody cares, but please do not criticize the DC which stay for me one of the nicest piece of HW developed around the Atari. And remember that at that time you had no chance of having an ARM proc with some firmware you could update etc. At that time you had to make it right the first time and directly in silicon! So please respect for Mr Richard Adams

May I ask you to put your source / docs on our forums as well (Developers Area). Might do so as well. People are likely to be looking there first...

Done.

[DrCoolZic]

I just realize that it is not very useful to get a Stream checker if you do not have a KryoFlux ...
Therefore to get you interested I provide here two files:
- track 00 of populous (using Rob Nothern protection)
- and a dummy test raw file that contains information from a KryoFlux system using firmware 2.0 (courtesy of IFW) (use the -n option of the program on this file)

[ijor]

c) We have analysed dumps made with the DC and found most of the data present being processed; it looks like that due to memory constraints, unprotected data was saved as processed MFM, only some parts look like they are genuine raw. This for sure was a great a achivement back in the day, but since it's 2011 now, we'd like to store the complete disk as unmodified flux timing found on the original disk.

This is not accurate. The DC lets you save the data at the level you want. You can save the whole disk at the flux transition level if you want. There was even a configuration option in the built-in software for this purpose.

Back in the day it wasn't common to use whole flux transition level though. And the built-in software had some limitations in this regard. Plus, if you wanted to save such an image, then of course it wouldn't fit on floppies, and back at the day most people didn't have a hard disk. But again, you could if you wanted.

[IFW]

Which just highlights the point: dumps made with it are unlikely to be flux level normally...

[ijor]

Which just highlights the point: dumps made with it are unlikely to be flux level normally...

Honestly, I don't understand the point you are trying to make. Yes, most dumps done back at the day didn' save the whole disk at the flux transition level. Probably most of the time no image was actually produced. The most common DC usage then, was likely just to make a single disk to disk copy (no "dump"). So?

The way dumps were performed back then (or for that matter, nowadays by average users) has no relation with "modern" dumps made explicitely for preservation purposes.

[DrCoolZic]

Doing images at flux transition level is IMHO not needed in most cases. For a normal non protected track performing normal read is just fine and will allow to make perfect copy. The obvious advantage (beyond the fact it takes less space) is that it does not requires any complex processing just some CRC checking. On the other hand in most (all) cases processing raw flux transitions requires a lot of processing to be useful. Writing back blindly flux level transitions will result in 99% of the cases in non working FD. Originally I thought that doing flux transition for all track would allow to copy any FD but as you all know this was a bad idea
If you look carefully at the DC documentation you will see that it provides several levels of details from plain normal read to flux transition. The hard part was to carefully select the right level for a specific disk (plus a specific level had lot of controls....)
But again I am not sure it is very useful to compare the DC with the KF
The thing I really like about the KF is the capability to capture data over several rotations and the capability to position precisely the data in respect to index. DC was requiring several passes to do this


Subsidiary question: I have a backup of all the protected FD I have bought some 20 years ago (so I never use the originals)
Would you be able to find a difference between the originals and the copies with KryoFlux? A blind test

[DrCoolZic]

I just want to illustrate that the information provided by both device are very similar.
As an example I have an output published several years ago (in my protection document) from the Analyze program that takes the output from the DC cartridge - game: populous

Code: Select all

= Information for Track=00 Head=0 - 6248 Bytes - length=200.043 ms
  Good/Bad ID=10/0 - Good/Bad Data=9/1 - Good/Bad Sync=20/0
  Average data block length=16502 us
  GAP1 59 bytes length=1913.21 us
  ---+------+---------------+------+------+--------------------------+---------
  Sct|GAP2  |ID             |GAP3a |GAP3b |DATA                      |GAP4
  #  |Bt Lgt|Pos Lgt CRC TV |Bt Lgt|Bt Lgt|Bt  Lgt   CRC TMV FZB Clk |Bt  Lgt
  ---+------+---------------+------+------+--------------------------+---------
    1|15 458|  2 223 OK    0|22 700|15 475|515 16435 OK    0   0 3.99| 30  954
    2|15 477| 21 222 OK    0|22 700|15 475|515 16437 OK    0   0 3.99| 30  955
    3|15 477| 40 222 OK    0|22 699|15 475|515 16439 OK    0   0 3.99| 30  958
    4|15 479| 60 223 OK    0|22 702|15 476|515 16408 OK    0   0 3.98| 30  957
    5|15 478| 79 223 OK    0|22 702|15 476|515 16422 OK    0   0 3.99| 30  954
    6|15 477| 98 222 OK    0|22 700|15 475|515 17206 BAD   2   0 4.18| 91 2923
    7|15 472|120 222 OK    0|22 699|15 474|515 16425 OK    0   0 3.99| 30  956
    8|15 477|139 222 OK    0|22 699|15 474|515 16413 OK    0   0 3.98| 30  959
    9|15 480|159 223 OK    0|22 703|15 477|515 16417 OK    0   0 3.98| 30  957
   10|15 478|178 223 OK    0|22 702|15 476|515 16424 OK    0   0 3.99| 30  957
  ---+------+---------------+------+------+--------------------------+---------
  GAP5 89 bytes length=2807.32 us

The same analysis done with my KFanalyzer yesterday

Code: Select all

KFAnalyze V0.1 (2011) - run Tue Sep 06 21:18:23 2011
  Input file: 'populous.raw' - Flags:

= Track Layout Information: 6240 Bytes - length=199.832 ms
  ID Good/Bad=10/0 - Data Good/Bad=9/1 - Sync Good/Bad =20/0

  GAP1 56 bytes length=1815.70 us
  --------+------------------+------+------+--------------------------+---------
  GAP2    |ADDR              |GAP3a |GAP3b |DATA                      |GAP4
   Bt  Lgt|Sct    Pos Lgt CRC|Bt Lgt|Bt Lgt|Bt  Lgt   CRC TMV BRD Clk |Bt  Lgt
  --------+------------------+------+------+--------------------------+---------
   15  458|  1   2273 223 OK |22 701|15 476|515 16422 OK    0   0 3.99| 30  954
   15  477|  2  21530 222 OK |22 700|15 476|515 16438 OK    0   0 3.99| 30  956
   15  478|  3  40802 223 OK |22 701|15 476|515 16446 OK    0   0 3.99| 30  956
   15  477|  4  60083 222 OK |22 699|15 474|515 16443 OK    0   0 3.99| 30  959
   15  479|  5  79363 223 OK |22 701|15 475|515 16431 OK    0   0 3.99| 30  957
   15  479|  6  98632 223 OK |22 701|15 476|515 17197 BAD   1   0 4.17| 87 2797
   15  481|  7 120510 222 OK |22 699|15 474|515 16405 OK    0   0 3.98| 30  955
   15  477|  8 139745 222 OK |22 699|15 474|515 16409 OK    0   0 3.98| 30  957
   15  478|  9 158987 223 OK |22 700|15 475|515 16412 OK    0   0 3.98| 30  956
   15  478| 10 178233 223 OK |22 701|15 475|515 16433 OK    0   0 3.99|117 3762
  --------+------------------+------+------+--------------------------+---------

Of course both test have been done on different FDrive so some correction could be done (note that some fields have been reordered) but as you can see the results are EXTREMELY close (of course )

By the way you can see a preliminary page on the KryoFlux on my site http://info-coach.fr/atari/hardware/dev ... yoflux.php

[ijor]

Doing images at flux transition level is IMHO not needed in most cases.

There are valid reasons to prefer flux transition level images, at least for the purposes of low level analysis. If it is worth to save them at that level in the long term, that's another matter. But extra information rarely hurts, and the additional storage space required is, nowadays, not really a big problem.

The thing I really like about the KF is the capability to capture data over several rotations and the capability to position precisely the data in respect to index. DC was requiring several passes to do this

You are talking about limitations that apply only to the built-in software.

[DrCoolZic]

Doing images at flux transition level is IMHO not needed in most cases.

There are valid reasons to prefer flux transition level images, at least for the purposes of low level analysis. If it is worth to save them at that level in the long term, that's another matter. But extra information rarely hurts, and the additional storage space required is, nowadays, not really a big problem.

I agree it really depends of the usage. For preservation and analysis it is certainly better to store info at the flux level but for duplication (the main purpose of the original DC SW) it is not required.

The thing I really like about the KF is the capability to capture data over several rotations and the capability to position precisely the data in respect to index. DC was requiring several passes to do this

You are talking about limitations that apply only to the built-in software.

Yes of course. It is possible to achieve the same with the standard DC SW but it requires several passes.
I know that you have reverse engineered the DC SW and in that case it should be possible to add this capability.
As we have discussed in the past it would be nice to contact Richard Adams to hopefully get the source code from him

I have just completed an alpha release of my KFAnalyze program that takes advantage of the capability of the KryoFlux hardware to record flux transition over several rotations to correctly detect fuzzy bits. I have tested this capability on quite different "kind" of fuzzy bits (border bit, timing violations, random flux,...) and they all works
I will post a new thread on this but you can have a look at http://info-coach.fr/atari/hardware/dev ... #kfanalyze to get more information.

[DrCoolZic]

Doing images at flux transition level is IMHO not needed in most cases.

.... But extra information rarely hurts, and the additional storage space required is, nowadays, not really a big problem.

Just for info I have completed a second batch of images. It is already over 2GB of disk space and soon it is going to be a problem (I am only about 10% )

[ijor]

Good job Jean.

Btw, I received my own Kryoflux some time ago. Unfortunately still didn't have much time to "play" with it. Will try to make some tests and write a small review, but I agree that the board manufacturing looks very professional.

[DrCoolZic]

Good job Jean.

Btw, I received my own Kryoflux some time ago. Unfortunately still didn't have much time to "play" with it. Will try to make some tests and write a small review, but I agree that the board manufacturing looks very professional.

Very Good - I might ask you some images

[ijor]

I could get some time to run a few tests with the Kryoflux board. It is a nice piece of hardware. As I said already, board manufacturing and assembling looks very professional.

Most of the issues I could find are rather minor and not too important. My only (more or less) serious complain is that, IMHO, the disk interface is not correctly designed for older DD 5.25 drives (should be fine with most newer HD 5.25 drives). This is at least true for the version I received. ST users would normally don't care about 5.25 disks, so this is probably not an issue for most ST users. It is mostly an issue for 8-bit users only.

Other than that, it is a fine and recommended product.

[galibert]

We do have two common ST emulators that run IPF files, but these are internal versions for testing only.

Note that MESS runs ipf files too for the ST. Of course, the rest of the emulation is not up to par yet, and I still have bugs in the wd1772.

And I still don't know where the write splice is for Chase HQ, hint hint

OG.

[ijor]

And I still don't know where the write splice is for Chase HQ ...

Just out of curiosity, why you do care about the track splice position? The splice point is important for write back, but shouldn't be much relevant for emulation purposes.

Also note that some tracks don't have a splice point. Or more precisely, they don't have an unique single point, they have multiple (sometimes many) splices.

[galibert]

And I still don't know where the write splice is for Chase HQ ...

Just out of curiosity, why you do care about the track splice position? The splice point is important for write back, but shouldn't be much relevant for emulation purposes.

Also note that some tracks don't have a splice point. Or more precisely, they don't have an unique single point, they have multiple (sometimes many) splices.

Well, Mess now uses a low-level magnetic zones representation for the floppies, with vectors of magnetic state (N/S/unmagnetized/damaged) and their angular size. The "floppy" class then answers questions such as "when is the next transition", and the wd1772 class handles all the pll stuff and everything. It actually makes the emulation somewhat simpler, as getting closer to the hardware often does.

But that means one has to be able to go from the dump to the magnetic state (to an irrelevant N/S inversion). IPF provides (simplifying) a 0/1 NRZI stream, i.e. 0= no transition, 1=transition. If the number of 1 bits is odd, then the ends don't match and you have to add or remove one inversion somewhere. The global track formatting splice tends to be a good place for that.

That place is usually at the index on the ST (and often on the Amiga), but not always. In IPF the position is defined indirectly through a combination of gap type, gap position and index position. In the Chase HQ ST case, the (inferred) rules end up putting the splice on the first 4489 of the first sector, with damaging consequences. It's unclear whether the rules are wrong or the image is slightly wrong (splice at the index would work in that case).

Of course, generating a "surface image" automatically from Pasti dumps is way, way more complex

OG.

[DrCoolZic]

But that means one has to be able to go from the dump to the magnetic state (to an irrelevant N/S inversion). IPF provides (simplifying) a 0/1 NRZI stream, i.e. 0= no transition, 1=transition. If the number of 1 bits is odd, then the ends don't match and you have to add or remove one inversion somewhere. The global track formatting splice tends to be a good place for that.

Very interesting. Do you know if what you describe for IPF apply to Stream file?

[galibert]

But that means one has to be able to go from the dump to the magnetic state (to an irrelevant N/S inversion). IPF provides (simplifying) a 0/1 NRZI stream, i.e. 0= no transition, 1=transition. If the number of 1 bits is odd, then the ends don't match and you have to add or remove one inversion somewhere. The global track formatting splice tends to be a good place for that.

Very interesting. Do you know if what you describe for IPF apply to Stream file?

Well, the stream files are a direct map of the transitions, so theorically their could should be even on a complete revolution. In practice it may be more complex than that because of the splice, which can put two transitions a little too close to tell, or represent an unformatted part with beautiful randomness associated. That would be a minor part of the issues with adding kryoflux format support to Mess.

OG.

[mr.vince]

IMHO, the disk interface is not correctly designed for older DD 5.25 drives (should be fine with most newer HD 5.25 drives).

We indeed engineered KryoFlux for usage with HD drives as these can also read DD disks. However, we have people that are successfully reporting using it with DD drives. So speaking of "correctly".... how can we make it better?

And I still don't know where the write splice is for Chase HQ, hint hint

I will point IFW here, but he's currently in C64 land and messing with Vorpal so... I got to find a way to wake him up...

[ijor]

Well, Mess now uses a low-level magnetic zones representation for the floppies, with vectors of magnetic state (N/S/unmagnetized/damaged) and their angular size. The "floppy" class then answers questions such as "when is the next transition", and the wd1772 class handles all the pll stuff and everything. It actually makes the emulation somewhat simpler, as getting closer to the hardware often does.

Very interesting.

Again just out of curiosity: Did you have a specific reason for using specific magnetic poles? Or you just wanted to go as much low level as possible?

[ijor]

We indeed engineered KryoFlux for usage with HD drives as these can also read DD disks.

Yes, HD drives can usually read most DD disks. But as you know, you should ideally use a DD drive for writing back. Also, most DD drives can read flippies, while almost no HD can.

Yeah, I know you have a solution for flippies. But it requires a custom modification to the drive that is far and beyond the possibilities of most users. For most users it is far more convenient to use a DD drive that can read flippies "natively".

And besides and regardless of the capabilities, I guess we agree it is always good to support more type of drives.

So speaking of "correctly".... how can we make it better?

As I told you by PM some time ago, IMHO, the electrical interface is not the most appropriate. In the first place I'd replace the '244 drivers with ones with higher sink current capability. Some older drives would make those drivers sink too much current. And it might be even worse if two drives are connected at the same time. Ideally also the buffers should be open drain, but that is not nearly as critical as being high current.

Older drives tend to be electrically noisy, some kind of filters are recommended. And some type of filters could also give you protection that it would be also useful.

The pullups on the inputs are also out of specs. They should be stronger (lower ressistance), at least at the RD signal, obviously the most critical input.

That's again, IMHO, what I can see on a basic inspection of the board and checking the schematics. I might have missed something.

[mr.vince]

Yes, HD drives can usually read most DD disks. But as you know, you should ideally use a DD drive for writing back. Also, most DD drives can read flippies, while almost no HD can. Yeah, I know you have a solution for flippies. But it requires a custom modification to the drive that is far and beyond the possibilities of most users. For most users it is far more convenient to use a DD drive that can read flippies "natively".

Agreed. But: Doesn't using a DD drive mean flipping the disk to read the back side, because of the track offset? You would lose index information in this case.

As I told you by PM some time ago, IMHO, the electrical interface is not the most appropriate. In the first place I'd replace the '244 drivers with ones with higher sink current capability. Some older drives would make those drivers sink too much current. And it might be even worse if two drives are connected at the same time. Ideally also the buffers should be open drain, but that is not nearly as critical as being high current.

Older drives tend to be electrically noisy, some kind of filters are recommended. And some type of filters could also give you protection that it would be also useful.

The pullups on the inputs are also out of specs. They should be stronger (lower ressistance), at least at the RD signal, obviously the most critical input.

Thanks for this. Much of the above is intentional, e.g. like pullups, which could be changed, but would draw more current, leading to much more drain on USB. We only have 500mA for a standard port, so we were picky about not drawing too much. People with a drive not working (which are very few and less than it sounds above) could just power the board and exchange a pullup resistor (actually one array of four, that's it).

We even have 8" drives working with it...

As for the filters - the drivers are pretty robust, and we do have a filtering resistor on the read line. The other filtering can be done in the digital domain, where you can use all kinds of stuff. If we'd have this in the chain, there'd be no way of turning it off or changing things.

[galibert]

Again just out of curiosity: Did you have a specific reason for using specific magnetic poles? Or you just wanted to go as much low level as possible?

Well, I used 'A' and 'B' rather than N/S in order to show how arbitrary they are. But yeah, I wanted to be as low level as reasonable (not analog, thanks), so that meant zones or reversals. And zones are easier to handle when you also want to have non-magnetized and damaged parts.

At that point it's low level enough so that we're pretty sure we won't need to change anything fundamental anymore.

OG.