SCP disk images !

[coolhaken]

I am sorry. Copy Mode = "index" means blindly, "splice" means non-blind. I wrote the wrong word.
So, for flyers80 and dlfrsilver, please set Copy Mode = "splice" then dump those games again.

The following is the answer wrote by my friend NRS:

But why you require those disks to be dumped in splice mode? In my experience, it is better to dump multiple revolutions blindly. Then by software you can perfectly read sectors that cross the index.

The opposite is true. Dumping "blindly" means reading from index to index, so you may have a few bits missing at the index. No software will be able to reconstruct a sector across the index signal if it has bits missing in its middle. I suppose the word "blind mode" is misleading in this regard. I noticed that very problem when comparing the SCP image of Boulder Dash 1 with the Kryoflux image --- it was not a matter of how to interpret the data, but simply that there were bits missing at the end.
As a sidenote, SCP's software is the only flux-level software I know of that actually stops reading directly at the index signal. Even the old Central Point Transcopy software for the Option Board read at least until the next sync mark after the index.

[ijor]

The following is the answer wrote by my friend NRS:

Please transmit this to your friend (it wouldn't harm to join the forum, we don't bite

The opposite is true. Dumping "blindly" means reading from index to index, so you may have a few bits missing at the index. No software will be able to reconstruct a sector across the index signal if it has bits missing in its middle.

I think you are misinterpreting the format of the dump in multiple revolutions. What you say is true when a single revolution was dumped. In that case, the data will indeed stop at the index, and you might miss a couple of bits.

But when dumping multiple revolutions, reading of the disk is continuous. Reading doesn't stop and start again for each revolution. If dumping, say, two revolutions, then reading will start on the first index, and will stop at the third. So you have your data under the index. Nothing is missing.

So I still recommend using blind index to index mode, just in multiple revolutions, of course. Use the "splice" detection by yourself. Usually you, that knows better the platform and the specific format, are in better position to judge where the true splice is.

As a sidenote, SCP's software is the only flux-level software I know of that actually stops reading directly at the index signal. Even the old Central Point Transcopy software for the Option Board read at least until the next sync mark after the index.

The Discovery Cartridge, that predates both the SCP and the Kryoflux by a couple of decades, did stop at the index by default. Reading under the index was an option. But because of hardware constrains, reading under the index was expensive, so it made sense. The Catweasel, that came a few years after the DC, but yet long before the SCP and the Kryoflux stopped by default at the opposite (raising) edge of the index. That was a reasonable compromise giving you some overlap data under the index without need to interpret sync marks (which, of course, is not universal).

[JimDrew]

Supercard Ami (released in 1987, based around work on did on the Copy II option board) used the same modes that SuperCard Pro does. INDEX mode uses a single revolution, starting and stopping on the index mark for the read. On a write, the same is true. SPLICE is indentical to INDEX mode when doing read, except you can choose the number of revolutions. So, as Ijor mentioned you could start on the first and stop on the 3rd, with the spooled data being contiguous. Writing with SPLICE is much different than INDEX. When you write with SPLICE mode, the first revolution is always written, and writing continues until the write splice is found, at which point the writing is terminated. It is certainly possible to write out a single revolution and make the data under the index correct, but this requires multiple re-writes and verifies. It's much easier to just have a couple of revolutions of data. All reads are done "blindly", and only certain (Commodore) disk formats are analyzed and verified. Since there are many various utilities (aufit, HxC, Keir's utils, a8RawConv, SamDisk, etc.) to do that for many different disk formats, I didn't see the need to re-invent the wheel.

[coolhaken]

But when dumping multiple revolutions, reading of the disk is continuous.

If that is true then I don't know why the cross-index sector does not decode correctly. I have a Kryoflux image of the same disk, though not necessarily of the same copy, which works without problems. Does the copy used during imaging even work on real hardware, as in successfully running the game? If so, does writing that SCP image back result in a working copy? Given that many of the PC disk images on that FTP server by "flyers80 " are quite obviously not from original disks, and that the disk itself has a 180K file system on a 720K disk (making it look like a single-sided 5.25 inch disk copied over to a double-sided 3.5 inch disk), it's entirely possible that the physical disk itself would fail the keydisk check on real hardare and that I am unfairly blaming SCP.

For the record, I have uploaded to the "/Incoming KF" directory of that FTP server you told me my Kryoflux image of that Boulder Dash 1 disk; the archive also includes a Transcopy image, created from those Kryoflux stream files, that works in DOSBox-TC (cycles=500).

Can you join the Atari Forum ?

What for? I have got nothing to ask.

[JimDrew]

By the way, if you dump from index to index, you are never going to be missing any data. That will spool 100% of the bitcells contained within a single revolution. The very first bitcell time and the very last bitcell time needs to be added together and replaces either the first or last bitcell time. The first and last times will be split by the index mark. It's easy enough to prove this by comparing the data dumped using INDEX (1 rev) and SPLICE (2+ revs) modes. I have an experimental option to "rotate" the track in the editor/analyzer, which lets you rotate the track data forwards or backwards (circular track buffer). You can read a track that was not created on the index mark, rotate the track data so the write splice appears at the end of the track, write the track back, and the track will be error free. So, the bitcell under the index mark is correct.

[ijor]

Does the copy used during imaging even work on real hardware, as in successfully running the game? If so, does writing that SCP image back result in a working copy?

Is he asking me? I don't have that disk, so I have no idea if the disk works or not.

But when dumping multiple revolutions, reading of the disk is continuous.

If that is true then I don't know why the cross-index sector does not decode correctly.

Exactly which sector on which track you can't decode correctly? I downloaded the SCP image and I can decode sector $0B on the first track "correctly", at the extent that is recorded on the disk surface. That sector has a fake sector size and it is truncated, so I can't perform a CRC verification on that sector. On that particular copy, the sector was truncated after $8D bytes or so. After these $8D bytes, it is clear that the track write splice is present. From those $8D bytes, on that specific image, about 7 bytes cross the index. That is, about $85 bytes of the sector are before the index.

If that is good or not, if that would make it work or not, I have no idea. But if it doesn't, it doesn't seem to be a problem with the imaging process.

I have a Kryoflux image of the same disk, though not necessarily of the same copy, which works without problems.

Downloaded that image ... It is obviously a different copy. In first place the truncated sector ($0B) is slightly longer. The track write splice appears after $93 bytes or so. That is perfectly normal as a consequence of different RPM when each copy was recorded. But the copy imaged with the SCP seems also that it was modified. At the very least sector $0A on track 0 has been written on a second pass and the content is not identical to the other copy imaged with the Kryoflux.

Can you join the Atari Forum ?

What for? I have got nothing to ask.

Oh, man! Nobody said that you should join the forum to "learn". Just that this exchange of ideas could be more direct and fluid.

[NewRisingSun]

Oh, man! Nobody said that you should join the forum to "learn". Just that this exchange of ideas could be more direct and fluid.

For your information, the reason I could not register is that this forum software chooses to consider my email address "@web.de" as a "Spammers Domain - Stop Forum Spam!" Apparently the forum administrators blindly trust this list (https://www.stopforumspam.com/spamdomainsandips), which as of this writing also bans all gmail.com addresses. It's bad forum administration policy to ban registrations from large free email providers such as gmail, web.de or gmx across the board, but then enable users to register using "throwawayemail.com" which is what I had to use to register here.

Is he asking me?

I'm not sure who I'm talking to anymore. "coolhaken" PMs me on the vogons.org message board saying that he's got an SCP image that doesn't work in the experimental build of DOSBox that I sent him, which supports protected disk images. I ask him questions about where the image comes from and how it was imaged, and next you know, my private messages to him get publicly posted on a discussion board. Just to give you some background.

Exactly which sector on which track you can't decode correctly?

The disk is protected by WayDisk Minder version 2 (PDF page +4, right column). It checks, among other things, that the data of sector 11 at the splice point is exactly as expected on cylinder 0, head 0.

By the way, if you dump from index to index, you are never going to be missing any data. That will spool 100% of the bitcells contained within a single revolution. The very first bitcell time and the very last bitcell time needs to be added together and replaces either the first or last bitcell time. The first and last times will be split by the index mark.

Information such as this should be in the SCP specification. As someone trying to understand the format, the specification and manual are somewhat vague on this question, hence my incorrect understanding that even multi-revolution images contained every revolution read separately. So, for each revolution, am I supposed to run

Code: Select all

cellLength[firstCellNumber] += cellLength[lastCellNumber--];

after parsing the data, or do I do that only for single-revolution images?

[ijor]

I'm not sure who I'm talking to anymore. "coolhaken" PMs me on the vogons.org message board saying that he's got an SCP image that doesn't work in the experimental build of DOSBox that I sent him, which supports protected disk images. I ask him questions about where the image comes from and how it was imaged, and next you know, my private messages to him get publicly posted on a discussion board. Just to give you some background

Ok. I understand. I'm known as Ijor. I specialize in copy protections for a few platforms, mainly Atari. I have no relation with whoever dumped that disk. Neither I am affiliated with SCP (developed and sold by Jimdrew) in anyway. But dealing with Atari disks, and as most Atari 8-bit disks are not aligned at all with the index, I think I have some experience decoding tracks "under the index".

The disk is protected by WayDisk Minder version 2 (PDF page +4, right column). It checks, among other things, that the data of sector 11 at the splice point is exactly as expected on cylinder 0, head 0.

Well, as I said, I could decode sector 11 across the index "correctly". As you surely knows, that sector has a fake sector size of 8K and it is truncated. Only $95 or $8D or so bytes were recorded. Then, you can't perform a CRC verification. And unfortunately that protection article you link doesn't offer much technical details.

So I don't know exactly what the protection expects from that sector. I do know that I decoded correctly the sector on that SCP image. Correct in the sense that I can decode what was recorded on that particular copy. I'll include a couple of hex dumps in a separate post.

By the way, if you dump from index to index, you are never going to be missing any data. That will spool 100% of the bitcells contained within a single revolution. The very first bitcell time and the very last bitcell time needs to be added together and replaces either the first or last bitcell time. The first and last times will be split by the index mark.

Information such as this should be in the SCP specification.

If I understand what Jim meant, he is talking about something inherent to dumping one revolution, not something specific to the SCP format. But honestly, I don't agree 100% with Jim here.

[NewRisingSun]

So I don't know exactly what the protection expects from that sector.

The keydisk checking code expects the sector to be as decoded from the Kryoflux image which you already wrote you had taken a look at. If there is a way of decoding the SCP image to yield that result, then it is a question of interpreting the SCP data; otherwise, the SCP image is merely a bad dump.

[NewRisingSun]

To be more precise about what the protection is looking for: from what I can tell (*) it counts the number of identical bytes at the beginning of the sector and uses the resulting count as part of a decryption key.
(*) The protection uses the Trace vector decoder method of obfuscating its code, so it's a bit tedious to find out what exactly is going on when debugging.

[ijor]

So I don't know exactly what the protection expects from that sector.

The keydisk checking code expects the sector to be as decoded from the Kryoflux image which you already wrote you had taken a look at. If there is a way of decoding the SCP image to yield that result, then it is a question of interpreting the SCP data; otherwise, the SCP image is merely a bad dump.

Again, you have to be more specific. The sector is truncated by the write splice. So after the point of truncation, what you decode is not very reliable in either image. That sector has a theoretical size of 8K. I assume that the code doesn't actually check 8K worth of bytes, does it? I assume it checks less.

So the question is how many bytes it checks. If it checks for the exact number of bytes that the Kryoflux image has before truncation ($93), then it would fail. But then it means the disk that was imaged with the SCP is bad, not that it is a bad dump.

Note that it is very possible that the number of bytes checked depends on the exact copy. That is, it is possible that after the original disk was recorded, after each copy, the duplication process checks how many bytes were actually written on that particular copy, and adjusts the keydisk checking code accordingly.

This is sector 11 on track 0/0 on the Kryoflux image you posted:

Code: Select all

00001980   FB 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   ûNNNNNNNNNNNNNNN
00001990   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019A0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019B0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019C0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019D0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019E0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019F0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
00001A00   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
00001A10   4E 4E 4E                                           NNN

And this is the same sector on the SCP image:

Code: Select all

00001990   .. .. .. .. .. .. FB 4E  4E 4E 4E 4E 4E 4E 4E 4E   ......ûNNNNNNNNN
000019A0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019B0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019C0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019D0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019E0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
000019F0   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
00001A00   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
00001A10   4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E   NNNNNNNNNNNNNNNN
00001A20   4E 4E                                              NN

You can see that I decoded across the index correctly. The sector on the SCP image is shorter, but this is because it was recorded like that. It's $93 bytes vs $8D bytes. After that, the write splice is present.

But the sector on the SCP image crosses the index. As I said in the previous post, from these $8D bytes, only $85 are before the index. But as you can see, I can decode the rest of the bytes across the index correctly.

Again, don't know if this will work or not. But decoding across the index is not the problem. At least not in this case.

[ijor]

To be more precise about what the protection is looking for: from what I can tell (*) it counts the number of identical bytes at the beginning of the sector and uses the resulting count as part of a decryption key.

I see. Then what I said in the previous post is not just possible but likely. If it really depends on the exact number of bytes, without any tolerance whatsoever, then it probably means that the key is generated after the copy was recorded, verifying exactly how many bytes resulted to be the same in this particular copy.

Which means that the difference between both images doesn't necessarily means one is bad.

(*) The protection uses the Trace vector decoder method of obfuscating its code, so it's a bit tedious to find out what exactly is going on when debugging.

Yeah. That is very common on the ST. It's a PITA if you don't have the tools. You need a good emulator that would output logs of the tracing.

[NewRisingSun]

Well, as I said, I could decode sector 11 across the index "correctly".

Take a look at page 14 of the "Kryoflux Stream Protocol V1.1". That explains exactly how the index times (which I assume match the "duration" field in the SCP track header) are to be interpreted relative to the flux times. Something like this would be desirable for SCP.

Using modified Kryoflux decoding code, I can easily get a clean decode from end the of rev 0 over the beginning of rev 1, but not from the end of rev 0 over the beginning of rev 0, or from the end of rev 1 over the beginning of rev 1. The flux in which the index occurs is considered part of the previous revolution. This works well for Kryoflux images, and should work for SCP images according to that statement by Jim, but I cannot get it to work. I could also (possible mis-) interpret Jim's statement as saying that the flux in which the index pulse occurs is split into two pulses, and so for multi-revolution images, I need to combine the last pulse duration of rev 0 with the first pulse duration of rev 1. Hence my question.

[ijor]

Using modified Kryoflux decoding code, I can easily get a clean decode from end the of rev 0 over the beginning of rev 1, but not from the end of rev 0 over the beginning of rev 0, or from the end of rev 1 over the beginning of rev 1.

I don't understand why you want to do that. If you can decode revs 0 and 1 together correctly, then you already have the data under the index, don't you?

Why are you trying to "join" end of rev 1 with start of rev 1 ??? Or are you just trying to implement code that will decode under the index even for images with a single revolution?

[NewRisingSun]

That, and because I am converting between file formats, including to those that can store only a single revolution, such as TransCopy or PRI. And given that it works perfectly with Kryoflux sources, it should work with SCP sources as well.

[ijor]

That, and because I am converting between file formats, including to those that can store only a single revolution, such as TransCopy or PRI. And given that it works perfectly with Kryoflux sources, it should work with SCP sources as well.

Well, I don't think that using a single revolution is foolproof in every case. I don't agree with Jim in this regard.

Using a single revolution would work only if you assume a fixed constant relation between flux transitions and index pulses. But due to small magnetic variations, and mainly due to mechanical/optical imprecision, there is some jitter. A given flux transition in one revolution might appear before the index pulse, but in the next revolution might appear after the index. Some drives have quite some jitter, specially 5.25 ones.

If you want to convert to a single revolution format, or build a circular buffer, then you will need to take (at least) two revolutions, perform pattern matching between the start of rev 0 and the start of rev 1 to detect and locate a possible index jitter. Not to difficult except on extreme cases where the track is unformatted and you read mostly noise. You can also achieve a similar result after decoding, but in some cases this might require at least 3 revolutions.

Btw, the people behind the Kryoflux agree with what I just said. So it is not something specific to SCP.

[NewRisingSun]

Well, with Kryoflux images, my approach has worked for every single image so far that uses sector-spanning protections --- I count eight such images that I imaged myself, too many for me to just have been lucky ---, so I beg to differ.

Btw, the people behind the Kryoflux agree with what I just said.

Link to source, please. The only statement I remember why Kryoflux people want multiple revolutions is to properly identify weak bits, as a single revolution may have a few of them randomly in the "1" state, thus making the detection method "illegal number of zero bits in a row" unreliable with just one revolution.

[ijor]

Btw, the people behind the Kryoflux agree with what I just said.

Link to source, please. The only statement I remember why Kryoflux people want multiple revolutions is to properly identify weak bits, as a single revolution may have a few of them randomly in the "1" state, thus making the detection method "illegal number of zero bits in a row" unreliable with just one revolution.

From this post in this very same forum:

http://atari-forum.com/viewtopic.php?f= ... 70#p263470

The absolute minimum number of revolutions you need to sample is 2 for commercial disks.
The reason is really simple: on some track(s) data may pass over the index signal. There is no foolproof way of reconstructing such data from a single revolution ... Add index signal jitter ... Another problem as I said is index signal jitter. Usually not so bad with most 3.5 drives, can get very bad with 5.25 drives. With a single revolution you risk not sampling the entire track if the index signal slightly varies (jitter).

[NewRisingSun]

So the reason that it always worked with my Kryoflux images is that my drive is very accurate, while the drive used to make that Boulder Dash 1 SCP image is not that accurate? I still need an explanation that accounts for that. Maybe Jim can add another two cents here.

[ijor]

So the reason that it always worked with my Kryoflux images is that my drive is very accurate, while the drive used to make that Boulder Dash 1 SCP image is not that accurate? I still need an explanation that accounts for that.

Honestly, I don't know, and I have no way to actually know. May be your drive has almost no jitter and the other has. May be your were lucky. May be they process single rev images slightly different (I don't have access to either firmware source to know). May be you are misinterpreting Jim comments and are processing the SCP image incorrectly. Or may be there is a combination of factors.

Btw, the SCP image has 40 tracks. If it was taken on a low density 360K drive, then it is very likely that is has much more jitter than a 3.5 drive. But again, I am just speculating. I have no way to be sure.

[NewRisingSun]

Well, let's just wait for Jim's response, then.

In the meantime, if you have the time, could you please take a look at the same FTP server, file "/SCP/IBM-PC/[SCP-IBM PC] JOE & MAC CAVEMAN NINJA (ELITE SYSTEM - EURO version) - IBM PC - (C) 1991 DATA EAST - ELITE SYSTEM LTD.7z"? I can decode almost all SCP images properly (except of course those with index-spanning sectors ), including 1.44M ones, but this one gives me a headache, as I cannot seem to extract any usable data from it. It takes forever in the HxCFloppyEmulator to load, and exporting to Kryoflux makes HxC freeze at track 1, and when I finally killed the process, I had a 60 GB file. (As before, I did not produce that image.)

[JimDrew]

By the way, if you dump from index to index, you are never going to be missing any data. That will spool 100% of the bitcells contained within a single revolution. The very first bitcell time and the very last bitcell time needs to be added together and replaces either the first or last bitcell time. The first and last times will be split by the index mark.

Information such as this should be in the SCP specification. As someone trying to understand the format, the specification and manual are somewhat vague on this question, hence my incorrect understanding that even multi-revolution images contained every revolution read separately. So, for each revolution, am I supposed to run

Code: Select all

cellLength[firstCellNumber] += cellLength[lastCellNumber--];

after parsing the data, or do I do that only for single-revolution images?

The information about the first/last bitcell time split should be obvious to anyone that would be writing software that deals with flux data. I guess I will mention this in the documentation.

Multi-revolution images do NOT have every revolution read separately. The revolutions are read contiguously, but the info for each revolution shows the offset for the start of each revolution's data.

The index pulse sensor is typically a hall sensor with 3.5" disks, and an optical with 5.25" disks. Even with 5.25" disks the optical sensor response time is in tens of nanoseconds and consistent, not microseconds and triggering randomly, so there should never be any type of problem getting the same data start position every time you read a revolution. The time factor (track duration) will change due to drive speed warble, but the index trigger is consistent. If you use the SCP's analyzer software you can view the number of bitcells read in the Flux Display window. The only time you will see the number of bitcells change is due to a write splice/weak bit, and that is typically by 1 or 2 bitcells.

I am not a C programmer, so I can't help you with your code. But if you would like an example how to control the SCP hardware and properly decode/encode the .scp image format, there is source code readily available that Keir Fraser wrote. It's located here:

https://github.com/keirf/Disk-Utilities

Keir has routines for reading, writing, decoding, and encoding. Basically everything you need for making a copier/imager.

[NewRisingSun]

The information about the first/last bitcell time split should be obvious to anyone that would be writing software that deals with flux data.

It's not obvious to me, and Ijor thinks that treating any particular revolution as a circular buffer would fail because of jitter.

[JimDrew]

I added more to my response while you were responding.

[NewRisingSun]

so there should never be any type of problem getting the same data start position every time you read a revolution. The time factor (track duration) will change due to drive speed warble, but the index trigger is consistent.

Well, I have your word, and my own experience with Kryoflux images, against Ijor and IFW (of the Kryoflux team) on this.

I guess I will just keep on trying. The image I am having problems with is "/Incoming SCP/Ibm-Pc by flyers80/[SCP-IBM PC] Boulder Dash.7z" on the FTP, should you want to take a quick look at it.

Edit: I just used HxC to convert my working Kryoflux image to SCP format, then used my SCP decoder to find that the Kryo->SCP converted image works nicely. My SCP decoder may be good after all, and I just may be wasting a lot of time on what might be a bad dump after all.