Imaging and image conversion under Linux or on real ST?

[Lando_C]

Hi!

I run steem under linux, and have my 1040 set up with a null modem cable, so I can use emulator.acc and XYZ.prg to transfer files to an from the linux box. This works nicely, but, most imaging tools are made for MS-DOS/windows, and now i stumbled on a problem:

The atariforum demo only comes in MSA format.. adn steem only reads .ST format images.

is there a MSA to ST conversion program available as source, linux binary or as a last resort real ST binary ? (but if I have to convert on the ST it means I have to zmodem the data twice across 19200 null modem - that's sloooooow)

[simonsunnyboy]

Steem reads and writes .MSA images as well!

You can convert msa to st and viceversa with a small tool written by Thothy: http://thothy.atari.org/

[daeghnao]

I'm working on a bit of imaging code to work with the zmodem protocol - I wanted to try something out to refresh my coding skills, and it looks like it'll be a useful application. Watch out over the next few weeks, all being well I'll get my code into a fit state...

[Mug UK]

The only code I could find (non-specific - I think - to a particular OS) is RAWRITE.

It's in Pascal so might be of some use or none at all!

[daeghnao]

As I understand it, rawrite just streams to/from the disk sector by sector to create or write an image. Linux already has this capability in its filesystem, the devices /dev/fd* represent the raw disk data. So, if your driver auto-detects the disk format (I think anything recent should do) and your disk controller can handle it, you can image disks on Linux just using 'dd' or 'cp'. Try 'minfo a:' first to check that your drive can read the disk. Unfortunately for me, I have very basic floppy facilities, so my PC can hardly read any of my ST disks.

[Mug UK]

The .ST format is basically that .. a raw dump of each track and sector.

Unfortunately, there's not much source code out there (freely available) to read in data off a floppy disk into an image

As for Linux-related stuff ... que? Talking to Windows user here

[daeghnao]

As for Linux-related stuff ... que? Talking to Windows user here

Doing one of these at a command prompt in Linux:

Code: Select all

cp /dev/fd0 image.st

Code: Select all

dd if=/dev/fd0 of=image.st bs=512

makes an image of whatever's in the drive that other OSs like to call drive A:

If you do it the other way round, it does the same job as rawrite, putting an image onto the disk - assuming the disk has the same format as the image. There are various ways of telling it what the format of the disk should be, but none of them are pretty, nor can I remember them off the top of my head.

[daeghnao]

The atariforum demo only comes in MSA format.. adn steem only reads .ST format images.

I've used steem with MSA images on Linux for quite some time now - some ideas that came to mind: do you have a recent version of steem? I don't know whether it had MSA support right from the start (probably did, knowing how industrious the authors were in getting it right). Also, is the msa file downloaded properly? Try using the 'file' command at the commandline, e.g.

Code: Select all

> file STRE34_2.MSA
STRE34_2.MSA: Atari MSA archive data, 10 sectors per track, 2 sided, starting track: 0, ending track: 79

Of course, this assumes that your magic database knows about msa files...

[Lando_C]

:oops:

Oopsie.. i had the files in the wrong dir.. steem runs MSA files just nicely =)

But, linux sure does raw floppy access.

Is there a detailed specification of the .ST and .MSA formats available?

[Mug UK]

Taken from the MSA2ST document by Damien Burke:

• .MSA FILE FORMAT
  ============
  
  For those interested, an MSA file is made up as follows:
  
  Header:
  
  Word ID marker, should be $0E0F
  Word Sectors per track
  Word Sides (0 or 1; add 1 to this to get correct number of sides)
  Word Starting track (0-based)
  Word Ending track (0-based)
  
  Individual tracks follow the header in alternating side order, e.g. a double
  sided disk is stored as:
  
  TRACK 0, SIDE 0
  TRACK 0, SIDE 1
  TRACK 1, SIDE 0
  TRACK 1, SIDE 1
  TRACK 2, SIDE 0
  TRACK 2, SIDE 1
  
  ...and so on. Track blocks are made up as follows:
  
  Word Data length
  Bytes Data
  
  If the data length is equal to 512 x the sectors per track value, it is an
  uncompressed track and you can merely copy the data to the appropriate track of the disk. However, if the data length value is less than 512 x the sectors per track value it is a compressed track.
  
  Compressed tracks use simple a Run Length Encoding (RLE) compression method.
  
  You can directly copy any data bytes until you find an $E5 byte. This signals a compressed run, and is made up as follows:
  
  Byte Marker - $E5
  Byte Data byte
  Word Run length
  
  So, if MSA found six $AA bytes in a row it would encode it as:
  
  $E5AA0006
  
  What happens if there's an actual $E5 byte on the disk? Well, logically
  enough, it is encoded as:
  
  $E5E50001
  
  This is obviously bad news if a disk consists of lots of data like
  $E500E500E500E500... but if MSA makes a track bigger when attempting to compress it, it just stores the uncompressed version instead.
  
  MSA only compresses runs of at least 4 identical bytes (after all, it would be wasteful to store 4 bytes for a run of only 3 identical bytes!). There is one exception to this rule: if a run of 2 or 3 $E5 bytes is found, that is stored appropriately enough as a run. Again, it would be wasteful to store 4 bytes for every single $E5 byte.
  
  The hacked release of MSA that enables the user to turn off compression
  completely simply stops MSA from trying this compression and produces MSA images that are completely uncompressed. This is okay because it is possible for MSA to produce such an image anyway, and such images are therefore 100% compatible with normal MSA versions (and MSA-to-ST of course).
  
  .ST FILE FORMAT
  ===========
  
  The file format of the .ST image files used by PaCifiST is simplicity itself;
  they are just straight images of the disk in question, with sectors stored in
  the expected logical order. So, on a sector basis the images run from sector 0 (bootsector) to however many sectors are on the disk. On a track basis the layout is the same as for MSA files but obviously the data is raw, no track header or compression or anything like that.
  
  .ST files can be written to real disks using DKC's STImage program for the PC, available from my web site (see top of this document). If you have .ST files on your real ST you can write them to a disk using ArghBlarg's STTODSK.TTP program, also available from my web site.
  
  You can also use MSAtoST's companion program, STtoMSA, to convert .ST files to MSA files. That is also available from my web site.

[thothy]

Is there a detailed specification of the .ST and .MSA formats available?

You might also want to check my MSA-to-ST converter from:
http://www.uni-ulm.de/~s_thuth/ix/msa-0.1.0.tar.gz

[ijor]

Linux already has this capability in its filesystem, the devices /dev/fd* represent the raw disk data.

Virtually every OS, including Windows can read "raw disk data" at the logical sector level. And this is what you are doing in Linux when accessing /dev/fd0.

dd isn't really part of the OS or the filesystem. It's a standard user mode program. If you wont you could easily write a dd version for Windows, and probably already exists one.

But the main problem is the disk geometry, which is required to translate from physical to logical sectors and viceversa. This is where Windows is severely limited.

So, if your driver auto-detects the disk format (I think anything recent should do)

I’m not sure about how the auto-detect capabilities of the Linux floppy driver. But there is no effective way to auto-detect the geometry of same ST disks. How you could detect, for example, the number of tracks if it’s not specified correctly in the boot sector?

and your disk controller can handle it

Seems there are some mythos about modern PC floppy controller. I’ve never seen or heard about any FDC that cannot read all ST disks. The only exception is when there is no actual FDC in the PC, such as USB floppy drives. Or some ST copy protected disks.

Unfortunately for me, I have very basic floppy facilities, so my PC can hardly read any of my ST disks.

Again, unless you have a special floppy (such as USB), then your problem is somewhere else. Either in the driver or in the need to manually set the floppy geometry.

[daeghnao]

Linux already has this capability in its filesystem, the devices /dev/fd* represent the raw disk data.

Virtually every OS, including Windows can read "raw disk data" at the logical sector level. And this is what you are doing in Linux when accessing /dev/fd0.

dd isn't really part of the OS or the filesystem. It's a standard user mode program. If you wont you could easily write a dd version for Windows, and probably already exists one.

For windows, you have rawrite and rawread for this kind of thing. The point I was making is that in Linux you don't necessarily use a specialised program, it exposes the raw data as a file and hence you can use a general-purpose tool (dd or cp) to manipulate that data.

But the main problem is the disk geometry, which is required to translate from physical to logical sectors and viceversa. This is where Windows is severely limited.

So, if your driver auto-detects the disk format (I think anything recent should do)

I’m not sure about how the auto-detect capabilities of the Linux floppy driver. But there is no effective way to auto-detect the geometry of same ST disks. How you could detect, for example, the number of tracks if it’s not specified correctly in the boot sector?

and your disk controller can handle it

Seems there are some mythos about modern PC floppy controller. I’ve never seen or heard about any FDC that cannot read all ST disks. The only exception is when there is no actual FDC in the PC, such as USB floppy drives. Or some ST copy protected disks.

Unfortunately for me, I have very basic floppy facilities, so my PC can hardly read any of my ST disks.

Again, unless you have a special floppy (such as USB), then your problem is somewhere else. Either in the driver or in the need to manually set the floppy geometry.

Hmm, some food for thought there. I had the impression from reading through some documentation that the driver would step through a list of possible geometries until it found one that allowed the disk to be read. I will investigate further when I have a moment; when I last tried this, I couldn't even read the bootsector of the particular disk I was trying it on, so I naturally assumed that the hardware wasn't up to it. Now that you've cleared that one up, I'll have a proper go at it when I next get a chance, maybe tomorrow night if I'm lucky.

[ijor]

For windows, you have rawrite and rawread for this kind of thing. The point I was making is that in Linux you don't necessarily use a specialised program, it exposes the raw data as a file and hence you can use a general-purpose tool (dd or cp) to manipulate that data.

Windows does this as well. It exposes any physical drive (and also any volume or partition) as a standard device that can be easily accessed from user space. For example, drive A: is “\\.\a:”

You don’t need any “specialized” tool. It just lacks a “dd” tool in the standard distribution (which again, it has nothing to do with the OS or the filesystem). A simple “copy like” program would work. I just tried and the integrated “DOS copy” does not for some reason, but it perfectly could.

Hmm, some food for thought there. I had the impression from reading through some documentation that the driver would step through a list of possible geometries until it found one that allowed the disk to be read.

I'd need to check the source to comment further on this. But my guess is that is not detecting it correctly.

I understand that there are specific ST drivers for Linux that are not part of the standard kernel. What I don't know for sure if the drivers are for hard disks only (some ST partitioning are not fully compatible with PC), or for floppies as well.

So it's possible that you need those drivers. But again, in some cases there is no effective alternative but to manual specify the geometry.

[daeghnao]

For windows, you have rawrite and rawread for this kind of thing. The point I was making is that in Linux you don't necessarily use a specialised program, it exposes the raw data as a file and hence you can use a general-purpose tool (dd or cp) to manipulate that data.

Windows does this as well. It exposes any physical drive (and also any volume or partition) as a standard device that can be easily accessed from user space. For example, drive A: is “\\.\a:”

You don’t need any “specialized” tool. It just lacks a “dd” tool in the standard distribution (which again, it has nothing to do with the OS or the filesystem). A simple “copy like” program would work. I just tried and the integrated “DOS copy” does not for some reason, but it perfectly could.

Ah, very interesting. I've had a poke around the MS site, it looks like these \\.\a: type things are called "volume devices" and appeared in Windows 2000. I did see reference to another syntax starting \\?\ and naming device and partition numbers, something from NT perhaps? Rawrite and rawread are fairly old programs, likely they predate the introduction of volume devices. Fascinating stuff, I'll have to see if there's a brief guide to these devices somewhere. Is there really no way to manipulate the geometry from the command line, though?

As for dd, there's http://uranus.it.swin.edu.au/~jn/linux/rawwrite/dd.htm and http://users.erols.com/gmgarner/forensics/ - both of these look like pretty good things to know about. So on a reasonably recent Windows, the general-purpose approach ought to be available too.

Hmm, some food for thought there. I had the impression from reading through some documentation that the driver would step through a list of possible geometries until it found one that allowed the disk to be read.

I'd need to check the source to comment further on this. But my guess is that is not detecting it correctly.

I may have been remembering the mtools documentation, now that I think about it. I do loathe having to go to the source to find out what things do, I remember a time when Linux was reasonably well-documented...

I understand that there are specific ST drivers for Linux that are not part of the standard kernel. What I don't know for sure if the drivers are for hard disks only (some ST partitioning are not fully compatible with PC), or for floppies as well.

So it's possible that you need those drivers. But again, in some cases there is no effective alternative but to manual specify the geometry.

I recall reading something about ST hard drive devices, I think you're onto something there. Will be interesting to see what I can make it do.

[thothy]

While creating disk images from my old ST disks under Linux, I also often had the problem that the auto-detection of /dev/fd0 failed. But there are other device files, with fixed geometries:

- Use /dev/fd0u720 for standard disks with 80 tracks, 9 sectors per track and 2 sides

- Use /dev/fd0u800 for disks with 80 tracks, 10 sectors per track and 2 sides

- ... etc ... (see the output of "ls /dev/fd0*" for more)

Another hint: Use "dd_rescue" instead of "dd" to create disk images, this tool can handle bad sectors in a better way.

[Lando_C]

Today (night actually..) I have watched parts of The Union Demo, The Mindbomb demo, Syntax Terror demo, Ooh crikey wot a scorcher and SoWatt, On my 520 STFM, real iron, not emulated. niiiice.

BUT the megafile 20 is filling up, despite I only have a nullmodem to fill it through..... NEED. MORE. DISKSPACE.



Mindboggling!

[daeghnao]

As promised, I have been investigating direct access to the floppy controller on Linux to try to image these disks. I've successfully managed to discover the size of the floppy by programming up a script that tries to read particular sectors. So, I can tell whether the disk has 9 or 10 sectors per track, for example.

However, even with the drive parameters set to what I've detected (and I've checked against what the real STE thinks the disk size is) I still can't read bootsectors on 10-sector disks. I can get data from 9-sector disks just fine, though. I haven't tried systematically accessing directly sector by sector to see what I can get to apart from the sectors that I probe for the size and the bootsector - that may be the next step, though.

So, current status - all the software seems to be working fine, but things are still not happy...

[daeghnao]

Been playing with floppies and Linux a bit more, interesting things are happening. I installed a second floppy drive and tried out the ten-sector disk that I was having trouble with. I had the same trouble as with the first drive, so that makes it unlikely to be a problem with the floppy itself.

Next, I wrote a script that iterates over the disk issuing explicit read commands to the controller, to read the contents of the disk. Trying this on the non-working 10-sector disk revealed what might have been a pattern in the sectors that can't be accessed. To check this, I looked for more 10-sector disks, and found one where I did seem to be able to read the bootsector. So, I ran my script on that disk, and I managed to get a flawless image from it.

So, my problem doesn't seem to be 10-sector disks in general, just particular ones. My hunch is that it's to do with either the drive or the program that originally formatted the disk. Next on my list of things to try out, is to see if all the disks that I can't read properly exhibit the same pattern of inaccessible sectors.

[beeka]

Windows NT (and 2000 onwards) are posix compliant, so they also expose the disk device as a file. Several years ago, I wrote a small app to read that into a file. This made nice images but no luck with non-standard formats went. It could even read the bootsector to determine the proper geometry for the disk (which worked) but could not make a proper image.

I thought the problems were with Windows... some think it is the controller chip. Could the problem also be the use of HD disk drives? Using HD disks in STs (as DD ones) is known to be problematic. Could it be that the PC drives are too precisely aligned and that the data is 'missed' on newer drives (except with some disks)?

Just a thought,

Steve.

PS: Having typed that, I realise that this might be nonsense... most standard DD disks I have tried worked fine. But I will put the thought out there anyway.

[daeghnao]

Latest in the saga of my old floppies... I was fiddling about with copying disks on the nice shiny STE, and I happened to make a disk that had the same trouble as my older 10-sector disks - a pattern of sectors that I can't read, like this:

Track 0 Sector 1
Track 1 Sector 7
Track 2 Sector 3
Track 3 Sector 9
Track 4 Sector 5
Track 5 Sector 1
Track 6 Sector 7
...

And so on throughout the disk (I just checked the whole disk through with raw drive accesses to make sure the pattern went all the way). It's exactly the same as the pattern of unavailable sectors on about 100 floppies that I have. They'll read fine in the STE, but not in the PC.

How did I make this disk? My old copy of Fastcopy Pro, that's how. Just to double-check, I downloaded 'brdlfrmt' from umich, formatted a disk to 10 sectors on the STE, checked it on the PC... no problems. Same disk formatted with Fastcopy Pro, the pattern of unreadable sectors arises. I think a large proportion of the floppies I have were created with Fastcopy 3, which might also produce the same trouble.

Now, is there anyone who can theorise on what the underlying cause is, and whether it's fixable in place?

[ggn]

Format skewing (also called quick format)?

Turn this off and re-format the disk.

George

[daeghnao]

Format skewing (also called quick format)?

Turn this off and re-format the disk.

George

It's "Fast Format" on my fcopy pro. And turning it off does make a readable disk. I'd expected the disk to always be skewed, no matter what. Anyway, now that I know what it is that's been stopping the disks from working, I can avoid it in the future. Thanks for the suggestion.

As for my existing disk collection, I can copy each one onto a non-skewed disk and then read the files on the PC, and that's a lot less hassle at the moment than transferring them over the serial line with zmodem protocol.

[Lautreamont]

Format skewing (also called quick format)?

Turn this off and re-format the disk.

George

It's "Fast Format" on my fcopy pro. And turning it off does make a readable disk.

As for my existing disk collection, I can copy each one onto a non-skewed disk and then read the files on the PC
[snip]

I've experienced the same problem,
but didn't figure out that FastCopy was guilty:

Of the 35 disk images I've done recently, only 12 were free of read errors
on my PC driver. I thought they were too old and felt lucky my STe could
still read them and copy them on newer disks. I even did the copies with
FastCopy III, and asked for a safe format just because I already had
read errors with those disks!

Good catch daeghnao!

[daeghnao]

I had some more problems with disks, but I think they were to do with a lot of my disks having had damaged sectors before I started using them (of course, there's no data written to the damaged part, as the formatting marked it as such). What I'm now doing is imaging to ramdisk with fcopy, then transferring with zmodem over a null modem link. Have found some interesting old code, which I'll hopefully renovate at some point. I'll write up a nice long article about the whole thing soon.