Data corruption with CosmosEx SD card (Falcon/SCSI)

[Jookie]

I activated verify on write for the driver without seeing any improvements whatsoever.

Ow drat, that should have helped.

How does the RAW mode work? How is the data buffered in the RPi? It appears to me that there is a cache involved (in the RPi) in which the data is actually correct and which is accessed when verify is activated, but when the data is later actually written to the SD-card it is corrupted or the cache is corrupted. Does this make sense? It's hard to explain otherwise why verify does not detect the corruption.

The RAW mode is related to USB drives - acting as a bunch of sectors to ST, so HDDRIVER (or any other RAW disk driver) can handle it at sector level. It's used when you plug USB drive to USB port, the sectors of that drive are accessed and forwarded to ST, or written from ST to it. There might be some buffering done there (in linux kernel), so after writing data to RAW USB drive (or even better - to USB drive in general, even in translated mode), you should use CE_MOUNT.PRG to unmount that drive before you remove it from USB drive, that should flush all the caches. It shouldn't be related to SD card as you mention the SD card there...

Btw., I got one promising fix for the SCSI transfer issues - that problematic device which failed the stress test a lot now seems to be working fine. I need to do more stress testing, data validity testing, and if it's fine, then I need to test it on the other device, which didn't suffer from this problem (or it suffered only very occasionally, so I didn't notice it). If the same passes on the better device, then I will consider this as a good fix and there will be an update of the Xilinx firmware for SCSI tomorrow or the day after that.
If it will fail (for some magical reasons), I have a different fix and code in an alternative git branch, that would require one week or so of finishing, and it would mean a change to all the devices (even ACSI ones), so I hope that the SCSI fix will be just fine because this is much more work

Jookie

[stimpy]

What termination resistor packs are you using on the CosmosEx? I measured them and they do not seem to be the standard SCSI 220/330 ohm resistor networks (200/330 ohm resistor packs are used internally in the Falcon as well). Could the termination of the CosmosEx be a problem?

Well, maybe, but my current experiments don't show that this could be the problem (or I'm measuring it wrong). Those are 100 (or 110 ohm) resistor networks, connected to LM317 stabilizer, acting together as active termination (instead of passive 220/330 ohm ones).

Does active termination need a sink/source driver?

[Jookie]

Does active termination need a sink/source driver?

No, it should be fine just handled by the CPLD...

[Jookie]

dhedberg, the fix seems to improve the situation, for more details on things in update see here:
http://www.atari-forum.com/viewtopic.ph ... 77#p306477

Please run the update on your device, wait at least 2 minutes if it doesn't get back to life sooner (Xilinx update takes aroun 50 seconds), then run the CE_TSTHD.PRG from config drive:
- C for - continous read test
- On fail - pause until key pressed? YES - to make it stop, so you could spot the issue
- On fail - do SCSI reset? - doesn't matter, but press N for NO
- Show asterisk after good result? - press Y if you want to see the '*' after each good read, or press N if you're fine with empty screen but higher stress (no echo on screen -> less time between commands -> more stress)
- if it doesn't fail soon / at all, then let it run for a hour or so
- if everything went ok for an hour, then run the '[D] - data from RPi validity check'
- same as previous - if nothing bad is happening, let it for hour, if there is some error, you can quit it and report sooner

Also there was a minor fix with large impact on USB RAW drives - a small mistake resulted in bad access to higher sectors on USB RAW drives, so this update could improve the USB RAW drives, too.

[dhedberg]

I ran the continous stress test for 1.5 hours => no errors reported.
After that I ran the RPi validity check for 2 hours (2000Mb) => no errors reported.
I wrapped everything up with copying a 1MB file from my IDE drive to the CosmosEX SD-card => data corruption.

[Jookie]

I ran the continous stress test for 1.5 hours => no errors reported.
After that I ran the RPi validity check for 2 hours (2000Mb) => no errors reported.

OK, this is great, as you had some errors before and now it seems to be better on the lowest level

I wrapped everything up with copying a 1MB file from my IDE drive to the CosmosEX SD-card => data corruption.

CosmosEx hardware failure? Has anybody else with a Falcon reported the same kind of problems?

This one (hopefully) last issue is related to handling SD card - got the same thing reported on ST through ACSI, so this is not Falcon / SCSI related - I need few more days for this one. If I remember correctly, it was fine before some time, but then there were some changes to the Hans firmware (which handles SD card) and now we got this... SO I would say - so far, so good, one more thing to go and we will be fine!

Thank you for your endurance so far, and also for reporting - helps me to solve things.

Jookie

[dhedberg]

This one (hopefully) last issue is related to handling SD card - got the same thing reported on ST through ACSI, so this is not Falcon / SCSI related - I need few more days for this one. If I remember correctly, it was fine before some time, but then there were some changes to the Hans firmware (which handles SD card) and now we got this... SO I would say - so far, so good, one more thing to go and we will be fine!

Yes, looks good. I'll reset the computer and re-run the tests a couple of times to make sure.

Thank you for your endurance so far, and also for reporting - helps me to solve things.

I'm in no hurry! I have a wife and two kids!
I'm happy to help and I'm thankful you're so committed to supporting the CosmosEx. It's one of the most awesome add-ons for the Atari ever made!

[alienkidmj12]

I just received my cosmos ex and I'm experiencing corruption also when copying from shared to local cf partition. I've replaced scsi cable and also tried powering via USB. Still no luck

[Jookie]

dhedberg, try to update now and see if it's any better.

You could also run the test: http://joo.kie.sk/?page_id=778

[dhedberg]

dhedberg, try to update now and see if it's any better.
You could also run the test: http://joo.kie.sk/?page_id=778

Things seems to have improved but there's still problems...

- I copied and decompressed a 800kB, 1.5MB, and 7MB ZIP file from the IDE to CosmosEx SD-card. It all seemed to work, no errors while copying or decompressing. However when running some of the decompressed executables I get errors. Especially Papyrus 5. I got a "Data on the disk in drive J: may be damaged" dialog and after that the whole partition was trashed. Tried this several times, same thing over and over again. I don't know what Papyrus 5 does exactly during start-up but it sure fails every time. Everything works fine from the IDE.

- Also, when trying to copy a larger file (12MB) from the IDE to the CosmosEx SD-card I get error dialogs "Data on the disk in drive J: may be damaged". When clicking retry button I get the dialog "An error occurred while copying a file". Tried 10 times, same thing every time.

I'm in the process of running the new SD-card tests now...

[Jookie]

I'm looking forward for your SD card test results, possibly testing more cards on your device would be nice.

That copying and data corruption aren't nice, I hoped for it to go away completely. What driver (and version) are you using for SD card and IDE drive? What Falcon you have - how much RAM, TOS version, accelerator?

[dhedberg]

Jookie, I did run the SD card tests 3+3 (W+V) times, with the same card, and all of the tests were successful. I'll run some more tests and perform some more testing, but it will probably have to wait until after X-mas.

I'm running HDDriver v9.1. Upgraded to v10.01, but have not had time to install it yet. TOS 4.04. 14MB RAM. One machine has the Nemesis accelerator installed (which includes a clock patch). The other machine is a clean C-Lab MK-II with no modifications.

[Malekko]

I can add myself to the list of cosmosex file corruption.
I'm loading floppy disks to a hard drive (Logic)
When I go to run it the L is missing (ogic) followed by gibberish.

[Malekko]

I'll also add the CE is new, updated OS and HDDRIVER 10.01

[dhedberg]

I can add myself to the list of cosmosex file corruption.
I'm loading floppy disks to a hard drive (Logic)
When I go to run it the L is missing (ogic) followed by gibberish.

Sounds a bit like a termination issue. Do you have other devices on the SCSI bus? Only the first device (which is the Falcon030 unless you have a C-Lab where you have to terminate the internal part of the SCSI bus yourself) and the last device on the bus should and must be terminated.

[Malekko]

Sounds a bit like a termination issue. Do you have other devices on the SCSI bus? Only the first device (which is the Falcon030 unless you have a C-Lab where you have to terminate the internal part of the SCSI bus yourself) and the last device on the bus should and must be terminated.

I am using a C-LAB and internal SCSI via that internal SCSI adapter and nothing else. The CE is terminated so shouldn't this be good?
Thanks

[dhedberg]

I would recommend you to put a terminator on the external SCSI connector. I had similar problems with my C-Lab MKII before I terminated the bus at both ends.

[Malekko]

I can say with 100% certainty that it isn't termination, unless the CE isn't properly terminated. I've tried on two falcons one using internal SCSI via an adapter and terminating the backside and also on another falcon using the CE externally.

[dhedberg]

I can say with 100% certainty that it isn't termination, unless the CE isn't properly terminated. I've tried on two falcons one using internal SCSI via an adapter and terminating the backside and also on another falcon using the CE externally.

OK, just wanted to make sure that termination was not the issue. Seems like there still are some unresolved issues with the CosmosEx on the Falcon. I have tried my unit on two different Falcons as well. Hopefully Jookie will eventually be able to find a solution. Things improved with the last firmware update, although corruption still occur, which unfortunately makes the SD-card and USB memories in RAW-mode pretty useless.

[Jookie]

dhedberg,
I've released an update - this one doesn't fix much about the SD card (anything else than extending the SD card read time out), but the TSTHD has now more things in it, so I would like to ask you to do the following:
- update to the latest version
- boot CE with SD card
- run configdrive\tests\ce_tsthd.tos
- see if 'SD error on WRITE and READ' are 0
- run the 'G' test, you can use option 'B' there which will do write and then read, so you don't have to remember the settings
- try larger file size, e.g. 7 MB
- see if the test (write and verify) went all OK
- then in the main menu of the test see if the 'SD error on WRITE and READ' are still 0
- I expect this all will be fine, as you mentioned last time that this passed
- then try the problematic things you mentioned - running that self extracting Papyrus, or the other things which caused issues
- if issues occure, run the ce_tsthd.tos again and see if the 'SD error on WRITE and READ' are still 0, or they are some higher than zero number

You mention there a couple of times your IDE drive, and I probably asked this, but are you running CosmosEx from external power supply, or from the Falcon power supply? If from internal PSU, it could be that the CE is draining too much power from your internal PSU and that could screw up data on one or the other (exxos mentioned CF cards being sensitive on voltage drop). It would be safer to try first the copying / extracting on / between the SD card and their partitions only, so we could rule out the influence of the other IDE drive, and then when this works fine, then I would try the copying / extracting between the IDE and SD - that would point to the those two not playing well together...

[Malekko]

You mention there a couple of times your IDE drive, and I probably asked this, but are you running CosmosEx from external power supply, or from the Falcon power supply? If from internal PSU, it could be that the CE is draining too much power from your internal PSU and that could screw up data on one or the other (exxos mentioned CF cards being sensitive on voltage drop). It would be safer to try first the copying / extracting on / between the SD card and their partitions only, so we could rule out the influence of the other IDE drive, and then when this works fine, then I would try the copying / extracting between the IDE and SD - that would point to the those two not playing well together...

I can eliminate the PSU theory, i have had the same results using the CE internally (Atari PSU) and externally (measuring 5.01V under load)

[Jookie]

In the mean time there has been an update which improves SD card access just by extending the read timeout, and also some other fixes, so some testing on your side and feedback would be appreciated, so I can try to fix it if the issue still appears.

[Malekko]

I've been running my SD card for 3 days without fail.
Today i get 3 new SanDisk 8gb 40mb/s i plan on trying to format, partition and launch programs with these. Assuming i can, i'll post detailed documentation on the process!

[dhedberg]

Jookie,

Sorry for this late reply...
My CosmosEx are powered by an external power source. I updated CosmosEx to the latest software versions and ran som tests:

Good news:
- Papyrus now decompresses and starts without errors and without trashing the partition. Did this 2 times without errors!
- The CE_TSTHD.TOS tests run without errors (file sizes tested: 1MB, 8MB and 9MB).
- Tried copying one of the 8MB test files generated by CE_TSTHD.TOS from CosmosEx SD to CosmosEx SD. Success! Did this 4 times without errors!

Bad news:
- Tried copying a 12MB file from IDE to SD. Got TOS error dialogs. CE_TSTHD.TOS reports SD error on WRITE: 0016
- Tried copying a 12MB file from IDE to SD. Got TOS error dialogs. CE_TSTHD.TOS reports SD error on WRITE: 0006
- Tried copying one of the 9MB test files generated by CE_TSTHD.TOS from SD to SD. Got TOS error dialogs. CE_TSTHD.TOS reports SD error on READ: 0006
- Tried copying one of the 9MB test files generated by CE_TSTHD.TOS from SD to SD. Got TOS error dialogs. CE_TSTHD.TOS reports SD error on READ: 0007
- Tried copying one of the 9MB test files generated by CE_TSTHD.TOS from SD to SD. Got TOS error dialogs. CE_TSTHD.TOS reports SD error on READ: 0015

Seems like increasing the timeout for the SD card in Hans helps, but the problems start when trying to read/write files larger than 9MB. If you take a look at the failing tests there are a few things we can learn from them.
1. When copying a file from SD to SD we get read errors (both in TOS dialogs and CE_TSTHD.TOS reported errors). SD read fails.
2. When copying a file from the IDE to the SD we get write errors (both in TOS dialogs and of course also by CE_TSTHD.TOS). IDE read is successful, SD write fails.
3. Both read and write from/to the SD card are problematic.
4. Errors only seem to happen when reading/writing large amount of data (over 8MB).

Let me know what you think.

Thanks,
Daniel

[Jookie]

dhedberg,

ok, so at least it's getting better now.

CE_TSTHD.TOS reports SD error on WRITE: 0016
CE_TSTHD.TOS reports SD error on READ: 0006

The (only) positive thing about these is, that Hans at least knows that it went wrong.

Seems like increasing the timeout for the SD card in Hans helps, but the problems start when trying to read/write files larger than 9MB.
...
4. Errors only seem to happen when reading/writing large amount of data (over 8MB).

Yes, so the more data is written, the bigger load is created on the SD card, and it might require longer time to finish writing the data (internaly) before it can continue with the processing of more data. Probably just a larger timeout (which is currently 3 seconds) would help, but then the question is - how long the Atari drivers (HDDRIVER, ICD Pro or whatever you are using) are willing to wait before terminating the current operation as unsuccessful.

The simplest thing you could try now is to take 2 (or more) other SD cards (other brands, other sizes) - if you have some spare, and retry the problematic cases with them - they might behave better (need less time to process small chunks of data), and also run the CE_TSTHD.TOS and data generating test on them - the difference for which you should be looking is that the better cards should show lower alphabet letters than the worse cards...

The workaround for this could also be to use translated drives for large files - the data generating test shows much less time needed when writing to e.g. shared drive (all 'B's).

So, could you try other SD cards and report?

Jookie