Moderators: Jookie, Moderator Team
Malekko wrote:I can report another working SD HD using SanDisk 8gb 40mb/s
Jookie wrote:Yes, so the more data is written, the bigger load is created on the SD card, and it might require longer time to finish writing the data (internaly) before it can continue with the processing of more data. Probably just a larger timeout (which is currently 3 seconds) would help, but then the question is - how long the Atari drivers (HDDRIVER, ICD Pro or whatever you are using) are willing to wait before terminating the current operation as unsuccessful.
The simplest thing you could try now is to take 2 (or more) other SD cards (other brands, other sizes) - if you have some spare, and retry the problematic cases with them - they might behave better (need less time to process small chunks of data), and also run the CE_TSTHD.TOS and data generating test on them - the difference for which you should be looking is that the better cards should show lower alphabet letters than the worse cards...
The workaround for this could also be to use translated drives for large files - the data generating test shows much less time needed when writing to e.g. shared drive (all 'B's).
So, could you try other SD cards and report?
dhedberg wrote:Have now tried 3 different cards. The cards are of different speeds and brands. The fastest card only give me B (80%) and C (20%) write time counts while the slowest give me B, E, H and I write time counts.
dhedberg wrote:The results for all three cards are exactly the same: Copy files 8MB or less works, but 9MB or more fails.
The CE_TESTHD.TOS tests succeed.
dhedberg wrote:To me it seems like the speed of the card is not crucial. In that case I would think that the fastest card would be able to copy larger files than the slower cards, but they all succeed for 8MB files but fail on 9MB files. So no obvious correlation there.
dhedberg wrote:Please increase the timeout in Hans to 5 seconds or so. It would be interesting to see what effect it would have.
dhedberg wrote:Just tried using KOBOLD to copy the larger files and it seems to work. Kobold is using quite a bit of memory for its read buffers while I'm pretty sure that TOS has very small buffers. Could it be that when TOS is copying a file it is causing a long sequence of very small writes to the SD-card?
dhedberg wrote:Would it be possible for Hans to collect and combine small writes into larger ones?
Jookie wrote:dhedberg wrote:Please increase the timeout in Hans to 5 seconds or so. It would be interesting to see what effect it would have.
I could do that, and I'm also thinking about another improvement related to this - wait undefinitelly, but keep checking for new requests from ST (Falcon) and cancel the current operation (which might take long) if some new request from ST (Falcon) comes - that could make the waiting even longer without getting stuck in a single operation once the Atari driver decides not to wait anymore.
Jookie wrote:dhedberg wrote:Would it be possible for Hans to collect and combine small writes into larger ones?
There's not enough RAM in Hans to do that - there's 20 kB of RAM, 6 kB is already used, so there's 14 kB free, and that wouldn't help much.
The more complicated way of doing that would be:
- let the SD card writes go to RPi instead of doing just immediate write (it has lots of power and free RAM)
- that one would do the buffering, merging of write operations
- Hans would keep just a list of dirty sectors, which are currently buffered in RPi, so in case a read operation for one of those sectors would come before write to SD, then the read would go from Hans instead
- then after enough of buffered data in RPi, there could be the combined larger write from RPi to Hans, which would pass it to SD card (RPi can't access the SD card directly)
Technically this is possible, but I'm not sure if it's worth the hassle if all you need to do is use Koboldand this could all end up with data being badly written, but it's still a possibility. This would not work in solo mode, when no Raspberry Pi is installed.
Jookie wrote:So what now? This could take weeks of development, while most of the users will probably never copy 10 MB file (or larger) on ST, altough I can imagine that this might happen with just lots of data copying, it doesn't have to be single large file to hit this issue. Using Kobold is an option, but it would be better to have it properly working.
dhedberg wrote:Please let me know if you make any changes that may improve things (longer timeout, smarter timeout, etc.).
dhedberg wrote:It would be good to confirm that TOS is actually writing lots of small chunks, and find out the size of these.
dhedberg wrote:I like the idea of managing a write buffer in the RPi to optimize the SD write performance/reliability. The resources are there (unless a lot of people are actually running the CosmosEx without the RPi)... lets use them!
dhedberg wrote:But please remember that when copying a file from SD to SD, I also got READ errors. I am not sure how these could possibly be explained?
dhedberg wrote:What's the speed of the TT SCSI interface compared to the F030? If data is transferred a lot faster on the Falcon that could explain the results.
dhedberg wrote:The TOS versions are different too. It would be of interest to see if the reading and writing during file copying use the same data block sizes.
Jookie wrote:According to this:
http://www.atari-wiki.com/?title=ACSI,_SCSI_and_IDE
It's 4MB/s on TT, and 2 MB/s on Falcon. But that's the maximum the computer can go, they both are limited by what CosmosEx can go, and that's around 1.1 MB/s on R/W when transfering to RPi (translated drives) and even slower - 0.6MB/s or so - on writing to SD card (being slowed down by SD card).
The other thing is - if the data validity checks tests for CE run fine, then I wouldn't blame the SCSI interface, as that would be ruled out by those test (although still could be, I still might be wrong on anything).
Jookie wrote:dhedberg wrote:The TOS versions are different too. It would be of interest to see if the reading and writing during file copying use the same data block sizes.
If the current debugging and logging methods won't help in solving the issue (in Hans), I will probably have to add some more logging in other places to see the block writes/reads in TOS (probably in TOS Rwabs() handler).
Jookie wrote:OK, got something...
On my Falcon, with 14 MB of ST-RAM, when the sectors of the file being read are nicelly one after another, the HDDRIVER issues a large read - 28144 sectors - 13.7 MB of data. That can no way fit in the 3 second timeout. Now the question is:
- does HDDRIVER make a large, but interrupted SCSI-10 read, and I'm missing implementation for those interrupts in transfer, or...
- does HDDRIVER or TOS just allocate all the free RAM, and then does huge un-interrupted read into that large buffer, and then large write, in which case I just need to base the timeout value on transfer size (e.g. 3 seconds for each MB transfered, or so). Jookie
dhedberg wrote:How much ST-RAM does your TT have?
Jookie wrote:The TT has 4 MB ST RAM, and 16 MB TT RAM. The TT RAM shouldn't influence this (even in Falcon), because if I remember correctly, DMA transfers can't be done to TT RAM, so only ST RAM is used for DMA transfers and thus hard disk transfers.
I've extended the timeout in Hans to be calculated from how many MB are transfered, but in debug mode of Hans the throughput is only 0.6 MB/s, and it makes the read time out on 11 seconds, as it's not able to push through 14 MB during those 11 seconds. So my guess is that HDDRIVER has 10 seconds timeout on transfers (which is probably not configurable at this moment in HDDRUTIL). With CE having throughput of 1.1 MB/s the 13.77 MB would take 12,5 seconds to transfer (in best case, in worse case when the SD card is being slow it could be double of that), so currently there's not much I can do in firmware about this.
I wrote an e-mail to Uwe S. (author of HDDRIVER), I'm waiting for his reply. There are at least these options:
1) let TOS think that it has only 4 MB of RAM - as you suggested, then it would do smaller transfers
2) remove 14 MB ST RAM from Falcon - one of your suggestions, but it's a bad solution to not use your ST RAM when you already have it
3) set TOS to limit the size of transfers to something less than 13.7 MB (maybe 4 MB max transfer size would be better) - if it's possible somehow
4) configure HDDRIVER (through HDDRUTIL) to split such large transfers into a smaller ones (e.g. 4 MB chunks) - Uwe would have to add this one
5) configure HDDRIVER (through HDDRUTIL) to have larger timeout for large transfers - either based on transfer size (like I did in Hans), or single fixed value - Uwe would have to add this one
6) do nothing
I don't like the options 2) and 6).
Options 1) and 3) are TOS configuration thing, although I'm not sure if 3) is done in some utility or possible at all.
Options 4) and 5) require that either Uwe tells us that there's already a setting for that in HDDRUTIL / HDDRIVER, or that he would add this feature, which might result in everybody affected buying newer version of HDDRIVER. Or he might tell us that he won't add this, and then there are only previous options available as resolution to this.
Jookie wrote:Just to keep you updated - I didn't manage to do much during the last week - got both kids sick, now they are getting better.
Jookie wrote:Uwe responded quickly (you know, no kids), it seems that everything is fine from his side. I don't see the obvious issue, I have to think this through, so in the mean time I've tested this large file copy on RAW USB flash drive (with HDDRIVER) and it suffers in similar manner - I didn't expect such large transfers on ST, so now I have to fix it. I hope that while it might be easier to fix it on RAW USB flash drive, I might find the issue of SD card, too...
Jookie wrote:dhedberg wrote:How much ST-RAM does your TT have?
The TT has 4 MB ST RAM, and 16 MB TT RAM. The TT RAM shouldn't influence this (even in Falcon), because if I remember correctly, DMA transfers can't be done to TT RAM, so only ST RAM is used for DMA transfers and thus hard disk transfers.
Jookie
anodyne wrote:The TT can do DMA into TT RAM; the Falcon can't. The DMA chip in the Falcon is essentially the ST one, while the DMA chip in the TT allows 32-bit addressing.
Jookie wrote:I'm looking at the same issue with RAW (USB) drive to make sure it's not SD card issue, and it happens there, too. The current thing that I'm stuck on (and still examining) is that after 2 or 3 MB of transfered data the ACK line (which is controlled by Falcon) stays low and stays there - the Falcon stops receiving data for some reason. I thought it was due to the REQ line being too fast (it was around 65 ns, while SCSI-II spec said it should be at least 100 ns or more), so I've extended it to 240 ns, but the issue still happens (and the ACK line from Falcon gets stuck). There are still things I want to try, I didn't give up on this one (although it lasts more than what I expected and wished for).
Jookie wrote:Hello Daniel,
sorry for late reply and inactivity on this, but I got somehow stuck... I wanted to fix a related issue (the RAW drives and large transfer) and got stuck there too... So I wanted to add a simple feature to make myself happy with a little success, and I failed there too. I keep failing on everything I touch in the last weeks and months, and I'm starting to feel bit like an unlucky person.
I still want to fix the issue, but I have to kick my self into it - with so many things I failed in the last time, my motivation and mood are as low as they can be.
That ZIP file thing - does this happen even if you turn off the 'ZIP files as directories' feature through config? This feature might be clashing together with trying to create plain zip files... I didn't try that, still have to take a look at it.
Jookie
Users browsing this forum: No registered users and 0 guests