Issue with copying large directory trees from GEMDOS drive do ACSI partition

A forum about the Hatari ST/STE/Falcon emulator - the current version is v2.2.0

Moderators: simonsunnyboy, thothy, Moderator Team

User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

I discovered an issue with GEMDOS drives (mapped host directories). Am using such a drive to exchange files via FTP between my PC and Hatari on my Raspberry Pi. The host directory is mapped to GEMDOS drive K. When I try to copy large directory trees from K to an ACSI partition, only a portion of the tree is copied.

The problem occurs only when reading from the GEMDOS drive. Copying from an ACSI partition to the GEMDOS drive works as expected, but copying the other way or from and to the GEMDOS drive leads to an incomplete copy. Even deleting large trees on the GEMDOS drive is incomplete.

You can find a test folder in the attached zip archive. The folder contains 313 subfolders with 795 files with 11345097 bytes. That is what I tried to copy yesterday. Don't wonder about the file and directory names and that the files only contain zeroes. It's only the same structure with the same file sizes.

This is what happens:

TEST folder contains: 313 folders, 795 files, 11345097 bytes

Copy GEMDOS -> ACSI: 65 folders, 327 files, 8917240 bytes
Copy GEMDOS -> GEMDOS: 65 folders, 327 files, 8917240 bytes
Copy ACSI -> GEMDOS: 313 folders, 795 files, 11345097 bytes (ok)

After trying to delete the folder on the GEMDOS drive, 2 folders, 5 files and 225547 bytes are left. GEM complains about not being able to delete "TEST".

I am running a self-compiled Hatari 2.2.1 on my Pi. But I can reproduce this behaviour on my Windows installation (Windows 10 64 bit, Hatari 2.2.1, downloaded from https://download.tuxfamily.org/hatari/2.2.1) as well as with an older Hatari version (2.0.0) on the Pi.
You do not have the required permissions to view the files attached to this post.
ThorstenOtto
Atari God
Atari God
Posts: 1166
Joined: Sun Aug 03, 2014 5:54 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by ThorstenOtto »

Unpacking the archive on linux, it says there are 800 files with 11570644 bytes. Could it be that there are some hidden directories? GEMDOS does not support the hidden flag on directories, only on files.
czietz
Hardware Guru
Hardware Guru
Posts: 1228
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by czietz »

Based on my understanding of the Hatari GEMDOS emulation code:

On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103
If these have been exhaused, it reuses an old host-side DTA cache entry - unconditionally:
https://git.tuxfamily.org/hatari/hatari ... .2.1#n2908
If TOS later tries to use an old DTA (on Atari side), this DTA will point to a host-side cache entry that has been overwritten.

This will happen with very deep directory trees like the one in the ZIP file.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

@ThorstenOtto: You are right with the number of files and the byte count. I checked the tree in my DOS box with "attrib /s /d", but there are no hidden or system files or directories. GEM's file info shows 313 folders / 795 files / 11345097 bytes on the GEMDOS drive, but 315 / 800 / 11570644 on the ACSI partition. That's weired.

@czietz: If I understand that correctly, this is truely an issue and should be fixed. In my opinion (apart from any TOS limits) there is no reason to use an array with a fixed limitation, especially such a low limitation, rather than a dynamic array with an initial size of 256 being remalloced when needed.

However, when I ran into this issue for the first time, I tried to use a ZIP archive instead. But it makes no difference between copying or unzipping a huge folder from a GEMDOS drive. So, my workaround right now is to copy ZIP files to the host directory, move them to an ACSI partition and unzip the content from there. But that is not very satisfying.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

czietz wrote:On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103
How do you crazy guy come up with something like that off the cuff? I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough). Thank you very much (once again :wink:).

:cheers:
czietz
Hardware Guru
Hardware Guru
Posts: 1228
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by czietz »

Count wrote:
czietz wrote:On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103
How do you crazy guy come up with something like that off the cuff?
Years of experience reviewing someone else's code... :)
Count wrote: I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough).
For sure, you're aware that this only makes the issue much less likely. It can still occur with a sufficiently complex directory tree. However, I can't come up with a perfect solution, either. There's imho no way for Hatari to know whether it can safely discard a host-side DTA cache entry.
ThorstenOtto
Atari God
Atari God
Posts: 1166
Joined: Sun Aug 03, 2014 5:54 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by ThorstenOtto »

That's still weird. Normally, the desktop will only use one DTA per directory level when copying files. But the archive surely does not contain such a deep structure (on SingleTOS, that won't be possible anyway, since it would overflow the limit for pathnames, even if the directories all have only 1 character in the name).
czietz
Hardware Guru
Hardware Guru
Posts: 1228
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by czietz »

For every Fsfirst a new cache entry is used. It doesn't matter that the Desktop reuses DTAs. See my test program over at hatari-devel.
User avatar
1st1
Atari Super Hero
Atari Super Hero
Posts: 881
Joined: Mon May 07, 2012 11:48 am

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by 1st1 »

I have seen that issue as well on my TT using MiNT and connected to my Windows server using NFS. It was easy to fix, use Kobold 3.5.
Power without the Price. It's not a bug. It's a feature. _/|\_ATARI

1040STFM in PC-Tower (PAK68/2, OvrScn, 4 MB, 1GB SCSI, CD-ROM...) * 3x Falcon 030 * 3x TT030 * many 260 /520/1040ST(F)(M)(+) * 520/1040STE * many Mega ST * 2x Mega STE * Stacy * STBook * 2x SLM605 * 3x SLM804 * SMM804 * SH 204/205 * Megafile 30/44/60 * SF314 * SF354 * 5x Pofo * PC3 * ...
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Ah, Kobold is a nice hint. I am using a Mupfel script right now, but Kobold should do the same job very much faster (within seconds and not minutes).
User avatar
1st1
Atari Super Hero
Atari Super Hero
Posts: 881
Joined: Mon May 07, 2012 11:48 am

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by 1st1 »

By the way, also WIndows 10 1909 file explorer gets in trouble when selecting hundreds of Gigabytes with hundred thousands of files in the folder, when moving that from one drive to another. "Robocopy" is my friend there... Every OS has it's limitations... Impossible? I had to move my ATARI, Amiga, DOS and C64 files to another new harddisk these days as the old one has quite bad SMART status since a few days...
Power without the Price. It's not a bug. It's a feature. _/|\_ATARI

1040STFM in PC-Tower (PAK68/2, OvrScn, 4 MB, 1GB SCSI, CD-ROM...) * 3x Falcon 030 * 3x TT030 * many 260 /520/1040ST(F)(M)(+) * 520/1040STE * many Mega ST * 2x Mega STE * Stacy * STBook * 2x SLM605 * 3x SLM804 * SMM804 * SH 204/205 * Megafile 30/44/60 * SF314 * SF354 * 5x Pofo * PC3 * ...
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

Count wrote:I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough)
Could you try whether the attached patch also fixes your issue?

It's proper fix for the issue; doesn't allocate new internal DTA entry if emulated code re-uses earlier DTA, and grows the DTA cache on demand.

(You can apply it to Hatari sources by doing following at top of the Hatari source directory: "patch -p1 < patchfile".)
You do not have the required permissions to view the files attached to this post.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Eero Tamminen wrote:Could you try whether the attached patch also fixes your issue?
Thank you for your patch. I gave it a try and the result is really interesting.

Preliminaries:
- 68030 TT at 16 MHz
- 4 MB ST RAM
- 8 MB TT RAM
- 68881 FPU
- German TOS 3.06
- F: partition on a hard disk image
- K: host directory

I created a huge folder on my host directory by copying one folder (with 316 subfolders and 802 files / 11,606,512 bytes) six times into it. The result was a folder containing 1,902 subfolders and 4,812 files / 69,639,072 bytes in total.

Copying that folder within K using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.
Copying that folder from K to F using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.

Copying that folder using Kobold from K to F succeeded on both Hatari versions.
Copying the folder copied by Kobold from F back to K using desktop drag&drop succeeded on both Hatari versions.

So, the result of your patch seems to be the same as increasing MAX_DTAS_FILES to 2048.

And to make it somewhat scary, here's one for those wearing a tin foil hat:
- 634 subfolders copied is exactly one third of 1,902.
- 1,604 files copied is exactly one third of 4,812.
- 23,213,024 bytes copied is exactly one third of 69,639,072
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

Count wrote: Copying that folder within K using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.
Copying that folder from K to F using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.
...
So, the result of your patch seems to be the same as increasing MAX_DTAS_FILES to 2048.
Thanks for testing and the detailed information about it!

As results are same with the host directory as with the disk image, it's not anymore a Hatari issue. Abort is due to some TOS v3 issue / limit.

You might try foldernnn.prg in AUTO folder to increase these limits, or using latest EmuTOS version.

Did TOS give any indication why the operation aborted; was one of the files too large, directory depth too large, or were there too many files or folders in total for it?

After TOS aborts, you might also try AltGr+Pause to invoke Hatari console debugger, and give "info gemdos" command to see what DTAs are in use by GEMDOS HD emulation. You could attach it here too (zipped if it's very large).
ThorstenOtto
Atari God
Atari God
Posts: 1166
Joined: Sun Aug 03, 2014 5:54 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by ThorstenOtto »

Eero Tamminen wrote:[ Abort is due to some TOS v3 issue / limit.
I think it is more related to how the desktop descends into subdirectories. Kobold, when copying to/from a drive that is not FAT (like the host directory in this case) also just uses GEMDOS functions. But it might call them in a different order than the desktop does. Maybe tracing the GEMDOS calls will help (although that migth be tedious with so many files)
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

"info gemdos" should give some idea about that.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

No, TOS didn't give any indications. It just stopped the operation and the dialog box was closed. But that amount of folders, files and bytes is far beyond what I really need. So, your patch is definitely a step forward.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Unfortunately, the debugger doesn't show up. I get a message about it in the status line, but that's all.
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

Count wrote:Unfortunately, the debugger doesn't show up. I get a message about it in the status line, but that's all.
On Windows, you need to start Hatari with the "-W" (or "--wincon") option to get console window for Hatari debugger. On Mac/Linux Hatari, debugger is on the console from which you started Hatari.
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Ok, I started Hatari from PuTTY and made it to the debug console. How can I redirect the information to a file?

These are the last lines:

Code: Select all

+ 2032: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001098
  - 0: 00000778
  Fsnext entry = 1.
+ 2033: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001098
  - 0: 00000778
  Fsnext entry = 1.
+ 2034: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001099
  - 0: 00000779
  Fsnext entry = 1.
+ 2035: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001099
  - 0: 00000779
  Fsnext entry = 1.
+ 2036: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001100
  - 0: 00000780
  Fsnext entry = 1.
+ 2037: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001100
  - 0: 00000780
  Fsnext entry = 1.
+ 2038: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001101
  - 0: 00000781
  Fsnext entry = 1.
+ 2039: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001101
  - 0: 00000781
  Fsnext entry = 1.
+ 2040: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001102
  - 0: 00000782
  Fsnext entry = 1.
+ 2041: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001102
  - 0: 00000782
  Fsnext entry = 1.
+ 2042: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001103
  - 0: 00000783
  Fsnext entry = 1.
+ 2043: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001103
  - 0: 00000783
  Fsnext entry = 1.
+ 2044: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001104
  - 0: 00000784
  Fsnext entry = 1.
+ 2045: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001104
  - 0: 00000784
  Fsnext entry = 1.
+ 2046: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001105
  - 0: 00000785
  Fsnext entry = 1.
+ 2047: /home/pi/host_fs/.system/Partitions/hatari_hostfs/k/TEST/TEST2/00000976/00001105
  - 0: 00000785
  Fsnext entry = 1.

Open GEMDOS HDD file handles:
- 64 (0x101e2ba): /home/pi/host_fs/.system/Partitions/hatari_hostfs/o/emulator.inf

Forced GEMDOS HDD file handles:
- None.
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

Was this with the 2048 DTA limit, or my own patch? According to your info command output excerpt, there are two DTAs are used for each file, and if this was when things stopped working, 2048 entries were used at that point.

Looking at Hatari sources, "info" command output is hard-coded to go to stderr, so to get that to a file, you need to redirect Hatari's stderr. On Linux you would do it like this:

Code: Select all

hatari [options]  2> stderr.txt
(After this you still see debugger prompt, but not what is the output from debugger commands, as that's redirected to the stderr.txt file.)
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Shame on me. I failed building the patched version of Hatari correctly. I didn't have in mind that a simple "make all" doesn't work. Due to this, I was still running my dirty hack rather than your patch. Unfortunately, I didn't check the file date. :?

Today I cleared the source tree and the cmake cache file, compiled with cmake and ran the correct version with your patch.

The patch works and my test folder gets copied completely. I diffed the two directories recursively using "diff -r TEST TEST2" on the host and got no differences reported.

Thank you for your great work, Eero! :cheers:
And please excuse my stupid mistake. :roll:
User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2161
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Eero Tamminen »

Count wrote: Today I cleared the source tree and the cmake cache file, compiled with cmake and ran the correct version with your patch.

The patch works and my test folder gets copied completely. I diffed the two directories recursively using "diff -r TEST TEST2" on the host and got no differences reported.
Folder gets copied completely? Do you mean that it works better than increasing MAX_DTAS_FILES to 2048?

Could you check from debugger "info gemdos" what's now the number of DTAs at the end of your test?

Thanks for testing!
User avatar
Count
Atari freak
Atari freak
Posts: 66
Joined: Sat Sep 16, 2017 9:15 am
Location: Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Count »

Yes, it works better. Increasing MAX_DTAS_FILES to 2048 lead to incomplete copies.
I attached a zip file with the "info gemdos" output.
You do not have the required permissions to view the files attached to this post.
Rustynutt
Atari Super Hero
Atari Super Hero
Posts: 794
Joined: Wed Mar 21, 2012 7:38 am
Location: Oregon

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Post by Rustynutt »

1st1 wrote:By the way, also WIndows 10 1909 file explorer gets in trouble when selecting hundreds of Gigabytes with hundred thousands of files in the folder, when moving that from one drive to another. "Robocopy" is my friend there... Every OS has it's limitations... Impossible? I had to move my ATARI, Amiga, DOS and C64 files to another new harddisk these days as the old one has quite bad SMART status since a few days...
Thanks for that (Robocopy).
Had a WIN7 500gb drive "accidentally" quick formatted with GB's of TOS archives and 1000's of folders. The directory got hosed, yet nearly all files were recoverable manually.
On top of that being a huge pain in the ass, didn't catch the WIN7 limitations until reading your post. Copying over the 4 hour recovery time of files, found destination folders corrupt.
Can imagine chasing something like this down buried beneath an emuator a real PIA.
:cheers:
Post Reply

Return to “Hatari”