Issue with copying large directory trees from GEMDOS drive do ACSI partition

A forum about the Hatari ST/STE/Falcon emulator - the current version is v2.2.0

Moderators: simonsunnyboy, thothy, Moderator Team

Count
Atari nerd
Atari nerd
Posts: 48
Joined: Sat Sep 16, 2017 9:15 am
Location: Castrop-Rauxel, Germany

Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Count » Sat Feb 15, 2020 4:23 pm

I discovered an issue with GEMDOS drives (mapped host directories). Am using such a drive to exchange files via FTP between my PC and Hatari on my Raspberry Pi. The host directory is mapped to GEMDOS drive K. When I try to copy large directory trees from K to an ACSI partition, only a portion of the tree is copied.

The problem occurs only when reading from the GEMDOS drive. Copying from an ACSI partition to the GEMDOS drive works as expected, but copying the other way or from and to the GEMDOS drive leads to an incomplete copy. Even deleting large trees on the GEMDOS drive is incomplete.

You can find a test folder in the attached zip archive. The folder contains 313 subfolders with 795 files with 11345097 bytes. That is what I tried to copy yesterday. Don't wonder about the file and directory names and that the files only contain zeroes. It's only the same structure with the same file sizes.

This is what happens:

TEST folder contains: 313 folders, 795 files, 11345097 bytes

Copy GEMDOS -> ACSI: 65 folders, 327 files, 8917240 bytes
Copy GEMDOS -> GEMDOS: 65 folders, 327 files, 8917240 bytes
Copy ACSI -> GEMDOS: 313 folders, 795 files, 11345097 bytes (ok)

After trying to delete the folder on the GEMDOS drive, 2 folders, 5 files and 225547 bytes are left. GEM complains about not being able to delete "TEST".

I am running a self-compiled Hatari 2.2.1 on my Pi. But I can reproduce this behaviour on my Windows installation (Windows 10 64 bit, Hatari 2.2.1, downloaded from https://download.tuxfamily.org/hatari/2.2.1) as well as with an older Hatari version (2.0.0) on the Pi.
You do not have the required permissions to view the files attached to this post.

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 981
Joined: Sun Aug 03, 2014 5:54 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby ThorstenOtto » Sat Feb 15, 2020 4:48 pm

Unpacking the archive on linux, it says there are 800 files with 11570644 bytes. Could it be that there are some hidden directories? GEMDOS does not support the hidden flag on directories, only on files.

czietz
Hardware Guru
Hardware Guru
Posts: 1137
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby czietz » Sat Feb 15, 2020 5:10 pm

Based on my understanding of the Hatari GEMDOS emulation code:

On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103
If these have been exhaused, it reuses an old host-side DTA cache entry - unconditionally:
https://git.tuxfamily.org/hatari/hatari ... .2.1#n2908
If TOS later tries to use an old DTA (on Atari side), this DTA will point to a host-side cache entry that has been overwritten.

This will happen with very deep directory trees like the one in the ZIP file.

Count
Atari nerd
Atari nerd
Posts: 48
Joined: Sat Sep 16, 2017 9:15 am
Location: Castrop-Rauxel, Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Count » Sat Feb 15, 2020 5:59 pm

@ThorstenOtto: You are right with the number of files and the byte count. I checked the tree in my DOS box with "attrib /s /d", but there are no hidden or system files or directories. GEM's file info shows 313 folders / 795 files / 11345097 bytes on the GEMDOS drive, but 315 / 800 / 11570644 on the ACSI partition. That's weired.

@czietz: If I understand that correctly, this is truely an issue and should be fixed. In my opinion (apart from any TOS limits) there is no reason to use an array with a fixed limitation, especially such a low limitation, rather than a dynamic array with an initial size of 256 being remalloced when needed.

However, when I ran into this issue for the first time, I tried to use a ZIP archive instead. But it makes no difference between copying or unzipping a huge folder from a GEMDOS drive. So, my workaround right now is to copy ZIP files to the host directory, move them to an ACSI partition and unzip the content from there. But that is not very satisfying.

Count
Atari nerd
Atari nerd
Posts: 48
Joined: Sat Sep 16, 2017 9:15 am
Location: Castrop-Rauxel, Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Count » Sat Feb 15, 2020 6:34 pm

czietz wrote:On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103

How do you crazy guy come up with something like that off the cuff? I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough). Thank you very much (once again :wink:).

:cheers:

czietz
Hardware Guru
Hardware Guru
Posts: 1137
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby czietz » Sat Feb 15, 2020 7:23 pm

Count wrote:
czietz wrote:On host side, Hatari can only keep track of 256 DTAs:
https://git.tuxfamily.org/hatari/hatari ... 2.2.1#n103

How do you crazy guy come up with something like that off the cuff?


Years of experience reviewing someone else's code... :)

Count wrote: I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough).


For sure, you're aware that this only makes the issue much less likely. It can still occur with a sufficiently complex directory tree. However, I can't come up with a perfect solution, either. There's imho no way for Hatari to know whether it can safely discard a host-side DTA cache entry.

ThorstenOtto
Atari Super Hero
Atari Super Hero
Posts: 981
Joined: Sun Aug 03, 2014 5:54 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby ThorstenOtto » Sun Feb 16, 2020 9:32 am

That's still weird. Normally, the desktop will only use one DTA per directory level when copying files. But the archive surely does not contain such a deep structure (on SingleTOS, that won't be possible anyway, since it would overflow the limit for pathnames, even if the directories all have only 1 character in the name).

czietz
Hardware Guru
Hardware Guru
Posts: 1137
Joined: Tue May 24, 2016 6:47 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby czietz » Sun Feb 16, 2020 9:48 am

For every Fsfirst a new cache entry is used. It doesn't matter that the Desktop reuses DTAs. See my test program over at hatari-devel.

User avatar
1st1
Atari Super Hero
Atari Super Hero
Posts: 871
Joined: Mon May 07, 2012 11:48 am

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby 1st1 » Sun Feb 16, 2020 6:22 pm

I have seen that issue as well on my TT using MiNT and connected to my Windows server using NFS. It was easy to fix, use Kobold 3.5.
Power without the Price. It's not a bug. It's a feature. _/|\_ATARI

1040STFM in PC-Tower (PAK68/2, OvrScn, 4 MB, 1GB SCSI, CD-ROM...) * 2x Falcon 030 32GB/14MB+ScrnBlstrIII * 2x TT030 73GB/20MB+Nova * 520/1040STFM * 520/1040STE * 260/520ST/+ * some Mega ST * 2x Mega STE 500MB/4MB+M.CoCo * Stacy * STBook * SLM605 * SLM804 * SLM605 * SMM804 * SH 204/205 * Megafile 30/44/60 * SF314 * SF354 * 5x Pofo * PC3

Count
Atari nerd
Atari nerd
Posts: 48
Joined: Sat Sep 16, 2017 9:15 am
Location: Castrop-Rauxel, Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Count » Mon Feb 17, 2020 6:07 pm

Ah, Kobold is a nice hint. I am using a Mupfel script right now, but Kobold should do the same job very much faster (within seconds and not minutes).

User avatar
1st1
Atari Super Hero
Atari Super Hero
Posts: 871
Joined: Mon May 07, 2012 11:48 am

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby 1st1 » Mon Feb 17, 2020 7:07 pm

By the way, also WIndows 10 1909 file explorer gets in trouble when selecting hundreds of Gigabytes with hundred thousands of files in the folder, when moving that from one drive to another. "Robocopy" is my friend there... Every OS has it's limitations... Impossible? I had to move my ATARI, Amiga, DOS and C64 files to another new harddisk these days as the old one has quite bad SMART status since a few days...
Power without the Price. It's not a bug. It's a feature. _/|\_ATARI

1040STFM in PC-Tower (PAK68/2, OvrScn, 4 MB, 1GB SCSI, CD-ROM...) * 2x Falcon 030 32GB/14MB+ScrnBlstrIII * 2x TT030 73GB/20MB+Nova * 520/1040STFM * 520/1040STE * 260/520ST/+ * some Mega ST * 2x Mega STE 500MB/4MB+M.CoCo * Stacy * STBook * SLM605 * SLM804 * SLM605 * SMM804 * SH 204/205 * Megafile 30/44/60 * SF314 * SF354 * 5x Pofo * PC3

User avatar
Eero Tamminen
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2050
Joined: Sun Jul 31, 2011 1:11 pm

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Eero Tamminen » Thu Apr 09, 2020 4:07 pm

Count wrote:I just recompiled Hatari after increasing MAX_DTAS_FILES from 256 to 2048 and everything is fine (even 1024 wasn't enough)


Could you try whether the attached patch also fixes your issue?

It's proper fix for the issue; doesn't allocate new internal DTA entry if emulated code re-uses earlier DTA, and grows the DTA cache on demand.

(You can apply it to Hatari sources by doing following at top of the Hatari source directory: "patch -p1 < patchfile".)
You do not have the required permissions to view the files attached to this post.

Count
Atari nerd
Atari nerd
Posts: 48
Joined: Sat Sep 16, 2017 9:15 am
Location: Castrop-Rauxel, Germany

Re: Issue with copying large directory trees from GEMDOS drive do ACSI partition

Postby Count » Thu Apr 09, 2020 7:11 pm

Eero Tamminen wrote:Could you try whether the attached patch also fixes your issue?

Thank you for your patch. I gave it a try and the result is really interesting.

Preliminaries:
- 68030 TT at 16 MHz
- 4 MB ST RAM
- 8 MB TT RAM
- 68881 FPU
- German TOS 3.06
- F: partition on a hard disk image
- K: host directory

I created a huge folder on my host directory by copying one folder (with 316 subfolders and 802 files / 11,606,512 bytes) six times into it. The result was a folder containing 1,902 subfolders and 4,812 files / 69,639,072 bytes in total.

Copying that folder within K using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.
Copying that folder from K to F using desktop drag&drop aborted after 634 folders and 1,604 files (23,213,024 bytes) on both Hatari versions.

Copying that folder using Kobold from K to F succeeded on both Hatari versions.
Copying the folder copied by Kobold from F back to K using desktop drag&drop succeeded on both Hatari versions.

So, the result of your patch seems to be the same as increasing MAX_DTAS_FILES to 2048.

And to make it somewhat scary, here's one for those wearing a tin foil hat:
- 634 subfolders copied is exactly one third of 1,902.
- 1,604 files copied is exactly one third of 4,812.
- 23,213,024 bytes copied is exactly one third of 69,639,072


Social Media

     

Return to “Hatari”

Who is online

Users browsing this forum: No registered users and 4 guests