exfat performance

https://github.com/MiSTer-devel/Main_MiSTer/wiki

Moderators: Mug UK, Zorro 2, Greenious, spiny, Sorgelig, Moderator Team

R4MS
Atari maniac
Atari maniac
Posts: 78
Joined: Sun Jan 13, 2019 6:54 pm

exfat performance

Postby R4MS » Sun Sep 22, 2019 2:15 pm

I was playing around while investigating the IDE problems in minimig and did some performance tests.

dd if=/dev/zero of=/media/fat/testfile.tmp bs=1k count=$((40*1024))

writes an empty 40MB file... on the exfat partition the speed is less than 400K per second. On an ext4 partition on the same SD card, I get over 10 MB/s even on long (800M) files with random content (/dev/urandom).

On the minimig side I do not see any speed difference for hardfiles (hdf) on both partition types and also the reproducible problems stay the same. So right now it is not a real problem, but with increasing performance of the cores it might become one.

To get a better performance when manipulating files on the Linux side, I did split the exfat partition and make an ext4 partition for all disk images. Those can easily be linked to /media/fat/, where MiSTer will find them.

Since removing the SD card from the de10-nano is quite a fiddly task, I would guess, that not a lot of people will plug the SD card into a Windows PC for file transfer.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5274
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: exfat performance

Postby Sorgelig » Sun Sep 22, 2019 2:47 pm

ext4 is native Linux FS, so it's expected to be more optimized with proper caches on different levels. Besides that exFAT is non-native, it's even unofficial. I've heard MS gave green light to officially integrate exFAT into the Linux. So may be soon we will see official and probably more efficient implementation of exFAT.
By the way, did you try to write to pre-allocated/existing file? Writing a new file has an overhead for FAT allocation and size update in catalog. With existing files content overwrite should be faster.
Also, performance depends on cluster size very much. It's strongly advised to use cluster size for exFAT not less than 16KB (32KB is better).

R4MS
Atari maniac
Atari maniac
Posts: 78
Joined: Sun Jan 13, 2019 6:54 pm

Re: exfat performance

Postby R4MS » Sun Sep 22, 2019 3:15 pm

Sorgelig wrote:By the way, did you try to write to pre-allocated/existing file? .

Yes, no difference with dd. But when copying a file, the speed can top 10MB/s.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5274
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: exfat performance

Postby Sorgelig » Sun Sep 22, 2019 4:11 pm

try bs=64k

R4MS
Atari maniac
Atari maniac
Posts: 78
Joined: Sun Jan 13, 2019 6:54 pm

Re: exfat performance

Postby R4MS » Sun Sep 22, 2019 4:29 pm

OK, you were right. Got a good 10 MB/s when using bs=64k. Still, I'll be using ext4 for data to exclude strange behavior of the file system.

Sorgelig
Ultimate Atarian
Ultimate Atarian
Posts: 5274
Joined: Mon Dec 14, 2015 10:51 am
Location: Russia/Taiwan

Re: exfat performance

Postby Sorgelig » Sun Sep 22, 2019 8:35 pm

I don't expect strange behaviour using vhd images as everything is done within a single file of fixed size.


Return to “MiSTer”

Who is online

Users browsing this forum: No registered users and 12 guests