Image and video viewers

[medmed]

I've found a memory leak - I'll fix it asap.

[medmed]

The library uses memalign to allocate memory, then it uses free.
But at first glance, the mem doesn't appear to be released.

May be someone know a related bug with it?

I'll try to do some tests later to see if the mem is released correctly.

[medmed]

Yes there's a problem freeing memory allocated via memalign() - The problem comes from free().
I've replaced memalign by Mxalloc and the memory is released as normal.

I tried to implement a custom memalign function but then Mfree(((void**)ptr)[-1]); produce a bus error.

Whatever It seems to work with a classic Mxalloc/Mfree (PSD read/write), don't know why they use memalign in their library.

I'll patch it again tomorrow.

Code: Select all

void* MallocAllocator::DoAllocate(size_t size, size_t alignment)
{
#if defined(__APPLE__)
    void *m = 0;
    size_t minAlignment = sizeof(void *);
    while (alignment > minAlignment) {
        minAlignment *= 2;
    }
    errno = posix_memalign(&m, minAlignment, size);
    return errno ? NULL : m;
#elif defined(__MINT__)
    // void* p1; // original block
    // void** p2; // aligned block
    // int offset = alignment - 1 + sizeof(void*);
    // if ((p1 = (void*)Mxalloc((size + offset),3)) == NULL)
    // {
    //    return NULL;
    // }
    // p2 = (void**)(((size_t)(p1) + offset) & ~(alignment - 1));
    // p2[-1] = p1;
    // return p2;
    return (void*)Mxalloc((size),3);
#elif defined(__GNUG__)
	return memalign(alignment, size);
#else
	return _aligned_malloc(size, alignment);
#endif
}

void MallocAllocator::DoFree(void* ptr)
{
#if defined(__MINT__)
    // Mfree(((void**)ptr)[-1]);
    Mfree(ptr);
#elif defined(__APPLE__) || defined(__GNUG__)
	free(ptr);
#else
	_aligned_free(ptr);
#endif
}

[mikro]

memalign() is not properly supported in mintlib. It is properly supported in my fork which uses dlmalloc allocator, feel free to try it out: https://github.com/mikrosk/mintlib/tree/dlmalloc.

I'd stay away from calling Mxalloc as that is a direct syscall which can then put a lot of strain to kernel's memory manager (we don't have virtual memory, mind you). The easiest solution for your problem is to keep two pointers, one aligned and one unaligned.

A bit more complex solution is to store the unaligned pointer as the first 4 bytes in the allocated buffer. I use this trick in ScummVM.

[ThorstenOtto]

The problem are not the two pointers (that is what posix_memalign in mintlib already does). The problem is that you cannot call free() on the returned pointer, and a lot of code assume that. See above example, where sometimes also a special function has to be called to free the pointer.

But i agree on the use of Mxalloc/Malloc. That causes lots of overhead in the kernel, and also some overhead due to the exception being raised every time.

Apart from that, there is rarely a reason to use posix_memalign at all (most of the time it is used to align buffers that are accessed with SSE instructions on other architectures), and you can just use malloc() instead.

[medmed]

memalign() is not properly supported in mintlib. It is properly supported in my fork which uses dlmalloc allocator, feel free to try it out: https://github.com/mikrosk/mintlib/tree/dlmalloc.

Thanks for the link mikro,

I'll try it to do some benchmark with video/audio library to see if I get better results.

Apart from that, there is rarely a reason to use posix_memalign at all (most of the time it is used to align buffers that are accessed with SSE instructions on other architectures), and you can just use malloc() instead.

Thanks Thorsten,

Tested with malloc and it seems ok.

EDIT:
malloc failed in some cases while Mxalloc succed.

[medmed]

memalign() is not properly supported in mintlib. It is properly supported in my fork which uses dlmalloc allocator, feel free to try it out: https://github.com/mikrosk/mintlib/tree/dlmalloc.

I'd stay away from calling Mxalloc as that is a direct syscall which can then put a lot of strain to kernel's memory manager (we don't have virtual memory, mind you). The easiest solution for your problem is to keep two pointers, one aligned and one unaligned.

A bit more complex solution is to store the unaligned pointer as the first 4 bytes in the allocated buffer. I use this trick in ScummVM.

I've just tried with mintlib_dlmalloc version using memalign(alignment, size) & free(ptr) and it's worst.

I think there's no real easy option to use Mxalloc - There's the best stability I have.

[medmed]

Hi,

Finally I used malloc:
- Mxalloc freeing well memory but others parts of memory are cleared - I don't understand why. So I loose my structs values and it's too instable.
- I've found an other implementation of memalign on the web. This version seems more stable than the one implemented on freemint. Whatever free don't released the whole memory.
- malloc also don't release all memory for large amount only, but it's the quicker solution I have.

Find below the diff (including memalign function) + psd_sdk_mint for 68k/CF.

[medmed]

Stable 0.7.5.2 revision published.

Be aware that depending the mokup selected, unpacking all PSD layers may require a lot of memory.

New:
- Read and write support for PSD pictures (Libpsd 0.9 replaced by Psd_Sdk)

Fix:
- Progress bar while opening PSD files.
- PSD Layers preview
- WebP alpha rendering (thanks to LP)
- Proposed width and height for resizing now fit the screen resolution (useful with MFD raw export)

Misc:
- Thumbs engine redesigned

Note: Due to bad implementation of memalign() in Freemint this function was replaced by malloc() in the pds_sdk library port.

[marss]

Thanks!!!

[arf]

[…]
- Mxalloc freeing well memory but others parts of memory are cleared […]. So I loose my structs values and it's too instable.

Are you really sure with that analysis? That would really make me wonder, as many programs rely on Mxalloc() and work perfectly fine.

[medmed]

Well unfortunately I'm pretty sure.

I don't know what the library do internally but it's what I've noticed.
It's what I've seen but as every thing is open source anyone can confirm that by himself.
Noticed that I think there's a good reason why psd_sdk uses memalign. I tried to patch with Mxalloc and there's the result:
Some of my structures data were lost if psd_sdk uses Mxalloc & Mfree instead of malloc/memalign & free: values set were modified by an overlapping memory write I guess.

I've wrote a function dedicated to debug that here :
- I've called this function to many places in the code to see when and why my structures values changed
- The changes were not always the same - and were random
I've check and recheck my code during hours before understand that.
- Then reswapping psd_sdk with malloc or memalign function solved the issue

N.B.: All structures (and everything else) in my code are allocated with Mxalloc - and everything works fine in it
I think it's library related functioning.

[mikro]

If by "library" you mean mintlib, it doesn't do anything, it just passes the parameters to the kernel.

While one can never rule out a problem in the kernel, I also find it very unlikely that Mxalloc would just by itself clear some other block's memory. You can try to record the number of calls with their sizes, then you could basically replay those calls and check after each call whether your previously allocated blocks still contain your prefilled value.

Most likely you will find out that everything's in order and that you have an illegal memory accesses in your code. mintlib's malloc allocates more memory at once (also using Mxalloc, by the way) so it could easily hide your illegal access: for instance if you malloc 2 KiB and get 4 KiB from mintlib, and then allocate 6, 10, 100, 1000, ... KiB, those remaining 2 KiB will never be used and broken code can write there whatever it wants and it will work fine.

[medmed]

By library I mean psd_sdk that uses memalign for a reason.

There's no illegal access with all the other formats & the same code is used for all of them, only using psd library overlap my already done Mxalloc so I don't really understand what to search. There's no Mfree until the application or image was closed.

I'm really disappointing guys - It's what I've noticed.

I use Mxalloc with my structures - I fill them with values ​​- I don't touch them anymore. I call some functions from psd_sdk then the initial values ​​has changed: There are cases where this happens - I just need someone who know the kernel architecture to investigate. I would advise not to accuse people who noticed weird behavior of memory access violation in their code without reproducing the scenario.
The Linux kernel still has bug fixes, but we don't. All right. Act like there is no such thing then.

BTW: I've also tried to alloc more memory than requested in psd_sdk lib with Mxalloc but same results.
I've built your fork (see it joined in previous posts) and let memalign and free (as by default) in psd_sdk - as you can see the whole memory (512mb) was eat.
whatever recording the number of calls with their sizes seems a good idea.

[shoggoth]

My pleasure. Can't wait I hope it'll work.
I don't know how pixels are oriented with et4000 -
For now the code take into account for 256 colors (via vq_scrninfo):
- Packed pixels rrrgggbb
- Interleaved planes
But not really tested for packed pixels as I played with Hatari or Aranym.
So tell me if everything will be okay or not, I will fix the problem if necessary.

One way to test 8bpp packed pixel modes on a Falcon is to e g use a 320x480x16bpp and pretend that it's 640x480x8bpp. Colours will look wrong, it's a bit blocky, but it can be good enough when testing.

[medmed]

My pleasure. Can't wait I hope it'll work.
I don't know how pixels are oriented with et4000 -
For now the code take into account for 256 colors (via vq_scrninfo):
- Packed pixels rrrgggbb
- Interleaved planes
But not really tested for packed pixels as I played with Hatari or Aranym.
So tell me if everything will be okay or not, I will fix the problem if necessary.

One way to test 8bpp packed pixel modes on a Falcon is to e g use a 320x480x16bpp and pretend that it's 640x480x8bpp. Colours will look wrong, it's a bit blocky, but it can be good enough when testing.

Thanks for the tips shoggoth,

The depack ARGB to 8bits chunky are already there, but I don't know what value is recovered for an et4000 in:
- screen_workstation_bits_per_pixel : work_out[4];
- screen_workstation_format : screen_vdi_info[0]; (via vq_scrninfo(vdi_handle, screen_vdi_info);)

[mikro]

I'm really disappointing guys - It's what I've noticed.

Don't feel disappointed - being sceptical is a natural scientific approach. :) So it's not like we don't believe you but as Carl Sagan used to say “extraordinary claims require extraordinary evidence”. (Free)MiNT is celebrating this year 30 years and in the whole history I don't remember one bug report similar to yours (there was one when invoking 'make' with a lot of files: under some circumstances the memory block list would become corrupt and started to fail on an assert; but that was in mintlib's malloc).

That doesn't mean that you aren't the first one who put FreeMiNT's memory allocator under real stress test and its symptoms are as described but you have to prove it. Until then, the most likely explanation remains a bug in your code (don't take it literally: it could be that psd_sdk was never tested without VM, on big endian, whatever... in my 20+ years career of a software developer and Atari coder I have seen quite a lot of weird things).

If you provide a small / isolated test case proving that Mxalloc is indeed doing something weird, I'll be more than happy to leverage it as a first class bug in the kernel. :)

[medmed]

I'm really disappointing guys - It's what I've noticed.

Don't feel disappointed - being sceptical is a natural scientific approach. So it's not like we don't believe you but as Carl Sagan used to say “extraordinary claims require extraordinary evidence”. (Free)MiNT is celebrating this year 30 years and in the whole history I don't remember one bug report similar to yours (there was one when invoking 'make' with a lot of files: under some circumstances the memory block list would become corrupt and started to fail on an assert; but that was in mintlib's malloc).

That doesn't mean that you aren't the first one who put FreeMiNT's memory allocator under real stress test and its symptoms are as described but you have to prove it. Until then, the most likely explanation remains a bug in your code (don't take it literally: it could be that psd_sdk was never tested without VM, on big endian, whatever... in my 20+ years career of a software developer and Atari coder I have seen quite a lot of weird things).

If you provide a small / isolated test case proving that Mxalloc is indeed doing something weird, I'll be more than happy to leverage it as a first class bug in the kernel.

Hi,

When I could I'll extract the while loop who extracts layers with psd_sdk and put some printf then we'll know
I've rechecked everything during hours to be sure. I would prefer to use Mxalloc.

[medmed]

I'm really disappointing guys - It's what I've noticed.

Don't feel disappointed - being sceptical is a natural scientific approach. So it's not like we don't believe you but as Carl Sagan used to say “extraordinary claims require extraordinary evidence”. (Free)MiNT is celebrating this year 30 years and in the whole history I don't remember one bug report similar to yours (there was one when invoking 'make' with a lot of files: under some circumstances the memory block list would become corrupt and started to fail on an assert; but that was in mintlib's malloc).

That doesn't mean that you aren't the first one who put FreeMiNT's memory allocator under real stress test and its symptoms are as described but you have to prove it. Until then, the most likely explanation remains a bug in your code (don't take it literally: it could be that psd_sdk was never tested without VM, on big endian, whatever... in my 20+ years career of a software developer and Atari coder I have seen quite a lot of weird things).

If you provide a small / isolated test case proving that Mxalloc is indeed doing something weird, I'll be more than happy to leverage it as a first class bug in the kernel.

Well I've wrote a small test case to reproduce this.
Then I've built psd_sdk with mxalloc and mfree and I've executed my test and no bug.
Then I've built mm_pic with the new built library and no bug.
Can it be Aranym?
I don't really understand... May be I'm drunk and I don't know that haha

But I know what I've seen - And I haven't change anything in my code so...

[mikro]

Are you running aranym-mmu ? I've noticed that aranym-jit can't be always trusted so even though it offers incredible performance, it's not worth it. Especially while developing something (plus you'll get nice check whether your code isn't doing something stupid). With today's CPU power... even aranym-mmu is enough. :)

[medmed]

Are you running aranym-mmu ? I've noticed that aranym-jit can't be always trusted so even though it offers incredible performance, it's not worth it. Especially while developing something (plus you'll get nice check whether your code isn't doing something stupid). With today's CPU power... even aranym-mmu is enough.

Thanks Mikro,

I have JIT set to yes + BootstrapArgs = MEM_PROT=YES DEBUG_LEVEL=1
Also I just noticed that I've this variable in my config file : FixedMemoryOffset = 51000000 and I don't know why, an OSX fix for using JIT I guess.

The binary name on OSX is

Code: Select all

./MacAranym\ JIT\ SDL2.app/Contents/MacOS/MacAranym\ JIT\ SDL2 

So may be I can change JIT=yes to JIT=no in the config file but I don't know how to run with mmu mode?

[ThorstenOtto]

I have JIT set to yes + BootstrapArgs = MEM_PROT=YES DEBUG_LEVEL=1

The JIT version does not support memory protection, they are mutually exclusive.

Also I just noticed that I've this variable in my config file : FixedMemoryOffset = 51000000 and I don't know why, an OSX fix for using JIT I guess.

Yes, thats the virtual address that the JIT version uses to access the atari memory. Sometimes that address is already used by the OS, or by shared libraries, and then it has to be adjusted. But most of the time the default should work.

So may be I can change JIT=yes to JIT=no in the config file but I don't know how to run with mmu mode?

There should be a separate executable (aranym-mmu) for this.

[Eero Tamminen]

memalign() is not properly supported in mintlib. It is properly supported in my fork which uses dlmalloc allocator, feel free to try it out: https://github.com/mikrosk/mintlib/tree/dlmalloc.
...

I've just tried with mintlib_dlmalloc version using memalign(alignment, size) & free(ptr) and it's worst.

Worst in which way? It works fine in ScummVM, which is a pretty complex C++ program. I profiled lot of different ScummVM game engines, and for some engines & their games that did a lot of allocations and frees (due to it being C++ code I think), it gave a significant speedup, and no slowdown in others.

[medmed]

I have JIT set to yes + BootstrapArgs = MEM_PROT=YES DEBUG_LEVEL=1

The JIT version does not support memory protection, they are mutually exclusive.

Also I just noticed that I've this variable in my config file : FixedMemoryOffset = 51000000 and I don't know why, an OSX fix for using JIT I guess.

Yes, thats the virtual address that the JIT version uses to access the atari memory. Sometimes that address is already used by the OS, or by shared libraries, and then it has to be adjusted. But most of the time the default should work.

So may be I can change JIT=yes to JIT=no in the config file but I don't know how to run with mmu mode?

There should be a separate executable (aranym-mmu) for this.

Thanks Thorsten - I'll search for such executable for osx.

[medmed]

memalign() is not properly supported in mintlib. It is properly supported in my fork which uses dlmalloc allocator, feel free to try it out: https://github.com/mikrosk/mintlib/tree/dlmalloc.
...

I've just tried with mintlib_dlmalloc version using memalign(alignment, size) & free(ptr) and it's worst.

Worst in which way? It works fine in ScummVM, which is a pretty complex C++ program. I profiled lot of different ScummVM game engines, and for some engines & their games that did a lot of allocations and frees (due to it being C++ code I think), it gave a significant speedup, and no slowdown in others.

Hi Eero,

Here a resume:
- By default the library psd_sdk uses memalign() and free.
It conducted to a memory leak, memory wasn't entirely released.
- I tried to implement a custom memalign() grabbed from old linux source,
it seemed to release more memory than the default memalign() implementation but not a lot
- malloc() do not released all allocated memory randomly - I don't know if it's again due to my environment
- Then I've tried mint_dlalloc with memalign and nothing was released. All memory was consumed
-> Perhaps I had to use a memalign_free() function - I've used free()
- Mxalloc() released all allocated memory but I had other part of memory who seemed cleared
- I've tried to reproduce that but it seems to work now

Do you use memalign for scummVM?

EDIT: I'll retry again as with Mxalloc.