SLB 'devkit' ?

[vido]

@vido:

How much work/hard would be to adapt sources that Vision would be compilable with the gcc?
I am asking this because native coldfire version for the Vision would be really great!

A LOT:
First a remark: VISION has been designed in early 90s with 1 big requirement and an assumption:
- Be fast, so use 680x0 code when it comes to CPU intensive stuff such as bitplane to index conversions, bitpmap zooms, LZW inflate/deflate, etc.
- Do not consider portability, i.e. an int is always 16bit, quite a crazy assumption nowadays...
So to be gcc compliant:
- rework all files to deal with int vs WORD
- for Coldfire, it would be even MUCH worse as all 68000 and 68020 routines would have to be ported to C.. And there are tons of these...

I know Jean but I was more aiming to get answer from Thorsten. We already talked about that
But I see this more as step by step procedure as Vision as it is is already working on the FireBee and you already did first step by using SLB and LDG modules which are already coldfire code and that made Vision usable on the FireBee! Thank you for that again!
About the 68000 and 68020 routines ... they are working as they are and for the beginning if you just rework C part to be compiled with the gcc there would be another improvement for the FireBee. Regarding routines ... they can be reworked one by one if you or someone else find the time and will
And ... I am not pushing anything! I am just asking for the opinion as I am not the coder myself ...

[ThorstenOtto]

Why do you expect a different answer from me? It's Jean's project, so he has to decide. I only helped out to support the changed versions of zView modules (mainly because i was guilty to cause extra work for him ;D ).

So i only looked at small portions of the code so far. Generally, porting C-code from/to gnu-c should be doable, but is still quite some work, The problem here are the assembler sources mostly, and that are usually parts you don't want to touch too often unless you have to. It's easy to make mistakes there, so that would require a lot of testing afterwards. And not all developers have a firebee at hand to do such tests.

you already did first step by using SLB and LDG modules which are already coldfire code

That's only true for the modules that come with zView. The dozens of modules from Lonny are coldfire-compatible in the sense that they will work with the 68k emulation, but it's not pure coldfire code.

[vido]

Thanks Torsten for the answers!

I really didnt expect much different answers from Jean except your opinion about C code adapting to the gcc.
True about the modules and that already helps a lot!

Just a note also to the other developers ... there are some developers FireBees that can be lended to the coders who would develop/port their software to the FireBee for some time.

[JeanMars]

Hi Thorsten,

You might have missed this one as a page wrap occurred right after:
I had a quick look at the modifications you made, 2 comments:
- Yes please I would prefer to have a ZIP containing all updated source archive
- I see many static dependencies in new sources regarding jpg, png and tiff for new SLB format. Why this? The caller should not be aware of this, does that mean that every time a new dynamic module is added, VIsion/zView has to be modified to be aware? This is not the case for LDG ones.
And a new one:
- when re-implementing qsort/bsearch instead of pointing to official 'C' lib?

Cheers,
Jean

[ThorstenOtto]

Oops sorry, yes i missed that. You can find the archive here

The static dependencies are the SLB structures, and small wrapper functions to load the 3rd party libraries like pnglib.This is because pnglib etc. needs functions from the C-library, and are not directly used by the application, but only by the codec that needs it. Moving those structures to the codecs would mean i have to link in the c-library in the codecs again.#

does that mean that every time a new dynamic module is added, VIsion/zView has to be modified to be aware?

No, that would only be the case if it needs a new shared lib, and then only if that lib is also used by other modules. Very unlikely currently, most larger freely available, image loading libraries are already done. And in other cases (like gif for example) they are statically linked.

There are a few that are currently known, but there is no import library for Pure-C yet. Those are ifdef'd out in the source currently. The complete list is defined in https://github.com/th-otto/zview/blob/3 ... .h#L73-L80

- pnglib: done
- zlib: done. needed by pnglib, freetype, and pdf (in zView only)
- jpeg: done, but the codec also needs exif, which is not ready yet
- tiff: not ready yet. This will get a bit tough, due to its size, and because it uses 64bit integers which are not available in Pure-C
- lzma: same as tiff.
- exif: not yet finished, but working on it so the jpeg codec can be used
- bzip2: done, but only needed by freetype currently, which is used by pdf
- freetype: done, but only needed by pdf

when re-implementing qsort/bsearch instead of pointing to official 'C' lib?

Because the qsort function in pcstdlib expects Pure-C calling convention for the comparison function, while it is actually defined in the library using cdecl calling. Same for bsearch. Eventually, they could be moved to the codecs and/or libraries instead, having a bit of duplicate code, but avoiding such problems.

[ThorstenOtto]

Exif library is now also available at http://tho-otto.de/sharedlibs.php. Attached is a small patch that activates the jpeg module for vision, based on the previous patches. The archive also contains the changed files, and the import libraries for jpeg and exif. Note that the jpeg import library is a stripped down version of what is normally available, since it would otherwise conflict with the static version you currently link to. You will still need the jpeg8d.slb from my archives, though.

[JeanMars]

Hi Thorsten,

thanks for your efforts on this, I'm a bit confused; I don't want to waste your time as I'm not sure I will implement new zView SLB modules in Vision.
Let's take a step back:
zView has the legacy LDG module system. They are working with some drawbacks such as:
- memory consumption as common libs (mintlib, zlib,exif,...) are not shared
- module itself cannot be used by other apps
I believe you main motivations to design this SLB zview plugin model are to solve these 2 drawbacks.
However, the current design implies 2 other drawbacks:
- More complexity on the app side dev/maintenance as many regular 'C' functions have to be exported, which adds even more complexity when it comes to deal with PureC, sometimes I'm thinking app is also a plugin for a SLB module
- App has dependencies to shared SLB modules when an image module depends on it (such as exif, jpg, tiff, png). That really bugs me as the app shall know nothing about module dependencies (in my opinion of course)

I would suggest a different approach:
- for memory consumption:
- there is a light 'mintlib' that can be linked against the module to move from ~100KB to ~10KB. Assuming no need to printf or so which is a fair assumption for a module I guess -or- just use PureC
- There should be anyway some kind of smart module loading management on the app side: app should not load each and every module, just dynamically load the ones that are about to be used. And in case memory gets low, app can decide to unload less used modules
- for library sharing: assuming we have this smart memory module management, the duplicated modules for non shared libraries would just impact the size on the disk, which is I believe not a big deal

Note that I like the possibility for the image module to call back to the app, but to me it shall not be mandated, I see this possibility more for a helper function for the module such as sending a log information to the app.

What do you think?
Maybe we should open a new discussion on this as it goes beyond the SLB devkit...
Cheers,
Jean

[ThorstenOtto]

I'm not sure I will implement new zView SLB modules in Vision.

But you have done that already And Lonny is almost finished converting his modules, so once they are available, you could drop instead the LDG module support.

I believe you main motivations to design this SLB zview plugin model are to solve these 2 drawbacks.

Yes, basically.

adds even more complexity when it comes to deal with PureC

Yes, for Gnu-C, that would have been much easier. And actually the code that i've hacked into Vision should be part of some library/devkit that comes along with zView. But afaik, Vision is the only app that makes use of zView modules, so for now that should suffice. And such efforts are not uncommon, they are two different languages somehow, so it is comparable to something like creating C-bindings for Perl or other languages.

App has dependencies to shared SLB modules when an image module depends on it (such as exif, jpg, tiff, png).

Yes, that is really not so nice. But i don't think there is a easier way to handle this, other than "hiding" that code in a separate library.

there is a light 'mintlib' that can be linked against the module to move from ~100KB to ~10KB.

Basically that is what Lonny already has done, by using Pure-C for his modules. But there are are severe problems with this:

• in zView at least, the png library was used 3 times at least, zlib 4 times at least, and jpeg 2 times. If you look at the size of SLBs (which don't even have mintlib linked), that means ~900K of memory&code wasted. If you run the original zView, you will see that it takes almost 8MB of memory, even without any images opened yet (and that is with just the codecs provided by zview, not counting any extra ones). That has been dropped to ~1.5MB with the current version. So we are talking about several MB here, not just a few KB. IMHO that was way too much, even for "high-end" Ataris.
• There is a conceptual problem when linking any c-library into the modules. Any resources allocated (memory, open files etc) are done on behalf of the application that uses it, but maintained statically in the modules. So the application has no way of freeing these resources when unloading the modules.

Assuming no need to printf

Unfortunately that assumption is wrong. Almost all libraries have some logging mechanism, and that is mostly using printf style functions.

app should not load each and every module, just dynamically load the ones that are about to be used.

Agreed, we have talked about that already. But it is not as easy as one may think, especially the condition whether a library in still in use.

a helper function for the module such as sending a log information to the app.

Yes, that's one of the things on the todo list for zView. There is currently sometimes no indication to the user why a image could not be loaded. OTOH, you don't want to have to click away dozens of alert boxes when opening a browser window.

[Cyprian]

Assuming no need to printf

Unfortunately that assumption is wrong. Almost all libraries have some logging mechanism, and that is mostly using printf style functions.

what about adding a dummy printf function to a light 'mintlib'?

[JeanMars]

Hi Thorsten,

But you have done that already And Lonny is almost finished converting his modules, so once they are available, you could drop instead the LDG module support.

With your help Actually, zView LDG modules are working pretty nicely except for memory consumption I would agree but this can be mitigated a lot by implementing a smart module loader.
With zView SLB plugin design, it pushes back the plugin dependencies complexity to the application side, which I really don't like. I know there are not so many candidates for implementing zView SLB plugin support but if you add complexity here, that's not good advertising

in zView at least, the png library was used 3 times at least, zlib 4 times at least, and jpeg 2 times. If you look at the size of SLBs (which don't even have mintlib linked), that means ~900K of memory&code wasted. If you run the original zView, you will see that it takes almost 8MB of memory, even without any images opened yet (and that is with just the codecs provided by zview, not counting any extra ones). That has been dropped to ~1.5MB with the current version. So we are talking about several MB here, not just a few KB. IMHO that was way too much, even for "high-end" Ataris.

8MB because all modules are loaded When zView is up, no module should be loaded. Unbeatable
BTW, just noted that Vision takes about 1.2MB when started and that includes all internal image modules like TIFF, JPEG, TGA, GIF (much more limited of course than open source libs) and classic Atari formats plus all editing/catalog/filter stuff. You sure that you can't improve memory consumption more in zView (see attached top command)?

There is a conceptual problem when linking any c-library into the modules. Any resources allocated (memory, open files etc) are done on behalf of the application that uses it, but maintained statically in the modules. So the application has no way of freeing these resources when unloading the modules.

Each module has/should have a terminate function called by the app to free all allocated stuff when module is about to get unloaded. Anyway if a module does not free its own stuff, SLB model does not help here.

Agreed, we have talked about that already. But it is not as easy as one may think, especially the condition whether a library in still in use.

? Right after an image is loaded, call the image module memory manager to unload the most unused modules.
I tried an extreme test with Vision: unload a zvLDG after an image is loaded, created a catalog of more than 1000 files with 100 LDGs, no problem.

Unfortunately that assumption is wrong. Almost all libraries have some logging mechanism, and that is mostly using printf style functions.

As mentioned by Cypran, just define printf to void in this case as a printf would be ugly anyway. Here is the main interest for the app to provide a printf-like logging function.

Don' get me wrong, I'd be happy to implement a modern plugin model in Vision, my point is that the current SLB one is not mature enough to me; I've spent a lot of time in Vision to remove this kind of dependencies regarding GUI/image module/plugins and I'm not ready to break this...

Cheers,
Jean

[ThorstenOtto]

8MB because all modules are loaded

Yes, but that was with only the 12 modules supplied by zview.

You sure that you can't improve memory consumption more in zView (see attached top command)?

Not much. Only thing that should be changed is to delay loading the dependent libs unil they are needed, currently that is already done when initializing the module, eg. when gathering the list if extensions. Your screenshot also shows that most of them use exactly 16384 bytes. I'm currently not sure why it is twice as much as expected, but mint will round any memory allocation up to a multiple of 8k, that is something you cannot change (and does not make sense anyway when using MP). I have to check though why zveureka and zvjpg seem to use more than this, maybe there is a leak.

Each module has/should have a terminate function called by the app to free all allocated stuff when module is about to get unloaded. Anyway if a module does not free its own stuff, SLB model does not help here.

That's true. But the main problem here are not the modules itself, but the libraries they use. The modules itself usually do only 1-3 allocations, that is manageable. What's difficult to check are the allocations of jpeg/tiff etc., you can never be sure that they really release all their memory. They are just not designed that way, because typically on unix a shared lib is only unloaded when the program is about to exit.

As mentioned by Cypran, just define printf to void in this case as a printf would be ugly anyway.

That's not doable. The libraries are designed to have the same interface as the static ones, since they are also used by the commandline tools that come with them, like tiffcp, tiffcrop, pnginfo etc. If they would only be used by zView/vision, i would not even need to export all the available functions. But the goal here (especially for pnglib and zlib) was to have a direct replacement for the static versions.

the current SLB one is not mature enough to me

Well i've just started, there is always room for enhancements

[mikro]

zView has the legacy LDG module system. They are working with some drawbacks such as:
- memory consumption as common libs (mintlib, zlib,exif,...) are not shared
- module itself cannot be used by other apps

If you are interested about LDG pitfalls and possible solutions, feel free to read this: https://docs.google.com/document/d/1d8M ... sp=sharing, I wrote it a few years ago. Basically LDG (in some future version) could solve the first and is able to solve the second (with the LDG manager in AUTO).

The first problem can be solved (and I had done it to some extent) with a context. I did it with libc (i.e. exported all libc functions and #define'd them like "#define printf context->printf"), it required basically no change to code which uses it (just included "my_libc_wrapper.h" instead of <stdio.h>)

It would be an interesting challenge to extend this concept, i.e. have some kind of dynamic context where you could "preload" all the libraries (libc, zlib, ...) and somehow transparently call it from the modules.

I share this only as a food for thought, basically Thorsten's solutions is already here and works. The only drawback is that it is not so easy to update / maintain, as the SLB frontend requires quite some changes.

[ThorstenOtto]

Basically LDG (in some future version) could solve the first and is able to solve the second (with the LDG manager in AUTO).

I don't agree with that. In your blog you state that there is no problem if some function inside a library calls malloc(), does some work, then calls free. But there are certain problems due to that:

• malloc does it own memory management, using a list or whatever that is statically maintained inside the library.
• Because of that management, the malloc() call may actually result in a GEMDOS Malloc() call, but the free() call does not neccessarily call Mfree(); it might just add the block to the free list.

The first problem can be solved (and I had done it to some extent) with a context. I did it with libc (i.e. exported all libc functions and #define'd them like "#define printf context->printf"), it required basically no change to code which uses it (just included "my_libc_wrapper.h" instead of <stdio.h>)

This is actually exactly what i am doing Only main difference: the plugin system of zView/vision requires the calls to be passed from the 3rd-party libraries through the codecs to the application.

Most of the problems here arise from the two different languages/libraries: the SLBs are compiled by GNU-C, but Vision is compiled by Pure-C (and also the other way around: Lonnys codecs are compiled by Pure-C, but zView by GNU-C). So you cannot just point the printf calls inside the SLB to Pure-Cs printf library function, you need a small wrapper for that (not only for printf, but actually for all library functions you want to export). The other problem is the different "int" size of the two compilers (for "double" that is even worse, but luckily there are not many functions using it).

It would be an interesting challenge to extend this concept, i.e. have some kind of dynamic context where you could "preload" all the libraries (libc, zlib, ...) and somehow transparently call it from the modules.

That would be essentially the solution you would need to implement real shared libraries, like on unix. No hope for that i fear, that cannot work without a true virtual memory management, and especially it would require all applications to use the same C-Library, which would rule out all Pure-C programs. Not to speak about other languages like GFA.

The only drawback is that it is not so easy to update / maintain, as the SLB frontend requires quite some changes.

I haven't looked at or updated yet the zcodeclib which is also part of the repo, maybe that could solve some of the problems. But using that for vision would require changes there again...

Actually i don't expect that there are any other 3rd-party libraries added to this system in the near future. The only missing thing (for Pure-C) is tifflib (and its dependent lzma) currently. Theoretically that could already be implemented since Vision does not call any TIFF functions itself.

[ThorstenOtto]

The delayed loading of tiff etc. has now been implemented. Good news for you: no change is required on your side Just install the codecs from the latest build. Only zvtiff, zvpng and zvjpeg are affected.

[JeanMars]

Hi Thorsten,

not sure I get your point; does that mean I don't have to have any tiff/png/jpeg hard-coded dependency in Vision's code to use the SLB plugins?

[ThorstenOtto]

For png: no, because that was already done. For jpeg: that was already part of the last patch. For tiff: i still have to create the libraries for purec, but the other code is already there and just has to be activated.

[JeanMars]

Ah good news
So what does Vision need to do now? 'Just' export 'C' stdlib functions to zvSLB? Would be nice to get rid of this dependency too BTW...
Right now I'm working on the smart image module memory loader, I'll look into this after.

Cheers,
Jean

[ThorstenOtto]

Just apply the patches i already sent (if not already done). The exports are part of the 2nd patch (and also in the archive).

[JeanMars]

Hi Thorsten,

I still see dependencies to exifslb.lib and jpegslb8.lib in vision.prj.
And iizview.c still has these dependencies:
SLB *__CDECL plugin_slb_get(zv_int_t lib)
{
switch ((int)lib)
{
case LIB_PNG:
return slb_pnglib_get();
case LIB_Z:
return slb_zlib_get();
case LIB_JPEG:
return slb_jpeglib_get();
case LIB_EXIF:
return slb_exif_get();
#ifdef NOTYET
case LIB_TIFF:
return slb_tiff_get();
case LIB_LZMA:
return slb_lzma_get();
case LIB_BZIP2:
return slb_bzip2_get();
case LIB_FREETYPE:
return slb_freetype_get();
#endif
}
return NULL;
}

My understanding was these jpeg/tiff/lzma/exif were out of Vision. For png/zlib, this is not a big deal as Vision has native png support with png lib (through Vision static internal model, not zvLDG/zvSLB).

[ThorstenOtto]

Yes, that loading of the libraries is still done by the application. What i've changed recently is just that eg. tiff.slb is not loaded immediately anymore when you load the module (for example when scanning the directory), but only when needed. The current dependencies will remain, since that is the place where your applications library functions are hooked into the SLBs.

[JeanMars]

Hi Thorsten,

as I'm still reluctant to have this plugin_slb_get and related functions in Vision, can we do half-way:
- Vision will export regular 'C' functions to SLB plugin
- SLB plugin won't crash if an expected export is not present (function NULL) and report an error from plugin_set_imports call so that Vision can decide to not use it

This way, SLB with dependencies on other SLBs won't get loaded by Vision but other will.

Another question: does SLB plugin make a copy of zview_plugin_funcs passed in plugin_set_imports, or it just use the provided pointer? This is to know if this structure could be local instead of global.

[ThorstenOtto]

- Vision will export regular 'C' functions to SLB plugin

But that is already done.

- SLB plugin won't crash if an expected export is not present (function NULL) and report an error from plugin_set_imports call so that Vision can decide to not use it

That is what the interface_version is designed for. If the interface to the codecs has to be changed for whatever reason, this version will be increased, and Slbopen will already return an error. Checking the individual function pointers does not make sense, because i cannot suddenly return an error from functions like memcpy() instead of a pointer. Also, i cannot add those checks to the plugins that are not part of zview, like the ones from Lonny.

This way, SLB with dependencies on other SLBs won't get loaded by Vision but other will.

Again, that should already be the case. You just have to make sure that you don't call any plugin function if plugin_init() fails; i think this is currently not always the case, the get_option function seems to be called in GetDrvCaps.

it just use the provided pointer? This is to know if this structure could be local instead of global.

It uses only the pointer. But the structure is already locally defined in iiziew.c?


Only thing that i could to is to move some code out of iizview.c into a separate file, which hides those dependencies, and could be updated separately if needed.

[JeanMars]

Hi Thorsten,

But that is already done.

Well..not. I didn't add these pieces of code in Vision's sources because I'm not ready to have this kind of dependency in Vision.

My proposal here is to have some kind of trade-off between your SLB model, for which I don't want it to make these dependencies in Vision, and the possiblity for Vision to support SLB plugins when they don't have other SLB dependencies.

Again, that should already be the case. You just have to make sure that you don't call any plugin function if plugin_init() fails; i think this is currently not always the case, the get_option function seems to be called in GetDrvCaps.

iizv_slb_plugin_init does not do anything right now; this is where I'm supposed to call plugin_set_imports but I'd like to fill the zview_plugin_funcs only with functions that make sense (to me ).
Anyway, simply best practice here: if you find a NULL pointer to a function you expect, just fail the request, don't try to make a call. Just asking for some flexibility here for you & me to agree on.

Only thing that i could to is to move some code out of iizview.c into a separate file, which hides those dependencies, and could be updated separately if needed.

This is not the point, I don't want to hide these dependencies, I want to not have them

[ThorstenOtto]

but I'd like to fill the zview_plugin_funcs only with functions that make sense (to me ).

But that doesn't make sense. If you don't set some function, and the plugin fails if any is null, you won't be able to use any of them.

I also don't see the point. All the referenced functions are already there.

I want to not have them

Sorry, but then you have to stick to the old LDG modules, with the same libraries linked statically a few hundred times into the modules instead of only once in the application. The new modules just need those "dependencies", they cannot work without. It was one of the major goals of the new design.

I'm not saying that the new design is perfect, but it works, and much better than before.

[JeanMars]

Hi Thorsten,

But that doesn't make sense. If you don't set some function, and the plugin fails if any is null, you won't be able to use any of them.

I'm pretty sure all plugins don't need all exported functions. My suggestion was just to fail init if some function plugin knows it needs is not provided.

Sorry, but then you have to stick to the old LDG modules, with the same libraries linked statically a few hundred times into the modules instead of only once in the application.

Only if app decides to load them all Which is, to me, not something to do.

Basically, this new design, IMHO, is pushing back too much complexity on app side, has to know SLB dependencies regarding the modules, provide tons of usual standard functions to modules, deal with PureC/gcc difference, even reimplement qsort and bsearch primitives and I'm afraid of maintening this as well.

That's why for the time being I'll stick Vision to the LDG model.

Cheers,
Jean