LZ4 decoder (assembler)

All 680x0 related coding posts in this section please.

Moderators: simonsunnyboy, Mug UK, Zorro 2, Moderator Team

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

orionfuzion wrote:Hi everybody,

I jump on this thread to ask if someone has the ARJ mode7 unpack routine in ASM?

Thx,

Orion
Finally found in the ARJ Beta v9.97B package on demozoo: https://demozoo.org/productions/133652/

wietze
Captain Atari
Captain Atari
Posts: 306
Joined: Fri Mar 01, 2013 10:52 pm

Re: LZ4 decoder (assembler)

Post by wietze »

leonard wrote:
ggn wrote:Apart from the 4k binary Cyprian tried, has anyone done any other tests regarding pack ratio? I'm still favouring ARJ mode 4 and 7 for max pack ratio, but this looks like a great alternative for when depack speed is crucial.
Regarding packing ratio, if you're looking for best ratio ever, use shrinkler. But depacking time is really, really slow. For very good packing ratio and average depacking speed, use ARM mode 7 or UPX ( UPX is almost same packing ratio as ARJ m7 but with depacks 30% faster ). LZ4 has not good packing ratio but is extremly fast compared to other. Look at my atari-st depack benchmark result. First column is the packed size of 3 different binaries for test. Last columns show the number of "50hz" ticks to depack ( lower is faster ).

.l77 is LZ77
.lz is doynamite depacker
.upx UPX
.am7 ARJ beta -m7
.pft PackFire
.lz4 lz4 ( my fast depacker version )
.shk Shrinkler
DepackB_004_SC1425_fastest.png
Would you also include cranker in your benchmark?

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

wietze wrote:
leonard wrote:
ggn wrote:Apart from the 4k binary Cyprian tried, has anyone done any other tests regarding pack ratio? I'm still favouring ARJ mode 4 and 7 for max pack ratio, but this looks like a great alternative for when depack speed is crucial.
Regarding packing ratio, if you're looking for best ratio ever, use shrinkler. But depacking time is really, really slow. For very good packing ratio and average depacking speed, use ARM mode 7 or UPX ( UPX is almost same packing ratio as ARJ m7 but with depacks 30% faster ). LZ4 has not good packing ratio but is extremly fast compared to other. Look at my atari-st depack benchmark result. First column is the packed size of 3 different binaries for test. Last columns show the number of "50hz" ticks to depack ( lower is faster ).

.l77 is LZ77
.lz is doynamite depacker
.upx UPX
.am7 ARJ beta -m7
.pft PackFire
.lz4 lz4 ( my fast depacker version )
.shk Shrinkler
DepackB_004_SC1425_fastest.png
Would you also include cranker in your benchmark?
Hi Wietze,

I jump on you comment for my own purpose :)
I guess you ported the Amiga version to the ST?
Can you provide me with this packer and the source of the unpack routine?

Thx,

Orion

User avatar
ggn
Atari God
Atari God
Posts: 1258
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Post by ggn »

Orion,

You can grab arj mode 7 packer and unpacker with source and windows binaries from the other orion_'s (with underscore!) website: http://onorisoft.free.fr/jag/bjlpak.zip

As for cranker, I'm sure that last time I tried, I built it with standard cli tools and gcc, it worked first time.
is 73 Falcon patched atari games enough ? ^^

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

ggn wrote:Orion,

You can grab arj mode 7 packer and unpacker with source and windows binaries from the other orion_'s (with underscore!) website: http://onorisoft.free.fr/jag/bjlpak.zip

As for cranker, I'm sure that last time I tried, I built it with standard cli tools and gcc, it worked first time.
Thx mate.

The Cranker compression program was indeed already ported to various OSes and the unpacking routine, which is pure 68k code, can be reused as is on ST.
But the compression ratio is disappointing. UPX is clearly better (nrv2b).

As for arj mode 7, it has the best compression ratio, but the unpacking routine requires 13KB of free space which does not cope with my constraints...

So I have the choice between UPX and Atomik 3.5 for my "project"...
Atomik 3.5 is not that bad, it is slightly better than UPX/nrv2b in most cases, even if the unpacking routine has a bigger footprint and is a bit slower.

I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?

-- Orion

User avatar
ggn
Atari God
Atari God
Posts: 1258
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Post by ggn »

orionfuzion wrote:I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?
Try https://github.com/ggnkua/Atari_ST_Sour ... /Hakim/UPX
is 73 Falcon patched atari games enough ? ^^

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

ggn wrote:
orionfuzion wrote:I'm currently trying to modify the UPX compression program to enable the packing of data files.
Did someone already work on that?
Try https://github.com/ggnkua/Atari_ST_Sour ... /Hakim/UPX
Yep, I saw the corresponding post on this forum (15 years ago !).
But I don't like the approach as UPX packs the dummy PRG header together with the actual data.
Thus I'm working on modifying UPX sources directly in order to enable data files packing.
I will post the patch here ;)

-- Orion

User avatar
Zorro 2
Administrator
Administrator
Posts: 2217
Joined: Tue May 21, 2002 12:44 pm
Location: Saint Cloud (France)
Contact:

Re: LZ4 decoder (assembler)

Post by Zorro 2 »

Hi all !

Great news that people want to contribute if this little part of UPX which begins here : http://www.atari-forum.com/viewtopic.php?f=16&t=1060

I worked with Hakim about issues there are few years ago in ASM and GFA....

I did it but don't forget to notice :
- small or bigger header option should be used and know how to decompress (add in the source !)
- which number of version of UPX used to compress ?
- and of course parameters used (--nrv2b, --best, -9k, etc.)

@orion : it should be interesting to see you to code in ASM :mrgreen: :mrgreen: :mrgreen:
Member of NoExtra Team

User avatar
troed
Atari God
Atari God
Posts: 1456
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: LZ4 decoder (assembler)

Post by troed »

ripupx.s removes the dummy PRG header. I modified the code to support latest UPX versions and -small etc for Closure.

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

troed wrote:ripupx.s removes the dummy PRG header. I modified the code to support latest UPX versions and -small etc for Closure.
I dislike Hakim's approach which is to add a fake PRG header before the actual data in order to fool UPX, because that fake PRG header is packed together with the data and it may be a problem in my case (this is the "dummy" header I mentioned, not the one added by UPX to decompress the exe which is removed by ripupx.s).

@Zorro2 : I will code in C to modify UPX ;)
As for the ASM, just wait for my next crack, it's already 5000 lines of 68k code!
Why so much code for a single crack? ;)

-- Orion

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

Hi mates,

Please find attached a patch (zipped) that enables the packing of raw data files using the UPX compression program.
Thus there is no more need to insert a fake PRG header to fool UPX for compressing data files.

Apply the patch to the source directory of the latest UPX version (3.95).
Run make all and use the resulting src/upx.out binary.

To compress a data file, just run UPX as follows:
$ upx <your standard compression options> --data

For instance:
$ upx --nrv2b --small test.dat -otest.upx --data

To unpack a compressed data file, use the standard "-d" option:
$ upx -d test.upx -otest.dat

Enjoy ;)

-- Orion
You do not have the required permissions to view the files attached to this post.
Last edited by orionfuzion on Mon Apr 01, 2019 9:49 pm, edited 1 time in total.

User avatar
troed
Atari God
Atari God
Posts: 1456
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: LZ4 decoder (assembler)

Post by troed »

Awesome! Will definitely help smoothing out my build chain :)

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

Cool, another packer to my tool chain.

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

WOW... I just switched from Atomik 3.5 to UPX nrv2b for the packing of my next crack/fileversion (2 floppies), I got the same compression ratio but UPX decrunches at the speed of light :)

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

Anybody knows how to modify the depacker routine to use a1 as destination address of unpacked data? I'm having some troubles :(

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

I think that I've fixed it...

F*ck!, not yet.

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

Done!

If someone is interested I can post the code later, you just have to remove some part that copy the packed data at the end of the buffer.

User avatar
ggn
Atari God
Atari God
Posts: 1258
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Post by ggn »

swapd0 wrote:Done!

If someone is interested I can post the code later, you just have to remove some part that copy the packed data at the end of the buffer.
Declaring interest in the source! Is yours based on hakim's, modified to be a0->a1?

[EDIT]I use --ultra-brute to compress so maybe the default unpack routine won't work with my case...
is 73 Falcon patched atari games enough ? ^^

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

I'm using the following command to pack the data files.
upx -q --nrv2b --small --data infile -ooutfile

a0 packed data
a1 destination addresss

Code: Select all

;UPX unpack routine (NRV2B only!) (--best)
;==================
;
;Usage:		a0 = start of data
;			a1 = destination address
;
;Associated files:	Make_prg.s	Make a "program" out of data-file
;					-> Data can be packed with PC UPX
;
;			Ripupx.s	Takes out "unnecessary" part out of
;					datafiles, works only with UPX v.1.24
;
;			Unp_test.s	A test program for finding out a
;					suitable decrunch space (usually $2e)
;
;Compared to other existing Atari ST packers; Atomik, Pack-Ice, Pack-Fire,
;Jam Pack, PP Packer, Automation packer, Speedpacker 3 etc., UPX v.1.24
;offers better compression and much faster decompression.
;
;UPX v.1.24 doesn't work well with the files smaller than 1024 bytes,
;usually they are left unpacked.
;
;This source alongside the associated files can be distributed under the
;terms of GNU General Public License.
;
;And just as the UPX, this source and the associated sources and their
;binaries comes with ABSOLUTELY NO WARRANTY. So better make those backups!
;
;More info about UPX:	http://upx.sourceforge.net
;

dec_space	equ	$2e		;at least $2e, no odd values!

do_upx:
	movem.l	d0-d7/a0-a6,-(sp)
	cmp.l	#"UPX!",(a0)
	bne	no_upx

	lea	$20(a0),a0
	moveq	#-1,d7
	moveq	#-$80,d0
	moveq	#-$68,d6
	moveq	#-1,d5
	lsl.w	#5,d6
	bra.s	l33ee8
l33ee6:
	move.b	(a0)+,(a1)+
l33ee8:
	add.b	d0,d0
	bcc.s	l33ef4
	bne.s	l33ee6
	move.b	(a0)+,d0
	addx.b	d0,d0
	bcs.s	l33ee6
l33ef4:
	moveq	#1,d1
	moveq	#0,d2
l33ef8:
	add.b	d0,d0
	bne.s	l33f00
	move.b	(a0)+,d0
	addx.b	d0,d0
l33f00:
	addx.l	d1,d1
	add.b	d0,d0
	bcc.s	l33ef8
	bne.s	l33f0e
	move.b	(a0)+,d0
	addx.b	d0,d0
	bcc.s	l33ef8
l33f0e:
	subq.l	#3,d1
	bcs.s	l33f1c
	lsl.l	#8,d1
	move.b	(a0)+,d1
	not.l	d1
	beq.s	l33f62
	move.l	d1,d5
l33f1c:
	add.b	d0,d0
	bne.s	l33f24
	move.b	(a0)+,d0
	addx.b	d0,d0
l33f24:
	addx.w	d2,d2
	add.b	d0,d0
	bne.s	l33f2e
	move.b	(a0)+,d0
	addx.b	d0,d0
l33f2e:
	addx.w	d2,d2
	tst.w	d2
	bne.s	l33f4e
	addq.w	#1,d2
l33f36:
	add.b	d0,d0
	bne.s	l33f3e
	move.b	(a0)+,d0
	addx.b	d0,d0
l33f3e:
	addx.w	d2,d2
	add.b	d0,d0
	bcc.s	l33f36
	bne.s	l33f4c
	move.b	(a0)+,d0
	addx.b	d0,d0
	bcc.s	l33f36
l33f4c:
	addq.w	#2,d2
l33f4e:
	lea	0(a1,d5.l),a3
	move.l	d5,d1
	sub.l	d6,d1
	addx.w	d7,d2
	move.b	(a3)+,(a1)+
l33f5a:
	move.b	(a3)+,(a1)+
	dbf	d2,l33f5a
	bra.s	l33ee8

l33f62:
	bsr.s	set_copy

	move.l	a0,d0
	bsr.s	bit_test
	move.l	d0,a0
loopix2:
	move.b	-(a4),-(a0)
	cmp.l	a3,a4
	bne.s	loopix2

no_upx:
	movem.l	(sp)+,d0-d7/a0-a6
	rts

set_copy:
	lea	store(pc),a3
	lea	dec_space(a3),a4
	rts

bit_test:
	btst	#0,d0
	beq.s	equal
	addq.l	#1,d0
equal:
	rts

store:
	ds.b	dec_space

User avatar
ggn
Atari God
Atari God
Posts: 1258
Joined: Sat Dec 28, 2002 4:49 pm

Re: LZ4 decoder (assembler)

Post by ggn »

Thanks for the source but I'm afraid I'm going for maximum compression here due to space constraints. So nrv2b simply won't cut it :).
is 73 Falcon patched atari games enough ? ^^

User avatar
swapd0
Obsessive compulsive Atari behavior
Obsessive compulsive Atari behavior
Posts: 111
Joined: Thu Dec 13, 2007 8:56 pm

Re: LZ4 decoder (assembler)

Post by swapd0 »

I thought that with nrv2b you get the best compression ratio, at least I've better ration than lzo, lz77 & lz78. But with small files I need to use other algorithm.

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

Here is my nrv2b unpacking routine (which works with files packed as data using my upx patch):

Code: Select all

	;
	; UPX unpacking macros
	;

ADDBITS	macro
	add.b 	d0,d0   	; sets Z, C and X
	endm

	; Get 1 byte, then get 1 bit into both C and X.
FILLBITS	macro
	; Note: we shift the X flag through,
	; thus must init d0.b with $80.
	move.b	(a0)+,d0
	addx.b	d0,d0   	; sets C and X
	endm

GETBIT	macro
	ADDBITS
	bne.s	\@
	FILLBITS
\@
	endm

	;
	; dec_nrv2b()
	;
	; UPX NRV2B unpacking routine by Markus Oberhumer,
	; Laszlo Molnar & John Reiser.
	;
	; In:
	;   a0: start address of packed data
	;   a1: destination address of unpacked data
	;   (dest. and src. areas shall not overlap)
	;
	; Out:
	;   d0: unpacked data size
	;
dec_nrv2b:
	movem.l	d0-a6,-(a7)

	; Check if file is packed with UPX.
	cmp.l	#"UPX!",(a0)
	; File is not packed with UPX, return.
	bne	decompr_end

	; Store unpacked file size into saved d0.
	move.l	8(a0),(a7)
	; Skip UPX header.
	lea	32(a0),a0

	moveq.l	#-1,d5
	moveq.l	#-128,d0	; d0.b = $80
	moveq.l	#-1,d7
	moveq.l	#-$68,d6	; 0xffffff98
	lsl.w	#5,d6   	; 0xfffff300 = -0xd00
	bra.s	decompr_loop

decompr_literal:
	move.b	(a0)+,(a1)+
decompr_loop:
	GETBIT
	bcs.s	decompr_literal
decompr_match:
	moveq.l	#1,d1
	moveq.l	#0,d3
decompr_l1:
	GETBIT
	addx.l	d1,d1
	GETBIT
	bcc.s	decompr_l1

	subq.l	#3,d1
	bcs.s	decompr_get_mlen ; last m_off
	lsl.l	#8,d1
	move.b	(a0)+,d1
	not.l	d1
	beq.s	decompr_end
	move.l	d1,d5

decompr_get_mlen:
	GETBIT
	addx.w	d3,d3
	GETBIT
	addx.w	d3,d3
	tst.w	d3      	; addx doesn't set the Z flag
	bne.s	decompr_got_mlen
	addq.w	#1,d3

decompr_l2:
	GETBIT
	addx.w	d3,d3
	GETBIT
	bcc.s	decompr_l2

	addq.w	#2,d3

decompr_got_mlen:
	lea	0(a1,d5.l),a2

	; Must use sub as cmp doesn't affect the X flag.
	move.l	d5,d1
	sub.l	d6,d1
	addx.w	d7,d3

	move.b	(a2)+,(a1)+
.copy:	move.b	(a2)+,(a1)+
	dbra	d3,.copy

	bra.s	decompr_loop
decompr_end:
	movem.l	(a7)+,d0-a6
	rts
I'm using it for my file version of Son Shu Si.

User avatar
Mr Nours
Captain Atari
Captain Atari
Posts: 239
Joined: Mon Jun 17, 2002 11:10 am
Location: Montpellier, France
Contact:

Re: LZ4 decoder (assembler)

Post by Mr Nours »

Thanks Orion, i just tested patched upx an depack routine within or mjj demo system. Works fine!
______
Fuzion, the best french Atari CD crew ->The Fuzion Shrine!
ST emulation and more ->Emulation Atari ST(fr)!

orionfuzion
Atariator
Atariator
Posts: 28
Joined: Fri Nov 11, 2016 1:57 pm
Location: Paris, France
Contact:

Re: LZ4 decoder (assembler)

Post by orionfuzion »

Mr Nours wrote:Thanks Orion, i just tested patched upx an depack routine within or mjj demo system. Works fine!
:cheers:

-- Orion

User avatar
Cyprian
10 GOTO 10
10 GOTO 10
Posts: 1921
Joined: Fri Oct 04, 2002 11:23 am
Location: Warsaw, Poland

Re: LZ4 decoder (assembler)

Post by Cyprian »

swapd0 wrote:I'm using the following command to pack the data files.
upx -q --nrv2b --small --data infile -ooutfile

a0 packed data
a1 destination addresss
which UPX do you use?
I've tried UPX 3.96w on Windows 10 and it shows me:

Code: Select all

>upx -q --nrv2b --small --data test.bin -otest.upx
upx: unrecognized option '--data'
Usage: upx [-123456789dlthVL] [-qvfk] [-o file] file..
Portfolio / Lynx II / Jaguar / TT030 / Mega STe / 800 XL / 1040 STe / Falcon030 / 65 XE / 520 STm / SM124 / SC1435
SDrive / PAK68/3 / Lynx Multi Card / LDW Super 2000 / XCA12 / SkunkBoard / CosmosEx / SatanDisk / UltraSatan / USB Floppy Drive Emulator / Eiffel / SIO2PC / Crazy Dots / PAM Net
Hatari / Steem SSE / Aranym / Saint
http://260ste.appspot.com/

Post Reply

Return to “680x0”