How to recognize the compression algorithms with your eyes

[Vido]

Firmwares for SOHO routers are using LZMA to pack firmwares, there is tool called binwalk, its written in python and if you using python 2 and have python-lzma installed binwalik will validate found headers an false-positives will be excluded from search, something simmilar could be done for quickbms I think, its idea

[DavidDineen]

Good. Anyway I have added view zstd to the list in the first post since it uses a phenq results 1 week magic number that allow to guess it at 100%.

Was looking for zstd compression recognising technique and was about to ask in a thread. But here I found 100% success technique. Thanks for mentioning the magic number, this solves my problem I was facing in my current project.

[aluigi]

In case someone is interested in some statistics, I have collected the most used compression algorithms used in my 2'023 quickbms scripts.
Some games uses zip-like archives and that means some numbers may be inflated, some scripts cover multiple formats and some algorithms may be invoked twice in few scripts just because they are for more than one format.

Code: Select all

zlib    545
deflate 119
lzma    87
xmemdecompress (lzx) 61
lzss    56
lzo1x   49
gzip    49
lz4     44
lz77wii 21
oodle   21
bzip2   16
dk2/EA  15
zstd    10
custom  10

[GHFear]

Thanks! This has been very helpful lately
Just want to add how I spot RefPack/dk2 nowadays.
First off...
0xXX = random value.
RefPack/dk2 compression mostly looks like this "0xXX\0xFB" (where the 0xFB seems to be the tell tale sign of refpack.) and then right after is the Decompressed Size as a 4byte 32 bit integer / long. (to further confirm that it in fact is refpack/dk2.)
So all in all it looks like this "0xXX\0xFB\0x00\0x1B\0x62\0x1C"

Depending on the strength of the compression the strings can still be sorta readable like this: "c:\datatemp\inter\PS2\Neutral\Cha
pâs\Hogwarts\ZoneãONE_HW_Viaduct_E
.nãnce_DD\LefTextur�'àET_BURNâOOK_PAPER.ss€"

If you see this pattern, use comtype dk2

[baloons34]

Thanks! This has been very helpful lately
Just want to add how I spot RefPack/dk2 nowadays.
First off...
0xXX = random value.
RefPack/dk2 compression mostly looks like this "0xXX\0xFB" (where the 0xFB seems to be the tell tale sign of refpack.) and then avis sur phenq france right after is the Decompressed Size as a 4byte 32 bit integer / long. (to further confirm that it in fact is refpack/dk2.)
So all in all it looks like this "0xXX\0xFB\0x00\0x1B\0x62\0x1C"

Depending on the strength of the compression the strings can still be sorta readable like this: "c:\datatemp\inter\PS2\Neutral\Cha
pâs\Hogwarts\ZoneãONE_HW_Viaduct_E
.nãnce_DD\LefTextur�'àET_BURNâOOK_PAPER.ss€"

If you see this pattern, use comtype dk2

Thank you so much, all this info is so useful. I have spent hours on Google trying to get things straight for my new project and then found this thread which is just full of information. The zstd code is exactly what I was not getting right. You saved my time!

[GHFear]

Thank you so much, all this info is so useful. I have spent hours on Google trying to get things straight for my new project and then found this thread which is just full of information. The zstd code is exactly what I was not getting right. You saved my time!

I think you meant to thank aluigi

This Guy!

[aluigi]

[baloons34]

Thank you so much, all this info is so useful. I have spent hours on Google trying to get things straight for my new project and then found this thread which is just full of information. The zstd code is exactly what I was not getting right. You saved my time!

I think you meant to thank aluigi

This Guy!

Hahahaha yes you are right, I must thank alugi

[baloons34]

:D

Thanks a lot Alugi