Compiling QuickBMS on OS X

austin · Post by **austin** » Sun Oct 02, 2016 9:05 pm

Using gcc on OS X to compile quickbms_src

it is spitting out a lot of warnings and errors at the moment. I have suppressed some by modifying the Makefile:

Code: Select all

EXE      = quickbms
# OS X aliases gcc/g++ to clang. force make to gnu compilers here (from Homebrew installation)
CC      = /usr/local/Cellar/gcc/6.2.0/bin/gcc-6
CXX      = /usr/local/Cellar/gcc/6.2.0/bin/g++-6
# -m32 because QuickBMS has been tested only on 32bit systems and gives problems using 64bit native code
# add -I flag for <malloc.h> .. on OS X its in malloc/ folder.
CFLAGS   += -m32 -s -O0 -fstack-protector-all -fno-unit-at-a-time -fno-omit-frame-pointer -I/usr/include/malloc/
CDEFS   += -DDISABLE_UCL -DDISABLE_MCRYPT -DDISABLE_TOMCRYPT
CLIBS   += -lstdc++ -ldl -lcrypto -lssl -lz -lbz2 -lstdc++ -lm -lpthread -llzo2
PREFIX   = /usr/local
BINDIR   = $(PREFIX)/bin
SRC      = $(EXE).c

all:
   $(CC) $(SRC) $(CFLAGS) -o $(EXE) \
   \
   -D_7ZIP_ST \
   \
   compression/*.c* encryption/*.c* extra/mem2mem.c libs/lzma/LzmaDec.c libs/lzma/Lzma2Dec.c libs/lzma/Bra86.c libs/lzma/LzFind.c libs/lzma/LzmaEnc.c libs/lzma/Lzma2Enc.c libs/mspack/*.c libs/ppmd/*.cpp libs/aplib/depacks.c libs/brieflz/depacks.c libs/brieflz/brieflz.c compression/jcalg1_static.lib libs/zziplib/*.c libs/bcl/*.c libs/szip/*.c libs/lzhl/*.cpp libs/tdcb/*.c libs/libkirk/*.c libs/7z_advancecomp/*.cc libs/iris/*.cpp libs/old_cabextract/lzx.c libs/mrci/*.cpp libs/lz4/*.c libs/snappy/snappy.cc libs/snappy/snappy-c.cc libs/snappy/snappy-stubs-internal.cc libs/snappy/snappy-sinksource.cc libs/mmini/mmini_huffman.c libs/mmini/mmini_lzl.c libs/clzw/lzw-dec.c libs/clzw/lzw-enc.c libs/lzlib/lzlib.c libs/blosc/blosclz.c libs/gipfeli/*.cc libs/liblzg/src/lib/decode.c libs/liblzg/src/lib/encode.c libs/liblzg/src/lib/checksum.c libs/doboz/*.cpp libs/amiga/* libs/sphlib/c/*.c libs/shadowforce/*.cpp libs/zstd/common/*.c libs/zstd/compress/*.c libs/zstd/decompress/*.c libs/zstd/dictBuilder/*.c libs/zstd/legacy/*.c -Ilibs/zstd -Ilibs/zstd/common -Ilibs/zstd/legacy libs/azo/unAZO.cpp libs/azo/Decoder/MainCodeD.cpp libs/azo/Common/x86Filter.cpp libs/nintendo_ds/*.c libs/ctw/*.c libs/grzip/libgrzip.c libs/heatshrink/heatshrink_decoder.c libs/heatshrink/heatshrink_encoder.c libs/libzling/*.cpp  -Ilibs/ecrypt/include -Ilibs/libcsc -D_7Z_TYPES_ libs/libcsc/csc_dec.cpp libs/libcsc/csc_filters.cpp libs/libcsc/csc_memio.cpp -DDENSITY_FORCE_INLINE=inline -Drestrict=__restrict__ libs/density/src/*.c libs/density/src/spookyhash/src/*.c libs/brotli/dec/*.c libs/brotli/common/*.c libs/libbsc/adler32/adler32.cpp libs/libbsc/bwt/bwt.cpp libs/libbsc/coder/coder.cpp libs/libbsc/coder/qlfc/qlfc.cpp libs/libbsc/coder/qlfc/qlfc_model.cpp libs/libbsc/filters/detectors.cpp libs/libbsc/filters/preprocessing.cpp libs/libbsc/libbsc/libbsc.cpp libs/libbsc/lzp/lzp.cpp libs/libbsc/platform/platform.cpp libs/libbsc/st/st.cpp libs/shoco/shoco.c libs/ms-compress/src/*.cpp libs/lzjody/lzjody.c libs/lzjody/byteplane_xfrm.c disasm/disasm.c disasm/cmdlist.c disasm/assembl/assembl.c libs/mydownlib/mydownlib.c libs/TurboRLE/trlec.c libs/TurboRLE/trled.c libs/TurboRLE/ext/mrle.c libs/lhasa/lib/*_decoder.c libs/lhasa/lib/crc16.c libs/dipperstein/*.c libs/liblzf/lzf_d.c libs/liblzf/lzf_c_best.c libs/zopfli/*.c libs/lzham_codec/lzhamcomp/*.cpp libs/lzham_codec/lzhamdecomp/*.cpp libs/lzham_codec/lzhamlib/*.cpp -Ilibs/lzham_codec/include -Ilibs/lzham_codec/lzhamcomp -Ilibs/lzham_codec/lzhamdecomp -DLZHAM_ANSI_CPLUSPLUS libs/dmsdos/*.c libs/tornado/Tornado.cpp libs/tornado/Common.cpp libs/PKLib/*.c extra/mybits.c libs/lz5/lz5*.c libs/ppmz2/*.c* libs/libdivsufsort/*.c libs/xxhash/*.c extra/xalloc.c libs/lzfse/src/*.c  libs/hsel/myhsel.cpp libs/hsel/HSEL.cpp \
   \
   $(CLIBS) $(CDEFS)

install:
   install -m 755 -d $(BINDIR)
   install -m 755 $(EXE) $(BINDIR)/$(EXE)

.PHONY:
   install

using OS X El Capitan Version 10.11.6

any help would be appreciated

aluigi · Post by **aluigi** » Mon Oct 03, 2016 10:53 am

The first error is the missing LZO library: lzo/lzo1.h
You need it, or you can add -DDISABLE_LZO to skip it.

The second is a bug in libzpaq that apparently requires -Dunix or it thinks you are windows.

The 99,9% of the other errors are related to assembly and in-line assembly.

The last errors are related to lzham and the usage of "_is_pod".

What I can do on my side is:

for lzo nothing because it's a requirement and it's used in many games, so install it or use -DDISABLE_LZO
I can report the libzpaq problem to the author or add a commented variable in the Makefile containing both -Dunix and -I/usr/include/malloc/
skip yamb256/yamb.c if it's __APPLE__
skip all the (useless) amiga compressions if it's __APPLE__
lzham already checks __APPLE__ so I don't know why you get __is_pod instead of "std::__is_pod<T>::__value", it's a third-party library so the solution is probably just skipping it

austin · Post by **austin** » Mon Oct 03, 2016 9:51 pm

aluigi wrote:The first error is the missing LZO library: lzo/lzo1.h
You need it, or you can add -DDISABLE_LZO to skip it.

The second is a bug in libzpaq that apparently requires -Dunix or it thinks you are windows.

The 99,9% of the other errors are related to assembly and in-line assembly.

The last errors are related to lzham and the usage of "_is_pod".

What I can do on my side is:
for lzo nothing because it's a requirement and it's used in many games, so install it or use -DDISABLE_LZO
I can report the libzpaq problem to the author or add a commented variable in the Makefile containing both -Dunix and -I/usr/include/malloc/
skip yamb256/yamb.c if it's __APPLE__
skip all the (useless) amiga compressions if it's __APPLE__
lzham already checks __APPLE__ so I don't know why you get __is_pod instead of "std::__is_pod<T>::__value", it's a third-party library so the solution is probably just skipping it

Thanks for the fast response. I installed LZO, defined unix like you said. To skip yamb algorithm and amiga compressions - what's the easiest way to do that? As for the lzham issue, it looks like it is using "std::__is_pod<T>::__value" but there is an issue with it still:

Code: Select all

libs/lzham_codec/lzhamdecomp/lzham_traits.h:71:33: error: expected ')' before '__is_pod'
    #define LZHAM_IS_POD(T) std::__is_pod<T>::__value

EDIT: It actually wants "__is_pod", and not "std::__is_pod<T>::__value", so scratch the lzham issue.

aluigi · Post by **aluigi** » Tue Oct 04, 2016 4:50 am

Very good, I guess that __is_pod thing can be solved with a -D_is_pod=__is_pod without source code modification

Regarding amiga it's enough to remove "libs/amiga/*" from the Makefile and remove all the "QUICK_COMP_CASE(AMIGA_" lines in perform.c (they are included in a #if defined).

For yamb.c it's probably more easy to just delete all the code from line 440 till 611.

To recap, the additional line in the Makefile necessary to compile on OSX is:

Code: Select all

-I/usr/include/malloc/ -Dunix -D_is_pod=__is_pod

austin · Post by **austin** » Tue Oct 04, 2016 2:51 pm

Ok, I removed amiga and yamb using the methods you gave. That cleared up the log to look at the sneaky errors :_)

#1
on __APPLE__, sys/ptrace.h uses different signal names for ptrace(...
I added this to io/process.c on line 13 to fix it

Code: Select all

    #ifdef __APPLE__
        #define PTRACE_TRACEME      PT_TRACE_ME
        #define PTRACE_PEEKTEXT     PT_READ_I
        #define PTRACE_PEEKDATA     PT_READ_D
        #define PTRACE_PEEKUSER     PT_READ_U
        #define PTRACE_POKETEXT     PT_WRITE_I
        #define PTRACE_POKEDATA     PT_WRITE_D
        #define PTRACE_POKEUSER     PT_WRITE_U
        #define PTRACE_CONT         PT_CONTINUE
        #define PTRACE_KILL         PT_KILL
        #define PTRACE_SINGLESTEP   PT_STEP
        #define PTRACE_GETREGS      PT_GETREGS
        #define PTRACE_SETREGS      PT_SETREGS
        #define PTRACE_GETFPREGS    PT_GETFPREGS
        #define PTRACE_SETFPREGS    PT_SETFPREGS
        #define PTRACE_ATTACH       PT_ATTACH
        #define PTRACE_DETACH       PT_DETACH
        #define PTRACE_GETFPXREGS   PT_GETFPXREGS
        #define PTRACE_SETFPXREGS   PT_SETFPXREGS
        #define PTRACE_SYSCALL      PT_SYSCALL
        #define PTRACE_SETOPTIONS   PT_SETOPTIONS
        #define PTRACE_GETEVENTMSG  PT_GETEVENTMS
        #define PTRACE_GETSIGINFO   PT_GETSIGINFO
        #define PTRACE_SETSIGINFO   PT_SETSIGINFO
    #endif

#2
In libs/lzham_codec/lzhamdecomp/lzham_platform.cpp line 65:
"__asm {int 3};" was giving syntax errors
the correct function call to invoke debug/trap signal on OS X is "__builtin_trap();"
so it becomes:

Code: Select all

void lzham_debug_break(void)
{
#if LZHAM_USE_WIN32_API
   DebugBreak();
#elif (TARGET_OS_MAC == 1) && (TARGET_IPHONE_SIMULATOR == 0) && (TARGET_OS_IPHONE == 0)
   __builtin_trap();
#else
   assert(0);
#endif
}

The last error I'm getting is coming from libiconv headers themselves and not quickbms. So that's confusing. see:

Code: Select all

/usr/local/opt/libiconv/include/iconv.h:139:55: error: expected ';', ',' or ')' before 'uc'
 typedef void (*iconv_unicode_char_hook) (unsigned int uc, void* data);
                                                       ^~
/usr/local/opt/libiconv/include/iconv.h:144:3: error: unknown type name 'iconv_unicode_char_hook'
   iconv_unicode_char_hook uc_hook;
   ^~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:155:61: error: expected ';', ',' or ')' before '*' token
               void (*write_replacement) (const unsigned int *buf, size_t buflen,
                                                             ^
/usr/local/opt/libiconv/include/iconv.h:157:15: error: expected ';', ',' or ')' before 'void'
               void* callback_arg,
               ^~~~
/usr/local/opt/libiconv/include/iconv.h:164:28: error: expected ';', ',' or ')' before 'code'
              (unsigned int code,
                            ^~~~
/usr/local/opt/libiconv/include/iconv.h:198:3: error: unknown type name 'iconv_unicode_mb_to_uc_fallback'
   iconv_unicode_mb_to_uc_fallback mb_to_uc_fallback;
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:199:3: error: unknown type name 'iconv_unicode_uc_to_mb_fallback'
   iconv_unicode_uc_to_mb_fallback uc_to_mb_fallback;
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:216:52: error: expected ';', ',' or ')' before 'namescount'
 extern void iconvlist (int (*do_one) (unsigned int namescount,
                                                    ^~~~~~~~~~
/usr/local/opt/libiconv/include/iconv.h:219:24: error: expected ';', ',' or ')' before 'void'
                        void* data);

EDIT: The iconv syntax error goes away when I use clang which makes sense. But then quickbms source is full of syntax errors because it prefers gcc (clang doesn't support nested functions so a lot breaks)

Can I also remove calls to iconv without quickbms breaking/losing major functionality ?

aluigi · Post by **aluigi** » Tue Oct 04, 2016 7:24 pm

#1
Well done

I will use your fix for sure.

#2
Try to add -D__builtin_trap() to the Makefile

#libiconv
Can you try to comment the "#define USE_LIBICONV" in quickbms.c?

austin · Post by **austin** » Tue Oct 04, 2016 9:52 pm

libiconv solution worked

Now come linker errors..

First, because we removed amiga compression, I had to comment out these lines (unz.c:5590)

Code: Select all

extern int UnSquash(unsigned char *, int);// __asm__("UnSquash");
int amiga_unsquash(unsigned char *in, int insz, unsigned char *out) {
    int     i;

    // copy to out setting the correct endianess
    for(i = 0; i <= (insz - 4); i += 4) {
        *(u32 *)(out + i) = ntohl(*(u32 *)(in + i));
    }

    return UnSquash(out, insz);
}

Next, because you're forcing 32bit compilation, and my local lzo and openssl libs are compiled as 64 bit, they get ignored by ld.
I downloaded the latest source of openssl and lzo, and built the libs as 32 bit, then changed quickbms Makefile to use them.

And now I have quickbms compiled and running on OS X

. Thanks again for your help

austin · Post by **austin** » Wed Oct 05, 2016 3:14 am

My goal was to compile quickbms to run the compression type scanner. It worked - data is lzss compressed

Here's comtype_scan2.bat re-written as a shell script:

Code: Select all

help() {
   echo ""
    echo "QuickBMS comtype scanner 2 (version 0.1.1b)"
    echo "by Luigi Auriemma"
    echo "e-mail: aluigi@autistici.org"
    echo "web:    aluigi.org"
    echo ""
    echo "you must specify:"
    echo "- the path of comtype2_scan.bms"
    echo "- the input file compressed with the unknown algorithm"
    echo "- the output folder where placing the unpacked files"
    echo ""
    echo "example:"
    echo "  comtype_scan2 c:\comtype_scan2.bms c:\dump.dat c:\output_folder [max_size]"
    echo ""
    echo "if an algorithm doesn't return immediately press CTRL-C"
    echo "and answer 'n' [no] when will be asked to \"terminate batch job\""
    echo ""
}

if [ $# -ne 3 ]; then
    help
    exit 1
fi

for i in {1..700}; do
   quickbms -a $i $4 $1 $2 $3
done

aluigi · Post by **aluigi** » Wed Oct 05, 2016 8:57 am

Excellent

aluigi · Post by **aluigi** » Wed Oct 05, 2016 10:41 am

Ah, I successfully compiled on OSX too

There will be a commented step-by-step in the next Makefile for OSX to make the life easier.

ZenHAX

Compiling QuickBMS on OS X

Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X

Re: Compiling QuickBMS on OS X