About Compiling QuickBMS In Windows

[Brolijah]

Intro
I finally got this to work under Windows, but because of the amount of time I spent trying to make this work, I am compelled to share my experiences and comments on doing so. I don't want others to waste nearly as much time as I did trying to make this work.

I was initially unsure whether to make this thread under Tutorials or here. I settled for here because I wanted to also make some comments on the build scripts and compatibility. Prior to simply asking in the QBMS Errors thread if there was a Windows specific build script, I could find no mention of there being one on aluigi.altervista.org nor in this forum. Nor could I find any mention of a user having attempted to compile under Windows.

I'll spare most the experience of wandering in the dark trying to build using the makefile prior, but much of the changes I had to make with the makefile were present in that batch script already. For future reference to curious individuals: Don't try to build under Windows with the Makefile. It's hell and you'll waste more time chasing dubious compilation errors than you may want to admit.

For convenience, I'll quote aluigi's build script here:

I use a .bat file for doing the whole compiling job on Windows using gcc 4:
http://aluigi.org/bms/compa.bat

I also made some minor edits to the script for convenience, readability, and a couple comments about compilation:

compile_quickbms.zip

Some of the following might be common sense solutions for some readers, but I'd rather be thorough and complete than not.

Potential Pitfalls
1. Missing includes to library headers shipped with MinGW.

Code: Select all

src\quickbms.c:172:14: fatal error: openssl/crypto.h: No such file or directory
     #include <openssl/crypto.h>

For some reason this occurred with all my MinGW installations. Headers and libs located in the opt directory aren't in the search paths by default (ie OpenSSL). As such, you may want to manually add opt\include and opt\lib paths to the compilation script.

2. Missing includes for mcrypt and tomcrypt.
Add these include paths to the compilation script. (My modified script already has this addition.)

Code: Select all

-Isrc\libs\libmcrypt -Isrc\libs\libtomcrypt

3. Several "skipping incompatible" library errors while linking.
You're trying to use an x86_64 toolchain + libraries. Install the i686 toolchain and try with that.

4. Failure to link xcompress.lib

Code: Select all

ld.exe: xcompress.lib(obj/i386/lci.obj): warning: line number count (0x2) exceeds section size (0x1)
xcompress.lib(obj/i386/lci.obj): could not read symbols: Invalid operation
collect2.exe: error: ld returned 1 exit status

Don't use GCC 7 or newer. The library xcompress.lib will be incompatible with the binutils that are shipped with GNU GCC 7 and 8. If you're hellbent on using latest possible GCC toolchain, then use GCC 6.4.0. It will also work with GCC 5 and 4.

5. Undefined reference to `BCrypt_init`
It's possible that bcrypt.h has been included by another file (possibly wincrypt.h). I've observed this in both GCC 5 and GCC 6. So, unless you're Mr. Aluigi himself or you actually are attempting to build and test on a Win98 machine... It should be satisfactory enough to make these two changes and add -lbcrypt to the compilation script:

Code: Select all

// In file /src/myenc.c : Comment out the mybcrypt include and use MinGW's
//#include "extra/mybcrypt.h"
#include <bcrypt.h>
// [ ... About 110 lines below ... ]
//if(!BCrypt_init()) return -1;  // The comment on this line in-file suggests only needed for Win98

Final Words
Could you put the batch script in an easy-to-find location... Like the page for quickbms on your site.

[aluigi]

Yeah compiling quickbms on Windows is complicated and it's a sort of "card castle" where everything is balanced and works on the building environment that I use for it.

Compiling on Windows is ever painful with all programs (except helloworld.c) so this is not really a surprise, various versions of VS, mingw/gcc/cygwin and so on... really painful.

1) I guess that #include is mandatory anyway. Did you cross-compile or compiled natively?

2) weird and interesting, I will review that

3) yes definitely, DO NOT TRY x86_64

4) this is interesting and also partially easy to bypass since a portion of the code from xcompress is available with source code (used on Linux)

5) this is really weird but the mybcrypt.h wrapper works well, do you get any specific error?
Here my BCrypt_Init is linked and perfectly called at runtime (just checked) and all the API referenced.
I don't have a sample ready to test and share unfortunately.

Ah, in case someone is curious... yes there are some users of quickbms who run it on Win98

[Brolijah]

(The list was just to answer any potential questions for people who come across these things as I did. Didn't mean them to sound like I was asking those questions here.)
Note 1 is just... weird. You'd think headers and libraries installed with MinGW would all be in GCC's default search path. Yet, /opt/include and /opt/lib never was for any of my installations. MinGW puts openssl in there and not the main include path. I compiled natively using MinGW-w64 GCC toolchains.
Note 4... I couldn't find much on xcompress. Didn't realize some of it was open-source.
Note 5. GCC complains about two things: (1) warning implicit declaration of BCrypt_init in myenc.c and (2) undefined reference to BCrypt_init at link time. I did think it very strange that GCC thought the call to BCrypt_init was an implicit declaration although it had already been defined with a body in the header... If you feel willing to look at the disassembly, I can attach the object file for quickbms.o. The actual function for BCrypt_init is just gone. (The easy check should've been find where "bcrypt.dll" is referenced before LoadLibrary, but the string is literally nowhere.)

[Brolijah]

Attached below is the object file I compiled for quickbms.c using the normal code in myenc.c.
BCrypt_init should have been compiled into this object by inclusion. "quickbms.c" includes "myenc.c", and "myenc.c" includes "mybcrypt.h". Except by my search, it's nowhere. In fact, the function call is undefined according to IDA.

[aluigi]

Maybe your gcc already has __BCRYPT_H__ defined since it has been already loaded and mybcrypt.h is ignored.

Anyway with a vanilla Mingw using gcc 4 x86 on Windows (native) there are no issues and it compiles on the fly without spending hours with errors.
I remember the old t-dragon toolchain (but probably older than 2015) on which it worked perfectly too, ever on the fly using compa.bat.

[Brolijah]

Maybe your gcc already has __BCRYPT_H__ defined since it has been already loaded and mybcrypt.h is ignored.

Oh god I can't believe I didn't try to test that scenario. I tried a stupid little test and I see now bcrypt is already included by another header.

Code: Select all

In file included from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/wincrypt.h:764:0,
                 from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/windows.h:95,
                 from C:/mingw-w64/i686-6.4.0-win32-dwarf-rt_v5-rev0/mingw32/i686-w64-mingw32/include/winsock.h:10,
                 from C:\mingw-w64\i686-6.4.0-win32-dwarf-rt_v5-rev0\mingw32\opt\include/openssl/dtls1.h:71,
                 from C:\mingw-w64\i686-6.4.0-win32-dwarf-rt_v5-rev0\mingw32\opt\include/openssl/ssl.h:1699,
                 from src\quickbms.c:173:

That answers that, I guess. I'll update the OP to reflect this. I believe someone, someday, will stumble across this thread, and I hope it'll be of some use to them.
As a side memo, the whole reason why I set out to compile quickbms myself was because I and a couple others on Discord wanted the fixed ALLZ code built in with quickbms. Thank you again for your help and this multi-useful tool.

[aluigi]

Wow, compiling the whole quickbms just for ALLZ?
Wasn't enough your stand-alone tool for doing that job?

[Brolijah]

Two reasons: It's just more convenient. Do you remember the Jump Stars ALAR archive type? That's also one of those Aqualead formats. The guys who were researching that game got stumped when they got to ALLZ from what I read on some 15 year old forum posts. Since games using ALAR are also likely using ALLZ or vice-versa, we could just use one tool for both.
The second reason is the Zanki Zero people's files are not packed in ALAR files... rather they're just ALLZ files stacked back to back. No header to them or anything, which is weird. So they need to split up those files then decompress. A time consuming process that could probably be scripted with just one tool.

[aluigi]

Very interesting.

I hope to make a simple step-by-step for an error-free compiling once I have some free time.

After all one day I had to open this Pandora's box
Being the only author and maintainer of QuickBMS everything will stop once I will stop working on it, so it's necessary to have a plan B ready in place.

My only request is to avoid forks, the situation is already a mess with the older versions of the tool, therefore forks are definitely my nightmare.
I'm ready and available to implement ANY feature and crazy idea that has a sense and works (and doesn't take time to be implemented), I'm very open on that side.

[aluigi]

As promised, this is the simple step-by-step for compiling quickbms on Windows.
There are NO requirements and it's meant for everybody.

1)
Download gcc 4.9.4
https://sourceforge.net/projects/mingw- ... z/download

2)
unzip gcc on your disk, maybe in c: (it will be c:\mingw32)

3)
Download the current quickbms dependencies
http://aluigi.org/bms/quickbms_dependencies.zip

4)
unzip the include folder in c:\mingw32\i686-w64-mingw32

5)
unzip the lib folder in c:\mingw32

6)
download the quickbms source code and extract the src folder in c:\quickbms (it will become c:\quickbms\src)
http://aluigi.org/papers/quickbms_src.zip

7)
download compa.bat and put it in c:\quickbms
http://aluigi.org/bms/compa.bat

8)
open compa.bat with a text editor and:
- remove line 16: set TEMP=z:\temp
- remove line 17: set TMP=z:\temp
- replace all the occurrencies of c:\mingw with c:\mingw32

9)
run the console (cmd.exe)

9)
go in the quickbms directory and set the path of the Mingw compiler:

Code: Select all

set PATH=c:\mingw32\bin

10)

Code: Select all

compa.bat

11)
for compiling quickbms_4gb_files.exe

Code: Select all

compa.bat -DQUICKBMS64

Notes:

* quickbms 0.10.0 has a compiling bug in src\utils.c at lines 440 and 442, you can just delete them because they are unused.

* there is also the BCrypt problem reported by Brolijah, it's enough to remove line 1431 in src\myenc.c and add -lbcrypt after compa.bat:

Code: Select all

compa.bat -DQUICKBMS64 -lbcrypt

* if you need to compile it fast for your tests remove "-s -O2" from compa.bat

* gcc 8.1.0 works perfectly too:
https://sourceforge.net/projects/mingw- ... z/download

* this step-by-step will be fixed after I will release quickbms 0.10.1

[aluigi]

I have manually patched xcompress.lib for setting the line number field of lci.obj and ldi.obj to zero
The patched version is already included in quickbms_dependencies.zip
For the tech guys:

Code: Select all

           original_xcompress.lib (MD5 8f4165d4d9a173c06bc00dc5a3ab3aab)
  Offset   |        patched_xcompress.lib
-----------------------------
000A40D0   02       00
0010B4AC   02       00

[spiritovod]

Could you please check compilation instruction? Current one doesn't seem to work with 0.10.1 sources, probably due to outdated compa.bat. I've tried to compile it "as is" with gcc 8.1.0, following your instruction (including BCrypt fix), but aside from expected warnings resulting exe doesn't appear (only some strange small quickbms_xxxxx.exe in temp folder).

[aluigi]

Do you remember what's the exact error reported during compiling?

You can just dump the whole output with "compa 2> dump.txt" and then grepping the error in the file.

[aluigi]

Just an update.
The current beta of quickbms 0.11 is now fully compatible with gcc 10 and clang (on Android too).

There are some notes about gcc 10:

1) it's necessary to add -lbcrypt to the compilation command (for example compa.bat -lbcrypt)

2) there is a problem of multiple definition between libtomcrypt and openssl: sha512_224_init, sha512_256_init, poly1305_init

3) the executables generated by gcc 10 are twice the size of those made by gcc 4

Currently point 2 is a problem and I don't know what may be the best solution.
swprintf has been completely removed from utils.c because it caused a bug in gcc 10 (swprintf is a portability mess), for example when used to build the unicode mode for fopen it only took the first character, the result was a file open in text mode (r instead of rb) with tons of problems.

This is the link to the full source code + exe:
https://aluigi.org/beta/quickbms_beta.zip

compa.bat has been updated too:
https://aluigi.org/bms/compa.bat

gcc 4 is still my preferred choice for quickbms btw but I'm happy to have fixed many things spotted by testing other compilers and versions.