Possible next features of QuickBMS

[aluigi]

Great. I have just relased QuickBMS 0.6.7 that contains all the fixes of the beta plus some other things.

[einstein95]

Okay, so I'm using this which should work, but the name just comes out with EEEE instead of the Japanese text.

Code: Select all

codepage 932
getct NAME string 0x2C
log MEMORY_FILE 0 0
put NAME unicode MEMORY_FILE
goto 0 MEMORY_FILE
get NAME unicode MEMORY_FILE

I'm an idiot, I never realised that if the string isn't in utf-16, the unicode type doesn't work. WHat I was wanting to do is to decode a string that is in Shift JIS and decode it properly, an example in python is like name.decode('shift-jis').encode('utf-8'). I take it there's currently no way to do this or am I just confusing myself?

[aluigi]

So you have just to add "codepage utf8" before the last "get NAME ..." to force the new conversion.
It should work but I'm not 100% sure.

[einstein95]

Okay, I've figured out what's happening, but not how to fix it.
I have strings that contain text like テ stored as 83 65 in bytes. Converting this to unicode in a memory file is converting it as 00 83 00 65, then it's decoded as 00 83 00 65 using a codepage.
What I'd like to request is a more direct way of conversion, where it'd take the existing bytes, in this example 83 65, and decode them using the specified codepage.

[aluigi]

I understand, but I don't think I can help much on this topic because it's quite confusing.
Maybe try to give a real exact and full example of JIS string and the utf8 one you expect.

[einstein95]

Okay, this is what I've got:
In Shift JIS: 45 47 47 8B 96 91 F8 2E 54 58 54
I expect to get EGG許諾.TXT from it (in UTF-8: 45 47 47 E8 A8 B1 E8 AB BE 2E 54 58 54).

[aluigi]

I'm checking the utf8_to_utf16_chr function used by quickbms for "put VAR unicode" and the codepage is correct (932) but the output of the wide chars is 0x30fb.
That's the problem.
Do you have any programming suggestion for the conversion jis->utf16 + utf16->utf8?

I was testing the iconv tool and it does a great job in doing the conversion: iconv -f SHIFT-JIS -t UTF8 -c jis.dat > text.txt

[aluigi]

I'm still investigating why it doesn't work because a simple MultiByteToWideChar 932 does the job.
I will keep you update.

[aluigi]

I guess I have to rewrite all the unicode functions from scratch mainly to adopt a "standard" style.
Regarding the codepage conversions I was thinking about something like "Set VAR utf8 VAR" limiting the output codepage only to utf8 with the source one decided by the Codepage command.
It may be an idea.

[einstein95]

I guess I have to rewrite all the unicode functions from scratch mainly to adopt a "standard" style.
Regarding the codepage conversions I was thinking about something like "Set VAR utf8 VAR" limiting the output codepage only to utf8 with the source one decided by the Codepage command.
It may be an idea.

That's a good idea and what I was thinking
The only issue that may be a problem is for people reimporting stuff that uses non-standard characters, but that's probably not the case.

[Ekey]

I would like to see the implementation of:

Debug log to file (in a more expanded form :B), while unpacking archives debug log saved in same directory.

Code: Select all

debuglog

btw: Luigi, maybe it's time to make QuickBMS Studio with GUI interface?

Like

[aluigi]

What you mean exactly with debuglog? An example of the output you expect?

Nice idea a QuickBMS Studio but it's too much for me (time/effort), anyway it's possible to make a step-by-step debugger and would be not difficult.
Personally I use ever -V for debugging together with "less" (the unix/cygwin tool for showing the output page-per-page and line-per-line) and works perfectly for my needs.
Try it and I'm sure you will appreciate it.

[Ekey]

In QuickBMS as you know have that command debug to display information. Would be great if this info will also is saved in a log file.

[aluigi]

Oh, I understand.
Can't you use the stderr redirector?
quickbms ... 2> log.txt

[Ekey]

I have own tool for redirect console info, but after ~30sec it stops logging. I do not know what the problem :/

[aluigi]

I have just made a simple script that may be useful with the comtype_scan2 and encryption_scan scripts:
http://aluigi.org/bms/hexdump_scanner.bms

It's enough to launch quickbms specifying the folder used with comtype_scan2 and it will automatically scan the folder displaying the hexadecimal dump of the files ordered for their size, from the biggest to the smallest.
The visualization covers the first 0x100 bytes of the file.

Example:

Code: Select all

comtype_scan2.bat comtype_scan2.bms dump.dat c:\output_folder
quickbms hexdump_scanner.bms c:\output_folder

The idea is to make the visualization process very quick without using a hex editor to open every file.

Hope it helps.

[TGE]

Any chances of offering floating point parsing? I know QuickBMS uses all integers internally, so I was thinking you could maybe make it output an array of 2 ints/int64s, an int of the whole number and of the decimals behind the period/comma. This would make for some easy model to obj conversion. Perhaps even more so than hex2obj!

[aluigi]

Unfortunately that's quite difficult.
Probably you "may" have some luck with the sprintf and scanf operators of the String command.
Just an example that may give you an idea of what I mean and two different solutions:

Code: Select all

set VAR string "123.456"

string VAR s "%d.%d" VAR1 VAR2
print "%VAR1% %VAR2%"
string VAR3 p "%d.%d" VAR1 VAR2
print "%VAR3%"

string VAR s "%f" VAR1
print "%VAR1%"
string VAR3 p "%f" VAR1
print "%VAR3%"

[TGE]

I know that numbers are rounded up so doesn't that mean the fractional part of a number is at some point stored in the memory?

[aluigi]

I guess that the scanf operator ('s') of the previous example (the 3rd output) is just what you refer here.
1123477881 -> 0x42f6e979 which is just the 32bit representation of "123.4560012817383"
You can use my calcc32 tool for performing these tests.