Can AlphaTab dynamically detect GP file encoding while importing?

[lannka]

Hi there, I plan to use AlphaTab to render .gp* files uploaded by users. Soon I noticed that many old files are not in Unicode, of which titles and comments are rendered in gibberish. I was able to get some of the files correct by manually numerating possible encodings (such as "gbk", "big5") in the importer.encoding setting.

So a work around to the issue can be to ask users to select encoding before upload. However, I don't think today on the internet an average user would understand what "encoding" is. (It seems to be a pretty old thing that was already fixed by UTF-8?).

So I'm looking for a more automatic solution. My thoughts:

• is the encoding a piece of information come along with the .gp* file that AlphaTab can read and use? I played the official GuitarPro software on Mac, unfortunately found the same problem. So I'm afraid that was something missing in the old gp file format.
• maybe encoding can be detected using some heuristics?

Thoughts?

[Danielku15]

Hi.

First a bit background:

Guitar Pro 3-5 files had a binary format which did not change much over time beside the fact that it was extended with new content. Some official documentation about version 4 was once published: http://dguitar.sourceforge.net/GP4format.html (I guess it was contributed by Arobas to the public at some point.) It also documents the data type string but does not mention anything special about the encoding beside the hint that every character is 1 byte.

Guitar Pro in it's old days used the default system encoding for non-unicode programs. This setting is part of the windows regional settings:

This leads us to this issue that if you share GP5 files across machines with different settings, the information about the right encoding is lost.

Regarding the handling in alphaTab

As you have found correctly: alphaTab allows you to specify the encoding for string decoding for exactly this reason. But detecting the "right" encoding is not easy only only possible maybe via some heuristics which come at significant cost on decoding. We could take the first proper string read, and try to guess the encoding. Defining the right heuristics is hard:

• If you encounter ASCII range characters a western european Windows-1252 encoding becomes likely.
• If you have mostly characters in the C0-FF range, a Windows-1251 could be likely due to the use of the cyrillic alpha characters there. I guess they are not mixed with latin letters.
• For chinese encodings like GBK I have no idea what could be a reliable detection.

A starting point could be to look around in the web what mechanisms for detecting the encodings might be known already. I heard that Notepad++ might do a good job. I also found https://github.com/aadsm/jschardet which seem to be used in VSCode.

I could imagine adding a feature to alphaTab which would allow you integrating libraries like that. e.g. a callback which would allow you to decode the strings from the raw bytes on your own. Or something to just dynamically detect the encoding based on the first string read. For the second approach it the callback would need to return a TextDecoder compatible encoding.

Adapting the logic of a more open library like https://github.com/Ousret/charset_normalizer or https://github.com/PyYoshi/cChardet would also be a possibility in a small scale.

Long story short:

• Is it technically possible to detect the encoding?
  Yes. For some cases it seems to work well. But it comes at cost of performance.
• Will alphaTab add such a detection?
  No. It is unlikely that I will add this directly to alphaTab unless it is contributed.
• Will alphaTab add a mechanism to take over the detection/decoding?
  This sounds like the most reasonable option to me. It's currently not my focus to add such a functionality but I'm also happy to accept contributions here.

[lannka]

Thanks for the investigation. I will update here if I found a reasonable detection mechanism.

In my project, I will do the detection at uploading time where performance cost could be absorbed. I'm even considering to normalize all file formats to AlphaTex for storage. Does that sound reliable? What's your vision of AlphaTex?

[Danielku15]

For storage I would maybe rather recommend using the Guitar Pro 7 exporter which is available since 1.2. It is well tested and allows users also to open it in other software supporting it. Normally new features mainly come in via the Guitar Pro 7 file format and then I adapt alphaTex to also have a syntax for it. That usually also means that things are rather supported in Guitar Pro than alphaTex.

Once there will be an alphaTex exporter I maybe will add some more automated tests which will ensure that really all model elements are supported. As most people will (very likely) use third party input formats, alphaTex was rather extended "on demand".