Here are some details that I learned about the current implementation (inherited from Enigmail).
The code does several "trial and error" attempts when processing OpenPGP messages, because it isn't certain which encoding is the correct one to use.
First, the message is extracted from the window's DOM, some fixes are potentially applied to its structure, and an attempt is made to process it.
If this fails, the code will attempt to retry with different encodings.
There are retries numbered 1, 2, 3,
If the initial encoding is already UTF-8, then it will skip retry 1.
Retry 1 (potentially skipped), uses the message that was extracted (and adjusted), and processes it using
If retry 1 doesn't work, then the code decides that the text that was extracted from the DOM will not be used for the further attempts.
Retry 2 (msgDirectDecrypt/msgDirectCallback) will use the message URI and a stream listener to obtain the raw message. It then converts the raw message using EnigmailData.convertFromUnicode(msgText, "UTF-8") and process it.
If retry 2 failed, it do retry 3.
Retry 3 calls EnigmailData.convertToUnicode(msgText, "UTF-8") (in my understanding that reverts the conversion from retry 2, going back to the raw message source), and will process it again.
All of these steps are done without waiting for the results. I've added some async declaractions and added some await statements, which was insufficient (by itself) to fix it.
The problem is that retry 2 cannot be easily awaited. It uses setTimeout(0) to continue the processing of the data that is obtained from the stream listener. My attempt to avoid the setTimeout(0) call resulted in a deadlock.
The two tests that are failing currently are processed in retry 3. That's why it's difficult to wait for the result.
I had the thought that we potentially could rewrite the retry attempts. Maybe we can find a more direct way to obtain the raw message data, without having to rely on setTimeout(0).
However, when experimenting, I noticed that the code makes an (apparently) incorrect decision.
For the messages that work with retry 3, I noticed that actually retry 1 works for them, too!
For our test messages, which are already in UTF-8, the code decides to skip retry 1.
However, that seems like an incorrect decision.
The text that we extract from the DOM is probably in the unicode encoding - a result of whatever we have converted from the source and rendered for display.
Retry 1 takes unicode input and converts it to UTF-8.
I've changed the code to always allow retry (and never skip it).
This causes our test messages to be work in retry 1. This change avoids that we reach the setTimeout(0) code from retry 2, resulting in more direct execution.
These changes (async/await, don't skip retry 1) were still insufficient to fix the tests.
Waiting for the end of the processing was still necessary.
Because I couldn't get Magnus' suggestion to work, I went back to my original idea, using the attribute, and using utils.waitFor.
With this change, I got a working try run:
I've spent a lot of time on this issue, and I feel it's not worth it to spend more time on it right now.
I would like to suggest that we accept this workaround for now, even if it isn't the "perfect" approach.
I recommend to take this fix to get the tests going, and plan follow-up work to get this into better shape.
As part of the patch I switched the processing of retry 2 and retry 3. I think it makes sense to avoid one conversion, so if we reach retry 2, it will start with the raw code, and only does a conversion for retry 3.