chevron_left pEp for Thunderbird

Encoding problems with thunderbird

  • forum 16 replies
  • last reply by aanno Feb. 19, 2022
  • link
    Members 7 posts
    Sept. 22, 2021, 6:48 a.m.
    link

    Hello,

    I'm using pep on Thunderbird 91.1.0 on Fedora Linux 34 for a few days now. At present I'm affected by encoding problems when reading (my pep processed) messages that I have not encountered before. So I suspect that pep is the culprit (but I'm not sure). Hence the question is: Is there a easy way to rule out pep?

    This is the begin of one of the affected messages (an reply) in raw format

    Message-ID: 3dbd0242-642a-0f42-d8dc-************@gmx.de
    Date: Tue, 21 Sep 2021 16:25:33 +0200
    MIME-Version: 1.0
    User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
    Thunderbird/91.1.0
    Subject: Re: AW: Einladung
    Content-Language: de-DE
    To: <deleted>
    References: PAXPR08MB6928F1B927491912320AAB9994DD9@************.eurprd08.prod.outlook.com
    e5a464e8-195f-951f-cb05-************@gmx.de
    AM0PR08MB5412F9C5AA1859847E403ACEAAA19@*************.eurprd08.prod.outlook.com
    From: <deleted>
    X-Pep-Version: 2.1
    In-Reply-To: AM0PR08MB5412F9C5AA1859847E403ACEAAA19@*************.eurprd08.prod.outlook.com
    Content-Type: multipart/mixed; boundary="286f89d8204472f648a592327a326a96"

    --286f89d8204472f648a592327a326a96
    Content-Type: text/plain; charset="utf-8"
    Content-Transfer-Encoding: quoted-printable

    Hallo Tanja,

    vielen Dank f=C3=83=C2=BCr die Einladung

    <rest of reply>

    This is a reply in German and should look like

    without encoding problems. But it shows up like this in thunderbird

    As you can see, the is a 'ü' (u Umlaut) that is encoded in a wrong way.

    Kind regards,

    aanno

  • link
    Team
    Sept. 29, 2021, 9:46 p.m.
    link

    Well, f=C3=83=C2=BCr is already wrong. To encode the the letter ü, two bytes are used in UTF-8, not four. So the question is where the mojibake comes from. Was the message you received encrypted? Are you using "secure server" (not "store messages securely") so the message got decrypted permanently? Does the sender see the sent message correctly?

    In general, encoding in pEp is working and has been working for a long time now. Our developers are German and Spanish, so we would have noticed encoding problems rather quickly. You can send yourself a message. Does that have a problem?

  • link
    Members 7 posts
    Oct. 2, 2021, 2:27 p.m.
    link

    Dear @joerg ,

    despite of what I've written, I'm now convinced that the encoding problem is caused by the pep thunderbird plugin: If I deactivate it, the problem is all gone.

    With pep activated, I've written an email to myself (body: 'test encoding ÄÖÜäöüß'). Encoding is wrong with pep. If I deactivate pep for this mail only (i.e. thunderbird plugin is still active), the encoding is all right.

  • link
    Team
    Oct. 4, 2021, 10:31 a.m.
    link

    Hmm, this is really surprising. pEp is using their own software in-house for a year or so, we're writing in German and Spanish, and I haven't seen an encoding problem in a while. It's possible that there is some incompatibility on Linux(?). @sva, do you see any encoding issues in your daily usage?

  • link
    Team
    Oct. 4, 2021, 10:34 a.m.
    link

    2021-10-04 12_33_14-Window.png

    2021-10-04 12_33_14-Window.png

    PNG, 5.2 KB, uploaded by joerg on Oct. 4, 2021.

  • link
    Team
    Oct. 4, 2021, 1:47 p.m.
    link

    Actually, I was very used to TB encoding problems in the past, but ever since I use pEp (and i am a pre-tester since ~a year now) i cant remember having seen any of them any more. We also did some brief tests now, but aren't able to reproduce anything like that. @aanno does it happen on a regular basis, or just with a/some particular communication partners? Can you send us an email to support@pep.security (maybe in German) with lots of Umlauts or similar and we reply back with the same, so that we can find a first track towards a way to reproduce on our side?

    Also we haven't been able to test on Fedora yet (its not officially supported, so if we dont find anyone in the "family" its impossible then anyway) - do you happen to have any other fedora machine in your environment (ideally with a TB 78), and encounter the same issue there, maybe? That would be very helpful to know, too. Thanks!

  • link
    Team
    Oct. 4, 2021, 1:59 p.m.
    link

    I can offer a "remote" VNC session.

  • link
    Team
    Oct. 4, 2021, 2:26 p.m.
    link

    Thanks Jörg - but lets first check on this please, @aanno

    => if its true that it only happens with some particular communication partner(s) it would be great if you can ask them to drop a message to support@, too, thanks!

  • link
    Members 7 posts
    Oct. 6, 2021, 6:21 a.m.
    link

    I've now tried pep on a Windows 10 machine with Thunderbird 91.1.2. No encoding problems here! Will try another Linux system (Ubuntu) soon and send an emal to support as well.

  • link
    Members 7 posts
    Oct. 6, 2021, 8:52 a.m.
    link

    Hello, I tried a freshly-installed, older Ubuntu (20.04) as virt-manager/kvm VM without problems, i.e. pEp works there. Then I tried a freshly installed Fedora 35-beta VM. With this VM, I get the same encoding problem like on my Fedora 34 machine.

    Hence, it seems to have something to do with Fedora! I will now post an Mail to support from the Fedora 35 VM.

  • link
    Team
    Oct. 6, 2021, 9:48 a.m.
    link

    Thanks for testing. The issue is that different Linux distros ship with different libraries. That's why the pEp package ships a "fat" binary were all relevant libraries are statically linked to avoid problems with the underlying platform. So what you're reporting is surprising but not totally unexpected. I think our QA team needs to test this in Fedora.

  • link
    Team
    Oct. 8, 2021, 11:48 a.m.
    link

    Dear all, with support mails the issue could be confirmed, but it seems to be on Fedora only. As pEp.security does not officially support Fedora, QA cant test - but they did test on CentOS which is officially supported. "Unfortunately" everything went fine there, no encoding problems appeared yet, some more tests ongoing. @Shelburn as part of our voluntary testing team used to test on Fedora a bit, but he is unfortunately away from his devices for another 2 weeks.

    Anyone else around with a Fedora system?

  • link
    Members 7 posts
    Oct. 11, 2021, 7:47 a.m.
    link

    If there are instructions on how to debug pEp, maybe I'm able to step in!

    Perhaps it is possible to run the external pEp daemon(s) from a docker image instead from the 'normal' installation?!? This way, is (perhaps) possible to narrow the debug to (a) the thunderbird plugin or (b) the external pEp daemons.

    When I look for the (user) daemons of pEp, I see the following:

    $ systemctl list-units --user --type=service | grep -i pep
  pEp-mini-json-adapter.service                                        loaded active     running      Start pEp mini json adapter server (Debug)
  pEp-update-service.service                                           loaded activating auto-restart Start pEp update service
  • link
    Team
    Oct. 11, 2021, 8:16 a.m.
    link

    Not easy to debug, you'd have to check the traffic between the add-on and the adapter. Usually you need a debug version of the adapter for that.

    Besides, this is not a problem in the add-on, it's a problem internal to the adapter related to how it's built and the libraries involved in the build. Works on Windows, Mac and other Linux distros, so there is a build problem specific to Fedora.

    What you could do is to check the syslog for messages related to iconv, which is used for charset conversion, or something similar to:
    [pid 72970] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
    [pid 72970] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

    Basically any output from the pEp-mini-json-adapter process.

  • link
    Members 7 posts
    Oct. 19, 2021, 6:37 p.m.
    link

    While there is a pEp thunderbird documentation at https://www.pep.security/docs/thunderbird.html, the only technical description I found is https://dev.pep.foundation/Thunderbird .

    I have looked at /var/log/messages and at the journal, but didn't find anything about iconv/gconv/json in it.

    $ journalctl --user
...
Sep 07 14:10:18 redsnapper systemd[4740]: pEp-update-service.service: Scheduled restart job, restart counter is a>
Sep 07 14:10:18 redsnapper systemd[4740]: Stopped Start pEp update service.
Sep 07 14:10:18 redsnapper systemd[4740]: Started Start pEp update service.
Sep 07 14:10:18 redsnapper sh[57765]: /mnt/home/tpasch/.local/bin/pEp-update-service: error while loading shared >
Sep 07 14:10:18 redsnapper systemd[4740]: pEp-update-service.service: Main process exited, code=exited, status=12>
Sep 07 14:10:18 redsnapper systemd[4740]: pEp-update-service.service: Failed with result 'exit-code'.
Sep 07 14:10:20 redsnapper systemd[4740]: pEp-update-service.service: Scheduled restart job, restart counter is a>
Sep 07 14:10:20 redsnapper systemd[4740]: Stopped Start pEp update service.
Sep 07 14:10:20 redsnapper systemd[4740]: Started Start pEp update service.
Sep 07 14:10:20 redsnapper sh[57768]: /mnt/home/tpasch/.local/bin/pEp-update-service: error while loading shared >
Sep 07 14:10:20 redsnapper systemd[4740]: pEp-update-service.service: Main process exited, code=exited, status=12>
Sep 07 14:10:20 redsnapper systemd[4740]: pEp-update-service.service: Failed with result 'exit-code'.
Sep 07 14:10:22 redsnapper systemd[4740]: pEp-update-service.service: Scheduled restart job, restart counter is a>
Sep 07 14:10:22 redsnapper systemd[4740]: Stopped Start pEp update service.
Sep 07 14:10:22 redsnapper systemd[4740]: Started Start pEp update service

    There are some adapter.log.xz files under ~/.pEp/core . But nothing interesting it them as well.

    There is a /tmp/pEp-mini-json-adapter.<user>.log log file and this could be interesting. However, it does not seem to be suited for a public post...

  • link
    Team
    Oct. 20, 2021, 4:19 p.m.
    link

    Hey aanno,

    thanks for your effort to dive deeper into the problem!

    There is a way to get a more verbose log from the adapter with strace that might give you more insight:

    $ systemctl --user stop pEp-mini-json-adapter.service
    $ strace -f ~/.local/bin/pEp-mini-json-adapter -D

    Then you start Thunderbird just normal and you can check the log directly.

    If you want, you can send parts of the logs which you find interesting encrypted to support@pep.security and we'll forward it someone in the dev-team who offered to have a quick look, nonetheless its not supported :)

  • link
    Members 7 posts
    Feb. 19, 2022, 11:04 a.m.
    link

    Button line: At present thunderbird is at 91.5.0 on fedora 35. I'm using thunderbird with pEp 1.1.114. With this combination, the issue is gone, i.e. pEp works as expected. No encoding problems any more.