1634836 - libotr not integrated into build

Status: ASSIGNED

(Thunderbird :: Build Config, task, P3)

Description

[ISHIKAWA, Chiaki]

Short summary.:

We need to load libotr libary for full testing since early April.
And for that matter, I think we will need libotr as part of TB runtime for ordinary user when the trunk hits the general user base?

Long summary:

I have checked the local log of thunderbird mochitest running FULL DEBUG version of TB.

In it, I noticed several dozens lines of console.log.

There, "Cannot load required OTR library" error started to happen sometime between March 12 and April 12, 2020.
I don't see it in the old log before March 12.

(I did not have time to look into this fully because of other serious local build failures.
Bug 1633092, Bug 1630345)

These are the lines. The number at the beginning of the line shows the frequency of occurrence.

========================================
console.log: errors and warnings.
========================================

console.log:

      1	 console.log: "socket closed with status 804b0002"
      1	 console.log: "socket open on port 34605"
==>  48	 console.log: (new Error("Cannot load required OTR library", "resource:///modules/OTRLib.jsm", 75))
      2	 console.log: check_no_attachment_size: node.size-><-
      2	 console.log: check_no_attachment_size: node.size->​<-
      1 # console.log: === JS Bridge: Starting server
      1 # console.log: === Mozmill: Seen window chrome://messenger/content/messenger.xul

The error seems to be generated BETWEEN tests.
That is, the error line seems to be printed BEFORE a test file is executed.
(BUT NOT ALL the test files, mind you.)

I thought this ought to be visible in other people's try-comm-central tryserver jobs.
But I was surprised when I notice that someone else's job picks up and load the library without a problem?
(I have not been able to run tryserver job for a mysterious reason for about a couple of weeks on and off and so have not tried that much yet after the local build fiasco.)

E.g.: search for OTR in the following live_backing.log from
https://firefoxci.taskcluster-artifacts.net/YFFPAUmZQTieuEORoYHAhw/0/public/logs/live_backing.log

[task 2020-05-01T05:18:12.916Z] 05:18:12     INFO - GECKO(1114) | console.debug: "Successfully loaded OTR library libotr.so.5 from /builds/worker/workspace/build/application/thunderbird/libotr.so.5"

So there is something different between tryserver (try-comm-central) and my local PC.

I checked my linux installation's packaging situation. I am using Debian GNU/Linux.
In the repository, sure enough, there is libotr5.
So I installed it.: I did not have it locally until today.

ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ apt-cache search libotr
golang-github-twstrike-otr3-dev - Go implementation of the OTR 3 protocol
keysync - Syncs OTR identities between the different chat programs
libotr5 - Off-the-Record Messaging library
libotr5-bin - toolkit for Off-the-Record Messaging library
libotr5-dev - Off-the-Record Messaging library development files
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ su
Password:
root@ip030:/NREF-COMM-CENTRAL/mozilla/comm# apt-get install libotr5-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libotr5
Suggested packages:
  libotr5-bin
The following NEW packages will be installed:
  libotr5 libotr5-dev
0 upgraded, 2 newly installed, 0 to remove and 17 not upgraded.
Need to get 159 kB of archives.
After this operation, 511 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://debian-mirror.sakura.ne.jp/debian testing/main amd64 libotr5 amd64 4.1.1-3+b1 [86.8 kB]
Get:2 http://debian-mirror.sakura.ne.jp/debian testing/main amd64 libotr5-dev amd64 4.1.1-3+b1 [72.1 kB]
Fetched 159 kB in 0s (674 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Selecting previously unselected package libotr5:amd64.
(Reading database ... 313047 files and directories currently installed.)
Preparing to unpack .../libotr5_4.1.1-3+b1_amd64.deb ...
Unpacking libotr5:amd64 (4.1.1-3+b1) ...
Selecting previously unselected package libotr5-dev:amd64.
Preparing to unpack .../libotr5-dev_4.1.1-3+b1_amd64.deb ...
Unpacking libotr5-dev:amd64 (4.1.1-3+b1) ...
Setting up libotr5:amd64 (4.1.1-3+b1) ...
Setting up libotr5-dev:amd64 (4.1.1-3+b1) ...
Processing triggers for libc-bin (2.30-4) ...
root@ip030:/NREF-COMM-CENTRAL/mozilla/comm# exit

I have rerun configure and build to see if the error goes away.
Well, sure enough, the erro rseems to be gone (!)
Now I get in the local log.:

 0:30.73 GECKO(1262690) console.debug: "Successfully loaded OTR library libotr.so.5 from system's standard library locations"

Observation:

Does this mean that the TB developer's guide is missing a dependency information concerning libtr(and 5 at that)?
Or for that matter, shouldn't the configure properly picks up the missing dependency and warns the developer and stops there?
I think it should.
(And it ought to suggest to run |./mach bootstrap|and |make bootstrap should install it in that case, also?)

(BTW, the long lasting adagio about the build/test infrastructure ought to FAIL the test that has javascript errors, or fails to load
some test files still stands. They are not successful in the ordinary sense of the "success" at all.
Unless they fail, nobody becomes aware of these errors promptly.)

I am a bit confused though, why the tryserver picked up libotr.a from /builds/worker/workspace/build/application/thunderbird/libotr.so.5.
This is NOT obviously a standard library location under linux.
Was it placed it intentionally by the tryserver farm in advance?
(In this case, obviously, the configure and friends misses to check the presence of this library before proceeding.)
Or was it produced by TB build process??? Unlikely, I suppose.

Then come to think of it, maybe we need to require the libotr library to be loaded on their linux PCs?
Otherwise, this library won't be usable for TB and error will be produced.

I checked TB's object directory and there are only the following third party libraries being produced on my linux environment.
From this, I think TB's script assumes that libotr library comes from system package.
This assumption does not hold under Debian at least as my experience shows.

/NEW-SSD/moz-obj-dir/objdir-tb3/third_party:
total used in directory 32 available 120.6 GiB
drwxr-xr-x 6 ishikawa ishikawa 4096 4月 28 10:04 .
drwxr-xr-x 57 ishikawa ishikawa 12288 4月 28 12:30 ..
drwxr-xr-x 3 ishikawa ishikawa 4096 4月 28 10:06 msgpack
drwxr-xr-x 3 ishikawa ishikawa 4096 4月 28 10:06 prio
drwxr-xr-x 2 ishikawa ishikawa 4096 4月 28 10:04 python
drwxr-xr-x 3 ishikawa ishikawa 4096 4月 28 10:04 sqlite3

Hope this helps.

TIA

Comment 1

[Rob Lemley [:rjl]]

The source is in /comm/third_party/libotr. It depends on libgpg-error and libgcrypt also found in that directory.
The integration with the build system is admittedly very poorly done, and does not take into consideration local developers. I'd like to improve that and make it a proper part of the build.
Alternatively, the build script that Taskcluster uses to build it could be made more portable or replaced with something that local developers can easily work with. Then the build could just refer to the prebuilt binaries and copy them into place.
The biggest issue is Windows. None of these libraries were written with Windows in mind, and they do not build with Clang+MSVC runtime without modifications. The official Windows builds use MinGW that's included in Debian 10 to cross compile.

Comment 2

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Please don't use 'n/a' as a severity for bugs of type 'defect' . Should this be a task or enhancement instead?

Comment 3

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Rename rnpdefs.mozbuild to third_party_defs.mozbuild.

These functions will be used for compiling libotr in-tree so the name
no longer makes sense.

Comment 4

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Move IQuote function to third_party_defs.mozbuild.

Function is useful building more than just RNP itself.

Depends on D100581

Comment 5

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Create comm_util.configure to hold shared configure template functions.

Using these functions for building libotr as well as librnp, so
moving to a separate include file to ensure they will be available for each
configure file.

Depends on D100582

Comment 6

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Modifications to libgpg-error sources.

No changes are made to libgpg-error files, just additions. These are
in the form of generated header files that are used when targeting
Windows and the addition of moz.build and modification of Makefile.in
as noted.

Depends on D100583

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - niwcompat updates to support building libgpg-error.

Updates to the compatibility headers to support building libgpg-error with
MSVC style compilers such as clang-cl.

Depends on D100584

Comment 8

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Add otr.configure for building libotr in tree.

Depends on D100585

Comment 9

[Rob Lemley [:rjl]]

Attachment: Bug 1634836 - Allow building libotr dependencies in tree for testing.

This will enable building libotr in tree with the --enable-libotr flag in
mozconfig. It is automatically applied when run in CI.
At the moment, just libgpg-error is compiled to object files. libotr itself
will continue to be built in a separate toolchain job in CI until the
rest of the dependencies are ported to mozbuild.

Depends on D100586

Comment 10

[Rob Lemley [:rjl]]

Kai and Dave, CCing you both to get this bug on your radar. It's still a work in progress, and I'm not sure I'll be able to get this working. The changes here get libgpg-error building with mozbuild the same as we do librnp. libgcrypt should not be as bad hopefully since there's no crazy build time binaries getting build to build header files. Shiftmedia made the Windows support possible.

Comment 11

[Rob Lemley [:rjl]]

Side note... the mingw that we use to build libotr for windows does not support aarch64 on windows. This bug is one way to supporting OTR on win64-aarch64. IF it works.

Comment 12

[Rob Lemley [:rjl]]

I worked on getting libotr to build with MSVC as a step towards using Clang-cl. It builds, it links. When Thunderbird starts up it crashes almost immediately.
I haven't had much luck debugging. I believe the crash occurs at https://searchfox.org/comm-central/rev/b0122cf5c5e1f96f2c2a8c1eaf4c8ae76e3bf89b/third_party/libotr/src/privkey.c#707 with the fgets call. (https://crash-stats.thunderbird.net/report/bp-29e1a069-2843-4456-9500-a7dd81220228#tab-details). The error is EXCEPTION_ACCESS_VIOLATION_WRITE, trying to read into memory it doesn't have access to?