1513000 - Update to Openh264 plugin release v1.8.1

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, enhancement, P1)

Description

[Dan Minor [:dminor]]

We want to update to v1.8.1 of the Openh264 plugin.

Comment 1

[Andreas Pehrson [:pehrsons]]

I'm marking this P1 for fixing in 66.

Comment 2

[Dan Minor [:dminor]]

I've built local copies of 1.8.1 for Linux, OS X and Windows. Linux and OS X are working fine with the testing I've done so far. Windows is not, but perhaps I'm not building it properly. I'll enable logging and investigate further.

Nick, will you be able to help out with this update?

Comment 3

[Dan Minor [:dminor]]

I'm also not able to build a working copy of v1.7.1 on my Windows system, so I'm not going to worry about this too much for now. From looking at Bug 1284285 it looks like the usual process is for us to generate binaries, sign them, and then pass them to Cisco prior to getting WebRTC people involved in testing.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Dan, I'll redirect your request over to catlee for scheduling.

Last time we did this we updated the mozharness support for building OpenH264 but it's quite possible the world has shifted in the meantime. The Docs attachment on bug 1380605 details what was done, there are lots of manual steps there. Signing would certainly differ now, and it would be great if this was automated in taskcluster. The infrequent releases mean we haven't committed the time to do that. Are you interested in doing so ?

Comment 5

[Dan Minor [:dminor]]

Getting this fixed is a very high priority for us because it works around an issue where video coming from Safari freezes (Bug 1492038).

Unless there is someone available to start work on this right away, I think it makes sense for me to work on this. It will probably take me longer than someone familiar with the process and taskcluster, but on the other hand, I can get started this morning.

Comment 6

[Dan Minor [:dminor]]

I have linux32 and linux64 builds working and I'm looking at macosx64 cross compiles next.

Comment 7

[Dan Minor [:dminor]]

This is the try build where I'm seeing 403s trying to retrieve the OSX SDK: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbd9e9e7ca565bb9d919a4ce279c3f8983b9a441

I could also use help on uploading the artifacts in a way that they are only accessible to Mozilla employees.

Thanks!

Comment 8

[Jordan Lund (:jlund)]

dminor, callek, and myself met and discussed plan going forward. Given priority of this version we will not block on writing automation for this. Dan will look into trying to fix builds and callek will do some prep work next week over holidays. Assumption is we will roll this out at start of Jan once people are back from holidays and we have QA, signoffs, etc.

Plan on order:

Builds:
* linux* in working state
* macos hitting tooltool sdk download permission error. Dan will pass callek try link for help
* windows hitting build errors. Dan will look into more first
* build artifact permissions. callek will help make artifacts more private on try

Signing:
* this will be outside automation. callek to either work with secops for autograph or use old signing infra

Crash reporting:
* callek to look more into what we did for last openh264 version

Uploading:
* Dan to sync with cisco for how we want to upload builds once we have them


Publishing
* callek to create release blob
* dan to determine with nils how we want to roll this out. API is not changing so we could uplift this to release quickly. Probably want to test on nightlytest channel first.
* QA should sign-off before making live on release
* callek to update rules based on dan/nils rollout request

^ callek to update docs with goal of enlisting someone else from releng to do next version.

Comment 9

[Justin Wood (:Callek)]

(In reply to Dan Minor [:dminor] from comment #7)
> This is the try build where I'm seeing 403s trying to retrieve the OSX SDK:
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=fbd9e9e7ca565bb9d919a4ce279c3f8983b9a441

So this job is trying to contact tooltool via the public endpoint, since you have the proxy enabled you should just do like https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/build_pool_specifics.py#6 and specify the http:// path for it.

Comment 10

[Justin Wood (:Callek)]

(In reply to Dan Minor [:dminor] from comment #7)
> I could also use help on uploading the artifacts in a way that they are only
> accessible to Mozilla employees.

As far as artifacts, if you store them in a (exposed to worker) location named `private/...` they will only be available to moco employees + NDA. https://docs.taskcluster.net/docs/manual/design/namespaces#artifacts

This should be fine, and can be done in task definition like:

```
  artifacts:
    public/logs:
      path: /builds/worker/logs/
      expires: '2019-01-18T13:47:19.996Z'
      type: directory
    public/build:
      path: /builds/worker/artifacts/
      expires: '2019-01-18T13:47:19.996Z'
      type: directory
    private/openh264:
       path: /builds/worker/private_artifacts/   # This can be whatever you want
       expires: 'whatever-datetime'
       type: directory
```

This allows any file/folder placed in the `path` you specify to be exposed via taskcluster artifacts under "private/openh264/..." -- Feel free to reach out if this is unclear to you.

Comment 11

[Dan Minor [:dminor]]

Thank you for the suggestions. I'm now able to cross-compile with clang but not link, it looks like I will need to add a specialized Makefile for cross-compiling as it doesn't seem like we can pass in the correct options to the linker with the current setup.

Comment 12

[Dan Minor [:dminor]]

After a long delay, I've started working on this again. I have builds for all platforms now.

I had a few oddities with win64 though. If I attempt to upload a directory as an artifact, I get an exception and no logs [1]. Uploading a file works fine [2]. There seems to be something that expects an artifact at public/build [3]. I worked around that last one by creating an empty directory there, but it seems weird. Any suggestions on these would be appreciated.

I had to make some modifications to the openh264 Makefiles to get clang on windows working, so I'll have to get those upstreamed. I also have a bunch of warnings that would be nice to fix. Upstream supports clang on Android, but support was added after 1.8.1, so we'll have to target a newer version, or get the support backported.

Next step would be to get the symbols available. Just to confirm, before I spend time on this, we do use those somewhere?

Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=be9bbce7a7a15a4c36fdf2668be676e358560dff&selectedJob=225465463
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=434ca218c0f25c805147523a58b3dd5fc1b3fbc8&selectedJob=225502484
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c763d0e4688d33e0e5a07a025f9354a8ddda5dc2&selectedJob=225489870

Comment 13

[Justin Wood (:Callek)]

(In reply to Dan Minor [:dminor] from comment #12)

I had a few oddities with win64 though. If I attempt to upload a directory as an artifact, I get an exception and no logs [1]. Uploading a file works fine [2]. There seems to be something that expects an artifact at public/build [3]. I worked around that last one by creating an empty directory there, but it seems weird. Any suggestions on these would be appreciated.

Leaving n-i so I can look into this part before I disperse myself (I'll have to look more on this part on monday)

Next step would be to get the symbols available. Just to confirm, before I spend time on this, we do use those somewhere?

I was previously asked to be sure I uploaded symbols to our symbol server for openh264, I suspect Socorro reports on them somehow. I don't have any further insight than that as I don't do crash analysis nor do I know which teams would have seen openh264 crash reports to tell us about them.

If I need to go digging in my old bugmail for further details I can, but that is what I know offhand.

Comment 14

[Pete Moore [:pmoore][:pete]]

(In reply to Dan Minor [:dminor] from comment #12)

After a long delay, I've started working on this again. I have builds for all platforms now.

I had a few oddities with win64 though. If I attempt to upload a directory as an artifact, I get an exception and no logs [1].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=be9bbce7a7a15a4c36fdf2668be676e358560dff&selectedJob=225465463

The log shows:

[taskcluster:error] &errors.errorString{s:"Rel: can't make build\\openh264\\artifacts relative to /build/openh264/artifacts"}
[taskcluster:error] Rel: can't make build\openh264\artifacts relative to /build/openh264/artifacts

This is due to the task payload declaring the following artifact:

    {
      "name": "private/openh264",
      "path": "/build/openh264/artifacts",
      "type": "directory"
    },

The path is relative to the task directory, so shouldn't have a leading / (i.e. I think you need "build/openh264/artifacts"). I think if you remove it, that should fix the problem.

Uploading a file works fine [2]. There seems to be something that expects an artifact at public/build [3]. I worked around that last one by creating an empty directory there, but it seems weird. Any suggestions on these would be appreciated.

[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=c763d0e4688d33e0e5a07a025f9354a8ddda5dc2&selectedJob=225489870

Similarly, there is an artifact declared in the task payload:

    {
      "name": "public/build",
      "path": "public/build",
      "type": "directory"
    }

This tells the worker that it should find a directory there for uploading. If you remove this from task definition, that should solve the problem.

Comment 15

[Justin Wood (:Callek)]

Thanks for looking pete!

Comment 16

[Dan Minor [:dminor]]

Hi Pete,

Thank you for having a look! Removing the forward slash from my artifact path did fix things for using a directory, strange that it worked for a file.

The "public/build" artifact is not a part of my kind.yml file [1], it must be coming from somewhere else, but I can work around that.

[1] https://hg.mozilla.org/try/file/434ca218c0f2/taskcluster/ci/openh264-plugin/kind.yml

Comment 17

[Dan Minor [:dminor]]

Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=762348d38f37cbb2079af16dd008a8c48796f8c7

It turns out that all builds look for "public/build", but it's only an error on windows. I'll just create an empty directory for that.

I'm waiting on an upstream pull request: https://github.com/cisco/openh264/pull/3095. Once that lands, we'll have to decide whether we want to ask for a backport the Android clang support to 1.8.1 or maintain the differences (just one Makefile) as a local patch.

I've done some limited testing of the plugins built on treeherder on my Linux and Windows systems and they look good so far.

Comment 18

[Dan Minor [:dminor]]

I discussed it with Nils yesterday and we don't want to delay this any further. I've produced a backport of the required patch and my pull request and updated kind.yml to point to that for the Android builds. I've forked it to github.com/dminor/openh264. If that's a concern, we can probably move it to the mozilla organization without too much trouble.

Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae1dfc36a29f85c263f2a557736b45abb01159b6

I've disabled running the gtests. The tests won't run on the Linux builders due to a glibc mismatch, won't run on Windows without getting git added to the build machines, OS X is now a cross-compile and we never ran them on Android anyway. We do local testing of the builds anyway, so I don't think this is a huge concern.

Comment 19

[Dan Minor [:dminor]]

Attachment: Bug 1513000 - Updates to mozharness openh264 scripts and configs; r=callek!

Comment 20

[Dan Minor [:dminor]]

Attachment: Bug 1513000 - Add taskcluster configuration for building openh264 plugin; r=callek!

Depends on D19817

Comment 21

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - Add private/openh264/* artifact access scopes to Decision task for level1+ repos. r=tomprince

Comment 22

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - Do symbol uploads for openh264. r=tomprince

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ddb817eed3bc
Updates to mozharness openh264 scripts and configs; r=callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/471db22054e7
Add taskcluster configuration for building openh264 plugin; r=callek

Comment 24

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/ddb817eed3bc
https://hg.mozilla.org/mozilla-central/rev/471db22054e7

Comment 25

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - Sign openh264 binaries. r=tomprince

Comment 26

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - Create action to trigger Openh264. r=tomprince

Comment 27

[Elliot]

when we can get the new FF with openh264 1.8.1?

Comment 28

[Justin Wood (:Callek)]

(In reply to Elliot from comment #27)

when we can get the new FF with openh264 1.8.1?

The best answer I can give at this time is "soon" we're pretty close to being able to finalize what is needed here, there will be some Quality Assurance steps to make sure nothing is wrong of course along the way.

Comment 29

[Elliot]

which nightly build I can test? I want to make sure Webex webapp can work normally. thanks.

Comment 30

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - expose revision as an attribute on the openh264 jobs. r=tomprince

Comment 31

[Justin Wood (:Callek)]

Attachment: Bug 1513000 - expose revision as an attribute on the openh264 jobs. r=tomprince

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ce8f26f311b
expose revision as an attribute on the openh264 jobs. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/81a50b35b482
Sign openh264 binaries. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/26021d8ebb27
Create action to trigger Openh264. r=tomprince

Comment 33

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/9ce8f26f311b
https://hg.mozilla.org/mozilla-central/rev/81a50b35b482
https://hg.mozilla.org/mozilla-central/rev/26021d8ebb27

Comment 34

[Tom Prince [:tomprince]]

Attachment: Bug 1513000: [openh264] Adjust `create_tasks` call in openh264 action for Bug 1528362; r?Callek

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a88b39ffb1f1
[openh264] Adjust `create_tasks` call in openh264 action for Bug 1528362; r=Callek

Comment 36

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/a88b39ffb1f1

Comment 37

[Intermittent Failures Robot]

1 failures in 2981 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• gecko-decision: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513000&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 38

[Justin Wood (:Callek)]

Debugging Symbols are on our server now, for all platforms as of this writing.

Comment 39

[Justin Wood (:Callek)]

Status Update: Binaries all sent over for hosting, there seems to be some issues (internally) with Android, which may prompt a new build. We plan to ship the desktop versions as-is at this time. Once we are able to serve them to users we'll update this bug so that interested people can follow along.

Comment 40

[Jan Beich]

Is there any plan to tag v1.8.1 for those who build the plugin from source?

Comment 41

[Dan Minor [:dminor]]

(In reply to Elliot from comment #29)

which nightly build I can test? I want to make sure Webex webapp can work normally. thanks.

Hi Elliot, this is now available on the 'nightlytest' update channel. You need to edit the channel-prefs.js file in your Firefox installation and change the value there to be 'nightlytest'. If you already have a copy of the openh264 plugin, you will want to remove the 'media.gmp-manager.lastCheck' pref in about:config to force it to do check for updates. Otherwise, I think it only checks for updates once a day. Please let us know if you run into any problems.

Comment 42

[Elliot]

(In reply to Dan Minor [:dminor] from comment #41)

(In reply to Elliot from comment #29)

which nightly build I can test? I want to make sure Webex webapp can work normally. thanks.

Hi Elliot, this is now available on the 'nightlytest' update channel. You need to edit the channel-prefs.js file in your Firefox installation and change the value there to be 'nightlytest'. If you already have a copy of the openh264 plugin, you will want to remove the 'media.gmp-manager.lastCheck' pref in about:config to force it to do check for updates. Otherwise, I think it only checks for updates once a day. Please let us know if you run into any problems.

still not get the 1.8.1 plugin as I followed your steps.

Comment 43

[Justin Wood (:Callek)]

(In reply to Elliot from comment #42)

(In reply to Dan Minor [:dminor] from comment #41)

(In reply to Elliot from comment #29)

which nightly build I can test? I want to make sure Webex webapp can work normally. thanks.

Hi Elliot, this is now available on the 'nightlytest' update channel. You need to edit the channel-prefs.js file in your Firefox installation and change the value there to be 'nightlytest'. If you already have a copy of the openh264 plugin, you will want to remove the 'media.gmp-manager.lastCheck' pref in about:config to force it to do check for updates. Otherwise, I think it only checks for updates once a day. Please let us know if you run into any problems.

still not get the 1.8.1 plugin as I followed your steps.

You should do the following:

• Exit Firefox completely
• Change the .js file that holds the update prefs (in the application folder, not profile) for app.update.channel to be 'nightlytest' instead of 'nightly' and save this file
• Launch firefox
• delete 'media.gmp-manager.lastCheck' preference in about:config
• (maybe restart firefox, not sure how often we would check for openh264 otherwise)

and you should get 1.8.1, if you are on Desktop. If you do not and still end up with 1.7.1 I'd like to know what OS you are on and some more details so I can try to track down why.

Comment 44

[Justin Wood (:Callek)]

This was just rolled out to Nightly users, and will remain Nightly Only for now. Android is not updated yet.

Comment 45

[Dan Minor [:dminor]]

For all platforms, Firefox 67 and later will have 1.8.1. I'm going to call this done.

Comment 46

[Greg ( Hank1 )]

Hello, i dunno if this is the right place but would it be possible to update to 2.0.0 https://github.com/cisco/openh264/commit/5f76764444393f333b6a6e3d64f72ee454b3f612

Comment 47

[Hank Peng]

Greg, the major change in 2.0.0 is the B-Frame support. It's rarely used in RTC, so the Cisco Openh264 team didn't ask Firefox to pick up this release.

Comment 48

[Greg ( Hank1 )]

(In reply to Hank Peng from comment #47)

Greg, the major change in 2.0.0 is the B-Frame support. It's rarely used in RTC, so the Cisco Openh264 team didn't ask Firefox to pick up this release.

ok.

Comment 49

[Arthur K. (he/him)]

2.1.0 has been released: https://github.com/cisco/openh264/commit/032f97db96827f9c4c183de5bc3794fce7b76bf6

Comment 50

[Arthur K. (he/him)]

And now the release notes:

v2.1.0

• Experimentally support for multi-thread decoding(default disabled,and may result in random problems if enabled)
• Assembly optimization for loongson platform
• Update meson version to 5
• Some minor bug fixes

Considering 2.0.0 was out since last July and skipped over for current FF, maybe about time and in time for 76?

Comment 51

[Dan Minor [:dminor]]

(In reply to Arthur K. from comment #50)

And now the release notes:

v2.1.0

• Experimentally support for multi-thread decoding(default disabled,and may result in random problems if enabled)
• Assembly optimization for loongson platform
• Update meson version to 5
• Some minor bug fixes

Considering 2.0.0 was out since last July and skipped over for current FF, maybe about time and in time for 76?

I've filed Bug 1619988 for this.