1576244 - `ValueError: zstandard Python package not available` on releng-hardware

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Posix OS, defect)

Description

[Nick Alexander :nalexander [he/him]]

I want to use modern features of our fetch tasks, namely repackaging (see Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1571589). Right now, repackaging produces only .tar.zst. That means that consumers need to have Python's zstandard installed... and the Linux 64 releng-hardware devices don't appear to have it. See https://treeherder.mozilla.org/#/jobs?repo=try&revision=51843fba863369122e2638215aa0816ce5064b39&selectedJob=263229365.

Can we get it added quickly, or should I grow repackaging for additional output types?

Comment 1

[Nick Alexander :nalexander [he/him]]

In fact, this is universal: zstandard isn't available on Linux, macOS, Win 10, or the Android bitbar hosts :(

Comment 2

[Mike Hommey [:glandium]]

How can we make this happen?

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

on ec2 windows workers, zstandard was installed in bug 1569899 by running these commands during ami creation:

C:\mozilla-build\python3\python3.exe -m pip install --upgrade pip==19.2.1
C:\mozilla-build\python3\python3.exe -m pip install --upgrade zstandard==0.11.1

:markco has bug 1570711 open for the installation on windows hardware via puppet.

maybe :dividehex, :aerickson &/or :dhouse can advise on what needs to happen on osx, android & linux

Comment 4

[Andrew Erickson [:aerickson]]

I'll make a bug for myself for android-hw (the bitbar images). Wander handles the android-em packet.net images.

We should have a requirements.txt or equivalent checked into the tree that contains the list of python dependencies we'd like pre-installed on all workers (subject to taskcluster/releng/relops approval). Whenever we build images we could reference that.

Comment 5

[Wander Lairson Costa]

(In reply to Andrew Erickson [:aerickson] from comment #4)

I'll make a bug for myself for android-hw (the bitbar images). Wander handles the android-em packet.net images.

We should have a requirements.txt or equivalent checked into the tree that contains the list of python dependencies we'd like pre-installed on all workers (subject to taskcluster/releng/relops approval). Whenever we build images we could reference that.

In Packet, we have only Python 2.7, should I install Python 3?

Comment 6

[Andrew Erickson [:aerickson]]

(In reply to Wander Lairson Costa [:wcosta] from comment #5)

In Packet, we have only Python 2.7, should I install Python 3?

If it's easy, I'd definitely say yes. I think RelOps considers python3 a requirement in images now.

Comment 7

[:dhouse]

macos 10.14 now has zstandard 1.3.8 (as of yesterday): https://bugzilla.mozilla.org/show_bug.cgi?id=1580502

Comment 8

[Wander Lairson Costa]

It is done for packet machines!

Comment 9

[Greg Mierzwinski [:sparky]]

I'm seeing this issue on linux machines (causing a perma-fail) and it seems to be an intermittent for windows hardware: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f4fd6bdf922f00fba33e9e147ce3d127171e001b&selectedJob=272641844

Comment 10

[Intermittent Failures Robot]

6 failures in 4850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform breakdown:

• linux64: 3
• linux64-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1576244&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 11

[Andrew Erickson [:aerickson]]

status update:

[x] android-bitbar
[x] linux-packet
[x] linux
[x] mac
[x] windows