Remove text nodes in the update xml served to clients running on 2019-01-03-22-05-33 and 2019-01-04-09-32-21

VERIFIED FIXED

Status

defect
P1
normal
VERIFIED FIXED
8 months ago
7 months ago

People

(Reporter: rstrong, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Bug 1517150 made it so the client didn't properly handle text nodes in the update xml served by balrog. The patch has been backed out but to recover these clients it would be great if balrog could serve the updates to the affected clients or all if that is easier without text nodes.
catlee, sorry about this. :(
Flags: needinfo?(catlee)
Nick,

What do you think about balrog eliminating text nodes in the returned xml output?

I am aware this makes it harder for humans to read. I am asking before thinking about writing a patch myself because it doesn't look like it would be a zero effort patch: https://github.com/mozilla/balrog/blob/bb7d2fa5049ff03d3acdf6f6abc3ca36f8a3703e/auslib/web/public/client.py#L223
Flags: needinfo?(nthomas)
These responses aren't meant to be human readable, so it's fine if we strip out the newlines/whitespace.
Flags: needinfo?(catlee)
Is it helpful for me to stay on the stuck nightly (I am right now) to verify a fix? Or is anyone else already filling that role?
Axel, for now, if you can keep this nightly, this would be better indeed :)
It is possible to reproduce by installing a build from
https://archive.mozilla.org/pub/firefox/nightly/2019/01/2019-01-03-22-05-33-mozilla-central/

I also verified that removing the unnecessary newlines and spaces from the xml file fixed this. Also, the tests didn't add the unnecessary newlines and spaces which is why the tests didn't catch this.
I downloaded a new one, but also kept a copy of the broken nightly around to verify this bug.
Are Nightly's still disabled ?  I don't see any new build running this morning.
Updates aren't disabled. I installed https://archive.mozilla.org/pub/firefox/nightly/2019/01/2019-01-01-09-47-42-mozilla-central/ and it updated to the build that includes the backout.
It looks like nightly builds weren't triggered per treeherder.
What's the status of this bug?

A lot of users are still stuck on the old builds and it's making a bit of a mess of our dashboard: https://metrics.mozilla.com/webrender/dashboard.html and hurting our nightly testing more generally.

[(u'20190101094742', 27),
 (u'20190101212914', 37),
 (u'20190102094850', 86),
 (u'20190102213721', 153),
 (u'20190103094209', 368),
 (u'20190103150357', 52408),
 (u'20190103220533', 9204),
 (u'20190104093221', 307),
 (u'20190104132412', 833),
 (u'20190104214806', 2483),
 (u'20190105123739', 1575),
 (u'20190105215256', 1002),
 (u'20190106095006', 389)]
I'll look into removing whitespace for those two particular nightlies. Making that change more broadly will require us to check any tools that parse the xml, eg update and final verify. Anything using grep etc is likely to break. It's also more user friendly to anyone debugging updates to have whitespace, but Firefox pretty print it regardless this is a non-issue.

re comment #12, the 52k on 20190103150357 seems to be the 65.0b8 beta. Of the rest ~9500 of ~16500 is stuck (57%) so this is still worth addressing ASAP. We'll need to write and deploy a fix, taking care to not impact any releases which might be happening this week.
Assignee: nobody → nthomas
Flags: needinfo?(nthomas)
Priority: -- → P1

Landed at https://github.com/mozilla/balrog/commit/461c8f331e2081dd53ba38ccb06f26077b9a2433.

Verified in Balrog dev with a mac build querying https://aus5.dev.mozaws.net/update/6/Firefox/66.0a1/20190103220533/Darwin_x86_64-gcc3/en-US/nightly/Darwin%2017.7.0/ISET:SSE4_2,MEM:16384/default/default/update.xml. Note that Firefox does pretty print, so you need a view-source: prefix or use wget/curl/etc.

Next step stage and prod.

We didn't have an opportunity to deploy with 65.0b9 being built today; will try again tomorrow but will have to find a quiet time around other releases.

Depends on: 1518387

The fix is now deployed. Pike let me know that he was able to update to 20190108101613, which was the latest nightly at that point, and I have also updated (on Mac).

Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
See Also: → 1518681

I was stuck the last days with firefox nightly and today got the first update again. I guess this changes fixed it for me. thx.

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.