Bug 1517150 made it so the client didn't properly handle text nodes in the update xml served by balrog. The patch has been backed out but to recover these clients it would be great if balrog could serve the updates to the affected clients or all if that is easier without text nodes.
catlee, sorry about this. :(
Nick, What do you think about balrog eliminating text nodes in the returned xml output? I am aware this makes it harder for humans to read. I am asking before thinking about writing a patch myself because it doesn't look like it would be a zero effort patch: https://github.com/mozilla/balrog/blob/bb7d2fa5049ff03d3acdf6f6abc3ca36f8a3703e/auslib/web/public/client.py#L223
These responses aren't meant to be human readable, so it's fine if we strip out the newlines/whitespace.
8 months ago
Is it helpful for me to stay on the stuck nightly (I am right now) to verify a fix? Or is anyone else already filling that role?
Axel, for now, if you can keep this nightly, this would be better indeed :)
It is possible to reproduce by installing a build from https://archive.mozilla.org/pub/firefox/nightly/2019/01/2019-01-03-22-05-33-mozilla-central/ I also verified that removing the unnecessary newlines and spaces from the xml file fixed this. Also, the tests didn't add the unnecessary newlines and spaces which is why the tests didn't catch this.
I downloaded a new one, but also kept a copy of the broken nightly around to verify this bug.
Are Nightly's still disabled ? I don't see any new build running this morning.
Updates aren't disabled. I installed https://archive.mozilla.org/pub/firefox/nightly/2019/01/2019-01-01-09-47-42-mozilla-central/ and it updated to the build that includes the backout.
It looks like nightly builds weren't triggered per treeherder.
What's the status of this bug? A lot of users are still stuck on the old builds and it's making a bit of a mess of our dashboard: https://metrics.mozilla.com/webrender/dashboard.html and hurting our nightly testing more generally. [(u'20190101094742', 27), (u'20190101212914', 37), (u'20190102094850', 86), (u'20190102213721', 153), (u'20190103094209', 368), (u'20190103150357', 52408), (u'20190103220533', 9204), (u'20190104093221', 307), (u'20190104132412', 833), (u'20190104214806', 2483), (u'20190105123739', 1575), (u'20190105215256', 1002), (u'20190106095006', 389)]
I'll look into removing whitespace for those two particular nightlies. Making that change more broadly will require us to check any tools that parse the xml, eg update and final verify. Anything using grep etc is likely to break. It's also more user friendly to anyone debugging updates to have whitespace, but Firefox pretty print it regardless this is a non-issue. re comment #12, the 52k on 20190103150357 seems to be the 65.0b8 beta. Of the rest ~9500 of ~16500 is stuck (57%) so this is still worth addressing ASAP. We'll need to write and deploy a fix, taking care to not impact any releases which might be happening this week.
Assignee: nobody → nthomas
Priority: -- → P1
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.