1599385 - bouncer nazgul on ESR68 behaves weirdly

Status: VERIFIED FIXED

(Release Engineering :: Release Automation: Bouncer, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We seem to be having an issue with the Fennec nazgul job every once in a while and only on ESR68.

A while ago our Fennec release failed - https://firefoxci.taskcluster-artifacts.net/IJ_sWhHDRrurjSSVyDX8MQ/0/public/logs/live_backing.log

Performing a POST request to https://bounceradmin-default.prod.mozaws.net/api/product_add/ with kwargs {'timeout': 60, 'data': {'product': 'Fennec-68.2.1', 'ssl_only': 'true'}} was supposed to add that entrance in bouncer, but the follow-up call 2019-11-12 19:54:27,942 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products/> shows no record of it. It worked smooth on existing legacy bouncer so I’m thinking it could be something related to the version of Fennec or something. Also, betas still work smoothly on nazgul so it’s something isolated to ESR68 as I can tell of it.

@jbuck did a database dump https://mozilla.slack.com/archives/CJJ8U1S3B/p1574105469006500 and seems like we have very oudated entries, for some reason.

Earlier last night, the Fennec beta bouncer job failed for the same reason - https://firefox-ci-tc.services.mozilla.com/tasks/HzZbsVPYQWGq0e9hjQOZEQ

First thoughts:
a) we need to double-check to understand why is the database out of date. Potentially resync it?
b) I recall we only landed support for Nazgul in bouncer-submission so I don't understand how come it runs for locations in ESR68. Worth checking if we haven't uplifed wrong logic on ESR68.

Comment 1

[Jeremy Orem [:oremj]]

How would you like to approach this? Is it possible it's out of date, because the automation doesn't update it?

I think the only way to truly validate this is to write a script that periodically checks nazgul bouncer vs production bouncer. I can work on this, but it will likely be a week or two out.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Jeremy Orem [:oremj] from comment #1)

How would you like to approach this? Is it possible it's out of date, because the automation doesn't update it?

I think the only way to truly validate this is to write a script that periodically checks nazgul bouncer vs production bouncer. I can work on this, but it will likely be a week or two out.

That seems reasonable. But quick question - can't we just sync databases directly? Like we create a t0 moment where we migrate everything from production to Nazgul and be done with it. Would that work?

Comment 3

[Jeremy Orem [:oremj]]

This should be fixed with https://github.com/mozilla-services/bouncer-admin/pull/67

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

I think we might have a fallout on nazgul side. Beta is now broke too. More in https://firefoxci.taskcluster-artifacts.net/bBeAg7BIQhSxZ3dm_exADQ/0/public/logs/live_backing.log. Also esr68 is broken, supposably for the same reasoning https://firefox-ci-tc.services.mozilla.com/tasks/Ug5fyr0ET8uGE9mZjOeNlQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FUg5fyr0ET8uGE9mZjOeNlQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log

On the other hand, I think we should ditch and exit if we receive a 500 status code. It's weird to see that product_show returns 500 and bouncerscript continues with the logic. I'll follow-up with a patch for that.

Comment 5

[Jeremy Orem [:oremj]]

This should be fixed up now:

# curl http://localhost/api/product_show?product=Fennec-68.0.2
<?xml version="1.0" encoding="utf-8"?><products><product id="11794" name="Fennec-68.0.2"/></products>[root@ip-172-31-39-66 ~]#


[root@ip-172-31-39-66 ~]# curl http://localhost/api/product_show?product=Fennec-68.0.2
<?xml version="1.0" encoding="utf-8"?><products><product id="11794" name="Fennec-68.0.2"/></products>[root@ip-172-31-39-66 ~]# curl http://localhost/api/product_show?product=Firefox-73.0
<?xml version="1.0" encoding="utf-8"?><products><product id="12412" name="Firefox-73.0"><language locale="ach"/><language locale="af"/><language locale="an"/><language locale="ar"/><language locale="ast"/><language locale="az"/><language locale="be"/><language locale="bg"/><language locale="bn"/><language locale="br"/><language locale="bs"/><language locale="ca"/><language locale="ca-valencia"/><language locale="cak"/><language locale="cs"/><language locale="cy"/><language locale="da"/><language locale="de"/><language locale="dsb"/><language locale="el"/><language locale="en-CA"/><language locale="en-GB"/><language locale="en-US"/><language locale="eo"/><language locale="es-AR"/><language locale="es-CL"/><language locale="es-ES"/><language locale="es-MX"/><language locale="et"/><language locale="eu"/><language locale="fa"/><language locale="ff"/><language locale="fi"/><language locale="fr"/><language locale="fy-NL"/><language locale="ga-IE"/><language locale="gd"/><language locale="gl"/><language locale="gn"/><language locale="gu-IN"/><language locale="he"/><language locale="hi-IN"/><language locale="hr"/><language locale="hsb"/><language locale="hu"/><language locale="hy-AM"/><language locale="ia"/><language locale="id"/><language locale="is"/><language locale="it"/><language locale="ja"/><language locale="ja-JP-mac"/><language locale="ka"/><language locale="kab"/><language locale="kk"/><language locale="km"/><language locale="kn"/><language locale="ko"/><language locale="lij"/><language locale="lt"/><language locale="lv"/><language locale="mk"/><language locale="mr"/><language locale="ms"/><language locale="my"/><language locale="nb-NO"/><language locale="ne-NP"/><language locale="nl"/><language locale="nn-NO"/><language locale="oc"/><language locale="pa-IN"/><language locale="pl"/><language locale="pt-BR"/><language locale="pt-PT"/><language locale="rm"/><language locale="ro"/><language locale="ru"/><language locale="si"/><language locale="sk"/><language locale="sl"/><language locale="son"/><language locale="sq"/><language locale="sr"/><language locale="sv-SE"/><language locale="ta"/><language locale="te"/><language locale="th"/><language locale="tl"/><language locale="tr"/><language locale="trs"/><language locale="uk"/><language locale="ur"/><language locale="uz"/><language locale="vi"/><language locale="xh"/><language locale="zh-CN"/><language locale="zh-TW"/></product></products>

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Today's bouncer nazgul worked like a charm - https://firefox-ci-tc.services.mozilla.com/tasks/foC8dhWKQhuUn4PNPadv8g \o/