Closed Bug 1508703 (bmo-release-20181120.2) Opened 7 years ago Closed 7 years ago

push updated bugzilla.mozilla.org live: mozillabteam/bmo:release-20181120.2

Categories

(bugzilla.mozilla.org :: Infrastructure, defect)

Product:
bugzilla.mozilla.org
Component:
Infrastructure
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
bmo-push accept production

People

(Reporter: dylan, Assigned: bobm)

References

Details

bug 1505793 : Add triage owner in /rest/bug bug 1506754 : Group Membership report "include disabled users" doesn't seem to work bug 1328665 : Two issues with Project Review form for RRAs bug 1505050 : make the request nagging script more robust bug 1504325 : Mozilla Gear Request form broken: The requested format gear does not exist with a content type of html. bug 1351116 : github issue links are not expanded in parentheses bug 1503270 : PhabBugz daemon should update revision status from draft to needs-review when fixing new revisions bug 1505831 : Add nick field to all user objects including *_detail returned by /rest/bug bug 1493996 : Stop using YAHOO.lang.*, remove String.prototype.trim polyfill bug 1489706 : Add section to show_bug displaying the status of associated revisions bug 1480042 : Request queue dropdown badge shows empty names for some display names bug 1507539 : Missing import statements in Push.pm bug 1507851 : Revision panel on show_bug should not appear for logged out users bug 1507901 : Phab revision panel in show_bug failing to display due to type check error for Bugzilla_api_token bug 1507867 : splinter review links in emails are missing host/scheme, rendering them unusable bug 1507518 : Allow title parameter value in url to be the title of the page bug 1508695 : Incorrect or missing tracking flags on search results in REST API { "build" : "https://circleci.com/gh/mozilla-bteam/bmo/16287", "commit" : "6b9745c6dd927411ad890a22905c44b3d2829ad3", "source" : "https://github.com/mozilla-bteam/bmo", "version" : "20181120.2" }
Pushed to dev.
Status: NEW → ASSIGNED
status-bmo-push: --- → development
tracking-bmo-push: --- → verify?
ran perl checksetup.pl -t -p, output below. No problems. Adding new table oauth2_access_token... Adding new table oauth2_access_token_scope... Adding new table oauth2_auth_code... Adding new table oauth2_auth_code_scope... Adding new table oauth2_client... Adding new table oauth2_client_scope... Adding new table oauth2_refresh_token... Adding new table oauth2_refresh_token_scope... Adding new table oauth2_scope... Creating __lbheartbeat__... Creating js/yui.js... Creating js/yui3.js... Creating skins/yui.css... Creating skins/yui3.css... Creating version.json... New parameter: report_secbugs_teams Invalid parameter: webdotbase Adding new column 'is_hidden' to the 'longdescs_activity' table... Adding foreign key: oauth2_access_token.client_id -> oauth2_client.id... Adding foreign key: oauth2_access_token.user_id -> profiles.userid... Adding foreign key: oauth2_access_token_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_access_token_scope.access_token -> oauth2_access_token.access_token... Adding foreign key: oauth2_auth_code.client_id -> oauth2_client.id... Adding foreign key: oauth2_auth_code.user_id -> profiles.userid... Adding foreign key: oauth2_auth_code_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_auth_code_scope.auth_code -> oauth2_auth_code.auth_code... Adding foreign key: oauth2_client_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_client_scope.client_id -> oauth2_client.id... Adding foreign key: oauth2_refresh_token.client_id -> oauth2_client.id... Adding foreign key: oauth2_refresh_token.access_token -> oauth2_access_token.access_token... Adding foreign key: oauth2_refresh_token.user_id -> profiles.userid... Adding foreign key: oauth2_refresh_token_scope.refresh_token -> oauth2_refresh_token.refresh_token... Adding foreign key: oauth2_refresh_token_scope.scope_id -> oauth2_scope.id... checksetup.pl complete.
Production deployment pushed. Snap shot taken, DB not yet migrated.
status-bmo-push: staging → production
tracking-bmo-push: accept → verify?
Output of checksetup.pl in production: Adding new table oauth2_access_token... Adding new table oauth2_access_token_scope... Adding new table oauth2_auth_code... Adding new table oauth2_auth_code_scope... Adding new table oauth2_client... Adding new table oauth2_client_scope... Adding new table oauth2_refresh_token... Adding new table oauth2_refresh_token_scope... Adding new table oauth2_scope... Creating __lbheartbeat__... Creating js/yui.js... Creating js/yui3.js... Creating skins/yui.css... Creating skins/yui3.css... Creating version.json... Invalid parameter: webdotbase Adding new column 'is_hidden' to the 'longdescs_activity' table... Adding foreign key: oauth2_access_token.client_id -> oauth2_client.id... Adding foreign key: oauth2_access_token.user_id -> profiles.userid... Adding foreign key: oauth2_access_token_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_access_token_scope.access_token -> oauth2_access_token.access_token... Adding foreign key: oauth2_auth_code.client_id -> oauth2_client.id... Adding foreign key: oauth2_auth_code.user_id -> profiles.userid... Adding foreign key: oauth2_auth_code_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_auth_code_scope.auth_code -> oauth2_auth_code.auth_code... Adding foreign key: oauth2_client_scope.scope_id -> oauth2_scope.id... Adding foreign key: oauth2_client_scope.client_id -> oauth2_client.id... Adding foreign key: oauth2_refresh_token.client_id -> oauth2_client.id... Adding foreign key: oauth2_refresh_token.access_token -> oauth2_access_token.access_token... Adding foreign key: oauth2_refresh_token.user_id -> profiles.userid... Adding foreign key: oauth2_refresh_token_scope.refresh_token -> oauth2_refresh_token.refresh_token... Adding foreign key: oauth2_refresh_token_scope.scope_id -> oauth2_scope.id... checksetup.pl complete.
We're seeing HTTP 502 for the following very long query: https://bugzilla.mozilla.org/rest/bug?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&include_fields=_all%2C_custom%2C_extra&keywords=sec-critical%2Csec-high%2Csec-moderate&limit=0&keywords_type=anywords&product=Core&product=Core+Graveyard&product=DevTools&product=DevTools+Graveyard&product=Extend+Firefox+Graveyard&product=Firefox&product=NSS&product=Toolkit&query_format=advanced&resolution=FIXED&resolution=WONTFIX&resolution=DUPLICATE&resolution=INCOMPLETE It appears the internal (application) timeout is happening, so let's try increasing that first. Then we should see instead a nginx or ELB timeout, and that can be fixed too. So the relevant application settings: MOJO_HEARTBEAT_TIMEOUT=240 MOJO_INACTIVITY_TIMEOUT=240 If the nginx / elb timeout can match that would be even better, of course.
This looks good. We might still want to adjust timeouts, but part of what is going on right now is an echo of the Firefox 52 event.
tracking-bmo-push: verify? → accept
There is a pretty bad bug with comment rendering, which is reverted in release-20181120.3. That will be built by morning, hope we can get it out before CA wakes up.
The query in bug 1508703#c8 normally yields short of 4k bugs, but now I need to use limit=500 to work around the 502 error. My metrics project is currently blocked hard by this, so I am somewhat eager for a fix.
(In reply to Christiane Ruetten [:cr] from comment #9) > The query in bug 1508703#c8 normally yields short of 4k bugs, but now I need > to use limit=500 to work around the 502 error. My metrics project is > currently blocked hard by this, so I am somewhat eager for a fix. The problem is the now (correct) tracking flag behavior results in either taking too much time or too much memory. I have a few ideas for a fix and I'll file a bug to track that.
What sort of time frame are we looking at?
(In reply to Christiane Ruetten [:cr] from comment #11) > What sort of time frame are we looking at? We're deploying a fix presently, see: https://manage.statuspage.io/pages/76k9j8n4y3zt/incidents/j1b6s5qfjkdt
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Did you back out bug 1508695? The situation reverted to the one three days ago. The 502 is gone, but now the custom fields are missing again.
Flags: needinfo?(dylan)

I think this might be resolved now.

Flags: needinfo?(dylan)
You need to log in before you can comment on or make changes to this bug.