Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Single locale release updates should not race each other

RESOLVED FIXED

Status

Release Engineering
Balrog: Backend
RESOLVED FIXED
2 years ago
11 months ago

People

(Reporter: rail, Assigned: vjoshi)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

We see a lot of races in funsize similar to this:

l) locale A requests data_version
2) locale B requests data_version
3) locale B updates the release blob
4) locale A fails to update the release blob

They probably don't update the same data in the blob, so we can probably find a work around for this.
Created attachment 8687173 [details] [diff] [review]
balrog_metrics-tools-4.diff

Let's start with some basic request metrics for now.

We can walk the TC tasks using the index, fetch their logs, process with something like https://github.com/etsy/logster (would require some custom parser) and submit to graphite.
Attachment #8687173 - Flags: review?(catlee)
Assignee: nobody → rail
BTW, this is an issue for l10n repacks as well know. It's all suddenly started on Tue Nov 3, see https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=bc9c6e996006&filter-searchStr=l10n&exclusion_profile=false

I wonder if it is somehow related to all recent  DB/webhead migrations...
According to the webapp/db PHX1-SCL3 dashboard, aus migrated on Monday, Oct 26
Created attachment 8687210 [details] [diff] [review]
balrog_metrics-tools-5.diff

req.elapsed is not that good from what I see. It's usually less than a second while the surrounding log lines are 10-15 secs away. Not sure why, let's go with old school way.
Attachment #8687173 - Attachment is obsolete: true
Attachment #8687173 - Flags: review?(catlee)
Attachment #8687210 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8687173 - Attachment is obsolete: false

Updated

2 years ago
Attachment #8687210 - Flags: review?(catlee) → review+
Comment on attachment 8687210 [details] [diff] [review]
balrog_metrics-tools-5.diff

https://hg.mozilla.org/build/tools/rev/58900072a047
Attachment #8687210 - Flags: checked-in+
Depends on: 1224674
Looks like this is related to the aus migration, which according to the dashboard happened on Oct 26.

It starts with the push from 25th (nightly builds on 26th):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=d53a52b39a95&filter-searchStr=update-%20balrog

The whole picture can bee seen at this link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=update-&fromchange=f8283eaf5aad

We are interested in jobs ending with "u" (en-USu, 1.1u, 5u, etc) with "HTTP 400" errors in the summary.

On Oct 29 we stopped funsize (see https://bugzilla.mozilla.org/show_bug.cgi?id=1219879#c3 and https://bugzilla.mozilla.org/show_bug.cgi?id=1220252#c3)

It was turned back on Nov 2, Mon (https://bugzilla.mozilla.org/show_bug.cgi?id=1220857). Nov 2 and 3 were very terrible - tons of balrog submission errors.

Sounds like we degraded after the migration, but not sure where: networking, db, TC-to-VPN delays...
Attachment #8687173 - Attachment is obsolete: true

Updated

2 years ago
See Also: → bug 1224698
Back to the pool - I don't think that I can look at this until after Mozlando.
Assignee: rail → nobody

Comment 8

2 years ago
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 16

Platform breakdown:
* windows8-32: 6
* linux32: 6
* windows8-64: 3
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2015-11-20&endday=2015-11-20&tree=all

Comment 9

2 years ago
83 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 83

Platform breakdown:
* windows8-32: 28
* linux32: 22
* linux64: 18
* windows8-64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2015-11-16&endday=2015-11-22&tree=all
Varun is actively working on this!
Assignee: nobody → varunj.1011

Comment 11

a year ago
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/18047698e064bff5af1459323fe943daf6c5175a
bug 1223872: merge blob updates on server when safe to do so (#93). r=bhearsum
67 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 67

Platform breakdown:
* linux64: 33
* linux32: 30
* windowsxp: 3
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-08&endday=2016-07-08&tree=all
67 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 67

Platform breakdown:
* linux64: 33
* linux32: 30
* windowsxp: 3
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-04&endday=2016-07-10&tree=all
This is in production I think, but we still hit some issues around old_data_version. eg over the weekend 

eg https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-searchStr=l10n&exclusion_profile=false&selectedJob=4293214
https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-central#L24037

Could you investigate Varun ?
(Assignee)

Comment 15

a year ago
(In reply to Nick Thomas [:nthomas] from comment #14)
> This is in production I think, but we still hit some issues around
> old_data_version. eg over the weekend 
> 
> eg
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-
> searchStr=l10n&exclusion_profile=false&selectedJob=4293214
> https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-
> central#L24037
> 
> Could you investigate Varun ?

Yes, I'm on it!
(In reply to Nick Thomas [:nthomas] from comment #14)
> This is in production I think, but we still hit some issues around
> old_data_version. eg over the weekend 
> 
> eg
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-
> searchStr=l10n&exclusion_profile=false&selectedJob=4293214
> https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-
> central#L24037
> 
> Could you investigate Varun ?

It's a bit confusing right now actually. CloudOps is running code with this, but we haven't moved admin traffic over to them yet. The WebOps admin box is still running older code, so this is effectively not in production yet. We should be cutting over admin later this week, so hopefully we'll have this in production sometime next week.
That makes more sense. Perhaps we should update the webops admin soon, so that we're not changing hosting and code when we swap over to CloudOps.
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 21

Platform breakdown:
* osx-10-10: 11
* windows8-64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-11&endday=2016-07-11&tree=all
113 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 113

Platform breakdown:
* linux32: 54
* linux64: 36
* osx-10-10: 13
* windowsxp: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-12&endday=2016-07-12&tree=all
66 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 66

Platform breakdown:
* linux32: 33
* linux64: 32
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-13&endday=2016-07-13&tree=all
206 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 206

Platform breakdown:
* linux32: 89
* linux64: 71
* osx-10-10: 24
* windowsxp: 12
* windows8-64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-11&endday=2016-07-17&tree=all
This landed in production awhile ago. Thanks Varun!
Status: NEW → RESOLVED
Last Resolved: 11 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.