Single locale release updates should not race each other

RESOLVED FIXED

Status

Release Engineering
Balrog: Backend
RESOLVED FIXED
2 years ago
9 months ago

People

(Reporter: rail, Assigned: vjoshi)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
We see a lot of races in funsize similar to this:

l) locale A requests data_version
2) locale B requests data_version
3) locale B updates the release blob
4) locale A fails to update the release blob

They probably don't update the same data in the blob, so we can probably find a work around for this.
(Reporter)

Comment 1

2 years ago
Created attachment 8687173 [details] [diff] [review]
balrog_metrics-tools-4.diff

Let's start with some basic request metrics for now.

We can walk the TC tasks using the index, fetch their logs, process with something like https://github.com/etsy/logster (would require some custom parser) and submit to graphite.
Attachment #8687173 - Flags: review?(catlee)
(Reporter)

Updated

2 years ago
Assignee: nobody → rail
(Reporter)

Comment 2

2 years ago
BTW, this is an issue for l10n repacks as well know. It's all suddenly started on Tue Nov 3, see https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=bc9c6e996006&filter-searchStr=l10n&exclusion_profile=false

I wonder if it is somehow related to all recent  DB/webhead migrations...
(Reporter)

Comment 3

2 years ago
According to the webapp/db PHX1-SCL3 dashboard, aus migrated on Monday, Oct 26
(Reporter)

Comment 4

2 years ago
Created attachment 8687210 [details] [diff] [review]
balrog_metrics-tools-5.diff

req.elapsed is not that good from what I see. It's usually less than a second while the surrounding log lines are 10-15 secs away. Not sure why, let's go with old school way.
Attachment #8687173 - Attachment is obsolete: true
Attachment #8687173 - Flags: review?(catlee)
Attachment #8687210 - Flags: review?(catlee)

Updated

2 years ago
Attachment #8687173 - Attachment is obsolete: false

Updated

2 years ago
Attachment #8687210 - Flags: review?(catlee) → review+
(Reporter)

Comment 5

2 years ago
Comment on attachment 8687210 [details] [diff] [review]
balrog_metrics-tools-5.diff

https://hg.mozilla.org/build/tools/rev/58900072a047
Attachment #8687210 - Flags: checked-in+
(Reporter)

Updated

2 years ago
Depends on: 1224674
(Reporter)

Comment 6

2 years ago
Looks like this is related to the aus migration, which according to the dashboard happened on Oct 26.

It starts with the push from 25th (nightly builds on 26th):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=d53a52b39a95&filter-searchStr=update-%20balrog

The whole picture can bee seen at this link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=update-&fromchange=f8283eaf5aad

We are interested in jobs ending with "u" (en-USu, 1.1u, 5u, etc) with "HTTP 400" errors in the summary.

On Oct 29 we stopped funsize (see https://bugzilla.mozilla.org/show_bug.cgi?id=1219879#c3 and https://bugzilla.mozilla.org/show_bug.cgi?id=1220252#c3)

It was turned back on Nov 2, Mon (https://bugzilla.mozilla.org/show_bug.cgi?id=1220857). Nov 2 and 3 were very terrible - tons of balrog submission errors.

Sounds like we degraded after the migration, but not sure where: networking, db, TC-to-VPN delays...
(Reporter)

Updated

2 years ago
Attachment #8687173 - Attachment is obsolete: true

Updated

2 years ago
See Also: → bug 1224698
(Reporter)

Comment 7

2 years ago
Back to the pool - I don't think that I can look at this until after Mozlando.
Assignee: rail → nobody

Comment 8

2 years ago
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 16

Platform breakdown:
* windows8-32: 6
* linux32: 6
* windows8-64: 3
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2015-11-20&endday=2015-11-20&tree=all

Comment 9

2 years ago
83 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 83

Platform breakdown:
* windows8-32: 28
* linux32: 22
* linux64: 18
* windows8-64: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2015-11-16&endday=2015-11-22&tree=all
Varun is actively working on this!
Assignee: nobody → varunj.1011

Comment 11

11 months ago
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/18047698e064bff5af1459323fe943daf6c5175a
bug 1223872: merge blob updates on server when safe to do so (#93). r=bhearsum

Comment 12

11 months ago
67 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 67

Platform breakdown:
* linux64: 33
* linux32: 30
* windowsxp: 3
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-08&endday=2016-07-08&tree=all

Comment 13

11 months ago
67 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 67

Platform breakdown:
* linux64: 33
* linux32: 30
* windowsxp: 3
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-04&endday=2016-07-10&tree=all

Comment 14

11 months ago
This is in production I think, but we still hit some issues around old_data_version. eg over the weekend 

eg https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-searchStr=l10n&exclusion_profile=false&selectedJob=4293214
https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-central#L24037

Could you investigate Varun ?
(Assignee)

Comment 15

11 months ago
(In reply to Nick Thomas [:nthomas] from comment #14)
> This is in production I think, but we still hit some issues around
> old_data_version. eg over the weekend 
> 
> eg
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-
> searchStr=l10n&exclusion_profile=false&selectedJob=4293214
> https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-
> central#L24037
> 
> Could you investigate Varun ?

Yes, I'm on it!
(In reply to Nick Thomas [:nthomas] from comment #14)
> This is in production I think, but we still hit some issues around
> old_data_version. eg over the weekend 
> 
> eg
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> central&revision=bbb29a9b88dd680dbb59577cbe4dc6e58d117100&filter-
> searchStr=l10n&exclusion_profile=false&selectedJob=4293214
> https://treeherder.mozilla.org/logviewer.html#?job_id=4293214&repo=mozilla-
> central#L24037
> 
> Could you investigate Varun ?

It's a bit confusing right now actually. CloudOps is running code with this, but we haven't moved admin traffic over to them yet. The WebOps admin box is still running older code, so this is effectively not in production yet. We should be cutting over admin later this week, so hopefully we'll have this in production sometime next week.

Comment 17

11 months ago
That makes more sense. Perhaps we should update the webops admin soon, so that we're not changing hosting and code when we swap over to CloudOps.

Comment 18

11 months ago
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 21

Platform breakdown:
* osx-10-10: 11
* windows8-64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-11&endday=2016-07-11&tree=all

Comment 19

11 months ago
113 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 113

Platform breakdown:
* linux32: 54
* linux64: 36
* osx-10-10: 13
* windowsxp: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-12&endday=2016-07-12&tree=all

Comment 20

11 months ago
66 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 66

Platform breakdown:
* linux32: 33
* linux64: 32
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-13&endday=2016-07-13&tree=all

Comment 21

10 months ago
206 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 206

Platform breakdown:
* linux32: 89
* linux64: 71
* osx-10-10: 24
* windowsxp: 12
* windows8-64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1223872&startday=2016-07-11&endday=2016-07-17&tree=all
This landed in production awhile ago. Thanks Varun!
Status: NEW → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.