1395128 - schedule publishing in balrog fails: Changes may not be scheduled in the past

Status: RESOLVED DUPLICATE of bug 1389312

(Release Engineering :: Release Automation: Other, defect, P1)

Description

[Johan Lorenzo [:jlorenzo]]

Similar to bug 1389312 and bug 1388330, "schedule publishing in balrog" failed in 56.0b7[1]. The technical reason is the following: 

> 02:28:45     INFO -  Balrog request to https://aus4-admin.mozilla.org/api/scheduled_changes/rules
> 02:28:45     INFO -  Data sent: {u'comment': None, 'change_type': 'update', u'fallbackMapping': u'Firefox-56.0b6-build2', u'buildID': None, u'backgroundRate': 100, u'product': u'Firefox', u'instructionSet': None, u'mapping': 'Firefox-56.0b7-build2', u'alias': u'firefox-beta', u'headerArchitecture': None, u'buildTarget': None, 'data_verison': 336, u'osVersion': None, u'distVersion': None, u'mig64': None, u'data_version': 336, 'when': 1504022400000, u'priority': 90, u'locale': None, u'version': None, u'update_type': u'minor', u'memory': None, u'distribution': None, u'rule_id': '32', u'channel': u'beta'}
> 02:28:45     INFO -  "POST /api/scheduled_changes/rules HTTP/1.1" 400 116
> 02:28:45     INFO -  Caught HTTPError: {"status": 400, "type": "about:blank", "detail": "Changes may not be scheduled in the past", "title": "Bad Request"}

The human reason is that we got delayed in signing off builds. 

I worked it around by submitting a new task manually[2]. For the record, I had to change the timestamp at 2 places in the task definition.

I wonder if we shouldn't just change the worker to take the current timestamp (on beta), instead of the what was defined in ship-it. However, I don't know if this value is used elsewhere, though

[1] https://tools.taskcluster.net/groups/xzIcLIHqSL2rEcHsUf9j9Q/tasks/hXXWT9JxRAuENUXvkL-pCQ/runs/1/logs/public%2Flogs%2Flive_backing.log.gz
[2] https://tools.taskcluster.net/groups/xzIcLIHqSL2rEcHsUf9j9Q/tasks/G1L8zR45TGGbhz4QOe4g5A/details

Comment 1

[[e-mail] (:bhearsum)]

I don't think hardcoding special behaviour for Beta is the best fix here -- this is something that can happen on any channel if the Task runs after the release eta has passed.

My suggestion is to change the balrog submitter to take the more recent of "release eta" and "5 seconds from now". This should fix the issue for all channels, and will ensure that release eta is respected whenever possible on Beta.