Closed Bug 1333876 Opened 7 years ago Closed 7 years ago

scheduled change history sometimes fails to load because of IndexError

Categories

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: ninad101)

References

Details

(Whiteboard: [lang=python][ready])

Found this while trying to debug bug 1333874.

We get this Traceback when trying to load the history for a recent Scheduled Change:
IndexError: list index out of range
  File "flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "flask/views.py", line 149, in dispatch_request
    return meth(*args, **kwargs)
  File "auslib/admin/views/scheduled_changes.py", line 219, in get
    r.update(cond[0])

I think this means that the the conditions history & regular scheduled changes history are getting out of sync in their data_version. This shouldn't be possible - otherwise it's very difficult to see what the conditions for a scheduled change where at any given point in time.

Haven't dug into this, but a couple of possibilties to investigate:
1) Retention policy script may be deleting some rows from one or both of the history tables
2) mergeUpdate may not cause an update to the conditions table

Could be something else, but those are my two current theories.
Priority: P2 → P1
This can be reproduced on http://localhost:8080/scheduled_changes/rules/14
with the latest production dump on 27 January 2017
On querying DB I found that  the rules_scheduled_changes_conditions_history and rules_scheduled_changes_history have different numbers of columns with different change_id for same sc_id for the scheduled_history which have this problem
This is true for sc_id=12 and sc_id=14

Output for sc_id=14 bellow:

mysql> select * from   rules_scheduled_changes_conditions_history  where sc_id=14;
+-----------+---------------------+-------+-------------------+-------------------+------------------+--------------+---------------+---------------+
| change_id | changed_by          | sc_id | telemetry_product | telemetry_channel | telemetry_uptake | data_version | when          | timestamp     |
+-----------+---------------------+-------+-------------------+-------------------+------------------+--------------+---------------+---------------+
|        32 | raliiev@mozilla.com |    14 | NULL              | NULL              |             NULL |         NULL |          NULL | 1481646376281 |
|        33 | raliiev@mozilla.com |    14 | NULL              | NULL              |             NULL |            1 | 1481821200000 | 1481646376282 |
+-----------+---------------------+-------+-------------------+-------------------+------------------+--------------+---------------+---------------+
2 rows in set (0.00 sec)

mysql> select * from  rules_scheduled_changes_history where sc_id=14;
+-----------+---------------------+-------+---------------------+----------+--------------+--------------+---------------+-----------------------+---------------------+------------------+--------------+--------------+--------------+------------------+--------------+-------------+----------------+-------------------------+-------------------+------------------+-------------------------+---------------------------------------------------------+----------------+-----------------+-------------------+---------------+----------------------+-------------+
| change_id | changed_by          | sc_id | scheduled_by        | complete | data_version | base_rule_id | base_priority | base_mapping          | base_backgroundRate | base_update_type | base_product | base_version | base_channel | base_buildTarget | base_buildID | base_locale | base_osVersion | base_systemCapabilities | base_distribution | base_distVersion | base_headerArchitecture | base_comment                                            | base_whitelist | base_alias      | base_data_version | timestamp     | base_fallbackMapping | change_type |
+-----------+---------------------+-------+---------------------+----------+--------------+--------------+---------------+-----------------------+---------------------+------------------+--------------+--------------+--------------+------------------+--------------+-------------+----------------+-------------------------+-------------------+------------------+-------------------------+---------------------------------------------------------+----------------+-----------------+-------------------+---------------+----------------------+-------------+
|        38 | raliiev@mozilla.com |    14 | NULL                |        0 |         NULL |         NULL |          NULL | NULL                  |                NULL | NULL             | NULL         | NULL         | NULL         | NULL             | NULL         | NULL        | NULL           | NULL                    | NULL              | NULL             | NULL                    | NULL                                                    | NULL           | NULL            |              NULL | 1481646376277 | NULL                 | insert      |
|        39 | raliiev@mozilla.com |    14 | raliiev@mozilla.com |        0 |            1 |          145 |            90 | Firefox-50.1.0-build2 |                 100 | minor            | Firefox      | NULL         | release      | NULL             | NULL         | NULL        | NULL           | NULL                    | NULL              | NULL             | NULL                    | default release rule updated by buildbot, DO NOT DELETE | NULL           | firefox-release |               107 | 1481646376278 | NULL                 | update      |
|        42 | raliiev@mozilla.com |    14 | raliiev@mozilla.com |        1 |            2 |          145 |            90 | Firefox-50.1.0-build2 |                 100 | minor            | Firefox      | NULL         | release      | NULL             | NULL         | NULL        | NULL           | NULL                    | NULL              | NULL             | NULL                    | default release rule updated by buildbot, DO NOT DELETE | NULL           | firefox-release |               107 | 1481821219924 | NULL                 | update      |
+-----------+---------------------+-------+---------------------+----------+--------------+--------------+---------------+-----------------------+---------------------+------------------+--------------+--------------+--------------+------------------+--------------+-------------+----------------+-------------------------+-------------------+------------------+-------------------------+---------------------------------------------------------+----------------+-----------------+-------------------+---------------+----------------------+-------------+
Working on this today.
Assignee: nobody → bhat.ninadmb
Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/a2f13d36ae7d1b1131a633ffe6b9d61c9c911fcd
Bug 1333876 - scheduled change history sometimes fails to load because of IndexError (#255). r=bhearsum
Depends on: 1341435
Thank you for fixing this Ninad!
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: Release Engineering → Release Engineering Graveyard
You need to log in before you can comment on or make changes to this bug.