850945 - Firefox 20.0b5 on ship-it can be marked as Ready even after the release started

Status: RESOLVED INCOMPLETE

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Rail Aliiev [:rail]]

See http://people.mozilla.org/~raliiev/sattap/d572f718.png

Comment 1

[[e-mail] (:bhearsum)]

Nothing in release runner logs about this:
2013-03-13 16:22:35,735 - INFO - updating status for Fennec-20.0b5-build1 to Running sendchange command
change sent successfully
2013-03-13 16:22:36,305 - INFO - mark as completed Fennec-20.0b5-build1
2013-03-13 16:22:36,406 - INFO - updating status for Firefox-20.0b5-build1 to Running sendchange command
change sent successfully
2013-03-13 16:22:37,052 - INFO - mark as completed Firefox-20.0b5-build1
Buildbot version: 0.8.7p1
Twisted version: 12.3.0

I'm supposed to be getting direct access to the production server logs, I'll look at them when I do.

Comment 2

[[e-mail] (:bhearsum)]

So, these rows are in a state where they're not marked as ready, but *are* marked as complete:
mysql> select ready, complete from fennec_release where name='Fennec-20.0b5-build1';
+-------+----------+
| ready | complete |
+-------+----------+
|     0 |        1 |

Which explains why they show up the way they do on ship it. I have no explanation as to how they got into this state though....I looked over the release runner logs again and I don't see anything in there suggesting that they got marked as failed. There's also no way to mark something as not ready from the web interface. I wonder if there was some db event around this time and we lost some data?

One thing that would've been helpful is to have changes noted in the app log - I'll see if I can make that happen.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: log changes

Comment 4

[[e-mail] (:bhearsum)]

I've done all the debugging I can - I manually marked these as ready in the db.

Comment 5

[[e-mail] (:bhearsum)]

I asked IT: DB to look into whether there was a database issue. Tracked in bug 852995.

Comment 6

[[e-mail] (:bhearsum)]

Unfortunately, db ops didn't have logs that far back. Leaving open for now because this is so recent, but we should close next month if we haven't seen it again.

Comment 7

[[e-mail] (:bhearsum)]

Haven't seen this again.