1230451 - Data sync from Nucleus broken for Thunderbird 38.4.0 release notes

Status: VERIFIED FIXED

(Websites :: Nucleus, defect)

Description

[Josh Mize [:jgmize]]

Originally reported in bug 1227868, but this was not due to a database-specific replication issue. Unlike the situation in bug 1228083, which affected only the sync in scl3, this issue affected all clients. The specific error I found in the logs of all the bedrock clients was:

django.db.utils.IntegrityError: duplicate key value violates unique constraint "rna_release_product_version_key"
Key (product, version)=(Thunderbird, 38.4.0) already exists.

After looking through the logs, it looks like the Release with id 372 at one point had that product and version combination, but then was changed to a version of "reuseme", and then release id 373 was later created with version 38.4.0. Because of a race condition in the data sync protocol, the change of version for release id 372 came *after* the attempted creation of release id 373, which violated the database uniqueness constraints and caused the rnasync process to fail. 

Given time and opportunity there are several things I would like to do to improve the data sync protocol used between nucleus and bedrock, but due to other priorities at the present time I will have to live with taking occasional manual corrective actions, and this bug tracks the specific actions needed to fix the Thunderbird 38.4.0 release notes.

Comment 1

[Josh Mize [:jgmize]]

In an iPython shell on nucleus:

     In [3]: Release.objects.filter(product='Thunderbird', version='38.4.0').values('id')
     Out[3]: [{'id': 373L}]
     In [4]: r = Release.objects.filter(product='Thunderbird', version='38.4.0').get()

     In [5]: r.notes()
     Out[5]:
     ([<Note: Fixed issue where messages moves of multiple messages from a maildir folder to an mbox folder failed.>,
     <Note: Various [security fixes][1]

     [1]: https://www.mozilla.org/security/known-vulnerabilities/thunderbird/#thunderbird38.4>],
     [])
     In [9]: r.note_set.values('id')
     Out[9]: [{'id': 786367L}, {'id': 786368L}]

In a python shell on bedrock in scl3:

     >>> Release.objects.filter(product='Thunderbird', version='38.4.0').values('id')
     [{'id': 373L}]
     >>> r = Release.objects.get(id=373)
     >>> r.note_set.all()
     []
     >>> Note.objects.filter(id=786367)
     [<Note: Fixed issue where messages moves of multiple messages from a maildir folder to an mbox folder failed.>]
     >>> n = _.get()
     >>> n.releases.all()
     [<Release: Thunderbird reuseme  Release>]
     >>> n.releases.values('id')
     [{'id': 372L}]

bedrock on the us-west deis cluster:
     >>> r = Release.objects.get(product='Thunderbird', version='38.4.0')
     >>> r.id
     373
     >>> r.note_set.all()
     []
     >>> n = Note.objects.get(id=786367)
     >>> n.releases.values()
     [{'product': u'Thunderbird', 'bug_list': u'', 'created': datetime.datetime(2015, 11, 23, 19, 44, 4, tzinfo=<UTC>), 'text': u'', 'release_date': datetime.datetime(2$
     >>> n.releases.values('id')
     [{'id': 372}]
     >>> n2 = Note.objects.get(id=786368)
     >>> n2.releases.values('id')
     [{'id': 372}]

quick fix: update modified timestamps in nucleus:
     In [1]: r = Release.objects.get(product='Thunderbird', version='38.4.0')

     In [2]: r.modified
     Out[2]: datetime.datetime(2015, 12, 3, 20, 53, 35, tzinfo=<UTC>)

     In [3]: r.save()

     In [4]: r.modified
     Out[4]: datetime.datetime(2015, 12, 4, 6, 44, 31, 248179, tzinfo=<UTC>)

     In [5]: for n in r.note_set.all():
     ...:     print(n.id, n.modified)
     ...:     n.save()
     ...:
     (786367L, datetime.datetime(2015, 11, 24, 17, 36, 19, tzinfo=<UTC>))
     (786368L, datetime.datetime(2015, 12, 1, 21, 17, 15, tzinfo=<UTC>))

This was only necessary because bedrock does not yet have the latest version of rnasync: a --force option was added in https://github.com/mozilla/rna/pull/54 which could have been used on each bedrock deployment to resolve this issue. Filed bug 1230452 to track that upgrade.

https://www.mozilla.org/en-US/thunderbird/38.4.0/releasenotes/ now has the notes matching those in nucleus.

Comment 2

[Wayne Mery (:wsmwk)]

jgmize, thank you.
The release notes are indeed now correct