1367054 - add support for memory in SYSTEM_CAPABILITIES field

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement, P1)

Description

[[e-mail] (:bhearsum)]

We'll need this to do 32 -> 64 bit Windows updates. bug 1367050 has a couple of ideas about to support doing < and > querying while inside the SYSTEM_CAPABILITIES field.

Comment 1

[Chris Peterson [:cpeterson]]

We will need this functionality when we migrate 32-bit Firefox users to 64-bit as part of the Firefox 56 update process (September 26).

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Chris Peterson [:cpeterson] from comment #1)
> We will need this functionality when we migrate 32-bit Firefox users to
> 64-bit as part of the Firefox 56 update process (September 26).

Shouldn't be an issue. I'll probably be picking this up in the next month or so.

Comment 3

[[e-mail] (:bhearsum)]

Example update URL pulled from recent Balrog logs:
https://aus5.mozilla.org:443/update/6/Firefox/55.0a1/20170605030204/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0%20(x64)(noBug1296630v1)(nowebsense)/SSE3,16331/default/default/update.xml

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum (:bhearsum) from comment #3)
> Example update URL pulled from recent Balrog logs:
> https://aus5.mozilla.org:443/update/6/Firefox/55.0a1/20170605030204/
> WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.
> 0%20(x64)(noBug1296630v1)(nowebsense)/SSE3,16331/default/default/update.xml

In bug 1373367 we're going to change the URLs to prefix the values in the SYSTEM_CAPABILITIES field. We'll need to support the above as well as:
https://aus5.mozilla.org/update/6/Firefox/55.0a1/20170605030204/WINNT_x86_64-msvc-x64/en-US/nightly/Windows_NT%2010.0.0.0%20(x64)(noBug1296630v1)(nowebsense)/ISET:SSE3,MEM:16331/default/default/update.xml

Comment 6

[[e-mail] (:bhearsum)]

Attachment: add support for memory

Comment 7

[[e-mail] (:bhearsum)]

Nick smartly pointed out that the original patch cannot be safely landed in production (the code + db changes are incompatible). I've put up 3 new PRs that essentially spread the landing out into graceful steps (add new columns, switch to new columns, remove old column). More description in the PRs. No huge rush on looking at these Nick - we won't be doing a push this week because of ops availability.
https://github.com/mozilla/balrog/pull/348
https://github.com/mozilla/balrog/pull/349
https://github.com/mozilla/balrog/pull/350

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

All r+, nice patch wrangling.

Comment 9

[[e-mail] (:bhearsum)]

The majority of this is in production - we now have support for "memory" in the update ping. The only thing left to do is remove the old systemCapabilities column, which I expect to happen next week.

Comment 10

[[e-mail] (:bhearsum)]

Unfortunately, part 3 of this blew up when we tried to deploy today. It turns out that we do a "select * from rules where ..." when we get the rules for an update query. It appears that because the SQLAlchemy model in db.py still has a reference to systemCapabilities, it isn't happy if that column doesn't exist in the db. We end up with an exception like:
balrogpub_1    | [2017-07-26 19:28:50,978] ERROR in app: Exception on /update/3/Firefox/33.0/20141202185629/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/default/default/default/update.xml [GET]
balrogpub_1    | Traceback (most recent call last):
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
balrogpub_1    |     response = self.full_dispatch_request()
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
balrogpub_1    |     rv = self.handle_user_exception(e)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
balrogpub_1    |     reraise(exc_type, exc_value, tb)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
balrogpub_1    |     rv = self.dispatch_request()
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
balrogpub_1    |     return self.view_functions[rule.endpoint](**req.view_args)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 66, in wrapper
balrogpub_1    |     response = function(request)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/validation.py", line 293, in wrapper
balrogpub_1    |     return function(request)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/produces.py", line 38, in wrapper
balrogpub_1    |     response = function(request)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/response.py", line 85, in wrapper
balrogpub_1    |     response = function(request)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/decorator.py", line 42, in wrapper
balrogpub_1    |     response = function(request)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/connexion/decorators/parameter.py", line 195, in wrapper
balrogpub_1    |     return function(**kwargs)
balrogpub_1    |   File "./auslib/web/public/client.py", line 100, in get_update_blob
balrogpub_1    |     release, update_type = AUS.evaluateRules(query)
balrogpub_1    |   File "./auslib/AUS.py", line 62, in evaluateRules
balrogpub_1    |     fallbackChannel=getFallbackChannel(updateQuery['channel'])
balrogpub_1    |   File "./auslib/db.py", line 1676, in getRulesMatchingQuery
balrogpub_1    |     rules = cache.get("rules", cache_key, getRawMatches)
balrogpub_1    |   File "./auslib/util/cache.py", line 61, in get
balrogpub_1    |     value = value_getter()
balrogpub_1    |   File "./auslib/db.py", line 1666, in getRawMatches
balrogpub_1    |     return self.select(where=where, transaction=transaction)
balrogpub_1    |   File "./auslib/db.py", line 37, in convertRows
balrogpub_1    |     for row in fn(*args, **kwargs):
balrogpub_1    |   File "./auslib/db.py", line 356, in select
balrogpub_1    |     return trans.execute(query).fetchall()
balrogpub_1    |   File "./auslib/db.py", line 356, in select
balrogpub_1    |     return trans.execute(query).fetchall()
balrogpub_1    |   File "./auslib/db.py", line 199, in execute
balrogpub_1    |     return self.conn.execute(statement)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1358, in execute
balrogpub_1    |     params)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
balrogpub_1    |     compiled_sql, distilled_params
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1599, in _execute_context
balrogpub_1    |     context)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1592, in _execute_context
balrogpub_1    |     context)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 325, in do_execute
balrogpub_1    |     cursor.execute(statement, parameters)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
balrogpub_1    |     self.errorhandler(self, exc, value)
balrogpub_1    |   File "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
balrogpub_1    |     raise errorclass, errorvalue
balrogpub_1    | TransactionError: ('(OperationalError) (1054, "Unknown column \'rules.systemCapabilities\' in \'field list\'")',)

I suspect that we can either roll out the new code before migrating the db, or split this up yet again -- and remove the db.py refererence in the next deploy, and then finally drop the column in a subsequent one. I'll need to some more testing to be sure.

Comment 11

[[e-mail] (:bhearsum)]

I did some more testing and thinking about this, and all we need to do is make sure only the new code is running before doing the db migration next time. Right now we've got balrog stage running the old database + new code, and it's working fine. This lines up with the notes on https://wiki.mozilla.org/Balrog#Schema_Upgrades about destructive upgrades needing to happen after the code is fully deployed.

Comment 12

[Chris Peterson [:cpeterson]]

(In reply to Ben Hearsum (:bhearsum) from comment #11)
> I did some more testing and thinking about this, and all we need to do is
> make sure only the new code is running before doing the db migration next
> time. Right now we've got balrog stage running the old database + new code,
> and it's working fine.

Ben, what is the current state of Balrog support for checking client memory size? When we migrate 32-bit Firefox users to 64-bit (sometime during Beta 56 and then on the Release channel in a post-56.0 update, something like a 56.0.1), we would like to only migrate users with >= 1800 MB. 32-bit users with < 1800 MB should stay with 32-bit.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Chris Peterson [:cpeterson] from comment #12)
> (In reply to Ben Hearsum (:bhearsum) from comment #11)
> > I did some more testing and thinking about this, and all we need to do is
> > make sure only the new code is running before doing the db migration next
> > time. Right now we've got balrog stage running the old database + new code,
> > and it's working fine.
> 
> Ben, what is the current state of Balrog support for checking client memory
> size? When we migrate 32-bit Firefox users to 64-bit (sometime during Beta
> 56 and then on the Release channel in a post-56.0 update, something like a
> 56.0.1), we would like to only migrate users with >= 1800 MB. 32-bit users
> with < 1800 MB should stay with 32-bit.

We can do that. We support <, <=, >, and >= operations on memory in the latest code. This should be in production either this week or next.

Comment 14

[[e-mail] (:bhearsum)]

We finally got the final database migration for this in production.