692452 - creating a 2.6.x branch with a few hot fixes

Status: VERIFIED WONTFIX

(Cloud Services :: Server: Core, defect)

Comment 1

[Tarek Ziadé (:tarek)]

also, backported http://hg.mozilla.org/services/server-core/rev/a6dc1d61bb30

Comment 2

[Tarek Ziadé (:tarek)]

I believe we have everything we wanted in that branch.

Comment 3

[Tarek Ziadé (:tarek)]

rpm-2.6.2 tagged

Comment 4

[Pete Fritchman [:petef]]

(In reply to Tarek Ziadé (:tarek) from comment #3)
> rpm-2.6.2 tagged

to be consistent, shouldn't we tag it "rpm-2.6.2-1"?

Comment 5

[Pete Fritchman [:petef]]

and we also need a version bump in the rpm spec file

diff -r 18a9ad1ab1db Services.spec
--- a/Services.spec     Mon Oct 10 18:13:52 2011 +0200
+++ b/Services.spec     Mon Oct 10 15:43:37 2011 -0700
@@ -1,6 +1,6 @@
 %define name python26-services
 %define pythonname Services
-%define version 2.6
+%define version 2.6.2
 %define unmangled_version %{version}
 %define release 1

Comment 6

[Pete Fritchman [:petef]]

failing loadtest.  lots of these:

2011-10-10 19:57:18,197 ERROR [syncserver] Could not set items
2011-10-10 19:57:18,197 ERROR [syncserver] (KeyError) 110 None None

and these:
2011-10-10 19:57:19,368 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 304, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 187, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 221, in __call__
    response = self._dispatch_request(request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 273, in _dispatch_request
    response = self._dispatch_request_with_match(request, match)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 299, in _dispatch_request_with_match
    result = function(request, **params)
  File "/usr/lib/python2.6/site-packages/syncstorage/controller.py", line 223, in get_collection
    kw.get('sort'))
  File "/usr/lib/python2.6/site-packages/syncstorage/storage/memcachedsql.py", line 161, in get_items
    fields, filters, limit, offset, sort)
  File "/usr/lib/python2.6/site-packages/syncstorage/storage/sql.py", line 537, in get_items
    res = safe_execute(self._engine, query)
  File "/usr/lib/python2.6/site-packages/services/util.py", line 334, in safe_execute
    return engine.execute(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1787, in execute
    connection = self.contextual_connect(close_with_result=True)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1829, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 182, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 369, in __init__
    rec = self._connection_record = pool.get()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 213, in get
    return self.do_get()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 732, in do_get
    con = self.create_connection()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 147, in create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 253, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 319, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 82, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 249, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.6/site-packages/pymysql/__init__.py", line 93, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 514, in __init__
    self._connect()
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 667, in _connect
    self._get_server_information()
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 838, in _get_server_information
    self.server_charset = charset_by_id(self.server_language).name
  File "/usr/lib/python2.6/site-packages/pymysql/charset.py", line 173, in charset_by_id
    return _charsets.by_id(id)
  File "/usr/lib/python2.6/site-packages/pymysql/charset.py", line 21, in by_id
    return self._by_id[id]
DBAPIError: (KeyError) 110 None None

Comment 7

[Tarek Ziadé (:tarek)]

That does not look at all like the 2.6.1 branch deployed.

For instance, in the TB

File "/usr/lib/python2.6/site-packages/services/util.py", line 334, in safe_execute
    return engine.execute(*args, **kwargs)

but line 334 is: http://hg.mozilla.org/services/server-core/file/9373252ad415/services/util.py#l334

Comment 8

[Tarek Ziadé (:tarek)]

There seem to be a bug in the buildtool wrt branches.

For the time being here's how to build the server-core RPM : http://tarek.pastebin.mozilla.org/1352826

Comment 9

[Tarek Ziadé (:tarek)]

build issue filed at bug 693611

Comment 10

[Tarek Ziadé (:tarek)]

Stuff to try:

1. Check that the ldap pool gets back on track when the ldap server goes down or is restarted

* send load to frontend
* stop load
* restart ldap slave
* sleep(30)
* start load

2. check that the lifetime feature helps to correctly spreads the load between several ldap nodes when they are added or removed

3. check that when mysql goes down on high load, all 2013 errors are caught and the pool back on track

4. check that there are no more unicode error when the password is non-utf8 garbage, but a proper 401

5. check that the backend errors gives us the original request path + method

Comment 11

[Pete Fritchman [:petef]]

(In reply to Tarek Ziadé (:tarek) from comment #10)
> Stuff to try:
> 
> 1. Check that the ldap pool gets back on track when the ldap server goes
> down or is restarted
> 
> * send load to frontend
> * stop load
> * restart ldap slave
> * sleep(30)
> * start load

pass

> 2. check that the lifetime feature helps to correctly spreads the load
> between several ldap nodes when they are added or removed

fail

I have ldap_max_lifetime under [storage] set to 300, and it's not working. I restarted a slave in the ldap-slave pool and it got no traffic (for 30 minutes) until I manually kicked/restarted the gunicorn instances.

> 3. check that when mysql goes down on high load, all 2013 errors are caught
> and the pool back on track

I did the same test as #1; run load to get all the connection pools going, restart a bunch of MySQLs, and start up the loadtest again.

> 4. check that there are no more unicode error when the password is non-utf8
> garbage, but a proper 401

when you say "non-utf8 garbage", do you mean regular ASCII text that is the incorrect password?  If not, can you suggest a way to test this (or just test against stage)?

> 5. check that the backend errors gives us the original request path + method

LDAP Connector (disconnected) - uri: 'ldap://ldap-slave.web.scl2.stage.svc.mozilla.com'

I am seeing stuff like that along with a BackendError, but the entire TB itself doesn't have the original request URI/method in the log.

Comment 12

[Tarek Ziadé (:tarek)]

I think this branch went to the limbs

Comment 13

[James Bonacci [:jbonacci]]

OK with that. Marking as Verified.