Closed Bug 692452 Opened 10 years ago Closed 10 years ago

creating a 2.6.x branch with a few hot fixes

Categories

(Cloud Services :: Server: Core, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

VERIFIED WONTFIX

People

(Reporter: tarek, Assigned: tarek)

References

Details

(Whiteboard: [qa+])

No description provided.
Depends on: 680324
Depends on: 691720
Depends on: 692789
I believe we have everything we wanted in that branch.
rpm-2.6.2 tagged
(In reply to Tarek Ziadé (:tarek) from comment #3)
> rpm-2.6.2 tagged

to be consistent, shouldn't we tag it "rpm-2.6.2-1"?
and we also need a version bump in the rpm spec file

diff -r 18a9ad1ab1db Services.spec
--- a/Services.spec     Mon Oct 10 18:13:52 2011 +0200
+++ b/Services.spec     Mon Oct 10 15:43:37 2011 -0700
@@ -1,6 +1,6 @@
 %define name python26-services
 %define pythonname Services
-%define version 2.6
+%define version 2.6.2
 %define unmangled_version %{version}
 %define release 1
failing loadtest.  lots of these:

2011-10-10 19:57:18,197 ERROR [syncserver] Could not set items
2011-10-10 19:57:18,197 ERROR [syncserver] (KeyError) 110 None None

and these:
2011-10-10 19:57:19,368 ERROR [syncserver] Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/services/util.py", line 304, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/paste/translogger.py", line 68, in __call__
    return self.application(environ, replacement_start_response)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 147, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/webob/dec.py", line 208, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 187, in __notified
    response = func(self, request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 221, in __call__
    response = self._dispatch_request(request)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 273, in _dispatch_request
    response = self._dispatch_request_with_match(request, match)
  File "/usr/lib/python2.6/site-packages/services/baseapp.py", line 299, in _dispatch_request_with_match
    result = function(request, **params)
  File "/usr/lib/python2.6/site-packages/syncstorage/controller.py", line 223, in get_collection
    kw.get('sort'))
  File "/usr/lib/python2.6/site-packages/syncstorage/storage/memcachedsql.py", line 161, in get_items
    fields, filters, limit, offset, sort)
  File "/usr/lib/python2.6/site-packages/syncstorage/storage/sql.py", line 537, in get_items
    res = safe_execute(self._engine, query)
  File "/usr/lib/python2.6/site-packages/services/util.py", line 334, in safe_execute
    return engine.execute(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1787, in execute
    connection = self.contextual_connect(close_with_result=True)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1829, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 182, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 369, in __init__
    rec = self._connection_record = pool.get()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 213, in get
    return self.do_get()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 732, in do_get
    con = self.create_connection()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 147, in create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 253, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/pool.py", line 319, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 82, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 249, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.6/site-packages/pymysql/__init__.py", line 93, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 514, in __init__
    self._connect()
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 667, in _connect
    self._get_server_information()
  File "/usr/lib/python2.6/site-packages/pymysql/connections.py", line 838, in _get_server_information
    self.server_charset = charset_by_id(self.server_language).name
  File "/usr/lib/python2.6/site-packages/pymysql/charset.py", line 173, in charset_by_id
    return _charsets.by_id(id)
  File "/usr/lib/python2.6/site-packages/pymysql/charset.py", line 21, in by_id
    return self._by_id[id]
DBAPIError: (KeyError) 110 None None
That does not look at all like the 2.6.1 branch deployed.

For instance, in the TB

File "/usr/lib/python2.6/site-packages/services/util.py", line 334, in safe_execute
    return engine.execute(*args, **kwargs)

but line 334 is: http://hg.mozilla.org/services/server-core/file/9373252ad415/services/util.py#l334
There seem to be a bug in the buildtool wrt branches.

For the time being here's how to build the server-core RPM : http://tarek.pastebin.mozilla.org/1352826
build issue filed at bug 693611
Stuff to try:

1. Check that the ldap pool gets back on track when the ldap server goes down or is restarted

* send load to frontend
* stop load
* restart ldap slave
* sleep(30)
* start load

2. check that the lifetime feature helps to correctly spreads the load between several ldap nodes when they are added or removed

3. check that when mysql goes down on high load, all 2013 errors are caught and the pool back on track

4. check that there are no more unicode error when the password is non-utf8 garbage, but a proper 401

5. check that the backend errors gives us the original request path + method
(In reply to Tarek Ziadé (:tarek) from comment #10)
> Stuff to try:
> 
> 1. Check that the ldap pool gets back on track when the ldap server goes
> down or is restarted
> 
> * send load to frontend
> * stop load
> * restart ldap slave
> * sleep(30)
> * start load

pass

> 2. check that the lifetime feature helps to correctly spreads the load
> between several ldap nodes when they are added or removed

fail

I have ldap_max_lifetime under [storage] set to 300, and it's not working. I restarted a slave in the ldap-slave pool and it got no traffic (for 30 minutes) until I manually kicked/restarted the gunicorn instances.

> 3. check that when mysql goes down on high load, all 2013 errors are caught
> and the pool back on track

I did the same test as #1; run load to get all the connection pools going, restart a bunch of MySQLs, and start up the loadtest again.

> 4. check that there are no more unicode error when the password is non-utf8
> garbage, but a proper 401

when you say "non-utf8 garbage", do you mean regular ASCII text that is the incorrect password?  If not, can you suggest a way to test this (or just test against stage)?

> 5. check that the backend errors gives us the original request path + method

LDAP Connector (disconnected) - uri: 'ldap://ldap-slave.web.scl2.stage.svc.mozilla.com'

I am seeing stuff like that along with a BackendError, but the entire TB itself doesn't have the original request URI/method in the log.
Whiteboard: [qa+]
I think this branch went to the limbs
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
OK with that. Marking as Verified.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.