1030880 - import-buildbot-data.py fails with "TypeError: 'int' object is not iterable" in cursor.execute() after upgrade from MySQLdb 1.2.3c1 to 1.2.5

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Something happened around 09:12 PT that is causing TBPL to not pick up completed jobs. As a result, all trees were closed at 10:48 PT.

Comment 1

[:Atoll]

This line of code:

def add_run_to_db(run, db, overwrite):
    cursor = db.cursor()
    cursor.execute("SELECT count(*) FROM runs WHERE buildbot_id = %s", run.id)

works with MySQL-python 1.2.4, but fails under 1.2.5. More details coming.

Comment 2

[:Atoll]

(In reply to Richard Soderberg [:atoll] from comment #1)
> works with MySQL-python 1.2.4

works with MySQL-python 1.2.3 (sorry, misread)

Comment 3

[Jake Maul [:jakem]]

This was tracked down to be an upgrade to the MySQL-python package, from the RHEL6 stock 1.2.3 to a newly-packaged 1.2.5.

This caused the import script to throw this error:

[[e-mail]1 ~]# sh /data/genericrhel6/src/tbpl.mozilla.org/import-buildbot-data-cron.sh
Fetching http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz ...
Traversing runs and inserting into database...
Traceback (most recent call last):
  File "/data/genericrhel6/src/tbpl.mozilla.org/tbpl/dataimport/import-buildbot-data.py", line 423, in <module>
    main()
  File "/data/genericrhel6/src/tbpl.mozilla.org/tbpl/dataimport/import-buildbot-data.py", line 336, in main
    inserted_runs = do_recent(db, options.overwrite)
  File "/data/genericrhel6/src/tbpl.mozilla.org/tbpl/dataimport/import-buildbot-data.py", line 301, in do_recent
    return add_to_db("http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz", db, overwrite)
  File "/data/genericrhel6/src/tbpl.mozilla.org/tbpl/dataimport/import-buildbot-data.py", line 284, in add_to_db
    inserted_runs = filter(lambda x: x, [run if add_run_to_db(run, db, overwrite) else None for run in get_runs(j)])
  File "/data/genericrhel6/src/tbpl.mozilla.org/tbpl/dataimport/import-buildbot-data.py", line 231, in add_run_to_db
    cursor.execute("SELECT count(*) FROM runs WHERE buildbot_id = %s", run.id)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 187, in execute
    query = query % tuple([db.literal(item) for item in args])
TypeError: 'int' object is not iterable


The package has been rolled back, and puppet changed from ensure=>latest to ensure=>present.


Dropping prio since the problem is now resolved. Leaving open because we don't know why this change broke the job.

Comment 4

[Justin Wood (:Callek)]

Attachment: MySQLDB log and diff --stat from 1.2.3c1 --> 1.2.5

Comment 5

[Justin Wood (:Callek)]

Attachment: MySQLDb.diff

This is a diff from 1.2.3c1 --> 1.2.5

This issue blocks genericadm from upgrading mysqldb, I can't see anything obviously wrong with it --- ed can you peek and possibly help us identify why the upgrade broke tbpl so we can fix tbpl (and hopefully identify other places the code pattern could exist that the upgrade would also break)

Comment 6

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #5)
> This issue blocks genericadm from upgrading mysqldb, I can't see anything
> obviously wrong with it --- ed can you peek and possibly help us identify
> why the upgrade broke tbpl so we can fix tbpl (and hopefully identify other
> places the code pattern could exist that the upgrade would also break)

Sorry, I don't know MySQLdb well enough to say why this is happening.

Sheeri, I don't suppose you would mind taking a look?

They're attempting to update MySQLdb from 1.2.3c1 to 1.2.5 and are hitting the exception in comment 3, from this import-buildbot-data.py cursor.execute():

https://hg.mozilla.org/webtools/tbpl/file/5931379f7336/dataimport/import-buildbot-data.py#l231
def add_run_to_db(run, db, overwrite):
    cursor = db.cursor()
    cursor.execute("SELECT count(*) FROM runs WHERE buildbot_id = %s", run.id)

Comment 7

[Sheeri Cabral [:sheeri]]

How important is it to figure this out? Isn't tbpl going away in Q3?

However, it seems like "TypeError: 'int' object is not iterable" is pretty clear. I'm not sure why it works in one version and not the upgraded one, maybe the upgraded one cares more about the data types and says "you can't iterate over a number"? Is "args" an int here?

Comment 8

[Ed Morley [:emorley]]

(In reply to Sheeri Cabral [:sheeri] from comment #7)
> How important is it to figure this out? Isn't tbpl going away in Q3?

Ideally, yeah. I'm not too fussed; I don't know how urgently the others in this bug want to update the package.

> However, it seems like "TypeError: 'int' object is not iterable" is pretty
> clear. I'm not sure why it works in one version and not the upgraded one,
> maybe the upgraded one cares more about the data types and says "you can't
> iterate over a number"? Is "args" an int here?

'int not being iterable' is clear, but I don't understand how we get to that point (I didn't write this file, have just made the odd tweak here and there).

Anyway, I don't have a vested interest in this - I'll leave it to the others to follow up with you, if the package upgrade is urgent :-)

Comment 9

[Jake Maul [:jakem]]

The package update is not blocking anything. It can definitely wait for TBPL to die... or (more likely) simply wait for someone to request it be updated because their app needs the newer version.

Sounds like we've done our due diligence here. We looked at it, the fix was not obvious, nobody has a vested interest in digging into it further at this time, and it will resolve itself via obsolescence of TBPL soon anyway. That's good enough for me. I'll resolve this bug.