744405 - provide json support in talos to serialize results data

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Right now we are doing a lot of work to serialize our results for uploading.  We could simplify our code greatly by using json proper.  

My understanding is this cannot happen until we are on >python 2.4 (i.e python 2.7).

Comment 1

[Jeff Hammel]

the json module is part of the stdlib  in python 2.6.  There are two alternatives to waiting for this:

1. we can depend on the simplejson module which would then be deployed with puppet (I assumed).  It is my understanding this is already on all of the machines, but perhaps not for the python that talos uses?

2. Conceivably, we could follow the pattern for PyYAML in create_talos_zip.py:

http://hg.mozilla.org/build/talos/file/01c4a976841b/create_talos_zip.py#l26

I haven't tested this with the simplejson module but there is no reason it shouldn't work (unless they require compilation which IIRC they don't; it is optional)

So we shouldn't make python 2.7 a blocker.  If we need this sooner there are other ways to go

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, create_talos_zip.py is a good solution for this:)

Comment 3

[Armen [:armenzg]]

I use pythonWithSimpleJson() [1] in a couple of places:
  7836    command=[self.pythonWithJson(self.OS), 'talos_from_code.py', \

We have simplejson in *all* testing slaves.

[1]
http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l7518

Comment 4

[Jeff Hammel]

It was my understanding that we still can't run talos on this python because of the pywin32 dependency.  Is this correct, or can we start using talos+json now?

Comment 5

[Armen [:armenzg]]

IIUC you can verify this on the tools slaves that you guys have.
Some of the slaves have python 2.5 and others 2.6/2.7.
That function only points specifically to those versions so I can use json or simplejson for talos_from_code.py.

I do this for talos_from_code.py:
try:
    import json
except:
    import simplejson as json

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

for talos on win32, we use specify python2.4 which doesn't have simplejson or json.  All python 2.5+ is good though.

Comment 7

[Armen [:armenzg]]

On Windows I pass with that function this python:
C:\\mozilla-build\\python25\\python.exe
rather than what the talos-step runs:
C:\\Python24\\python.exe

This means that within the same job I used two different python versions.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

so we are looking to run json from inside the talos harness, I think we either need to add this support in via create_talos_zips.py or wait to upgrade to python 2.7

Comment 9

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #8)
> so we are looking to run json from inside the talos harness, I think we
> either need to add this support in via create_talos_zips.py or wait to
> upgrade to python 2.7

I agree, and while I'd love to think that python 2.7 is coming soon, realistically I think it is not.

The other fly in my ointment is that the latest simplejson does *not* work on python 2.4.  So the first task is to find a version of simplejson that works on python 2.4 just for create_talos_zip.py .

Comment 10

[Jeff Hammel]

Doing some detective work, it looks like simplejson==2.1.6 is the last version to support python 2.4:

http://stackoverflow.com/questions/9707147/syntaxerror-invalid-syntax-while-installing-simplejson-2-4-line-390-python-2

http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.6.tar.gz#md5=2f8351f6e6fe7ef25744805dfa56c0d5

I have imported this successfully in python 2.4 so I think we're good to go :)

Comment 11

[Jeff Hammel]

The only thing initially (there is also amo results, but let's ignore those now since we've been so good at ignoring them so far) that needs json/simplejson is DatazillaOutput

We can either:

A) import json/simplejson at the top of the file and mandate it
- or -
B) enclose the definition of DatazillaOutput in a try block a la AMOResults: http://hg.mozilla.org/build/talos/file/ef760c9dbc36/talos/output.py#l418

I really don't have a strong preference nor will either choice affect production or how users should run things. Any thoughts, :jmaher?

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I vote for top of the file.

Comment 13

[Jeff Hammel]

Attachment: point of reference: ts.json before utilizing simplejson

Comment 14

[Jeff Hammel]

Attachment: point of reference: tsvg.json before utilizing simplejson

Comment 15

[Jeff Hammel]

Attachment: use simplejson/json to serialize

Comment 16

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #15)
> Created attachment 637987 [details] [diff] [review]
> use simplejson/json to serialize

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=5eefb46c4b73

Comment 17

[Jeff Hammel]

This should also fix https://bugzilla.mozilla.org/show_bug.cgi?id=769484

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 637987 [details] [diff] [review]
use simplejson/json to serialize

Review of attachment 637987 [details] [diff] [review]:
-----------------------------------------------------------------

a good step forward!

Comment 19

[Jeff Hammel]

pushed: http://hg.mozilla.org/build/talos/rev/195293d7404c