968775 - ujson everywhere

Status: VERIFIED WONTFIX

(Socorro :: Backend, task)

Description

[K Lars Lohn [:lars] [:klohn]]

It's time for the next json migration. simplejson was fine back when we used Python 2.4.  We switched to the built in json when we adopted Python 2.6.  Now we've realized how slow that json is.  It is time to switch to ujson, already proven to be significantly faster...

Comment 1

[Peter Bengtsson [:peterbe]]

Where does this matter? Anywhere near database I/O the CPU slowness of the pure python one is dwarfed in comparison.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Peter Bengtsson [:peterbe] from comment #1)
> Where does this matter? Anywhere near database I/O the CPU slowness of the
> pure python one is dwarfed in comparison.

We should definitely profile and figure out how much this matters.. I would not be surprised if there was a significant savings here, especially with the newer processed JSON ("jdump")

Comment 3

[Lonnen :lonnen]

We already have ujson in our deps (https://github.com/mozilla/socorro/blob/master/requirements.txt#L76).

Profiling would be nice.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

ujson may not be our best choice as we're fairly heavily invested in "object_hook" and "JSONEncoder".  Neither of those features is supported in ujson.  I'm writing a work around that seems promising, but is resulting in the death-by-a-thousand-cuts problem.

I'm also comparing simplejson again.

Comment 5

[Peter Bengtsson [:peterbe]]

Anything C based is likely to be a MASSIVE win. I'm also curious how they all compare in their serializing of floating point numbers.

Comment 6

[K Lars Lohn [:lars] [:klohn]]

after working with this and doing some bench marking, there is no "win" for this effort.  I have  learned some interesting things, though.  

as peterbe points out the amount of time saved by using a faster json coder is dwarfed by the time it takes to do the PG, HB, RMQ, ES io.   

Since we use "object_hook" and that is not implemented in ujson, there is too much engineering effort involved for such a minuscule reward. 

"object_hook" is strangely expensive, but not by enough to justify the engineering effort to change it.

Unless anyone objects, I'm closing this bug as WONTFIX.

Comment 7

[Peter Bengtsson [:peterbe]]

doing your spring cleaning for you :)

Comment 8

[Matt Brandt [:mbrandt]]

Sounds like a double vote of wontfix confidence. Bumping to verified