1039940 - serialisation of objects for webservice responses is extremely slow

Status: RESOLVED FIXED

(Bugzilla :: WebService, defect)

Description

[:glob ✱]

serialisation of objects for webservice responses is extremely slow, due to many reasons:
- data is often serialised then thrown away
- calling filter_wants is slow, even with its results cached
- calling type is slow

i'm working on a different approach to object serialisation which appears to be promising.

Comment 1

[Frédéric Buclin]

I encountered the same problem too. For instance, Bug.search is slower than running the same query from the UI. This is IMO a major problem.

Which approach are you working on? Any WIP patch?

Comment 2

[:glob ✱]

this is a high priority bug for me, i have several approaches which i'm experimenting with.

the easiest fix is to ensure that fields which are expensive to serialise are only done when requested.  eg. product and component name, as well as any timestamp responses.  this provides a measurable gain for minimal effort.

i'm also working on a complete rewrite of the serialisation process which addresses a number of issues, such as:
- only serialise fields which are requested
- change $self->type calls to as_string, as_int, etc (method calls are slow, and there's no need to perform extra string comparisons to determine the type)
- even with caching, calling filter_wants for each field is slow; we can calculate the list of fields to return once


without custom field support i've managed to reduce the time it takes to serialise about 9700 bug objects from 18 seconds to less than 1 second.  custom fields adds another 3 seconds overall, however i haven't yet tackled optimising that code.


my plan here is to continue to work on this bug, but if it looks like the rewrite will take too long, i'll split this bug into two.  this bug can carry the more trivial but less dramatic fix, and the larger work can happen elsewhere.

Comment 3

[:glob ✱]

splitting the bug in two; bug 1061233 will track the more dramatic work.

Comment 4

[:glob ✱]

Attachment: 1039940_1.patch

parsing of dates is slow; skip that unless that field is required.

on my development system this reduced a query which involved the serialisation of 9000 bugs from 14s to 10s.

Comment 5

[Dylan Hardison [:dylan] (he/him)]

Comment on attachment 8487049 [details] [diff] [review]
1039940_1.patch

Review of attachment 8487049 [details] [diff] [review]:
-----------------------------------------------------------------

I was able to see a very slight difference in speed when excluding the creation_time and last_change_time with this patch applied vs. without. gnv

Comment 6

[Dylan Hardison [:dylan] (he/him)]

"gnv" being pentadactyl for "open quickmark v" to make sure I do the right thing after an r+

Comment 7

[Mark Côté [:mcote]]

Shouldn't have reset Target Milestone here.

Comment 8

[:glob ✱]

To ssh://[e-mail]/bugzilla/bugzilla.git
   cfb4ad9..164d2a8  master -> master