serialisation of objects for webservice responses is extremely slow

RESOLVED FIXED in Bugzilla 5.0

Status

()

--
major
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: glob, Assigned: glob)

Tracking

({bmo-big, bmo-goal, perf})

unspecified
Bugzilla 5.0
bmo-big, bmo-goal, perf
Bug Flags:
approval +

Details

Attachments

(1 attachment)

serialisation of objects for webservice responses is extremely slow, due to many reasons:
- data is often serialised then thrown away
- calling filter_wants is slow, even with its results cached
- calling type is slow

i'm working on a different approach to object serialisation which appears to be promising.

Comment 1

4 years ago
I encountered the same problem too. For instance, Bug.search is slower than running the same query from the UI. This is IMO a major problem.

Which approach are you working on? Any WIP patch?
Severity: normal → major
OS: Mac OS X → All
Hardware: x86 → All
Target Milestone: --- → Bugzilla 5.0
Target Milestone: Bugzilla 5.0 → ---
this is a high priority bug for me, i have several approaches which i'm experimenting with.

the easiest fix is to ensure that fields which are expensive to serialise are only done when requested.  eg. product and component name, as well as any timestamp responses.  this provides a measurable gain for minimal effort.

i'm also working on a complete rewrite of the serialisation process which addresses a number of issues, such as:
- only serialise fields which are requested
- change $self->type calls to as_string, as_int, etc (method calls are slow, and there's no need to perform extra string comparisons to determine the type)
- even with caching, calling filter_wants for each field is slow; we can calculate the list of fields to return once


without custom field support i've managed to reduce the time it takes to serialise about 9700 bug objects from 18 seconds to less than 1 second.  custom fields adds another 3 seconds overall, however i haven't yet tackled optimising that code.


my plan here is to continue to work on this bug, but if it looks like the rewrite will take too long, i'll split this bug into two.  this bug can carry the more trivial but less dramatic fix, and the larger work can happen elsewhere.
splitting the bug in two; bug 1061233 will track the more dramatic work.
Status: NEW → ASSIGNED
Created attachment 8487049 [details] [diff] [review]
1039940_1.patch

parsing of dates is slow; skip that unless that field is required.

on my development system this reduced a query which involved the serialisation of 9000 bugs from 14s to 10s.
Attachment #8487049 - Flags: review?(dylan)
No longer blocks: 1035804
Comment on attachment 8487049 [details] [diff] [review]
1039940_1.patch

Review of attachment 8487049 [details] [diff] [review]:
-----------------------------------------------------------------

I was able to see a very slight difference in speed when excluding the creation_time and last_change_time with this patch applied vs. without. gnv
Attachment #8487049 - Flags: review?(dylan) → review+
"gnv" being pentadactyl for "open quickmark v" to make sure I do the right thing after an r+
Flags: approval?
Target Milestone: --- → Bugzilla 5.0

Updated

4 years ago
Flags: approval? → approval+
Target Milestone: Bugzilla 5.0 → ---
Shouldn't have reset Target Milestone here.
Target Milestone: --- → Bugzilla 5.0
To ssh://gitolite3@git.mozilla.org/bugzilla/bugzilla.git
   cfb4ad9..164d2a8  master -> master
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.