Closed Bug 867320 Opened 9 years ago Closed 7 years ago

Create reference workloads for Calendar

Categories

(Firefox OS Graveyard :: Gaia::Calendar, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gaye, Assigned: jhylands)

References

Details

(Keywords: perf, Whiteboard: c= , p=5)

Attachments

(2 files)

This may not be the highest priority item (James mentioned that calendar performance is thought to be quite good in 1.1), but I still think it'd wonderful to build on the work we've done to prepopulate pathological amounts of mock data in other apps to test the calendar under heavy load. In particular, I'd like to be able to investigate and monitor the performance of

- Loading from indexeddb and rendering many calendar accounts from different providers
- Loading from indexeddb and rendering many events in different views (ie day, week, month)
- Syncing calendars with lots of data from Google and Yahoo to a clean local
Whiteboard: c=performance
A Pivotal Tracker story has been created for this Bug: http://www.pivotaltracker.com/story/show/49070995
Stephany Wilkes changed story state to started in Pivotal Tracker
Assignee: nobody → jhylands
Status: NEW → ASSIGNED
Keywords: perf
Whiteboard: c=performance → c= ,
Unlike many of the other applications that we have reference workloads for (ie contacts, dialer, sms, etc.) one of the primary data loading bottlenecks that we have in calendar is network sync (there's a lot of good talk about this on 840935 for email). Ideally, a realistic perf setup like this would give us

1. An account on a remote caldav server with lots of data so that we could test the performance of heavy load network operations.
2. Lots of prepopulated calendar data in indexeddb so that we could also test the performance of loading data from indexeddb and rendering it in the UI.

We also rely on navigator.onLine in calendar to tell us whether or not we should try doing things over the network, so we too (like email) could use a debug flag for offline mode in indexeddb.

Also, we're likely adding features in 1.2 like attendees that will change the schema some. This shouldn't necessarily stop us from pushing ahead and adding reference workloads now, but we should keep in mind that we'll have to update the indexeddb population routine to reflect those updates.

I spoke with James about this a while back and I suspect he may have some thoughts before we begin implementing things.
Flags: needinfo?(jlal)
We could start with something very simple like offline only events...

Also- if the goal is to test calendar performance _after_ sync we can create an account which we can "prime" by syncing but after the first sync it will never sync again which we could use for testing...
Flags: needinfo?(jlal)
Whiteboard: c= , → c= , p=5
Depends on: 900194
See comment on 900194 - this bug will be resolved as soon as the calendar databases are merged in m-c/master. Note that these calendar databases are not compatable with v1-train phones, so I will also add a check for that before this is resolved.
Attached file Pull Request - patch
Attachment #807906 - Flags: review?(anygregor)
Attachment #807906 - Flags: review?(anygregor) → review+
https://github.com/mozilla-b2g/gaia/pull/12340
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Backing out since calendar reference workloads are breaking calendar perf tests and this wasn't reviewed by a calendar peer https://github.com/mozilla-b2g/gaia/pull/29889
Comment on attachment 8601516 [details] [review]
[gaia] gaye:revert-867320 > mozilla-b2g:master

r=self... see comment above
Attachment #8601516 - Flags: review+
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 8 years ago7 years ago
Resolution: --- → FIXED
Resolution: FIXED → WONTFIX
(In reply to Gareth Aye [:gaye] from comment #9)
> Backing out since calendar reference workloads are breaking calendar perf
> tests and this wasn't reviewed by a calendar peer
> https://github.com/mozilla-b2g/gaia/pull/29889

Can you elaborate on this - why would this be breaking calendar perf tests? If that's the case then it seems like perhaps the calendar database upgrade code is broken? That seems like probably a more serious issue to solve.
Flags: needinfo?(gaye)
Calendar database upgrade code is not broken. We have a good arsenal of unit tests to protect that functionality (see https://github.com/mozilla-b2g/gaia/blob/master/apps/calendar/test/unit/db_test.js). The reference workloads were injecting data into indexedDb that would have borked _any_ version of calendar. The work here happened without Miller's or my involvement (or knowledge for that matter) and it broke our performance tests starting a very, very long time ago. We haven't been on datazilla for ages and we had no idea what was going on. :Eli helped me understand that calendar was crashing <=> the reference workloads were loaded prior to starting the app, so I've gone ahead and backed this out and now we're showing up on raptor! TBH, I am super confused about why some basic sanity checks weren't done when the patch authors touching makeReferenceWorkload.sh. At the very least making sure that the affected apps can launch...
Flags: needinfo?(gaye)
This happened ~1.5 years ago, I am pretty sure that the apps could launch? I also remember testing most apps with this, and I am fairly certain that I tested Calendar as well. 

FWIW - I think that these reference workload like things are a much better test of upgrade code over unit test. They are a lot less brittle and comparable to integration vs unit test.
fwiw I just tested it on a 1.3 device and they work fine for me.
Its a pretty bold statement if you say we didn't test this before landing without any evidence.
The Calendar app *used* to work with these reference workload scripts in place, but had broken once a patch landed back in December. I'm pretty sure it was this one:

https://github.com/mozilla-b2g/gaia/commit/0e429d970c160e580e19e61ad8ff5612de159f00
Resolution: WONTFIX → FIXED
Fixing collision, oops.
Resolution: FIXED → WONTFIX
Sorry was talking about the patch :Eli mentioned. It was easier (without knowing a lot about the reference workload code) to revert all of the calendar bits than try to revert the modifications to the calendar parts.
I agree that something like reference workloads is a better (read: less mocked) test of database upgrades than calendar's existing unit tests, but I was thinking of this more as a perf tool than a migration testing tool.
Pushing the DB works fine on 1.3 (where I can see the events in the calendar app) but breaks the app on current trunk. This sounds like an upgrade bug to me.
It may be an issue that it is simply pushing the data to the wrong place - we've had to do lots of updates to reference workloads as database locations have changed over the various versions. Its entirely possible that one change might have been missed.
Resolution: WONTFIX → FIXED
You need to log in before you can comment on or make changes to this bug.