1002842 - Video cold launch time regressed since 1.3, above 1000ms acceptance threshold

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Video, defect, P1)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Per http://mzl.la/1k6ZmHK, Video is currently at/above 1100ms for its load time. 

Our acceptance metric for performance is either no regression from last release, or within the responsiveness guidelines, whichever is greater.

This is a regression from 1.3's behavior (http://mzl.la/1jaJ2ZP) at ~750ms. Our responsiveness guideline for cold launch is 1000ms (https://wiki.mozilla.org/FirefoxOS/Performance/UserStories, Perception of Progress).

Cold launch of Video should be brought within 1000ms. Requesting 1.4+ due to importance of the app.

Comment 1

[John Hu [:johnhu][:johu][:醬糊小弟]]

This may be related to the tablet patch because we use some media queries in css.

CC fred and chens here.

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Tracking 997101 as a dependency, not a duplicate. The bug is being over 1000ms as an acceptance goal, and there are a number of avenues that can probably be taken to fix this. I'd prefer not to dupe the acceptance issue with an assumed root cause.

Comment 4

[Mike Lee [:mlee]]

Hema,

Please have someone on the media team look into this.

Mike

Comment 5

[Hema Koka [:hema]]

Punam, 

As discussed in standup today, please investigate for other bottlenecks while the dependency is also being worked on in parallel. 

Feel free to reach out to Johu and performance engineers if you have questions.

Thanks
Hema

Comment 6

[Punam Dahiya [:pdahiya]]

Tried b2gperf --delay=10 Video on 1.4 and 1.5/m-c builds
1.4 load-time

2014-05-01 16:54:41,301 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1131, mean:1342, std: 375, max:2330, min:990, all:1189,1132,1111,1123,1121,1902,1129,1132,1890,1107,1136,2330,1104,1113,1875,1122,1119,1896,1135,1105,1960,1130,1122,1885,1161,1120,1130,990,1152,1864

1.5 load-time
2014-05-01 15:42:08,238 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1079, mean:1080, std: 22, max:1162, min:1037, all:1162,1114,1079,1080,1089,1089,1079,1075,1059,1084,1042,1086,1082,1081,1066,1061,1087,1076,1066,1102,1037,1087,1079,1069,1125,1077,1065,1074,1075,1069

Hi John,
Per http://mzl.la/1k6ZmHK , gaia branch - master , device - hamachi, load time for video app dropped in 1000s on 28 april and stayed around that since in 1.5. I checked gaia side and i don't see any video patch in that time frame that could have that impact. Do you know any gecko improvement that landed this or last week since (~ 27 april) on master that could have reduced load time.

Also, please share if you have inputs on how to bring down the load - time for video app in 1.4. Thanks!

Comment 7

[Punam Dahiya [:pdahiya]]

> Hi John,
> Per http://mzl.la/1k6ZmHK , gaia branch - master , device - hamachi, load
> time for video app dropped in 1000s on 28 april and stayed around that since
> in 1.5. I checked gaia side and i don't see any video patch in that time
> frame that could have that impact. Do you know any gecko improvement that
> landed this or last week since (~ 27 april) on master that could have
> reduced load time.
> 

typo as you must have guessed, it's 1000ms

Comment 8

[John Hu [:johnhu][:johu][:醬糊小弟]]

We had found a performance issue on CSS media query, bug 997101. We used few of them to support tablet version. If we can reduce them with css pre-processor or use JavaScript to load them dynamicly, https://groups.google.com/forum/#!topic/mozilla.dev.gaia/dVM02zQkmTk, that may help.

Comment 9

[John Hu [:johnhu][:johu][:醬糊小弟]]

According to alive, the dropping down of load time at last week may be caused by a back out code of system app. It may goes back when the code land back.

Comment 10

[[e-mail]]

per Comment 8, would you consider to use media-query-matching method to filter out @media for phone-sized device at build time? https://github.com/gasolin/provecss#media-query-matching

Comment 11

[John Hu [:johnhu][:johu][:醬糊小弟]]

There is another alternative.

We may try to use GAIA_DEVICE_TYPE=phone and GAIA_DEVICE_TYPE=tablet within Makefile to eliminate one media query:

<link rel="stylesheet" type="text/css" href="style/video_tablet.css" media="(min-height: 768px) and (min-width: 768px)" />

Comment 12

[Punam Dahiya [:pdahiya]]

Ran b2gperf --delay=10 Video and below are the load times

1) hamachi m-c Buildid:20131211040203 (without tablet fixes)
2014-05-02 14:02:37,708 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:615, mean:830, std: 302, max:1270, min:592, all:612,1253,592,596,1258,608,847,1244,598,602,1270,605,627,1261,595,595,1240,604,609,1252,614,627,1253,597,618,1255,613,617,1240,599



2 ) hamachi m-c build: 20131215040201 (with bug 903920 - tablet fixes)
2014-05-02 14:38:02,848 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:722, mean:766, std: 141, max:1290, min:695, all:850,709,719,851,722,731,1271,715,712,1290,716,712,740,715,719,719,723,725,728,742,725,695,734,717,727,721,719,722,729,696


3) hamachi 1.4 build 20140505000201 with media queries
2014-05-05 11:23:42,919 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1155, mean:1160, std: 30, max:1260, min:1124, all:1248,1124,1136,1199,1141,1128,1178,1155,1177,1151,1126,1166,1155,1138,1156,1156,1168,1136,1260,1172,1163,1138,1168,1172,1153,1154,1157,1164,1152,1131



4) hamachi 1.4 build 20140505000201 without media queries - by removing below include from video app index.html
( <link rel="stylesheet" type="text/css" href="style/video_tablet.css" media="(min-height: 768px) and (min-width: 768px)" /> ) and make reset-gaia

2014-05-05 11:06:17,275 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1127, mean:1125, std: 30, max:1179, min:1037, all:1176,1119,1138,1142,1144,1037,1179,1081,1136,1128,1122,1100,1125,1147,1126,1179,1107,1118,1135,1128,1088,1116,1077,1165,1136,1104,1142,1096,1126,1148

As per load time in  1 and 2, after flatfish tablet changes load time was still below 1000 ms for Video app and indicates that tablet changes are not solely responsible for the ~1100 ms load time. I see Video statistics and performance testing related fixes(Bug 948095 , Bug 922609) that landed in video. Will check and update bug if those have an impact on load_time.

Comment 13

[Punam Dahiya [:pdahiya]]

> 
> 4) hamachi 1.4 build 20140505000201 without media queries - by removing
> below include from video app index.html
> ( <link rel="stylesheet" type="text/css" href="style/video_tablet.css"
> media="(min-height: 768px) and (min-width: 768px)" /> ) and make reset-gaia

John, can you pl. validate that removing the above include is enough to measure load time regression due to media query in video?


> 2014-05-05 11:06:17,275 B2GPerfRunner INFO    | Results for Video,
> cold_load_time: median:1127, mean:1125, std: 30, max:1179, min:1037,
> all:1176,1119,1138,1142,1144,1037,1179,1081,1136,1128,1122,1100,1125,1147,
> 1126,1179,1107,1118,1135,1128,1088,1116,1077,1165,1136,1104,1142,1096,1126,
> 1148
> 
> As per load time in  1 and 2, after flatfish tablet changes load time was
> still below 1000 ms for Video app and indicates that tablet changes are not
> solely responsible for the ~1100 ms load time. I see Video statistics and
> performance testing related fixes(Bug 948095 , Bug 922609) that landed in
> video. Will check and update bug if those have an impact on load_time.

Bug 948095 fix has been uplifted to 1.3. Bug 922609 helps track performance using PerformanceTestingHelper on cold launch. I tried reverting the patch on 1.4 to measure impact but the load time stays in ~1100 ms without 922609 fix. 
I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms), however failed to map it to any specific update in video app. John, do you know any updates in this time frame that could have impacted video app. Thanks

Comment 14

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to Punam Dahiya from comment #13)
> John, can you pl. validate that removing the above include is enough to
> measure load time regression due to media query in video?

I had asked the same question at bug 997101 and the reporter of it. We may test it.

> > As per load time in  1 and 2, after flatfish tablet changes load time was
> > still below 1000 ms for Video app and indicates that tablet changes are not
> > solely responsible for the ~1100 ms load time. I see Video statistics and
> > performance testing related fixes(Bug 948095 , Bug 922609) that landed in
> > video. Will check and update bug if those have an impact on load_time.

Thanks for this information. According to the finding of bug 977101, it should affect the cold launch time because we use some media queries in tablet to have totally different layout between landscape and portrait.

> 
> Bug 948095 fix has been uplifted to 1.3. Bug 922609 helps track performance
> using PerformanceTestingHelper on cold launch. I tried reverting the patch
> on 1.4 to measure impact but the load time stays in ~1100 ms without 922609
> fix. 
> I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms),
> however failed to map it to any specific update in video app. John, do you
> know any updates in this time frame that could have impacted video app.
> Thanks

Sorry, I don't know which updates landed at that period affecting video app. I think we should ask sheriff or performance team to help us to find them out.

Comment 15

[Hema Koka [:hema]]

> I saw increase in load time b/w jan 30th (~811 ms) and feb 8th (~1156 ms),
> however failed to map it to any specific update in video app. John, do you
> know any updates in this time frame that could have impacted video app.
> Thanks

Punam, Thanks so much for testing and narrowing down the regression date range on cold launch time. 

Mike, who is the best person to help further to nail down the root-cause. Per Punam's investigation, no changes went into video app between jan 30th and feb 8th (impact from using media query is being investigated on a dependent bug). Perhaps there are other changes during this period that are indirectly impacting video app. We will need your team's help here. 

Thanks
Hema

Comment 16

[Hema Koka [:hema]]

Mike, 

We need your team's help to make further progress on this bug. Thanks!

Comment 17

[Mike Lee [:mlee]]

Jon,

Please help Hema's team root cause this issue.

Thanks,
Mike

Comment 18

[Punam Dahiya [:pdahiya]]

Unassigning myself as i have run out of ideas . Waiting for performance team (comment #17) to provide input.

Comment 19

[Hema Koka [:hema]]

Jon/Mike -- we need your team's help to make further progress

thanks
hema

Comment 20

[Jon Hylands [:jhylands]]

I will be looking at this tomorrow - sorry, was in Mountain View for a week, and had to concentrate on power related issues while I was there.

Comment 21

[Jon Hylands [:jhylands]]

So I spent most of today trying to do a developer build with the right revisions to produce the "before" numbers (~800 ms) and so far I haven't had any luck. I will continue to work on this tomorrow.

Comment 22

[Jon Hylands [:jhylands]]

Punam, can you point me to the specific build you have for Hamachi that shows a ~811 ms video app load time using b2gperf? I've tried recreating developer builds from various points in January, and nothing I've tried has scored lower than 1046 ms.

Comment 23

[Punam Dahiya [:pdahiya]]

(In reply to Jon Hylands [:jhylands] from comment #22)
> Punam, can you point me to the specific build you have for Hamachi that
> shows a ~811 ms video app load time using b2gperf? I've tried recreating
> developer builds from various points in January, and nothing I've tried has
> scored lower than 1046 ms.

I agree it's hard to pin a buildid , but the b2gperf video app cold launch time showed significant increase from jan 30th to feb 8th

hamachi m-c jan 30th
BuildId: 20140130040201
2014-05-02 19:27:25,589 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:811,
mean:1014, std: 397, max:1832, min:775, all:923,828,1832,805,788,1823,810,814,872,804,775,812,893,813,1829,788,804,1781,800,808,1786,792,815,808,787,797,810,838,807,1783



B2GPerf 
hamachi m-c feb 8th 
BuildId:20140208040204:
2014-05-05 12:45:34,095 B2GPerfRunner INFO    | Results for Video, cold_load_time: median:1156, 
mean:1153, std: 36, max:1271, min:1092, all:1187,1230,1105,1139,1124,1165,1171,1157,1125,1171,1173,1143,1271,1128,1159,1145,1092,1121,1171,1137,1134,1177,1156,1119,1121,1109,1165,1185,1156,1167

Comment 24

[Jon Hylands [:jhylands]]

So, after a few days of trying to get consistent results, I finally started to get somewhere today. I was able to flash my Hamachi with a 1.3 build that gave 600-800ms results, and then reproduce the results using m-c daily builds from Jan 30 and Feb 8 (Note that I tried this last week and was getting 1000+ ms numbers, not sure why it changed now).

However, while trying to narrow down the results to a finer grained date range, I ended up having to reflash my phone (for the 5th or 6th time in the past few days) and the Teleweb tool crashed partway through flashing, and the phone appears to be bricked at this point.

I'm going to try and fix it, but with this being a holiday in the US, I don't hold much hope. I will be at a conference starting tomorrow morning for the rest of the week, so someone else will have to do this.

The location of the daily builds I used is here:

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/

Specifically, Jan 30: (755 ms)

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/2014/01/2014-01-30-16-02-00/

Feb 8: (1130 ms)

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng/2014/02/2014-02-08-16-02-01/

Mike, can you hand this off to someone else to investigate?

Comment 25

[Mike Lee [:mlee]]

Jon will continue to work on this now that he's returned from JSConf.

Comment 26

[Jon Hylands [:jhylands]]

I managed to unbrick my phone this morning using fastboot, and I've been continuing on this today.

Together with what I found last Monday, these are my results so far (using nightly builds):

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-b2g28_v1_3-hamachi-eng

Jan 31 (16:40) - median 692
Feb 8 (00:40) - median 697
Feb 16 (00:40) - median 694

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-hamachi-eng

Jan 30 (16:02) - median 755
Jan 31 (09:54) - median 955
Jan 31 (16:02) - median 969
Feb 1 (04:02) - median 1188
Feb 1 (16:02) - median 1217
Feb 2 (16:02) - median 973
Feb 3 (04:02) - median 979
Feb 4 (04:02) - median 1083
Feb 4 (16:02) - median 1282
Feb 5 (16:02) - median 1327
Feb 8 (16:02) - median 1130

I'm going to get the raw datazilla database dump from jeads for this time period, so hopefully that will help narrow things down a little more. From the above data, I would have to say there appear to be a couple issues - one that happened on Jan 30/31, and one that happened on Feb 4.

Comment 27

[Jon Hylands [:jhylands]]

I've done some more tests, and it looks like the 755 listed above on Jan 30 was out-of-the-ordinary:

Jan 15 (04:14) - median 947
Jan 20 (04:02) - median 962
Jan 27 (16:05) - median 1107
Jan 29 (16:02) - median 967

Given that almost all the values are between 950 and 1200, and the current m-c values (June 2) are about 1200, I think the direction we've been looking is not valid (i.e., the regression did not happen between Jan 30 and Feb 8 as previously reported).

It is very clear that there has been a major regression between 1.3 and now, but we will need to dig further to find it.

Comment 28

[Jon Hylands [:jhylands]]

Attachment: VideoColdLoad-January2014.png

Graph showing Video Cold Load time in January 2014, master, hamachi

Comment 29

[Jon Hylands [:jhylands]]

After analyzing the data from all of January (Jeads exported the January datazilla database for me), it appears that there are two main regression points, although the second one will have to wait until I can see February's data to see what the trend is. The first happens on or about Jan 16, where there is a 100 ms regression (750 -> 850), although by January 28 it was back down to ~780 for a few runs. On Jan 31, there was a large jump (790 -> 930), but its not clear until I get the Feb data if it is transitory spike or an actual regression.

Comment 30

[Jon Hylands [:jhylands]]

Attachment: VideoColdLoad-Regression.png

Graph showing 2 regressions (each > 100ms) in late Jan/early Feb for Video cold load times (hamachi, master)

Comment 31

[Jon Hylands [:jhylands]]

So, with this new data, its pretty clear I have to eat my words from command 27, and the interesting regressions did in fact happen between Jan 30 and Feb 8. I'm now going to attempt to create developer builds with the specific Gaia/Gecko revisions for each regression, and bisect them.

Comment 32

[Dave Huseby [:huseby]]

Attachment: requirements_1002482.txt

the pip requirements file that works for running the b2gperf video test on the first known bad revisions.

use this like so:

pip install -r requirements_1002842.txt

Comment 33

[Dave Huseby [:huseby]]

Here's the bisect for one of the regression ranges:

LKG Gecko: eb3bc59
LKG Gaia:  d74eeec
Results for Video, cold_load_time: median:1057, mean:1098, std: 148, max:1837, min:1006, all:1270,1092,1837,1085,1205,1178,1072,1023,1035,1061,1048,1111,1023,1054,1058,1049,1006,1032,1075,1038,1041,1028,1092,1053,1100,1089,1065,1056,1041,1030

FKB Gecko: f448244
FKB Gaia:  e6b5fc4
Results for Video, cold_load_time: median:1304, mean:1337, std: 101, max:1750, min:1245, all:1340,1349,1304,1305,1353,1261,1304,1363,1563,1460,1291,1750,1413,1340,1272,1283,1364,1286,1292,1272,1302,1264,1299,1265,1387,1286,1330,1329,1245,1247

FKB Gecko: f448244
LKG Gaia:  d74eeec
Results for Video, cold_load_time: median:1483, mean:1470, std: 112, max:1659, min:1260, all:1549,1608,1395,1571,1590,1483,1261,1473,1552,1635,1260,1509,1659,1487,1294,1472,1484,1512,1304,1458,1478,1537,1272,1548,1464,1483,1260,1464,1499,1548

LKG Gecko: eb3bc59
FKB Gaia:  e6b5fc4
Results for Video, cold_load_time: median:1086, mean:1092, std: 78, max:1340, min:1005, all:1331,1029,1030,1124,1103,1087,1105,1022,1090,1052,1026,1090,1060,1090,1052,1091,1088,1005,1029,1053,1019,1086,1058,1059,1087,1340,1224,1118,1150,1074

gecko regression

gecko: 59ded1d
Results for Video, cold_load_time: median:1076, mean:1329, std: 393, max:2438, min:1028, all:1189,1043,1029,1804,1104,1125,2438,1056,1058,1810,1065,1366,1924,1065,1051,1830,1038,1073,1818,1028,1074,1848,1079,1171,1836,1043,1041,1797,1042,1039
good

gecko: 4cfb8bb
Results for Video, cold_load_time: median:1126, mean:1359, std: 446, max:2435, min:1026, all:1202,1053,1039,1829,1130,1159,1123,1061,1063,2417,1056,1259,1850,1286,1070,2416,1109,1056,1815,1075,1055,1824,1228,1130,2435,1088,1062,1802,1059,1026
good

gecko: 78f7c38
Results for Video, cold_load_time: median:1102, mean:1372, std: 461, max:2461, min:1030, all:1232,1044,1041,1842,1151,1126,2461,1076,1078,1819,1107,1094,1795,1056,1030,1819,1099,1103,1891,1135,1082,1835,1086,1044,2420,1051,1064,2417,1079,1102
good

gecko: 21ca356
Results for Video, cold_load_time: median:1322, mean:1329, std: 72, max:1622, min:1251, all:1288,1317,1356,1257,1310,1301,1371,1340,1360,1316,1265,1622,1348,1483,1343,1355,1267,1281,1330,1325,1276,1347,1251,1265,1348,1389,1320,1266,1334,1258
bad

gecko: 5a5fa6d
Results for Video, cold_load_time: median:1326, mean:1324, std: 52, max:1521, min:1247, all:1369,1267,1336,1291,1399,1356,1294,1285,1294,1269,1338,1521,1360,1321,1300,1340,1332,1270,1353,1275,1285,1369,1279,1347,1247,1303,1342,1343,1363,1293
bad

gecko: 9b46866
Results for Video, cold_load_time: median:1084, mean:1248, std: 381, max:2416, min:1029, all:1238,1069,1047,1802,1126,1157,1819,1051,1076,2403,1029,1141,1091,1097,1070,1066,1038,1827,1054,1076,1088,1079,1072,1151,1088,1090,1081,2416,1040,1071
good

gecko: 3e8ad39
Results for Video, cold_load_time: median:1105, mean:1290, std: 370, max:2446, min:1006, all:1195,1084,1055,1815,1075,1126,1134,1056,1039,1818,1023,1400,1865,1111,1106,2446,1053,1006,1834,1077,1128,1826,1043,1057,1073,1061,1105,1859,1159,1088
good


5a5fa6d369ab2e3c430cb9e1a5e909e9922c4a3f is the first bad commit
commit 5a5fa6d369ab2e3c430cb9e1a5e909e9922c4a3f
Author: Fabrice Desré <[e-mail]>
Date:   Tue Jan 14 16:00:25 2014 -0800

    Bug 950266 - Re-enable the Nuwa process on B2G by default r=me

:040000 040000 ee714dd593b91ae3bbabf139a301cc601103a0bf 8d2cb676eead40c2c8966fcc0f7edaa973e3921f M      b2g
:040000 040000 9eedfb6d546b4159f1258b1f60dbc93ad1171f18 505b5fb5f9f5d61d83c679abcce42fc1af467c41 M      dom
:040000 040000 98db54314717da56e9be72226f4f86ad402f0bc0 1d41e67a6a5e335e4384b3cd41ca1bb9264be388 M      testing
:040000 040000 e1107a1151393292aba2cb757a708097f65b45f9 c7c26367ed629078fc1484abaacf8cf8dda0b2e2 M      toolkit

Comment 34

[Dave Huseby [:huseby]]

Here's the other regression:

LKG Gecko: a8d1e0f
LKG Gaia:  6c9fac8
Results for Video, cold_load_time: median:853, mean:1128, std: 423, max:1798, min:824, all:980,1794,840,847,1777,881,848,1771,824,847,1757,826,855,1767,840,840,1798,852,849,838,858,1760,827,831,1755,866,838,1787,849,863

FKB Gecko: 4652f1b
FKB Gaia:  f9013a0


FKB Gecko: 4652f1b
LKG Gaia:  6c9fac8
Results for Video, cold_load_time: median:873, mean:875, std: 40, max:1029, min:809, all:1029,879,959,888,927,849,885,875,845,864,891,846,878,872,855,849,883,843,865,885,860,881,831,867,860,809,893,879,827,905

LKG Gecko: a8d1e0f
FKB Gaia:  f9013a0
Results for Video, cold_load_time: median:991, mean:1140, std: 372, max:2329, min:944, all:1085,964,996,1824,1009,1004,1054,1014,947,959,987,944,1812,977,1126,1059,974,981,2329,964,978,1022,954,986,954,977,1041,1041,2267,975

gaia regression

gaia: 75122a8
Results for Video, cold_load_time: median:995, mean:1097, std: 319, max:2294, min:923, all:1201,957,961,1923,1013,1034,990,956,996,995,953,1006,1859,998,963,1040,923,979,972,992,1010,1078,1008,1011,2294,953,944,957,925,1024
bad

gaia: 869e51f
Results for Video, cold_load_time: median:879, mean:1084, std: 391, max:1822, min:823, all:989,1780,882,990,1773,892,838,1814,896,862,1779,910,827,1778,840,850,1782,834,871,840,823,876,830,839,833,841,1822,893,913,844
good

gaia: 6b4def0
Results for Video, cold_load_time: median:1022, mean:1239, std: 495, max:2501, min:946, all:1046,962,1024,2501,1049,1031,1039,1006,947,1791,961,995,2462,1041,951,2287,972,998,946,1053,1831,1010,1029,2304,961,1020,980,952,995,1026
bad

gaia: caf4dde
Results for Video, cold_load_time: median:1008, mean:1103, std: 304, max:2330, min:935, all:1077,935,959,1144,1009,1005,946,972,1015,1804,1005,971,1010,1129,1034,972,1011,958,1787,1024,1008,2330,1008,1039,1023,964,1008,1011,968,991
bad

caf4dde01228fe880bf93e151f0c2057f58bea41 is the first bad commit
commit caf4dde01228fe880bf93e151f0c2057f58bea41
Author: Kevin Grandon <[e-mail]>
Date:   Thu Jan 30 11:31:12 2014 -0800

    Bug 962655 - Use build-time preference to enable rocketbar r=vingtetun, yurenju

:100644 100644 e0cb7b4a0cd0573fa5772e5a75dad77925f022e7 a3c8aeea716dceff8fd3a2324caeababebcf7081 M      Makefile
:040000 040000 6c321ae23f26a20e692c4e61075a026e3fb19ede 8b6a00fc847bd24c7002bfaa06e35db94087e427 M      apps
:040000 040000 48a71e9e1aaf50b497ca2149d82ab0d50677c42e 5196eb22b5e00be17e03f1880b42e9a3f5ae3bfa M      build
:040000 040000 7a510590d6fa1256375db1ef3d6bd6cb56b727d3 2c60420d65162d9c2ddcd8320272e2dabc546ddf M      shared
:040000 040000 567ad65ba8ba36379103b790778be2f9608ed32b bb9214743c012902ecc03557542693a86c0c2e09 M      tests

Comment 35

[Jon Hylands [:jhylands]]

I've created two new bugs to handle the followups for these two regressions, and marked them both with 1.4?

Bug 1024923 (regression identified in comment 34)

Bug 1024928 (regression identified in comment 33)

Comment 36

[Ryan VanderMeulen [:RyanVM]]

Calling this "fixed" for tracking purposes though it seems to me that this bug as-filed isn't really fixed until the dependent bugs are also fixed.