Closed Bug 812396 Opened 7 years ago Closed 7 years ago

[perf] Gaia Build process needs to combine js content

Categories

(Firefox OS Graveyard :: Gaia, defect, P2)

x86
macOS
defect

Tracking

(blocking-b2g:-, blocking-basecamp:-, b2g18+ fixed, b2g18-v1.0.0 wontfix, b2g18-v1.0.1 fixed)

RESOLVED FIXED
B2G C3 (12dec-1jan)
blocking-b2g -
blocking-basecamp -
Tracking Status
b2g18 + fixed
b2g18-v1.0.0 --- wontfix
b2g18-v1.0.1 --- fixed

People

(Reporter: onecyrenus, Assigned: jlal)

References

Details

(Keywords: perf, Whiteboard: [nptob][FFOS_perf])

Attachments

(4 files)

We can save upwards of 1 second for startup on some apps if we minify and combine css. 

Running some basic tests on the e-mail app. I was able to observe a faster load time if I combined / minified the source code. 

Tool used was mince

1) http://code.google.com/p/mince/
2) created a .minceconf for the e-mail app. 
3) turned on show app startup time
4) modified index.html to load combined css/js
4) make install-gaia

I was able to observe a 5.3 second startup time go down to approx a 4.1 second startup time. This was without minification, just combining.
Summary: [perf] Gaia Build process needs to combine and minify → [perf] Gaia Build process needs to combine and minify js/css content
Blocks: c&c
Hm, there's an existing bug for doing this in Firefox desktop. Might have good information about the wins, and why they haven't done it yet.
Keywords: perf
Dietrich this is for gaia apps ?
Yes, I know. The Firefox desktop bug, IIRC, had lots of discussion of exactly how much time was saved, as well as discussion around the costs (loss of source code blame, unable to edit directly, etc).
There's also YUI minifier as an alternative. Not sure which is better.
Firefox desktop CSS minification: bug 513132

Firefox desktop JS minification: bug 524858
There are still some further improvements that could be done but here is a rough pass. 

what i have done here is to minify the js JSMin + Csstidy, and combine everything. 

Overall load times before 5300- 5400 ms 

Afterwards
Load times: 3189, 3100, 2900, (4100 -outlier explained below). 

There is a difference in load times being observed if the phone is active and loading the app freshly vs waking from sleep loading the app.  This is what puts you into the 4000ms range.
In bug 812643 I got some numbers on how much does it cost to load css/js for Settings app. It costs around 300-600ms. Would be worth to check with minified css/js.
in order to repeat you would need both directories.
blocking-basecamp: --- → ?
:dietrich i believe you have characterized this incorrectly, as it refers only to client side js / css minifying and not gecko side js/css minifying which is a seperate issue. 

Generally i can't see a situation where the consumer would need "unminified" css + js. 

I believe this should be part of the build process.
(In reply to dclarke@mozilla.com  [:onecyrenus] from comment #9)
> :dietrich i believe you have characterized this incorrectly, as it refers
> only to client side js / css minifying and not gecko side js/css minifying
> which is a seperate issue. 

I haven't characterized it at all. I'm only pointing at previous work where there might be something to learn, useful data and insight into the benefits and costs.

> Generally i can't see a situation where the consumer would need "unminified"
> css + js. 

Nobody is saying otherwise :)
The timing winnings in comment 6 are too good to pass up.
blocking-basecamp: ? → +
Ted/Kyle, can one of you help here?  David Clarke can provide the scripts he used to minify.
Flags: needinfo?(khuey)
Neither ted nor I know anything about the gaia build system.  Whoever implemented the gaia build system should tackle this.
Flags: needinfo?(khuey)
Vivien, can you find someone to take a look at this?
Assignee: nobody → 21
Priority: -- → P2
OK, If Vivien shouldn't own this, who does?
I'll take it.

David: how did you measure your timings? I'm trying to reproduce your win on Settings app, I took all CSS/JS files and minified them into one js and one css file.

My initial readings are:
 - firstPaint down from 1774 to 1622
 - DOMContentLoaded down from 1080 to 819
 - window.load down from 1471 to 1115.60

It seems to be a nice win, but not to the level that you described.

My plan is to introduce a new Makefile command "minify-res" that will be reducing all header CSS/JS files into single mini.css|js file per app at build time.
Assignee: nobody → gandalf
Status: NEW → ASSIGNED
Some of those numbers are ~20% difference or more! That's actually huge, and in those ranges of time, can make the difference between noticeable and not noticeable to a user.
Yeah, it's a nice thing to win ~20% :)

But overall, no, we're still in the noticable zone. 
I'm trying to pinpoint the reasons, because we seem to be able to get close to 200ms (with Browser app), while with other apps we're dancing between 600ms and 2000ms. Even wake up from sleep app takes over 600ms.

According to cognitive psychology research the human reaction time to visual cues is ~200ms so we're pretty far from that, but that's definitely worth it. Especially since it costs nothing to minify the files.

I'd also like to try (just for the science) to check how does it look if I inline JS/CSS into HTML file, but our CSP forbids that. Can I manually turn it off?

If I can get someone to review patch in bug 819228, that is the only blocker for me.
(In reply to Zbigniew Braniecki [:gandalf] from comment #18)
> Yeah, it's a nice thing to win ~20% :)
>

I guess this number should be taken with care. First it is unclear to me what are the root cause of the slowness when loading multiple files and if there is an underlying platform bug we should really fix it instead of workaround it inside Gaia. Every apps that will be install will suffer this.

Also 20% for the time of this application. This number is probably some variables multiply by the number of files. So it is probably bigger on bigger app, smaller on smaller app.
 
> But overall, no, we're still in the noticable zone. 
> I'm trying to pinpoint the reasons, because we seem to be able to get close
> to 200ms (with Browser app), while with other apps we're dancing between
> 600ms and 2000ms. Even wake up from sleep app takes over 600ms.

As I said on IRC you should not try to compete with the browser. This one is in-process.

The 600ms comes from the animation (see bug 780692).

> 
> According to cognitive psychology research the human reaction time to visual
> cues is ~200ms so we're pretty far from that, but that's definitely worth
> it. Especially since it costs nothing to minify the files.
>

I do not agree with this assumption. And I guess the bugs linked in https://bugzilla.mozilla.org/show_bug.cgi?id=812396#c5 are about that.



> I'd also like to try (just for the science) to check how does it look if I
> inline JS/CSS into HTML file, but our CSP forbids that. Can I manually turn
> it off?
>

Add a pref called "security.csp.enable" to false in profile/user.js. (And then |adb push profile/user.js /data/local/user.js|
 
> If I can get someone to review patch in bug 819228, that is the only blocker
> for me.

Can you add the code it blocks on this bug?
(In reply to Vivien Nicolas (:vingtetun) from comment #19)
> I guess this number should be taken with care. First it is unclear to me
> what are the root cause of the slowness when loading multiple files and if
> there is an underlying platform bug we should really fix it instead of
> workaround it inside Gaia. Every apps that will be install will suffer this.

In desktop firefox, we found a non-zero cost for loading/interpreting any js files. More files = more costly file IO. Better actually to have one larger file than many small files (See browser.js, and the move to omnijar).

Basically, the less bytes off the filesystem, the better. The less js to parse, the better. The latest and laziest possible load of js, the better. We must do all of the things, and in this tiny device, even very small % wins do matter.

A question - Are we utilizing the startup cache for these JS files? Is it possible?
I retested this with startup animations and csp turned off.
I tested against three apps: Settings (lots of text), SMS (small), and Homescreen (boot time).

Settings:
                 |  Default    | combined    |   minified
-------------------------------------------------------------
firstPaint       |  1167 (24)  |  1177 (64)  | 1154 (64)
DOMContentLoaded |   919 (35)  |   705 (111) | 668 (123)
window.load      |  1082 (38)  |  1095 (110) | 1104 (98)

SMS:
                 |  Default    | combined    |   minified
-------------------------------------------------------------
firstPaint       |  1167 (24)  |  1177 (64)  | 1154 (64)
DOMContentLoaded |   919 (35)  |   705 (111) | 668 (123)
window.load      |  1082 (38)  |  1095 (110) | 1104 (98)

Homescreen:
                 |  Default    | combined    |   minified
-------------------------------------------------------------
firstPaint       |  3795 (660) |  3711 (202) | 3628 (532)
DOMContentLoaded |  1180 (139) |   967  (73) | 1004 (117)
window.load      |  1241 (136) |  1278  (55) | 1253 (180)

It boils down to:

 - if we remove CSP from the picture, there's little cost to keep multiple resource files
 - there seem to be no benefit to minify files over combining them
 - there seem to be a stable, 200ms win in DOMContentLoaded, that does not translate to window.load or firstPaint
 - although there seem to be no change in app startup time, there is a measurable statistically significant improvement of ~100ms in Homescreen firstPaint and reduced variance (ie. there are no cases with slowest (>4000k) firstPaints).

Hypothesis about the last one: under heavy load (booting) limiting the resource channels benefits load time.

My recommendation is to:
 - focus on fixing startup animation and CSP penalties as they give most formidable benefits (~450ms and ~150ms respectively)
 - investigate combining resource files at build time. I have a WIP patch that I'll clean and attach here tomorrow
 - do not go for minification

The benefit of this approach would be that we could not only reduce the variance and slow-cases in booting of the homescreen (and potentially other apps under resource constrains), but also earlier DOMContentLoaded times which enable earlier DOM operations for apps that interfere with DOM at startup.
Minification might be a memory win.  Would be interesting to measure.
(In reply to Zbigniew Braniecki [:gandalf] from comment #21)
> I retested this with startup animations and csp turned off.

These results don't seem to agree with that was seen in comment 0 and comment 6.  Can you apply this methodology to the Email app startup time?

It would also be cool to test with the Maps app and Cut The Rope, since they're in a sense more realistic web apps.  Donovan should be able to help get you packages for them.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #23)
> (In reply to Zbigniew Braniecki [:gandalf] from comment #21)
> > I retested this with startup animations and csp turned off.
> 
> These results don't seem to agree with that was seen in comment 0 and
> comment 6.  Can you apply this methodology to the Email app startup time?

Sure, will do that tomorrow morning.

I believe that the reason why they don't agree is bcause of CSP being on. When I tested load times with CSP (bug 820196) the penalty for loading each link was significant. With CSP turned off it's mitigated.

Now, depending on how much we can shave off of CSP cost we may end up with this still affecting app startup time.
I also wanted to test how combining and minifying will affect system app which has plenty of links - 65 script links and 48 CSS links.
It is pretty massive in result. Not only does it load over 110 links (at boot time), it also becomes pretty massive in size:
 - JS is just short of 500kb of code
 - CSS is 30kb of code

Both can be minified: JS by 56% and CSS by 32% (using YUI minifier).

I was unable to try that because the system code heavily relies on the code being loaded slower than parsing DOM (bug 820684) and in result when I delay it till DOMContentLoaded I cannot reproduce normal behavior (I see Update Sceen and no JS bugs in console), but my guess is that minification of that should give us boottime boost.
Email app:
                 |  default    |    no anim   |  no anim/csp
-------------------------------------------------------------
firstPaint       |  4563 (9)   |  4227  (64)  | 4039 (1)
DOMContentLoaded |  3975 (53)  |  3810 (157)  | 3750 (4)
window.load      |  4035 (36)  |  3892 (153)  | 3399 (5)

                 |  default    |   combined   |  minified
-------------------------------------------------------------
firstPaint       |  4563 (9)   |  4620  (63)  | 4392 (42)
DOMContentLoaded |  3975 (53)  |  3850  (86)  | 3623 (17)
window.load      |  4035 (36)  |  3902 (101)  | 3853 (98)

                 | no anim/csp |   combined   |  minified
-------------------------------------------------------------
firstPaint       |  4039 (1)   |  4106   (4)  | 3894 (49)
DOMContentLoaded |  3750 (4)   |  3659   (4)  | 3522 (30)
window.load      |  3399 (5)   |  3246   (4)  | 3159 (68)

Summary:
 - combination alone does not give a win for this app in firstPaint
 - minification gives ~200ms on firstPaint
 - turning off anim once again wins ~400ms
 - turning off CSP another ~150ms
 - minification, turning off CSP and turning off animation combines well giving ~600ms win in firstPaint, ~450ms win in DOMContentLoaded and ~850ms win in window.load
(In reply to Zbigniew Braniecki [:gandalf] from comment #26)
> Email app:
>                  |  default    |    no anim   |  no anim/csp
> -------------------------------------------------------------
> firstPaint       |  4563 (9)   |  4227  (64)  | 4039 (1)
> DOMContentLoaded |  3975 (53)  |  3810 (157)  | 3750 (4)
> window.load      |  4035 (36)  |  3892 (153)  | 3399 (5)
> 
>                  |  default    |   combined   |  minified
> -------------------------------------------------------------
> firstPaint       |  4563 (9)   |  4620  (63)  | 4392 (42)
> DOMContentLoaded |  3975 (53)  |  3850  (86)  | 3623 (17)
> window.load      |  4035 (36)  |  3902 (101)  | 3853 (98)
> 
>                  | no anim/csp |   combined   |  minified
> -------------------------------------------------------------
> firstPaint       |  4039 (1)   |  4106   (4)  | 3894 (49)
> DOMContentLoaded |  3750 (4)   |  3659   (4)  | 3522 (30)
> window.load      |  3399 (5)   |  3246   (4)  | 3159 (68)
> 
> Summary:
>  - combination alone does not give a win for this app in firstPaint
>  - minification gives ~200ms on firstPaint
>  - turning off anim once again wins ~400ms
>  - turning off CSP another ~150ms
>  - minification, turning off CSP and turning off animation combines well
> giving ~600ms win in firstPaint, ~450ms win in DOMContentLoaded and ~850ms
> win in window.load

Please keep the values you give in this bug minification only. The rest are different bugs and are only noise here :)
(In reply to Vivien Nicolas (:vingtetun) from comment #27)
> (In reply to Zbigniew Braniecki [:gandalf] from comment #26)
> > Email app:
> >                  |  default    |    no anim   |  no anim/csp
> > -------------------------------------------------------------
> > firstPaint       |  4563 (9)   |  4227  (64)  | 4039 (1)
> > DOMContentLoaded |  3975 (53)  |  3810 (157)  | 3750 (4)
> > window.load      |  4035 (36)  |  3892 (153)  | 3399 (5)
> > 
> >                  |  default    |   combined   |  minified
> > -------------------------------------------------------------
> > firstPaint       |  4563 (9)   |  4620  (63)  | 4392 (42)
> > DOMContentLoaded |  3975 (53)  |  3850  (86)  | 3623 (17)
> > window.load      |  4035 (36)  |  3902 (101)  | 3853 (98)
> > 
> >                  | no anim/csp |   combined   |  minified
> > -------------------------------------------------------------
> > firstPaint       |  4039 (1)   |  4106   (4)  | 3894 (49)
> > DOMContentLoaded |  3750 (4)   |  3659   (4)  | 3522 (30)
> > window.load      |  3399 (5)   |  3246   (4)  | 3159 (68)
> > 
> > Summary:
> >  - combination alone does not give a win for this app in firstPaint
> >  - minification gives ~200ms on firstPaint
> >  - turning off anim once again wins ~400ms
> >  - turning off CSP another ~150ms
> >  - minification, turning off CSP and turning off animation combines well
> > giving ~600ms win in firstPaint, ~450ms win in DOMContentLoaded and ~850ms
> > win in window.load
> 
> Please keep the values you give in this bug minification only. The rest are
> different bugs and are only noise here :)

Forgot about my last comment.
Just to add more clarity the e-mail app has one large js file > 1MB that is loaded.  I imagine minification helps because the size drops ~ 600K, and that makes a significant difference.
Target Milestone: --- → B2G C3 (12dec-1jan)
Chris, do you still want this based on the comments post #23?
Flags: needinfo?(jones.chris.g)
Yes, I would take this for a 200ms startup win.

gandalf, we've landed a platform change that makes the load animation "free".  Can you rerun your test in comment 26 to see if we're still getting the same win from combine+minify?
Flags: needinfo?(jones.chris.g)
(In reply to Chris Jones [:cjones] [:warhammer] from comment #31)

> gandalf, we've landed a platform change that makes the load animation
> "free".  Can you rerun your test in comment 26 to see if we're still getting
> the same win from combine+minify?

Oh, that's awesome! I was planning to file a bug about animation cost. I'll check that! And we also landed CSP patch (bug 820196) so it makes sense to remeasure the impact.
The next thing here is bug 815523 which may have impact on this. Not sure how far we are from landing that.
gandalf, where you able to re-run the tests?  I think this particular bug is getting close to coming off the blocker list.
Flags: needinfo?(gandalf)
i agree with cjones, for 200ms, this would be really excellent and exciting.  However at this point, we wont' block v.1 release for this -- but would want it.  Moving this to a soft-blocker.
blocking-basecamp: + → -
tracking-b2g18: --- → +
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

This should be working fairly well. I made some changes to the old webapp-l10n.js file to reflect the changes. The downside of this approach is we don't get the benefits of these changes when the default locale is not used. 

I don't really want to start modifying index.html then restoring it back after but that would be the alternative approach.

I did some basic testing launching all the apps doing a quick once over.
We should definitely have some more eyes on this.
Attachment #702079 - Flags: review?(kaze)
Attachment #702079 - Flags: review?(21)
the perf impact for homescreen, template app, settings, sms and calendar:

                     pre-patch         patch       delta
-----------------|---------------|--------------|----------|
homescreen [c]   |     3397 (10) |    3125 (42) |  -271    |
homescreen [dom] |     1387 (14) |    1216 (82) |  -171    |
homescreen [doc] |     1513  (4) |    1333 (66) |  -180    |


                     pre-patch         patch       delta
-----------------|---------------|--------------|----------|
template [c]     |     858   (5) |    872  (13) |   +14    |
template [dom]   |     408  (14) |    432   (9) |   +23    |
template [doc]   |     421   (9) |    437   (9) |   +16    |


                     pre-patch         patch       delta
-----------------|---------------|--------------|----------|
settings [c]     |     2290 (47) |    2134 (44) |  -155    |
settings [dom]   |     1229 (25) |    1149  (7) |   -80    |
settings [doc]   |     1599 (44) |    1422 (29) |  -176    |

                     pre-patch         patch       delta
-----------------|---------------|--------------|----------|
sms [c]          |     1647 (14) |    1600 (23) |   -47    |
sms [dom]        |      897 (12) |    838  (21) |   -59    |
sms [doc]        |     1224 (17) |    1171 (22) |   -53    |

                     pre-patch         patch       delta
-----------------|---------------|--------------|----------|
calendar [c]     |     3032 (51) |    2461 (11) |   -571   |
calendar [dom]   |     2329 (57) |    1745 (11) |   -583   |
calendar [doc]   |     2555 (59) |    1982  (9) |   -573   |


No impact on template app as expected, significant impact on homescreen and settings and massive win on calendar app.
Flags: needinfo?(gandalf)
Assignee: gandalf → jlal
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

Redirecting review to Alexandre.
Attachment #702079 - Flags: review?(21) → review?(poirot.alex)
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

(In reply to James Lal [:lightsofapollo] from comment #36)
> Comment on attachment 702079 [details]
> Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603
> 
> This should be working fairly well. I made some changes to the old
> webapp-l10n.js file to reflect the changes. The downside of this approach is
> we don't get the benefits of these changes when the default locale is not
> used. 

Actually it is. The l10n code is fired with Makefile default values, and you call the js concatenation code on each html file generated by l10n. FYI, we are only creating one html file for the default locale, next to the original html file. The webapp-zip step is going to ship the default locale html file instead of the original html file.

So your approach is valid and I don't see major issue in it except the original issue behind l10n compile code: pollution of your repo. We would need an objdir in order to implement both of those operations sanely without entangling them.

Otherwise your patch looks good, the only thing I'd like to have changed is the handling of complex cases: mix of strict/non-strict and mixed script type. My call would be to avoid concatenation in those cases and print warning messages instead. So that we can keep the concatenation code simple and eventually tune application code in order to optimize them later. As said in pull request, I think it is really better to avoid optimizing complex cases instead of eventually introduce complex build time bugs.

Please request review again with changes or with your opinion if you are thinking otherwise.
Attachment #702079 - Flags: review?(poirot.alex)
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

After thinking about it I am not a big fan of trying to detect the mixed "use strict;" statements. It slows things down and could be error prone in some cases. I think the real issue we are avoiding is breaking apps because they where never written to have their files merged. To work around this I did the following:

- you must opt in to have your app's js files aggregated (I did this only for calendar so far)

- you can exclude specific files from the aggregation (for benchmarking/whatever) by adding an data-attribute ex: <script data-skip-optimize ... ></script>

I think this is the best bet, if you app is slow and you have a ton of files simply opt-in then test your app. Then we can opt-in or out and the people who know most about their apps can actually test it out for themselves.

----

Benchmarks from calendar:

load is window.onload
localize is window.addEventListener('localized')
dom is window.addEventListener('DOMContentReady')

I took measurements by injecting a script before all others (without defer) and enabling/disabling aggregation for calendar. This is taken over two reboots with 4 app launches/kills each.
  
            load      localize      dom
           ------ |  ---------- |  ------ |
  APPLIED:  1364  |    1187     |   1189  |
   BEFORE:  1763  |    1656     |   1660  |
Attachment #702079 - Flags: review?(kaze) → review?(poirot.alex)
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

The blacklist pattern looks like a safe way to go.
I tested calendar app and change default locale while looking at the generated zip, and everything looks good.
Attachment #702079 - Flags: review?(poirot.alex) → review+
We have a patch waiting to land, there is very little risk here as we have chosen a per-app opt-in model. For most apps there is significant gain by aggregating JS files (see calendar numbers). I think we should land this for calendar at least.
blocking-b2g: --- → tef?
This patch is for JS only, given that this seems to be IO overhead should I proceed to write a separate patch for the CSS assets? Some apps are using a _ton_ of css files (system comes to mind) and even simple apps use more then one because of building blocks.
Flags: needinfo?(21)
Summary: [perf] Gaia Build process needs to combine and minify js/css content → [perf] Gaia Build process needs to combine js content
blocking-b2g: tef? → -
blocking-basecamp: - → ---
Would take on approval given testing, but not blocking on it.
blocking-basecamp: --- → -
FWIW, the new rule is that only blockers can land. So it's not actually true that we'd take a patch no matter how well tested as long as this isn't marked as a blocker.
(In reply to Jonas Sicking (:sicking) from comment #45)
> FWIW, the new rule is that only blockers can land. So it's not actually true
> that we'd take a patch no matter how well tested as long as this isn't
> marked as a blocker.

Only blockers and patches with approval, right?
I'm honestly not sure any more. But until I get clarified otherwise, I'll go with that assumption.
Master is open again so I landed it there...

https://github.com/mozilla-b2g/gaia/commit/53c83ac323095856522bc00aa41d20fda1594224

This is not any v1.0X release but I honestly think the performance benefits are worth it considering where we are at now.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Not sure what needs info from me in this bug.
Flags: needinfo?(21)
Component: Builds → Gaia
Comment on attachment 702079 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/7603

Well-validated, long-baked patch that's a startup win for calendar, and a prerequisite for bug 837111 which is a win for other apps in addition.

(Sigh how did this slip through the cracks ...)
Attachment #702079 - Flags: approval-gaia-v1?(21)
Attachment #702079 - Flags: approval-gaia-v1?(21) → approval-gaia-v1+
This commit does not apply cleanly to v1-train.  If this patch depends on another bug, please comment here and I will retry when that bug is approved to land on all branches that this bug needs to land on.  If the merge conflict needs to be resolved by hand, the following commands could be a useful starting point:

cd gaia
git checkout v1-train
git cherry-pick -x -m1 53c83ac323095856522bc00aa41d20fda1594224
<resolve merge conflict>

#	both modified:      build/utils.js
Discussed in triage and we can take this on v1.0.1 so that perf tests can be run there - marking this npotb since it's build scripts/makefiles and we assume any bustage would be obvious (breaking builds) whereas the win is more perf test data.
Whiteboard: [nptob]
Whiteboard: [nptob] → [nptob][FFOS_perf]
v1-train: 69cfb9f7d00e5ee7886e40645240686091b6b57e
v1.0.1: 1a6e1cd7715a5192b32db4b7127c5ae5b8162a7a
You need to log in before you can comment on or make changes to this bug.