Open Bug 748248 (ringmark) Opened 8 years ago Updated 3 years ago

Pass ringmark test suite

Categories

(Testing :: General, defect)

52 Branch
defect
Not set

Tracking

(Not tracked)

REOPENED

People

(Reporter: mounir, Unassigned)

References

()

Details

No description provided.
Depends on: ringmark-ring0
Depends on: ringmark-ring1
Alias: ringmark
Depends on: 749367
No longer depends on: 749367
Given Ringmark's use of browser sniffing, I suggest this bug be taken with a grain of salt.
Update. Last May, in an attempt to further understand the above claim that Ringmark uses browser sniffing, I sent the following email to Ms2ger:

--------------------------------------------------------------

Hello, I've elected to contact you directly instead of adding unnecessary noise to the ticket filed on Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=748248) where you posted:
 
> Given Ringmark's use of browser sniffing, I suggest this bug be taken with a grain of salt.

Can you be more specific? The only "browser sniffing" that Ringmark ever used was to temporarily disable 2 tests that were incapable of running correctly in Opera. These tests have since been completely re-written to ensure that all modern browsers are capable of running them (this issue was iframes not firing an onload event in Opera).

The Ringmark tests aren't supposed to be "spec tests" - they are designed to mimic the way developers use web technology, as closely as is reasonably possible. 

I'm reaching out to you to help improve the tests


--------------------------------------------------------------

I have not received a response.


Unequivocally, Ringmark does not "browser sniff". If there are any questions or concerns or any need for clarification, please feel free to contact me either directly or via this ticket.
Blocks: b2g-v-next
David, Just a heads up—my company Bocoup is now a developer partner and I'm now working full time on Gaia development. As the defacto maintainer of Ringmark, I can attend to issues regarding the test suite with an increased degree of immediacy
Marking all tracking bugs which haven't been updated since 2014 as INCOMPLETE.
If this bug is still relevant, please reopen it and move it into a bugzilla component related to the work
being tracked. The Core: Tracking component will no longer be used.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
Product: Core → Core Graveyard
I'm reopening this, as we had a regression in Ringmark Level 0 with Firefox 52.

We should at least figure out a way to get Ringmark level 0 in our testing (automated or otherwise).

Even if we don't like it, phone manufacturers use it as a standard part of their testing.
Component: Tracking → General
Product: Core Graveyard → Testing
QA Contact: chofmann
Version: Trunk → 52 Branch
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Facebook has abandoned the project. https://github.com/facebookarchive/rng.io
> Facebook has abandoned the project. https://github.com/facebookarchive/rng.io

We still have major phone vendors using at least level 0 for testing, so we should not regress in that area.

We did have a regression - bug 1347164 - and had we been at least looking at this test, we would have found it.

So the request here is that we do level 0 testing, even knowing that Facebook is no longer maintaining it.
Seems like if there's valid tests within this suite, we should at a minimum be trying to get them included in the web-platform-tests or something?
(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)
> Seems like if there's valid tests within this suite, we should at a minimum
> be trying to get them included in the web-platform-tests or something?

ni'ing annevk on what it would take for this to happen. And overlap with existing tests, etc.
Flags: needinfo?(annevk)
Do other browsers care about these tests? If they do, adding them to web-platform-tests seems like a good idea and I recommend filing an issue there as a first step.

Migrating these tests would be quite a bit of work though by the looks of it. Are there resources for that? (I can certainly take a look at areas I'm familiar with and port those, once we've decided this is actually something worth doing.)

Note that we're getting closer to code coverage reporting too, which might help us make a decision one way or another.
Flags: needinfo?(annevk)
(In reply to Anne (:annevk) from comment #11)
> Do other browsers care about these tests? If they do, adding them to
> web-platform-tests seems like a good idea and I recommend filing an issue
> there as a first step.

https://github.com/w3c/web-platform-tests/issues/5167
Mike: you mentioned major phone vendors use the test suite. Could you elaborate about this in the github issue if that can help with convincing this is worthwhile?
Flags: needinfo?(mozilla)
Will do.
Flags: needinfo?(mozilla)
(In reply to Anne (:annevk) from comment #11)
> Do other browsers care about these tests? If they do, adding them to
> web-platform-tests seems like a good idea and I recommend filing an issue
> there as a first step.
> 
> Migrating these tests would be quite a bit of work though by the looks of
> it. Are there resources for that? (I can certainly take a look at areas I'm
> familiar with and port those, once we've decided this is actually something
> worth doing.)
> 

This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests)
> This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests)

I'm wondering how much overlap there is between Ringmark and the current web-platform-tests?

There's more information about web-platform-tests here:

https://developer.mozilla.org/en-US/docs/Mozilla/QA/web-platform-tests

Did you (Boucoup) ever do any automation around Ringmark, or was there an expectation of a manual test?
(In reply to Mike Kaply [:mkaply] from comment #15)
> > This is something that we at Bocoup are interested in committing to. (reason: we are the original authors of these tests)
> 
> I'm wondering how much overlap there is between Ringmark and the current
> web-platform-tests?

I would certainly hope that there was, but Ringmark was never intended to be exhaustive spec-centric testing, FB's instruction was to test features that were important to specific use cases where the mobile web was struggling against native. 

> 
> There's more information about web-platform-tests here:
> 
> https://developer.mozilla.org/en-US/docs/Mozilla/QA/web-platform-tests

I'm familiar, but thank you none-the-less :)

> 
> Did you (Boucoup) ever do any automation around Ringmark, or was there an
> expectation of a manual test?

The rng.io website is just running a set of generic tests in the Ringmark project and then doing something interesting with the results. This work was done about 5 years ago, and FB had a specific set of goals that it wanted us to achieve; at the time automation was not one of those goals, but I'm confident that these tests could easily be repurposed (given resources)
You need to log in before you can comment on or make changes to this bug.