Closed Bug 1295826 Opened 8 years ago Closed 6 years ago

[Tracking] Fix SeaMonkey's Testing infrastructure.

Categories

(SeaMonkey :: Testing Infrastructure, defect)

defect
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ewong, Assigned: ewong)

References

(Blocks 1 open bug)

Details

Attachments

(14 files, 8 obsolete files)

6.66 KB, patch
frg
: review+
Details | Diff | Splinter Review
9.73 KB, patch
frg
: review+
Details | Diff | Splinter Review
29.21 KB, patch
frg
: review+
Details | Diff | Splinter Review
882 bytes, patch
Details | Diff | Splinter Review
1.54 MB, patch
Details | Diff | Splinter Review
2.35 MB, patch
Details | Diff | Splinter Review
1.00 MB, patch
Details | Diff | Splinter Review
1.16 MB, patch
Details | Diff | Splinter Review
925.38 KB, text/plain
Details
4.50 KB, patch
Details | Diff | Splinter Review
3.21 KB, patch
Details | Diff | Splinter Review
848 bytes, patch
Details | Diff | Splinter Review
2.46 MB, text/plain
Details
62.29 KB, patch
frg
: review+
Details | Diff | Splinter Review
Right now, our tests are in a mess (we haven't caught up with the changes in how 
the tests are packaged and what's downloaded) and instead of getting the slaves
to build broken tests (when their processing can be used in building 
regular builds), the unittests should be disabled.
Attachment #8781803 - Flags: review?(bugspam.Callek)
Summary: Disable all tests until bug 1209378 is fixed. → [Tracking] Fix SeaMonkey's Testing infrastructure.
Bug 1209378 was morphed to a TB bug since it fixed their packageURL issue.
So morphing this bug to track our SeaMonkey testing infra.
Assignee: nobody → ewong
Attachment #8781803 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #8781803 - Flags: review?(bugspam.Callek)
Attachment #8782008 - Flags: review?(bugspam.Callek)
Comment on attachment 8782008 [details] [diff] [review]
[configs] disable unittests and l10n dep nightlies

going for a post-land-review
https://hg.mozilla.org/build/buildbot-configs/rev/3bcc86136b6b6e030d785be4ae6b93c4483783f1
Bug 1295826 - Disable unittests and l10n dep builds. r=post-land-review
(In reply to Edmund Wong (:ewong) from comment #5)
> https://hg.mozilla.org/build/buildbot-configs/rev/
> 3bcc86136b6b6e030d785be4ae6b93c4483783f1
> Bug 1295826 - Disable unittests and l10n dep builds. r=post-land-review

Backed out http://hg.mozilla.org/build/buildbot-configs/rev/a1eb00a0b64b for 
possibly disabling the nightlies.
Depends on: 1343800
Attachment #8782008 - Attachment is obsolete: true
Attachment #8782008 - Flags: review?(bugspam.Callek)
Attached patch [custom] add steps to run tests (obsolete) — Splinter Review
Attachment #8895686 - Flags: review?(frgrahl)
To note:

The way tests are run has changed *very* dramatically since our last successful test run and as 
such, I'm just gleaming what I can from the tests run from firefox and translating their 
steps to buildbot.  Back then, I wasn't even familiar with the tests so let it be known that
this new way of testing is also unfamiliar.

The steps as given 'worked' for mochitest-1 for linux32 (well, the tests ran... just not successfully and I'm not 
sure if it's due to how I'm running the tests or our binaries are not compatible with the tests or
something else).

These three patches are a starting point to get our tests running.

Once these patches are pushed and a successful debug build is done, we should know whether or not
more is needed.  (I am betting on "more is needed").
further rationale for the patches:

tooltool:
-  we need the tooltool manifests in tools because 1) we don't pull the source so we can't put the testing 
   manifests in suite/config, so putting them in tools/testing is the 'best' place since we do pull tools
- for linux*, we need both gtk3 and gcc tooltool packages (firefox doesn't use the gtk3/gcc tooltool
   packages .. why? because I'm guessing their Taskcluster images have proper gtk3/gcc built in..)
   to run xpcshell

buildbot-config:
  - the new steps require pip and the usage of pypi.pub.build.mozilla.org/pub.  (I'm assuming since it's
    public, I don't need permission to get the pypi packages).

buildbotcustom:
  - a new set of steps required:
     a) install a virtualenv
     b) install some requirements (requirements.txt)
     c) install mozbase requirements
     d) install marionette requirements
     e) install websocket requirements  [ requires being run in a mock env]
     f) set up the right tooltool packages
     g) mozinstall the application
     h) run test

That is the extent of the new steps.  I'm hoping to have them running so I can adapt future 
patches to fix whatever needs fixing.
Attached patch [custom] add steps to run tests (obsolete) — Splinter Review
marionette_reqs are to be installed after the unpack of tests as that's
where the configs are from.
Attachment #8895686 - Attachment is obsolete: true
Attachment #8895686 - Flags: review?(frgrahl)
Attachment #8895699 - Flags: review?(frgrahl)
to note:

# of chunks have changed.  In the past, there were 5 chunks.  Now, Firefox has 10.  Since
I'm not familiar with the testing part of things (serge, where are you?), I'll have to
play it by ear and will go for 10. (pending patches)
Comment on attachment 8895683 [details] [diff] [review]
[tools] add test_tooltool_manifests to testing/tooltool_manifest

For Fx the manifests seem to be located in testing\config\tooltool-manifests. Samy for our c-c products.

If nothing whould oppose it should imho be also done this way in tools to have an identical structure for all manifest files so should be testing\config\tooltool-manifests.

> testing/tooltool-manifests/linux32/releng.manifest

digests have blanks in them.

Overall: indent is not consistent between them. Some start always on column 1 and other are indented. Might be the same for the packages where you copied them already. Can live with it if you think not worth fixing.

r+ with digest errors above fixed and first question answered/fixed.

Thanks for tackling it.
Attachment #8895683 - Flags: review?(frgrahl) → review+
Comment on attachment 8895684 [details] [diff] [review]
[configs] add pip_server and test_tooltool_manifest_src entries

r+ with eventually the path for the manifests changed.
Attachment #8895684 - Flags: review?(frgrahl) → review+
Comment on attachment 8895699 [details] [diff] [review]
[custom] add steps to run tests

Just looked only more or less formally over it but looks good. Just lets try it and see what needs fixing afterwards.
Attachment #8895699 - Flags: review?(frgrahl) → review+
(In reply to Frank-Rainer Grahl (:frg) from comment #14)
> Comment on attachment 8895683 [details] [diff] [review]
> [tools] add test_tooltool_manifests to testing/tooltool_manifest
> 
> For Fx the manifests seem to be located in
> testing\config\tooltool-manifests. Samy for our c-c products.
> 
> If nothing whould oppose it should imho be also done this way in tools to
> have an identical structure for all manifest files so should be
> testing\config\tooltool-manifests.
> 

ah. sorry  should be manifests


> > testing/tooltool-manifests/linux32/releng.manifest
> 
> digests have blanks in them.

Will fix this.

> 
> Overall: indent is not consistent between them. Some start always on column
> 1 and other are indented. Might be the same for the packages where you
> copied them already. Can live with it if you think not worth fixing.
> 
> r+ with digest errors above fixed and first question answered/fixed.
> 
> Thanks for tackling it.

I'm not going to indent any of the manifests.  As you mentioned, the indentation
is inconsistent so either we indent the manifests, or we just have them
left justified.

Edmund
Severity: normal → critical
over irc, the decision was to do the indentation on the manifests.
Attachment #8895683 - Attachment is obsolete: true
Attachment #8896996 - Flags: review+
Attachment #8896996 - Attachment is obsolete: true
Attachment #8896998 - Flags: review?(frgrahl)
Comment on attachment 8897003 [details] [diff] [review]
[tools] add tooltool manifests for tests (w/o space in digests and move to under testing/config)

r+++
Attachment #8897003 - Flags: review?(frgrahl) → review+
Comment on attachment 8897005 [details] [diff] [review]
[configs] add tooltool manifest src paths for testing infra, and add pip server url

And another r+
Attachment #8897005 - Flags: review?(frgrahl) → review+
found major issues in the previous patch after I had pushed it.  (backed
out)
Attachment #8895699 - Attachment is obsolete: true
Attachment #8897712 - Flags: review?(frgrahl)
Comment on attachment 8897712 [details] [diff] [review]
[custom] add steps to run tests

Looks good but so did the last one :)

r+
Attachment #8897712 - Flags: review?(frgrahl) → review+
Attachment #8924810 - Attachment description: [tools] additional package required → [tools] additional package required (virtualenv) [checked-in]
Attached file mochitest_log.txt
This is the very *FIRST* mochitest-1 test run in over three years or more.
A lot of stuff have happened since then and the test run shows how
far we need to fix as after a bit, it crashes and produces a crapton
of tracebacks... *sigh*
Depends on: 1417311
No longer depends on: 1417311
Attached file timeout log
currently our tests are timing out.
(In reply to Edmund Wong (:ewong) from comment #36)
> Created attachment 8931180 [details]
> timeout log
> 
> currently our tests are timing out.

attached is the log..  a lot of assertions and not quite good looking results :(
With this patch, the tests "run" [the infrastructure is in place, just need
to fix the actual tests, which will require new bugs].  Since this bug is 
getting long in the tooth, once I've applied the reviewed patch, I'm
closing this bug. [just a heads up] (unless someone disagrees)
Attachment #8939357 - Flags: review?(frgrahl)
Comment on attachment 8939357 [details] [diff] [review]
[custom] proposed patch [checked-in]

Thanks
Attachment #8939357 - Flags: review?(frgrahl) → review+
Comment on attachment 8939357 [details] [diff] [review]
[custom] proposed patch [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/7e5d72dc0fd2
Attachment #8939357 - Attachment description: [custom] proposed patch → [custom] proposed patch [checked-in]
Closing bug as 'fixed' only because the infrastructure fix was the purpose.  The actual fixing
of the tests will be done in subsequent bugs.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.