1037084 - Move TPS test framework out of mozilla-central into its own repository

Status: RESOLVED WONTFIX

(Testing Graveyard :: TPS, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

While working with TPS over the last couple of months I see various problems by having this framework located in mozilla-central. I will describe it in more details below. But because of all of that, I want to propose that we move all the code to it's own repository e.g. on Github or git.mozilla.org.

Reasons why I propose this transition:

1. The TPS tests located in the tree are NOT run by buildbot, and also won't ever be run there. Reason is that we have full integration tests, which need network access to reach the real FXA servers. So it's not allowed at all.

2. While updating TPS we have to land the same changes on all the branches. As jgriffin pointed out, there can be situations that different branches will get different code, but to be realistic - will that really happen at this stage? Also if it will happen we could have conditional includes for the fxa and/or sync module. That would execute different code paths for each type of build. 

3. When landing changes on mozilla-central or even inbound nowadays, we have to wait for a new Firefox build. Once we got this we can test if the changes didn't break anything. Also tree closures can delay the landing of patches.

4. To run TPS you will need the full packaged tests. Meanwhile it's a large file to be downloaded, for only a couple of KB of data. Well, you can say that also the tests are included, but here we might also think of moving them out?

5. You cannot keep the virtual env for TPS over a couple of versions, because you don't know if something has changed, when you download the tests of the next build. So you always have to create the venv from fresh.

6. Easier version handling on PyPI. If we would keep the version of TPS on the different branches, I could not imagine how to handle it on PyPI.

7. Based on 5) and 6) we cannot create a pre-configured environment for TPS as what we do for Mozmill. That way we could drastically lower the amount of dependencies to be installed on the testing machines. Also we could see a drastic speed-up, because the venv doesn't have to be created for each job.

Here some cons:

1. We will loose the hard binding between the Firefox code and TPS, something similar as what we have with Mozmill. So far we don't really had a problem with the latter mentioned framework.

2. Risk in regressions and that fallback code is necessary. I consider this as not too much risky. If there are problems we could include modules based on the version as mentioned above.

Please let me know what you think about this proposal, and if we wanna do that. If you have concerns or other improvements to tell, please do so. All input is appreciated.

If we decide to do that transition, I would like to start it as soon as possible because it blocks us from finishing off the new Jenkins-CI for TPS on our puppetized hosts. If we discard that transition lots of requirements for puppet have to be fulfilled. See bug 1036321.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Putting needinfo on Jonathan and Edwin, to get their feedback.

Comment 2

[Andrei Eftimie]

This might indeed speed up development + test execution.
Would make sense to unbundle the TPS tests as well. The bundle is now at 100-150MB where we only use ~283KB

Comment 3

[Jonathan Griffin (:jgriffin)]

The biggest disadvantage is that if Sync goes through another major transition and you need to support both old and new, things will get messy quickly.

But, I can see there are a lot of advantages here, that probably outweigh this risk.

Comment 4

[Edwin Wong [:edwong]]

This tension exists with most services automation. I would say that we should organize ourselves for maximum efficiency now (next 6-12 months) versus building for 1-3 yrs out. This is a bit more 'lean' approach.  With that in mind, let's move it out of m-c.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Wonderful. Lets lets get started with it. I did a quick look in what is possible and the convert extension of hg (http://mercurial.selenic.com/wiki/ConvertExtension) might be able to let us strip off at least all the unwanted files from a current mozilla-central repository. With keeping the history we should then be able to convert it to a git repository.

So what we need is:

* Only keep the files and history related to TPS, which are located here (--filemap):
** http://mxr.mozilla.org/mozilla-central/source/testing/tps/
** http://mxr.mozilla.org/mozilla-central/source/services/sync/tps/

* Try to convert the existing author names to github users (--authormap)

* Reorganize the folders in the repo:
** /testing/tps => tps/
** /services/sync/tps/extensions => tps/extensions

We should be able to get rid of the mozmill extension and make it a dependency for the tps python package. But I would defer that to later, because it might be a bit more complicated.

I talked with Hal on IRC about this problem. He mentioned to me that he will have a look at those steps and what would be possible. So putting ni for him. Thanks in advance!

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hal, do you have any feedback meanwhile? We kinda would like to get this work started but are blocked on your feedback. Thanks

Comment 7

[hwine]

Thanks for the ping - my apologies that you needed to do it.

As I read comment #5, I believe you're trying to do is:
 a) remove TPS from mozilla-central
 b) move away from hg and mozilla resources to github use only

Moving off moco hosted resources means:
 - no more mxr/dxr indexing (afaik)
 - no direct pull of repo material onto the test slaves (this is a opsec goal, not yet fully implemented)

If I have the above correct, then I am definitely not a blocker. I think the following would work for you.
 - make sure no one else in the dev (especially sync-dev) community objects
 - do the extract as you've noted, with the options you want
 - convert to git with the hg-git extension
 - push to a new repo on github
 - commit the file deletions to m-c in coordination with sheriffs
 - profit!

I can help with the creation of the repo on github if it's in the main mozilla account.

Comment 8

[Mark Hammond [:markh] [:mhammond]]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #7)
> Moving off moco hosted resources means:
>  - no more mxr/dxr indexing (afaik)

While I won't try and make this block, this might be something we live to regret - if I'm changing sync and notice a TPS dependency, I'll explicitly call it out.  So this might cause "unexpected" problems due to me becoming unaware that TPS directly uses certain sync functions.  Expecting me (or anyone else touching sync code) to remember to do a search on github repos isn't realistic.  However, if the perceived benefits outweigh these risks, then go for it :)

Comment 9

[Richard Newman [:rnewman]]

N.B., CrossWeave (the original name for TPS) was originally hosted at http://hg.mozilla.org/automation/crossweave/, outside mozilla-central.

Bug 674097 moved it into mozilla-central, largely because the testing code (e.g., the TPS add-on) is tightly coupled to the Sync and Places APIs, but also because it was intended that TPS run as part of TBPL, so that when a commit inadvertently breaks Sync, the sheriffs will notice.

I encourage you to familiarize yourself with that bug, and the motivations around the original move -- talking to jgriffin if necessary -- before undoing it.

Comment 10

[Cosmin Malutan, [:cosmin-malutan]]

Those are the steps required for 
First of all you'll need to have the extensions installed and enabled:
  In your .hgrc file please add:
>[extensions]
>hgext.convert=
>hgext.bookmarks =
>hggit = 
  To install hggit run:
>easy_install hg-git

  Create a file map with which need to be included/excluded/renamed:
    I'll attache this
  Run the command below to remap the repository:
>hg convert --filemap files_map.ini mozmill-central tps
  move to newly created tps directory
  run 'hg up' so the changes you made would be visible
  Add bookmarks for the branches we have under the hg repository, aurora/beta/release/esr31
>hg bookmark -r default master
>hg bookmark -r mozilla-aurora aurora
>hg bookmark -r mozilla-beta beta
>hg bookmark -r mozilla-release release
>hg bookmark -r mozilla-esr31 esr31
  You have to bee SSH-ly authenticated to github, then push the changes:
>hg push git+ssh://[e-mail]:mozilla/tps.git
  The tps repository has to exists under mozilla account

Done, now you have the TPS code outside of mozilla-central and you can carefully remove it from there

Comment 11

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: dest.ini

This is the file for mapping the directories, we don't have to map the users because all the users have the same email address for both mercurial and github. so the steps from the comment above should be enough.

Comment 12

[Cosmin Malutan, [:cosmin-malutan]]

I pushed the converted repository to my github account so here is how it looks like:
https://github.com/cosmin-malutan/pages/commits/master

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Not blocked anymore. We simply have to find the time doing it.

Comment 14

[Cosmin Malutan, [:cosmin-malutan]]

Henrik, do we still care about this? I could help if you want.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Currently this is very low priority. Also because the tests are currently not working as expected due to the server issue via bug 1066493.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that TPS could use taskcluster in the future and basic configurations are part of the tree, it might be better to keep the tess in mozilla-central. In case of our Firefox UI tests we are also going away from our own CI instance and will move the tests to mozilla-central by next quarter.

I would vote for wontfix'ing this bug.

Comment 17

[Mark Hammond [:markh] [:mhammond]]

SGTM!