Closed Bug 1344805 Opened 3 years ago Closed 3 years ago

Get AWSY running on Windows

Categories

(Testing :: AWSY, enhancement)

enhancement
Not set

Tracking

(firefox54 fixed, firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox54 --- fixed
firefox55 --- fixed

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(2 files, 2 obsolete files)

Once bug 1272113 lands we need to get AWSY running on Windows in AWS.
We can test this before it lands though right?
I presume that we will be this testing prior to landing via try runs as pyang has done with the original AWSY move to Taskcluster bug for Linux.
Attached patch awsy-windows.patch (obsolete) — Splinter Review
Joel: Perhaps it would be obvious to you why sy is not even being invoked using this patch in https://treeherder.mozilla.org/#/jobs?repo=try&revision=2804ff01f2fa34eb8f0032f2cce1166f6f610c19 ?
Assignee: nobody → bob
Attachment #8850936 - Flags: feedback?(jmaher)
it is tier3:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=2804ff01f2fa34eb8f0032f2cce1166f6f610c19&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false

and there is an error:
Traceback (most recent call last):
  File "mozharness\scripts\awsy_script.py", line 172, in <module>
    awsy_test = AWSY()
  File "mozharness\scripts\awsy_script.py", line 56, in __init__
    super(AWSY, self).__init__(**kwargs)
  File "Z:\task_1490314230\mozharness\mozharness\base\vcs\vcsbase.py", line 125, in __init__
    super(VCSScript, self).__init__(**kwargs)
  File "Z:\task_1490314230\mozharness\mozharness\base\script.py", line 1789, in __init__
    rw_config = ConfigClass(config_options=config_options, **kwargs)
  File "Z:\task_1490314230\mozharness\mozharness\base\config.py", line 265, in __init__
    self.parse_args(args=option_args)
  File "Z:\task_1490314230\mozharness\mozharness\base\config.py", line 481, in parse_args
    options.config_files + options.opt_config_files, options=options
  File "Z:\task_1490314230\mozharness\mozharness\base\config.py", line 446, in get_cfgs_from_files
    all_cfg_files_and_dicts.append((cf, parse_config_file(cf)))
  File "Z:\task_1490314230\mozharness\mozharness\base\config.py", line 159, in parse_config_file
    execfile(file_path, global_dict, local_dict)
  File "mozharness\configs\awsy\windows_config.py", line 12, in <module>
    "virtualenv_python_dll": os.path.join(os.path.dirname(sys.executable), 'python27.dll'),
NameError: name 'sys' is not defined
doh! Thanks!
Comment on attachment 8850936 [details] [diff] [review]
awsy-windows.patch

Review of attachment 8850936 [details] [diff] [review]:
-----------------------------------------------------------------

beware of windows 10 as there could be OS changes there, but lets get tests running on there first!

::: testing/mozharness/configs/awsy/windows_config.py
@@ +41,5 @@
> +    "default_blob_upload_servers": [
> +        "https://blobupload.elasticbeanstalk.com",
> +    ],
> +    "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),
> +    "metro_harness_path_frmt": "%(metro_base_path)s/metro/metrotestharness.exe",

we don't do metro anymore
Attachment #8850936 - Flags: feedback?(jmaher) → feedback+
erahm: Any idea what the failure is here? https://treeherder.mozilla.org/logviewer.html#?job_id=86546342&repo=try&lineNumber=1514
Flags: needinfo?(erahm)
Do you think this is the escape issue on Windows that you mentioned?
(In reply to Bob Clary [:bc:] from comment #8)
> Do you think this is the escape issue on Windows that you mentioned?

Yes, it looks like the same issue.
Flags: needinfo?(erahm)
* adjusts windows \ in checkpoint file path in testing/awsy/awsy/test_memory_usage.py

* moves the test pages out of the awsy module directory since this caused windows to barf when doing a pip install awsy.

* uses the taskcluster windows configs as the starting point instead of the talos windows configs.

try runs:

https://treeherder.mozilla.org/#/jobs?repo=try&author=bclary@mozilla.com&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=7bb1bed7ef41e37a24768b189347904d6984edac&group_state=expanded

First run was windows 10 64bit only. Next was windows 10 64bit and linux 64 bit. The last one is windows 10 32bit, 64bit and linux 64bit.

Note the path too long errors unzipping t5pn.zip:
https://treeherder.mozilla.org/logviewer.html#?job_id=86661686&repo=try&lineNumber=742
Attachment #8850936 - Attachment is obsolete: true
Attachment #8851627 - Flags: review?(pyang)
Attachment #8851627 - Flags: review?(jmaher)
Attachment #8851627 - Flags: review?(erahm)
Comment on attachment 8851627 [details] [diff] [review]
awsy-windows.patch v2

Review of attachment 8851627 [details] [diff] [review]:
-----------------------------------------------------------------

nice :bc!
Attachment #8851627 - Flags: review?(jmaher) → review+
Comment on attachment 8851627 [details] [diff] [review]
awsy-windows.patch v2

Review of attachment 8851627 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/awsy/awsy/test_memory_usage.py
@@ +171,5 @@
>          self.logger.info("starting checkpoint %s..." % checkpointName)
>  
>          checkpoint_file = "memory-report-%s-%d.json.gz" % (checkpointName, iteration)
>          checkpoint_path = os.path.join(self._resultsDir, checkpoint_file)
> +        checkpoint_path = checkpoint_path.replace('\\', '\\\\')

Can you add a note about what we're doing here?
Attachment #8851627 - Flags: review?(erahm) → review+
Sure.

        # Escape the Windows directory separator \ to prevent it from
        # being interpreted as an escape character.
        checkpoint_path = checkpoint_path.replace('\\', '\\\\')
Comment on attachment 8851627 [details] [diff] [review]
awsy-windows.patch v2

Review of attachment 8851627 [details] [diff] [review]:
-----------------------------------------------------------------

Overall very nice patch.

Can you give a try if we can take exec and work well? thank!

::: testing/mozharness/configs/awsy/taskcluster_windows_config.py
@@ +9,5 @@
> +
> +config = {
> +    "virtualenv_python_dll": os.path.join(os.path.dirname(sys.executable), 'python27.dll'),
> +    "virtualenv_path": 'venv',
> +    "exes": {

I just took out this param in bug 1347678
Since we might not run on buildbot I think we can try and see if it looks more clear.
Attachment #8851627 - Flags: review?(pyang) → review+
Attached patch awsy-windows-exe.patch (obsolete) — Splinter Review
Is this what you meant?

I modeled this after an existing version of a taskcluster windows config, so I'm not sure it will work but I'll submit a try as soon as you confirm this is the change you are looking for.
Attachment #8852349 - Flags: feedback?(pyang)
Comment on attachment 8852349 [details] [diff] [review]
awsy-windows-exe.patch

Review of attachment 8852349 [details] [diff] [review]:
-----------------------------------------------------------------

ya I think it's worth to try if this config doesn't affect on tc either.  Thanks!
Attachment #8852349 - Flags: feedback?(pyang) → feedback+
Windows is busted without the exe values. The initial error is tooltool.py is not found. I'd rather keep the original settings now and if we really care, we can try to trim the values later, but I think Windows is special and requires the exe entries. I'm going to drop this last awsy-windows-exe.patch patch and go ahead and land.
Attachment #8852349 - Attachment is obsolete: true
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a1b8d7494145
Get AWSY running on Windows, r=jmaher,pyang,erahm.
In taskcluster/ci/test/tests.yml for the by-test-platform, instead of 

+                windows.*:

I think I should have had

+                windows.*/opt:

and I wonder in taskcluster/ci/test/test-platforms.yml since I already had in taskcluster/ci/test/test-sets.yml

windows-vm-tests:
    - awsy

that I didn't actually need to put awsy in test-platforms.yml under the windows7-32-vm/opt and windows10-64-vm/opt since they already included the windows-vm-tests.
Whiteboard: leave open
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f78955f157ad95d9fdad5d5a17baa21e1bb01f6f&filter-tier=1&filter-tier=2&filter-tier=3

linux64, win32, win64 sy for the windows tc builds. no linux32 since it is only buildbot on try as far as I can tell.
Flags: needinfo?(jmaher)
Attachment #8852423 - Flags: review?(jmaher)
Comment on attachment 8852423 [details] [diff] [review]
awsy-followup.patch

Review of attachment 8852423 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/test/test-sets.yml
@@ -125,5 @@
>  ##
>  # Test sets still being greened up in various ways
>  
>  windows-vm-tests:
> -    - awsy

why is this removed?
Attachment #8852423 - Flags: review?(jmaher) → review+
(In reply to Joel Maher ( :jmaher) from comment #23)
> Comment on attachment 8852423 [details] [diff] [review]
> awsy-followup.patch
> 
> Review of attachment 8852423 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: taskcluster/ci/test/test-sets.yml
> @@ -125,5 @@
> >  ##
> >  # Test sets still being greened up in various ways
> >  
> >  windows-vm-tests:
> > -    - awsy
> 
> why is this removed?

If we put awsy in the windows-vm-tests, then we will automatically get awsy run for both opt and debug. Removing awsy from windows-vm-tests leaves awsy defined for the windows7-32-vm/opt and the windows10-64-vm/opt sets.
Whiteboard: leave open
https://hg.mozilla.org/mozilla-central/rev/a59550ba1770
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Blocks: 1352606
Component: General → AWSY
You need to log in before you can comment on or make changes to this bug.