749421 - Allow test slaves to save and upload files somewhere, even from try

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

We currently have bug 642167 open for uploading minidumps to ftp.mozilla.org. However, there are some security issues with that. Chris Atlee and I talked about allowing files to be uploaded to a scratch machine that would be accessible to developers and to the test slaves. The data would only have to live there for a few days.

The main aspects of this seem to be:

1. Find a machine to use with enough disk space. Chris said that this is harder than it sounds, since the machine will be under a lot of load and the disk would fail quickly.

2. Allow test slaves to access the machine.

Writing code to actually upload files from the test slaves should be a separate bug.

So how hard is #1? Would it be easier to use something like Amazon S3? None of this data is valuable, so we just want something easy that won't require much maintenance. Backups and redundancy shouldn't be necessary.

Comment 1

[Chris Cooper [:coop] (he/him)]

This seems like an automation bug to me, rather than anything platform-specific since I assume we'll want this for _all_ the test platforms.

Comment 2

[Chris AtLee [:catlee]]

possibly we could use a yet-to-be-implemented blob store. either that or S3

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

I spoke with Taras about this request. According to him, this change should result in in developer months of saved time. (People currently have to resort to printf-debugging on try.) Taras tells me that Bill says it would've saved him 3 weeks on bug 803376. I would really like to reduce the time to fix GC bugs.

catlee - Can we talk about the feasibility of prioritizing this issue higher than P4?

Comment 4

[Lawrence Mandel [:lmandel] (use needinfo)]

I spoke with billm about his request. It seems like the minimal request is:

1. Find a machine, as per the description. billm suggested that we can turn on the file upload via a config param to start so as to minimize the disk usage and churn.

2. Modify the firewall or other premissions to allow the test machines to ftp (or somehow otherwise transfer) out the file to the machine from #1.

billm will take care of the code to actually upload the files once the test machines are configured to allow the transfer. 

billm - Do you have a bug on file for uploading the files?

catlee - Am I missing any required steps to make this happen? Does modifying the firewall/permissions for test machines require sec review?

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

(In reply to Lawrence Mandel [:lmandel] from comment #4)
> billm - Do you have a bug on file for uploading the files?

There's no bug yet, but I can file something.

Comment 6

[Lawrence Mandel [:lmandel] (use needinfo)]

catlee - Bump. Can this bug be included in releng Q1 activities.

Comment 7

[John Hopkins (:jhopkins)]

Ted, gps, catlee and I discussed briefly.

Files to be uploaded should be determined in-tree (and outside of the build configs).  A simple solution is for tests to place files in an agreed directory location that gets uploaded.

Test file uploads must not be able to overwrite or delete files in other builds.  Possible solutions include uploading files via a web service or post-upload script.

Test builds do not have access to any SSH keys at present.  Normally, try and non-try build slaves use different keys but since the test build slaves are run against try and non-try binaries, this is a security aspect we'll have to give some thought.

Comment 9

[Jonathan Griffin (:jgriffin)]

We need to upload files from test jobs in order to be able to use the structured logs that our test jobs will be generating soon, courtesy of chmanchester.  They don't need to go any place in particular, they just need to end up in some publicly-visible, reliable location.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Mechanism to automatically upload files from MOZ_UPLOAD_DIR after running a mozharness script

The BlobberMixin is to handle the virtualenv dependecies, dir creation and MOZ_UPLOAD_DIR definition. Still to be done is the branch specification which is currently set in the configs file - it should be taken from the buildbot configs along the way.

The blobuploader python script package lies here for the moment - http://people.mozilla.com/~mtabara/python/packages/

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 780517 [details] [diff] [review]
Mechanism to automatically upload files from MOZ_UPLOAD_DIR after running a mozharness script

Thanks for the patch!

>diff --git a/examples/blobber_usage.sh b/examples/blobber_usage.sh
>new file mode 100755
>--- /dev/null
>+++ b/examples/blobber_usage.sh
>@@ -0,0 +1,3 @@
>+#! /bin/sh
>+
>+echo 'Hello world!' > $MOZ_UPLOAD_DIR/file_one

I'm guessing this is just here for testing / proof of concept right?

If this file (in some non-"Hello World" variation) were to stick around in some form, a better location would either be external_tools
    http://hg.mozilla.org/build/mozharness/file/069dc44ccf51/external_tools
if we're planning to call it from scripts, or test/helper_files
    http://hg.mozilla.org/build/mozharness/file/069dc44ccf51/test/helper_files
if we're planning to call it from the nosetests.

examples/ is for non-functional scripts that were intended to show mozharness developers how to write mozharness scripts.
I'm not sure how useful it is today, since we have many more working examples in scripts/.
If we were to keep blobbertest.py in its current form, it might belong in examples/.

Unless there's a specific reason to use an external script, though, self.write_to_file() should do the trick here.
http://hg.mozilla.org/build/mozharness/file/069dc44ccf51/mozharness/base/script.py#l312

>+class BlobberMixin(VirtualenvMixin):
>+    """Provides mechanism to automatically upload files written in
>+    MOZ_UPLOAD_DIR to the blobber upload server at the end of the
>+    running script.
>+
>+    This is dependent on ScriptMixin.
>+    The testing script inheriting this class is to define in its
>+    configuration file the moz_upload_servers (e.g. URLs to blobber
>+    upload servers)
>+
>+    """

The name's amusing, but I'm not sure about it in an intuitive/predictable sense.
BlobUploadMixin might make more sense.

Also, if we took away the MOZ_UPLOAD_DIR env var hardcode, we could make this a generic non-mozilla-specific mixin that could live in mozharness.base.transfer.  The inheriting classes could then set the appropriate blobber_dir based on MOZ_UPLOAD_DIR or config or commandline option or w/e.

>+    #TODO: documentation about the Blobber Server on wiki
>+    def __init__(self, *args, **kwargs):
>+        requirements = [
>+            'blobuploader==0.1',
>+        ]
>+        super(BlobberMixin, self).__init__(*args, **kwargs)
>+        for req in requirements:
>+            self.register_virtualenv_module(req, method='pip')
>+
>+    @PostScriptAction('create-virtualenv')
>+    def _build_upload_directory(self, action, success=None):
>+        dirs = self.query_abs_dirs()
>+        self.blobber_dir = os.path.join(dirs['abs_work_dir'], 'blobber_upload_dir')
>+        self.mkdir_p(self.blobber_dir)
>+        self.env['MOZ_UPLOAD_DIR'] = self.blobber_dir

We shouldn't hardcode this.  We should either go by self.query_abs_dirs() or self.config, probably.
We probably should have a way to add things to query_abs_dirs() without having to override it completely every time, too.
That's outside the scope of this bug unless you want to take it on.

>+    @PostScriptRun
>+    def _upload_blobber_files(self):
>+    #TODO: the branch should be taken from self.config (buildbot configs)
>+        upload = self.query_python_path("blobberc.py")
>+        self.run_command([upload] +
>+                         self.config['moz_upload_servers'] +
>+                         self.config['moz_upload_branch'] +
>+                         ["-d", self.blobber_dir]
>+        )

This assumes we want to run this upload no matter what, every time we run the script.
That's good for production jobs, but would be very problematic for a developer who wanted to run the same script on their desktop, but without attempts at uploads, etc.

I think we should make this an action, rather than a @PostScriptRun.
I'm going to land a _post_fatal() hook where we can run stuff on every single production run.
Alternately, you could have a check in here to skip the upload if the config says to.

I definitely would like a self.info() before the upload happens, to log/inform what we're trying to do.

>diff --git a/scripts/blobbertest.py b/scripts/blobbertest.py

As above, this could live in examples/ if it's going to stick around.

>+    def run_me(self):
>+        dirs = self.query_abs_dirs()
>+        script = os.path.join(dirs['base_work_dir'], '..', 'examples', 'blobber_usage.sh')
>+        self.run_command([script], env=self.env)

This is where the self.write_to_file() could go.

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Blob upload mechanism sample for Desktop Unit Test integration

* Blob upload mechanism to automatically upload fils written in the MOZ_UPLOAD_DIR to the blobber server.
* The code is integrated in the Desktop Unit Test running script.
* the "upload_blobber_files" is an action performed subject to request in the running script
* Still to be done is to determine the branch along the way (buildbot configs most likely). For the moment the branch relies in the config file for the running script

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 781917 [details] [diff] [review]
Blob upload mechanism sample for Desktop Unit Test integration

This looks good, thanks!

We're going to have to modify each script that we want to use blob uploads.

Also, we can make this method always run via _post_fatal() once bug 898711 lands, or we can make it always run via the decorator, as long as we have a way to turn it off.  Having it be a part of the actions means we'll only run it if we haven't fatal()ed or raised an uncaught exception.  I think that's generally the desktop_unittest.py behavior, though, so this is probably good.

Comment 14

[(not currently active) Ted Mielczarek]

A simple first usage of this would be to fix bug 642167 by setting the MINIDUMP_SAVE_PATH environment variable to be the same as MOZ_UPLOAD_DIR (or a subdir of that, if that works, it doesn't matter much) for our unit test and Talos runs.

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Mechanism to upload from MOZ_UPLOAD_DIR and redefine MINIDUMP_SAVE_PATH

* Removed the hardcoded moz_upload_branch from config files into BlobUploadMixin (now taken from buildbot configs)
* Redefined the MINIDUMP_SAVE_PATH with the same path as MOZ_UPLOAD_DIR.

I am currently setting up a personal development master to test the patch within all scripts. 

Questions:
1. will MINIDUMP_STACKWALK be config-provided?
2. should we set MINIDUMP_SAVE_PATH as MOZ_UPLOAD_DIR always as it is now? Or should we do that only when MINIDUMP_SAVE_PATH is not config-provided?

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 783348 [details] [diff] [review]
Mechanism to upload from MOZ_UPLOAD_DIR and redefine MINIDUMP_SAVE_PATH

(In reply to Mihai Tabara [:mtabara] from comment #15)
> Created attachment 783348 [details] [diff] [review]
> Mechanism to upload from MOZ_UPLOAD_DIR and redefine MINIDUMP_SAVE_PATH
> 
> * Removed the hardcoded moz_upload_branch from config files into
> BlobUploadMixin (now taken from buildbot configs)

This is true for tests, and I *think* it's true for builds.
Hardcoding it to use the buildbot properties makes it even harder for someone to use this standalone, though.  We might want a way to specify it outside of buildprops.json (commandline option?)

> * Redefined the MINIDUMP_SAVE_PATH with the same path as MOZ_UPLOAD_DIR.

This is test-specific, but since you're specifying it in desktop_unittest.py, that's cool.

> Questions:
> 1. will MINIDUMP_STACKWALK be config-provided?
> 2. should we set MINIDUMP_SAVE_PATH as MOZ_UPLOAD_DIR always as it is now?
> Or should we do that only when MINIDUMP_SAVE_PATH is not config-provided?

So far we have it for every OS.  That might not be true for B2G emulator tests, etc., but that's a different script, so we're fine here.

Comment 17

[(not currently active) Ted Mielczarek]

(In reply to Mihai Tabara [:mtabara] from comment #15)
> Questions:
> 1. will MINIDUMP_STACKWALK be config-provided?

The binary used comes from the build/tools repo and is different per-platform:
http://hg.mozilla.org/build/tools/file/tip/breakpad/

I don't know how to actually answer that question, hopefully that helps.

> 2. should we set MINIDUMP_SAVE_PATH as MOZ_UPLOAD_DIR always as it is now?
> Or should we do that only when MINIDUMP_SAVE_PATH is not config-provided?

Yes, we should do that. Having minidumps get uploaded is way more useful than the existing behavior of just saving them off to a local directory, and should be the default.

(In reply to Aki Sasaki [:aki] (back August 5) from comment #16)
> > * Redefined the MINIDUMP_SAVE_PATH with the same path as MOZ_UPLOAD_DIR.
> 
> This is test-specific, but since you're specifying it in
> desktop_unittest.py, that's cool.

This should be used for Talos as well, FWIW. (It wouldn't be harmful to have it set everywhere, but I don't know if you dislike that for theoretical purity.)

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: File upload mechanism from MOZ_UPLOAD_DIR and MINIDUMP_SAVE_PATH

* removed hardcoded branch specs from config file to cmdline option along with the moz_upload_server
* successfully uploads to blob server for Desktop Unittest on dev master with Linux Machine

To be done ASAP:
* test uploads to blob server on Mac & Win machines
* find proper way to change buildbot-configs to call the scripts with the new cmdlines options

Questions:
1. Are we still willing to run 'upload-blobber-files' via the  _post_fatal()?
2. Should we use any try/except with default values in the BlobMixin for getting branch/server?

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 788366 [details] [diff] [review]
File upload mechanism from MOZ_UPLOAD_DIR and MINIDUMP_SAVE_PATH

Sorry about the delay, 24.0b1 wasn't playing nice.
 
> Questions:
> 1. Are we still willing to run 'upload-blobber-files' via the  _post_fatal()?

Sure, as long as there's a way to turn it off.

> 2. Should we use any try/except with default values in the BlobMixin for
> getting branch/server?

How about, if both moz_upload_branch and moz_upload_server are specified, do the blob upload; otherwise skip?

Otherwise we can specify both, and have a bool flag that we can flip on/off.  --blob-upload or --no-blob-upload maybe.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: File upload mechanism from MOZ_UPLOAD_DIR

* blob upload server lies in the configuration files as it's less likely to change often
* blob upload branch lies as a command line option (a different patch for tunning the buildbot-configs to do so is coming soon)

* blob upload mechanism works only if both branch and server lie in the self.config, otherwise it's being skipped. Therefore one can run this standalone locally  by skipping branch as cmdline option

* the upload_blobber_files is no longer a manually called action but triggered throughout the PostScriptRun decorator
* in the same time, should anything fail with == FATAL level, the action will also be triggered by _post_fatal()

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 790802 [details] [diff] [review]
File upload mechanism from MOZ_UPLOAD_DIR

>diff --git a/configs/unittests/linux_unittest.py b/configs/unittests/linux_unittest.py
>--- a/configs/unittests/linux_unittest.py
>+++ b/configs/unittests/linux_unittest.py
>@@ -119,9 +119,13 @@ config = {
>             "halt_on_failure": True,
>             "enabled": ADJUST_MOUSE_AND_SCREEN
>         },
>     ],
>     "repos": [{"repo": "http://hg.mozilla.org/build/tools"}],
>     "minidump_stackwalk_path": MINIDUMP_STACKWALK_PATH,
>     "minidump_save_path": "%(abs_work_dir)s/../minidumps",
>     "buildbot_max_log_size": 52428800,
>+    "moz_upload_server": [
>+         "-u",
>+         "http://10.134.48.49:8080",
>+    ],
> }

I don't want to bog down too much here, since I've made a suggestion and you've implemented it.
Looking closer, I'm thinking:

Are we only going to have one upload server specifiable?
If so, we can

    "moz_upload_server": "http://10.134.48.49:8080",

and you could keep the

    [["--moz-upload-servers"],
    {"dest": "moz_upload_server",
     "help": "Blob servers's location"
    }]

though you'd want it to be singular (--moz-upload-server).

You would need to specify the -u yourself.
If we need to be able to specify a list, we don't have a good way to override, though we could

    "default_blob_upload_servers": ["http://10.134.48.49:8080",],

    [["--blob-upload-server"],
    {"dest": "blob_upload_servers",
     "action": "extend",
     "help": "Blob servers's location",
    }]

And then when you're looking for the upload servers, it's

    blob_upload_servers = self.config.get("blob_upload_servers", self.config.get("default_blob_upload_servers"))

Make sense?

I also like renaming this to blob_upload_server(s) since moz_upload_server could be an ssh/rsync upload as well.

>     def upload_blobber_files(self):
>         upload = self.query_python_path("blobberc.py")
>         dirs = self.query_abs_dirs()
>+        self.info("Get the directory from which to upload the files.")
>         try:
>             blob_dir = dirs['abs_blob_upload_dir']
>         except KeyError:
>             self.fatal("Blob upload folder config is missing.")

You're fatal()ing here... maybe you should be checking to see if the upload branch and server are set first?
You can also do a similar check, either by

    if 'abs_blob_upload_dir' in dirs:

or

    if dirs.get('abs_blob_upload_dir'):

>+            branch = ["-b", self.config['moz_upload_branch']]
>+            server = self.config['moz_upload_server']

If we follow my above suggestion for specifying the servers, we can

    server = []
    for server in self.config['blob_upload_servers']:
        server.extend(['-u', server])

>+            self.info("Files are to be uploaded with %s corresponding branch at"
>+                      " the %s location." % (branch, server))

This is printing lists... is that the desired effect?

>+    def _post_fatal(self):
>+        self.upload_blobber_files()

This *might* not be necessary.
I think the PostScriptRun methods run every time, but I'm not entirely sure.  Can you insert a fatal() in the script and see if it uploads twice?

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: File upload mechanism from MOZ_UPLOAD_DIR

* Talked to catlee and concluded that ATM we are most likely going to use multiple hosts rather than a pound/load balancer unique-host
* upgraded the blobberc python packge (pypi) to support mutiple hosts (v. 0.2)
* ranamed branch/server cmdline args to be more consistent
* on/off blob mechanism is based on branch specification. Thus one can avoid passing <blob-upload-branch> argument and the mechanism is skipped
* there are default hosts in the config files for blob server which can be overwritten by passing <blob-upload-server> cmdline option
* eventually, the blob server will have a specific IP range from which to accept POST calls, thus we don't have to worry about developers sending files to the server while locally-testing
* Aki was right. Specific _post_fatal() in scripts is unnecessary as @PostScriptRun runs anyway and thus calls the action to handle blob uploading
* refactoring

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 792288 [details] [diff] [review]
File upload mechanism from MOZ_UPLOAD_DIR

This is looking good!

>+        self.debug("Check <blob-upload-branch> and <blob-upload-server> options.")
>+        if 'blob_upload_branch' in self.config:
>+            self.info("Blob upload gear active.")

Hm, I can see a situation where someone copies our commandline, but creates their own config file that doesn't contain default_blob_upload_servers.
You probably want to check for blob_upload_branch and (blob_upload_servers or default_blob_upload_servers).

Also, what behavior do we expect when the blob upload directory either doesn't exist or is empty?
Is that handled by the python package, since I don't see anything to handle that here?

Comment 24

[Mihai Tabara [:mtabara]⌚️GMT]

> Hm, I can see a situation where someone copies our commandline, but creates
> their own config file that doesn't contain default_blob_upload_servers.
> You probably want to check for blob_upload_branch and (blob_upload_servers
> or default_blob_upload_servers).

Good point. I will update this.
 
> Also, what behavior do we expect when the blob upload directory either
> doesn't exist 

So far the creation of the folder is handled by the mozharness script, along with the env var definition (MOZ_UPLOAD_DIR)
http://hg.mozilla.org/users/mtabara_mozilla.com/mozharness/file/2be7e10c59dc/scripts/desktop_unittest.py#l360
In BlobUploadMixin I'm fatal()-ing if the folder is not returned by the query_env(). Should I move the creation of the folder in the BlobUploadMixin?

> or is empty?

Hm .. good observation about this. ATM the client does not do anything in particular. Do you want me to add a `'if not files' then log and return something` statement there?

https://github.com/catlee/blobber/blob/master/scripts/blobberc.py#L88

Comment 25

[Aki Sasaki (not active)]

(In reply to Mihai Tabara [:mtabara] from comment #24)
> > Also, what behavior do we expect when the blob upload directory either
> > doesn't exist 
> 
> So far the creation of the folder is handled by the mozharness script, along
> with the env var definition (MOZ_UPLOAD_DIR)
> http://hg.mozilla.org/users/mtabara_mozilla.com/mozharness/file/2be7e10c59dc/
> scripts/desktop_unittest.py#l360
> In BlobUploadMixin I'm fatal()-ing if the folder is not returned by the
> query_env(). Should I move the creation of the folder in the BlobUploadMixin?

It's possible for us to fatal() before that line, in which case we'll try to call the PostScriptRun before it exists.

I think being able to gracefully handle the directory not existing is a good idea.

> > or is empty?
> 
> Hm .. good observation about this. ATM the client does not do anything in
> particular. Do you want me to add a `'if not files' then log and return
> something` statement there?
> 
> https://github.com/catlee/blobber/blob/master/scripts/blobberc.py#L88

Sure, that sounds good.  Alternately we could check for the existence of the directory and the presence of files inside that directory before calling the blob uploader.  I'm fine with either approach.

Comment 26

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: File upload mechanism from MOZ_UPLOAD_DIR

* safe check for both branch and server
* skipping blob upload mechanism for blob_dir corner cases
* refactoring

Comment 27

[Aki Sasaki (not active)]

Comment on attachment 792953 [details] [diff] [review]
File upload mechanism from MOZ_UPLOAD_DIR

I think this is good :)
Thanks for your patience!

Comment 28

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: overview - File upload mechanism from MOZ_UPLOAD_DIR

* overview of the blob mechanism
* tests run successfully on dev-master (for Linux machine)
* blob server host is URL-hardcoded ATM in the config files. I suggest waiting to see how blob upload mechanism is coming along with staging/production load and depending on that to later on grant him a (DNS) name :)

Comment 29

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Buildbot configs changeset

* Buildbot-configs modification

Comment 30

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: buildbot custom changeset

* Buildbot-custom modification

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 793075 [details] [diff] [review]
overview - File upload mechanism from MOZ_UPLOAD_DIR

This patch is slightly bitrotted, but it's not too hard to merge.

Comment 32

[Aki Sasaki (not active)]

Comment on attachment 793080 [details] [diff] [review]
buildbot custom changeset

I don't like this approach because it assumes commandline option order.
If we append something else after --blob-upload-branch, we have a broken commandline.

Instead of

>     ('mochitest-1', {
>         'use_mozharness': True,
>         'script_path': 'scripts/desktop_unittest.py',
>-        'extra_args': ['--mochitest-suite', 'plain1'],
>+        'extra_args': ['--mochitest-suite', 'plain1',
>+                       '--blob-upload-branch'],
>         'script_maxtime': 7200,
>     }),

Maybe

    ('mochitest-1', {
        'use_mozharness': True,
        'script_path': 'scripts/desktop_unittest.py',
        'extra_args': ['--mochitest-suite', 'plain1'],
        'blob_upload': True,
        'script_maxtime': 7200,
    }),

and

if mozharness_suite_config.get('blob_upload'):
    extra_args.extend(["--blob-upload-branch", branch_name])

?

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 793077 [details] [diff] [review]
Buildbot configs changeset

As above.

Comment 34

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Buildbot configs changeset

* changed to 'blob_upload' flag

Comment 35

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Buildbot custom changeset

* changed to 'blob_upload' flag used for both branch-level and suite-level

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 793717 [details] [diff] [review]
Buildbot custom changeset

You can probably make the conditional easier to read now that we're creating a True, a la

if branch_config.get('blob_upload') and suites.get('blob_upload'):
    test_builder_kwargs['mozharness_suite_config']['blob_upload'] = True

This works too though.

Comment 37

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: buildbotcustom.diff

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 793727 [details] [diff] [review]
buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/38c9f32e09c7

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 793716 [details] [diff] [review]
Buildbot configs changeset

https://hg.mozilla.org/build/buildbot-configs/rev/f341f28838ec

Comment 40

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: overview - File upload mechanism from MOZ_UPLOAD_DIR

Cloned buil/mozharness, patched

Comment 41

[Aki Sasaki (not active)]

Comment on attachment 793753 [details] [diff] [review]
overview - File upload mechanism from MOZ_UPLOAD_DIR

http://hg.mozilla.org/build/mozharness/rev/5adba582be8a

Comment 42

[Armen [:armenzg]]

Live in production.

Comment 43

[Armen [:armenzg]]

Merged to production branch of mozharness.
Live as of now.

Comment 44

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Unit test for BlobUploadMixin

* unit testing for BlobUploadMixin code

Comment 45

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Unit test for BlobUploadMixin + typo in BlobUploadMixin

* unit testing for BlobUploadMixin 
* small logging typo in BlobUploadMixin

Comment 46

[Aki Sasaki (not active)]

Comment on attachment 796330 [details] [diff] [review]
Unit test for BlobUploadMixin + typo in BlobUploadMixin

https://hg.mozilla.org/build/mozharness/rev/71e9d27e5e26

Comment 47

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/mozharness/rev/b4c544aea4cf for easier testing.

Comment 48

[John Hopkins (:jhopkins)]

In production.

Comment 49

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

* instead of 64encoding to log, the screenshots go directly as files in MOZ_UPLOAD_DIR from where are to be uploaded within the blob gear

Comment 50

[(not currently active) Ted Mielczarek]

Comment on attachment 800363 [details] [diff] [review]
dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

Review of attachment 800363 [details] [diff] [review]:
-----------------------------------------------------------------

The content of this patch looks okay, but I'd like it to be a bit cleaner for landing. Also, we shouldn't land this on mozilla-{inbound,central} until the blob upload stuff lands there, so you'll have to land this directly on Cedar I suppose.

Please generate a patch for checkin without the whitespace removal. jhammel is working on a large refactoring in bug 746243, and I don't want you to bitrot him any more than necessary.

Also, you can remove a lot more code while you're at it, so you might as well.

::: build/automation.py.in
@@ +639,3 @@
>  
> +    # Always write screenshots to a file - bug 749421 related
> +    imgoutput = 'file'

The comment isn't really necessary, and you can get rid of the imgoutput variable.

@@ +651,3 @@
>      try:
> +        tmpfd, imgfilename = tempfile.mkstemp(prefix='mozilla-test-fail-screenshot_',
> +                                              dir=parent_dir)

Since you're rewriting half of this anyway, you can replace this with:
with mozfile.NamedTemporaryFile(prefix=..., dir=...) as f:
  returncode = subprocess.call(utility + [imgfilename])

(and a bunch of the code below, since you don't need its stdout anymore etc)

Comment 51

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR (

* simplified the dumpScreen() method to always output to a file and write it down to MOZ_UPLOAD_DIR

Comment 52

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

* simplified dumpScreen() to write screenshots to file in MOZ_UPLOAD_DIR

Comment 53

[(not currently active) Ted Mielczarek]

Comment on attachment 801031 [details] [diff] [review]
dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

Review of attachment 801031 [details] [diff] [review]:
-----------------------------------------------------------------

Like I said before, please attach a patch without the whitespace cleanup for final checkin.

::: build/automation.py.in
@@ +624,5 @@
>      if self.haveDumpedScreen:
>        self.log.info("Not taking screenshot here: see the one that was previously logged")
>        return
>      self.haveDumpedScreen = True;
> +    # Need to figure out which OS-dependant tool to use

spelling: dependent

@@ +633,4 @@
>      elif self.IS_WIN32:
>        utility = [os.path.join(utilityPath, "screenshot.exe")]
> +    # Always write screenshots to a file - bug 749421 related
> +    imgoutput = 'file'

Drop the comment and the imgoutput variable.

@@ +636,5 @@
> +    imgoutput = 'file'
> +
> +    # Get dir where to write the file
> +    env = self.environment()
> +    parent_dir = env.get('MOZ_UPLOAD_DIR', '')

nit: I'd probably use None for the default value.

@@ +646,4 @@
>      try:
> +        tmpfd, imgfilename = tempfile.mkstemp(suffix='.png')
> +                                              prefix='mozilla-test-fail-screenshot_',
> +                                              dir=parent_dir)

In my previous review I suggested you should use mozfile.NamedTemporaryFile.

@@ +650,2 @@
>          os.close(tmpfd)
>          dumper = self.Process(utility + [imgfilename])

and also subprocess.call here.

Comment 54

[Jeff Hammel]

(In reply to Mihai Tabara [:mtabara] from comment #52)
> Created attachment 801031 [details] [diff] [review]
> dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR
> 
> * simplified dumpScreen() to write screenshots to file in MOZ_UPLOAD_DIR

I don't understand if/how the linux `screentopng` tool is writing to the file; am I missing something or does it need its output redirected?

Comment 55

[(not currently active) Ted Mielczarek]

(In reply to Jeff Hammel [:jhammel] from comment #54)
> I don't understand if/how the linux `screentopng` tool is writing to the
> file; am I missing something or does it need its output redirected?

I fixed that in bug 910363. That should have been added as a dep.

Comment 56

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

* simplified dumpScreen() to generate screenshots as files only to MOZ_UPLOAD_DIR

Comment 57

[(not currently active) Ted Mielczarek]

Comment on attachment 801885 [details] [diff] [review]
dumpScreen() simplified so that all screenshots go as files in MOZ_UPLOAD_DIR

Review of attachment 801885 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, this looks good! Are you going to land this on Cedar?

Comment 58

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Blobber user/passwd sample in buildbot-configs template

Comment 59

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Activate flag to get oauth.txt file with credentials in buildbotcustom

Comment 60

[Rail Aliiev [:rail]]

Comment on attachment 812393 [details] [diff] [review]
Blobber user/passwd sample in buildbot-configs template

https://hg.mozilla.org/build/buildbot-configs/rev/3c2235479ed4

Comment 61

[Rail Aliiev [:rail]]

Comment on attachment 812397 [details] [diff] [review]
Activate flag to get oauth.txt file with credentials in buildbotcustom

https://hg.mozilla.org/build/buildbotcustom/rev/1487b120649f

Comment 62

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Fix client to version

Comment 63

[Rail Aliiev [:rail]]

Comment on attachment 812401 [details] [diff] [review]
Fix client to version

https://hg.mozilla.org/build/mozharness/rev/0c9b4395bd09

Comment 64

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add authentication cmd line arg for blobUploadMixin

* Blob client is using SSL + basic http authentication to upload files to elastic beanstalk. Thus, when calling the client script a file containing the auth credentials is to be given ( https://github.com/MihaiTabara/blobuploader/blob/master/blobberc.py#L137 )

* should I add the config param as blobupload_config_options as well?

Comment 65

[Rail Aliiev [:rail]]

Attachment: puppet blobupload credentials

Comment 66

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Change blob server URL in mozharness config

* point to secure elasticbeanstalk new blob server

! Please do not yet land this

Comment 67

[Rail Aliiev [:rail]]

Comment on attachment 812668 [details] [diff] [review]
puppet blobupload credentials

https://hg.mozilla.org/build/puppet/rev/1acc0a3e89dd

Comment 68

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Fix blobuploader in mozharness to 0.995

Comment 69

[[e-mail] (:bhearsum)]

something here is in production

Comment 70

[Mihai Tabara [:mtabara]⌚️GMT]

Yes, this one.

https://hg.mozilla.org/build/buildbotcustom/rev/45508cbe175e

Thanks!

Comment 71

[Rail Aliiev [:rail]]

Comment on attachment 812408 [details] [diff] [review]
Add authentication cmd line arg for blobUploadMixin

remote:   https://hg.mozilla.org/build/mozharness/rev/3be96a6a2b16
remote:   https://hg.mozilla.org/build/mozharness/rev/56a9cb3e7686
remote:   https://hg.mozilla.org/build/mozharness/rev/7aed71c502bf

Comment 72

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Fix tests in mozharness for new blobuploader version

Comment 73

[Rail Aliiev [:rail]]

Comment on attachment 813167 [details] [diff] [review]
Fix tests in mozharness for new blobuploader version

https://hg.mozilla.org/build/mozharness/rev/e7d96b1fb10a

Comment 74

[John Hopkins (:jhopkins)]

in production

Comment 75

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Fix blobuploader in mozharness to 1.0b

Comment 76

[Rail Aliiev [:rail]]

Comment on attachment 816028 [details] [diff] [review]
Fix blobuploader in mozharness to 1.0b

https://hg.mozilla.org/build/mozharness/rev/b378d511e8a9

Comment 77

[cmtalbert]

What is remaining to be done here for this to be used in production?

Comment 78

[Mihai Tabara [:mtabara]⌚️GMT]

We are to announce it these days following which we'll be switching it on, gradually, branch by branch.

Comment 79

[Justin Wood (:Callek)]

something here is in mozharness production

Comment 80

[Mihai Tabara [:mtabara]⌚️GMT]

Yes, http://hg.mozilla.org/build/mozharness/rev/7d9425c91051#l28.13

@Justin: Thank you.

Comment 81

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Switch on blobber to try branch

Comment 82

[Rail Aliiev [:rail]]

Comment on attachment 818068 [details] [diff] [review]
Switch on blobber to try branch

https://hg.mozilla.org/build/buildbot-configs/rev/7e0e163eacb3

Comment 83

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: FIx python path in mozharness

Comment 84

[Rail Aliiev [:rail]]

Comment on attachment 818705 [details] [diff] [review]
FIx python path in mozharness

remote:   https://hg.mozilla.org/build/mozharness/rev/dc6e45cdc540
remote:   https://hg.mozilla.org/build/mozharness/rev/a3366e91318c

Comment 85

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Fix tests in mozharness for python path fixing

Comment 86

[Rail Aliiev [:rail]]

Comment on attachment 818760 [details] [diff] [review]
Fix tests in mozharness for python path fixing

https://hg.mozilla.org/build/mozharness/rev/dc82c4906da1

Comment 87

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Andron Panda test

Comment 88

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Enable blobber on ash

Comment 89

[Rail Aliiev [:rail]]

Comment on attachment 819163 [details] [diff] [review]
Enable blobber on ash

https://hg.mozilla.org/build/buildbot-configs/rev/d8af6632e054

Comment 90

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Andron Panda Talos test

Comment 91

[Joel Maher ( :jmaher ) (UTC -8)]

please allow file type .etl to be uploaded ( https://github.com/catlee/blobber/blob/master/blobber/config.py#L12 ).  This will be useful for talos xperf.

Comment 92

[Aki Sasaki (not active)]

Comment on attachment 819150 [details] [diff] [review]
Add blobber gear to Andron Panda test

> -            self.warning("We failed to create the shutdown file: str(%s)" % str(e))
> +            self.warning("We failed to create the shutdown file: str(%s)" % str(e))

I can't seem to figure out what's changed about this line?

Comment 93

[Justin Wood (:Callek)]

something here merged to production mozharness

Comment 94

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Android Panda tests

* completely forgot about configs and buildbot-configs
* works fine on staging master!

Comment 95

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Buildbot-configs patch to enable blobber for all test suites in Android Panda

* works fine on staging master

Comment 96

[Rail Aliiev [:rail]]

Comment on attachment 821978 [details] [diff] [review]
Add blobber gear to Android Panda tests

https://hg.mozilla.org/build/mozharness/rev/909b41bedf64

Comment 97

[Rail Aliiev [:rail]]

Comment on attachment 821979 [details] [diff] [review]
Buildbot-configs patch to enable blobber for all test suites in Android Panda

https://hg.mozilla.org/build/buildbot-configs/rev/4507910edc52

Comment 98

[Justin Wood (:Callek)]

something here is in production

Comment 99

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Android Panda Talos tests

* checked successfully on staging master

Comment 100

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tune buildbotcustom for Talos tests for branch blobber-enabled only

Comment 101

[Aki Sasaki (not active)]

Comment on attachment 822526 [details] [diff] [review]
Add blobber gear to Android Panda Talos tests

lgtm.

Comment 102

[Rail Aliiev [:rail]]

Comment on attachment 822526 [details] [diff] [review]
Add blobber gear to Android Panda Talos tests

https://hg.mozilla.org/build/mozharness/rev/46b4768c6f1f

Comment 103

[Rail Aliiev [:rail]]

Comment on attachment 822528 [details] [diff] [review]
Tune buildbotcustom for Talos tests for branch blobber-enabled only

https://hg.mozilla.org/build/buildbotcustom/rev/210636103d4a

Comment 104

[Rail Aliiev [:rail]]

in production

Comment 105

[Aki Sasaki (not active)]

Comment on attachment 822528 [details] [diff] [review]
Tune buildbotcustom for Talos tests for branch blobber-enabled only

Backed out since this may have caused windows talos bustage.

Comment 106

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tune buildbotcustom for Talos tests for branch blobber-enabled only

Comment 107

[Rail Aliiev [:rail]]

Comment on attachment 822528 [details] [diff] [review]
Tune buildbotcustom for Talos tests for branch blobber-enabled only

https://hg.mozilla.org/build/buildbotcustom/rev/ee9161ac33bb
back to default

Comment 108

[John Hopkins (:jhopkins)]

in production

Comment 109

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Talos script

Comment 110

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tune buildbotcustom for full Talos blobber integration

Comment 111

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Talos script

Comment 112

[Aki Sasaki (not active)]

Catlee: FYI, if we rename oauth.txt, we'll have to time it so ScriptFactory and these mozharness configs change at the same time.

Comment 113

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #112)
> Catlee: FYI, if we rename oauth.txt, we'll have to time it so ScriptFactory
> and these mozharness configs change at the same time.

This might have already sprung to your mind, but an alternative to timing this is to download the file to both places, make the mozharness change, and then stop downloading to oauth.txt in a second change later.

Comment 114

[Aki Sasaki (not active)]

Comment on attachment 824300 [details] [diff] [review]
Add blobber gear to Talos script

http://hg.mozilla.org/build/mozharness/rev/7ac9028f9132

Comment 115

[Aki Sasaki (not active)]

Comment on attachment 824299 [details] [diff] [review]
Tune buildbotcustom for full Talos blobber integration

http://hg.mozilla.org/build/buildbotcustom/rev/a9f6adc7dcbb

Comment 116

[Armen [:armenzg]]

mozharness merged & live in production.

Comment 117

[Aki Sasaki (not active)]

Attachment: fix_blobber

This is mostly whitespace+unused import cleanup.
I also found that we switched back to importing VirtualenvMixin instead of BlobUploadMixin here: http://hg.mozilla.org/build/mozharness/rev/b29095553ca1

This patch should fix all of the above.

Comment 118

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 825662 [details] [diff] [review]
fix_blobber

Review of attachment 825662 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for the heads-up Aki.

Comment 119

[Aki Sasaki (not active)]

Comment on attachment 825662 [details] [diff] [review]
fix_blobber

r+ over IRC.
http://hg.mozilla.org/build/mozharness/rev/d29c0525efd7

Comment 120

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Add blobber gear to Marionette script

* not sure I need to patch all those configs files? (I feel like win_config and prod_config would be enough) ?

Comment 121

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tune buildbotcustom to enable blobber for Marionette

Comment 122

[Aki Sasaki (not active)]

Comment on attachment 826017 [details] [diff] [review]
Add blobber gear to Marionette script

You're definitely touching more config files than needed for this pass, but it's probably harmless... when you add 'blob_upload': True in the various marionette test suites in http://hg.mozilla.org/build/buildbot-configs/file/144a6d68d917/mozilla-tests/b2g_config.py , you'll be set.

Comment 123

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tune buildbot-configs to enable blobber for Marionette

* thanks for the heads-up. Missed that :-)

Comment 124

[Aki Sasaki (not active)]

Comment on attachment 826079 [details] [diff] [review]
Tune buildbot-configs to enable blobber for Marionette

I think GAIA_UI can be toggled on as well; r=me with or without that.

Comment 125

[Aki Sasaki (not active)]

Comment on attachment 826017 [details] [diff] [review]
Add blobber gear to Marionette script

https://hg.mozilla.org/build/mozharness/rev/a966ddb206fb

Comment 126

[Aki Sasaki (not active)]

Comment on attachment 826079 [details] [diff] [review]
Tune buildbot-configs to enable blobber for Marionette

https://hg.mozilla.org/build/buildbot-configs/rev/a9112074f912

Comment 127

[John Hopkins (:jhopkins)]

in production

Comment 128

[Andrew McCreight [:mccr8]]

What is the status of this?  Is it usable yet?  I see the bug is still open, but there are also a bunch of patches that have landed so I'm not sure. :)

Comment 129

[Aki Sasaki (not active)]

Aiui, this is enabled for a number of test suites, on a subset of branches (ash, cedar, pine, try)

Comment 130

[Mihai Tabara [:mtabara]⌚️GMT]

The test suites enabled are: android (pandas and talos), desktop unit tests, marionette and Talos.
It's still restricted to (ash, cedar, pine, try) until we fix a bunch of yet-opened-issues.

See bugs: 938543, 938339.

Comment 131

[Andrew Halberstadt [:ahal]]

Attachment: Add blobber gear to b2g emulator unittest script

Comment 132

[Andrew Halberstadt [:ahal]]

Attachment: Tune buildbot-configs to enable blobber for b2g emulator unittests

Is there a way to set it up so we don't need to add this to each chunk?

Comment 133

[Aki Sasaki (not active)]

(In reply to Andrew Halberstadt [:ahal] from comment #132)
> Created attachment 832974 [details] [diff] [review]
> Tune buildbot-configs to enable blobber for b2g emulator unittests
> 
> Is there a way to set it up so we don't need to add this to each chunk?

Yes, change this line
http://hg.mozilla.org/build/buildbotcustom/file/8c8a2610f748/misc.py#l1996
to 

if branch_config.get('blob_upload') and suites.get('blob_upload', True)

I think we didn't do that because not every test script was ready for it, and sending the unknown commandline arguments might break things.

Comment 134

[Andrew Halberstadt [:ahal]]

Attachment: Add blobber gear to b2g desktop unittest script

Note the previous buildbot-config patch should enable blob_upload for this script as well.

Comment 135

[Andrew Halberstadt [:ahal]]

Comment on attachment 832964 [details] [diff] [review]
Add blobber gear to b2g emulator unittest script

https://hg.mozilla.org/build/mozharness/rev/0658d7a63efb

Comment 136

[Andrew Halberstadt [:ahal]]

Comment on attachment 833077 [details] [diff] [review]
Add blobber gear to b2g desktop unittest script

https://hg.mozilla.org/build/mozharness/rev/a166ab38d937

Comment 137

[Andrew Halberstadt [:ahal]]

Comment on attachment 832974 [details] [diff] [review]
Tune buildbot-configs to enable blobber for b2g emulator unittests

https://hg.mozilla.org/build/buildbot-configs/rev/b89a82d65690

Comment 138

[Andrew Halberstadt [:ahal]]

Missed an import in the b2g desktop changes, here's the follow-up fix: https://hg.mozilla.org/build/mozharness/rev/1f488900b827

Comment 139

[Justin Wood (:Callek)]

something[s] here made it to production

Comment 140

[Andrew Halberstadt [:ahal]]

Just making a note here.. when enabling blobber on new branches, please also do so for b2g_config.py which uses a separate (but equivalent) dictionary. Example: http://hg.mozilla.org/build/buildbot-configs/rev/8de5f739eab9

Comment 141

[Chris AtLee [:catlee]]

Any reason we shouldn't enable it on all branches?

Comment 142

[Mihai Tabara [:mtabara]⌚️GMT]

Can we (finally :-D) close this bug once bug 956035 is confirmed to not have damaged anything?

Comment 143

[(not currently active) Ted Mielczarek]

Woot! We should get attachment 801885 [details] [diff] [review] landed on mozilla-inbound.

Comment 144

[(not currently active) Ted Mielczarek]

Let's call this done. I'm going to move attachment 801885 [details] [diff] [review] to a new bug because I think the patch needs updating.

Comment 145

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Could somebody provide a quick summary of what got done here? This seems like a great feature, but I have no idea how to use it.

Comment 146

[(not currently active) Ted Mielczarek]

Executive summary: test runs will have an environment variable set (MOZ_UPLOAD_DIR) which is a path to a directory where you can write files. Any files in that directory (subject to a file extension whitelist) will be uploaded to a publicly-accessible server at the end of the run.

whitelist: https://github.com/catlee/blobber/blob/master/blobber/config.py#L12

Comment 147

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Excellent! What's the server? Or will that appear somewhere in the log at the end of the run or something?

Comment 148

[Jason Duell]

Can we put a summary set of instructions on how to use this new feature somewhere on the relevant wiki(s)?  I'm thinking these page

  https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests

  https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/XPCShell

  https://developer.mozilla.org/en-US/docs/Mochitest

Comment 149

[Chris AtLee [:catlee]]

I put it here:

https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing$compare?to=521397&from=507043