Closed Bug 1595492 Opened 6 years ago Closed 6 years ago

Mapper seems down.

Categories

(Release Engineering Graveyard :: Applications: Mapper, defect)

Product:
Release Engineering Graveyard
Component:
Applications: Mapper
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: emilio, Assigned: garbas)

References

Details

https://mapper.mozilla-releng.net/ seems to be returning 502s, which means searchfox is erroring.

Flags: needinfo?(dhouse)

Rok, can you check the mapper service? I'm guessing this is related to the TC migration, and related to this do we need a new client+auth for the vcs-sync push to the mapper?

Flags: needinfo?(dhouse) → needinfo?(rgarbas)

:dhouse I didn't have time to migrate mapper away from legacy taskcluster (since i needed to focus on tooltool and treestatus) and i'll make sure i move mapper to use new firefox-ci as soon as I can

(In reply to Rok Garbas [:garbas] from comment #2)

:dhouse I didn't have time to migrate mapper away from legacy taskcluster (since i needed to focus on tooltool and treestatus) and i'll make sure i move mapper to use new firefox-ci as soon as I can

Thanks, Rok!

:emilio is searchfox not updating or how is it affected by this being down? I need to start publishing the mozilla-central mappings to s3, and so I could quickly do that if there is an immediate need (and searchfox could pull the mappings file from s3) and we don't get the mapper service up soon.

Flags: needinfo?(emilio)

Searchfox uses the mapper as follows:

So those requests are erroring and thus the indexing stops. We could try to fetch them from s3 if needed, possibly...

Flags: needinfo?(emilio)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)

Searchfox uses the mapper as follows:

So those requests are erroring and thus the indexing stops. We could try to fetch them from s3 if needed, possibly...

I looked in that repo and updated for some new urls: https://github.com/mozsearch/mozsearch-mozilla/pull/60

Assignee: nobody → rgarbas

(In reply to Emilio Cobos Álvarez (:emilio) from comment #4)

Searchfox uses the mapper as follows:

So those requests are erroring and thus the indexing stops. We could try to fetch them from s3 if needed, possibly...

I started pushing the mapping to s3 for gecko-dev and gecko-projects, here:
https://moz-vcssync.s3-us-west-2.amazonaws.com/mapping/gecko-dev/git-mapfile.tar.bz2
https://moz-vcssync.s3-us-west-2.amazonaws.com/mapping/gecko-projects/git-mapfile.tar.bz2 (I have not reviewed which branch this is for)

These are the full mapfiles. So about 40mb uncompressed for gecko-dev. If this is unusable, or slow/other, I could break it into some time period or we could change how it is stored. The alternative I'm considering is that s3 allows SQL over csv/json content. So I could include the commit time as a column in the datafile and the s3 SQL may be able to SELECT from that, and we could go across multiple archives.

The full map files are what I need, so it'd be great if you can keep them. Is the URL stable or do you think it's going to change? Can I start using it?

(In reply to Dave House [:dhouse] from comment #6)

These are the full mapfiles. So about 40mb uncompressed for gecko-dev. If this is unusable, or slow/other, I could break it into some time period or we could change how it is stored. The alternative I'm considering is that s3 allows SQL over csv/json content. So I could include the commit time as a column in the datafile and the s3 SQL may be able to SELECT from that, and we could go across multiple archives.

The full mapfiles should be fine, especially hosted on S3 west-2 where searchfox lives. As a comment in fetch-hg-map.sh indicates, the motivating use of since was that the mapper service would sometimes fail and provide 503 errors. A flat-file on the same private network is great.

Oh, although we do have an explanation of how to to do try builds at https://github.com/mozsearch/mozsearch#testing-clang-plugin-changes that assumes there's a RESTy API for mapping a specific mercurial hash to a gecko-dev git hash.

(In reply to Marco Castelluccio [:marco] from comment #7)

The full map files are what I need, so it'd be great if you can keep them. Is the URL stable or do you think it's going to change? Can I start using it?

Please start using it. We can keep these urls (I don't see a need to change them unless we break the archives out by some partitioning, and we could always keep the full files in these places).

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #8)

(In reply to Dave House [:dhouse] from comment #6)

These are the full mapfiles. So about 40mb uncompressed for gecko-dev. If this is unusable, or slow/other, I could break it into some time period or we could change how it is stored. The alternative I'm considering is that s3 allows SQL over csv/json content. So I could include the commit time as a column in the datafile and the s3 SQL may be able to SELECT from that, and we could go across multiple archives.

The full mapfiles should be fine, especially hosted on S3 west-2 where searchfox lives. As a comment in fetch-hg-map.sh indicates, the motivating use of since was that the mapper service would sometimes fail and provide 503 errors. A flat-file on the same private network is great.

These files are getting uploaded as low-replication because I don't anticipate high numbers of downloads, and they'll cost less. With this we risk getting failed downloads sometimes. If that becomes a problem, let me know and we can make them standard s3.

Can you provide a comparable mapping for https://mapper.mozilla-releng.net/nss/mapfile/full as well?

If I try and access https://moz-vcssync.s3-us-west-2.amazonaws.com/mapping/nss/git-mapfile.tar.bz2 I just get an AccessDenied error (locally via Firefox).

Can you elaborate on the failed downloads risk profile? Because Windows builds are so slow (or were so slow? maybe the new taskcluster is better?), Searchfox just needs the revision that gets built by the nightly builds to show up in the mapfile within 3.5 hours. If you're talking about replication latency, I think the 3.5 hour cushion means we're fine. If you mean that there's a chance of random failures in the GET request, that's a different problem, although we can certainly add retries to curl/wget invocations.

Flags: needinfo?(dhouse)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #12)

Can you provide a comparable mapping for https://mapper.mozilla-releng.net/nss/mapfile/full as well?

If I try and access https://moz-vcssync.s3-us-west-2.amazonaws.com/mapping/nss/git-mapfile.tar.bz2 I just get an AccessDenied error (locally via Firefox).

Can you elaborate on the failed downloads risk profile? Because Windows builds are so slow (or were so slow? maybe the new taskcluster is better?), Searchfox just needs the revision that gets built by the nightly builds to show up in the mapfile within 3.5 hours. If you're talking about replication latency, I think the 3.5 hour cushion means we're fine. If you mean that there's a chance of random failures in the GET request, that's a different problem, although we can certainly add retries to curl/wget invocations.

I've added nss. I didn't see the mappings for that were used also:
https://moz-vcssync.s3-us-west-2.amazonaws.com/mapping/nss-dev/git-mapfile.tar.bz2

I'll change them to be standard uploads. I reviewed the s3 documentation and found that I was out-of-date; "REDUCED-REDUNDANCY" is deprecated and the similar option "ONEZONE_IA" does not warn of any chance of random request failures but charges for 30days minimum storage time per object. (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html#sc-infreq-data-access)

Flags: needinfo?(dhouse)

I'm now trying to update:

But I get a 404 when I apply the index.taskcluster.net/v1/task/ to firefox-ci-tc.services.mozilla.com/api/index/v1/task/ transform from https://github.com/mozsearch/mozsearch-mozilla/pull/60/files and end up with:

Aside: Note that because of bug 1574854 where sometimes 200 pages are returned that say "404" on them, trying to find the right URL is extra confusing. I thought just changing the domain name so I ended up at https://firefox-ci-tc.services.mozilla.com/v1/task/gecko.v2.mozilla-central.nightly.latest.firefox.linux64-opt/artifacts/public/build/target.jsshell.zip had fixed things, but it was in fact a 404 webpage, not anything useful.

Still not sure what's going on, but https://docs.taskcluster.net/docs/manual/using/artifacts#finding-the-latest-build seems to suggest we're missing a "routes" directive for jsshell to expose it? Maybe under https://searchfox.org/mozilla-central/source/.taskcluster.yml#117 ? (There's an irony in that because searchfox is broken I may be missing recent changes to that file. On the upside, the line number is guaranteed to be consistent until we fix the bug! ;)

Okay, so using wget on the old jsshell.zip URL gets us a post-redirect task link of https://queue.taskcluster.net/v1/task/JB-2Shx1S5qyYar9k0LvlQ/ which seems to suggest the most recent jsshell.zip comes from a build on 2019-09-18, suggesting that we've just been coasting by on build infrastructure that either no longer exists or is rarely built.

Okay, the right link is https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.linux64-opt/artifacts/public/build/target.jsshell.zip

I located a modern task, clicked on it to bring up the GUI, clicked "More..." under "Task Details" on the left and located the "Routes" section, where I then picked the first entry there, index.gecko.v2.mozilla-central.latest.firefox.linux64-opt, then chopped off the "index." prefix.

Apologies for the spam on my journey of taskcluster discovery!

I'm going to call this fixed as we experienced successful searchfox indexing runs today. The additional pull requests that were landed to this end were:

Thanks very much for everyone's assistance in getting searchfox operational again!

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Flags: needinfo?(rgarbas)

I'll reopen as the mapper is still down.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Can you use the map files in S3 as described in comment #6?

Flags: needinfo?(mcastelluccio)

I'm using those now, but partly relying on the mapper service for new commits (that is, I download the full map file and then use the mapper service from that moment on). I can fully rely on the S3 map files if you plan to drop the mapper.

Flags: needinfo?(mcastelluccio)

The new mapfiles are missing the current mozilla-central tip of https://hg.mozilla.org/mozilla-central/rev/a4ef4d6cdff03e76ed4471faea08e1071b1bf05c and the mozilla-beta bumper-bot https://hg.mozilla.org/releases/mozilla-beta/rev/4abc43dda5a6cf5bcb3850ece70b76a6cd7bdccd both of which correspond to today's nightlies. Or they were as of initial runtimes of 1.25 hours ago and 0.75 hours ago.

Flags: needinfo?(dhouse)

(In reply to Marco Castelluccio [:marco] from comment #21)

I'm using those now, but partly relying on the mapper service for new commits (that is, I download the full map file and then use the mapper service from that moment on). I can fully rely on the S3 map files if you plan to drop the mapper.

Yes, our plan is to decommission the mapper service.

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #22)

The new mapfiles are missing the current mozilla-central tip of https://hg.mozilla.org/mozilla-central/rev/a4ef4d6cdff03e76ed4471faea08e1071b1bf05c and the mozilla-beta bumper-bot https://hg.mozilla.org/releases/mozilla-beta/rev/4abc43dda5a6cf5bcb3850ece70b76a6cd7bdccd both of which correspond to today's nightlies. Or they were as of initial runtimes of 1.25 hours ago and 0.75 hours ago.

I'll review the uploads and mapper logs to see why these are missing.

(In reply to Dave House [:dhouse] from comment #24)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #22)

The new mapfiles are missing the current mozilla-central tip of https://hg.mozilla.org/mozilla-central/rev/a4ef4d6cdff03e76ed4471faea08e1071b1bf05c and the mozilla-beta bumper-bot https://hg.mozilla.org/releases/mozilla-beta/rev/4abc43dda5a6cf5bcb3850ece70b76a6cd7bdccd both of which correspond to today's nightlies. Or they were as of initial runtimes of 1.25 hours ago and 0.75 hours ago.

I'll review the uploads and mapper logs to see why these are missing.

The current mapfile has both of these hashes in it. I'm checking the logs, but it was probably behind on the conversion:

~$ bzgrep a4ef4d6cdff03e76ed4471faea08e1071b1bf05c ~/Downloads/git-mapfile.tar.bz2
Binary file /Users/house/Downloads/git-mapfile.tar.bz2 matches
~$ bzgrep 4abc43dda5a6cf5bcb3850ece70b76a6cd7bdccd ~/Downloads/git-mapfile.tar.bz2
Binary file /Users/house/Downloads/git-mapfile.tar.bz2 matches
Flags: needinfo?(dhouse)

(In reply to Chris AtLee [:catlee] from comment #23)

(In reply to Marco Castelluccio [:marco] from comment #21)

I'm using those now, but partly relying on the mapper service for new commits (that is, I download the full map file and then use the mapper service from that moment on). I can fully rely on the S3 map files if you plan to drop the mapper.

Yes, our plan is to decommission the mapper service.

OK, I'm stopping to use it then. I guess you could just WONTFIX this bug and not transition the mapper service to the new Taskcluster deployment, unless there are other users relying on it.

(In reply to Dave House [:dhouse] from comment #25)

(In reply to Dave House [:dhouse] from comment #24)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #22)

The new mapfiles are missing the current mozilla-central tip of https://hg.mozilla.org/mozilla-central

I found the problem. An error in the logs upload exited the process and prevented the mapping upload from running earlier. I've changed the script to proceed if there are errors from the s3 archive+upload processes (using set -e, so I've added || true on the uploads). So that problem will not repeat.

(In reply to Dave House [:dhouse] from comment #27)

(In reply to Dave House [:dhouse] from comment #25)

(In reply to Dave House [:dhouse] from comment #24)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #22)

The new mapfiles are missing the current mozilla-central tip of https://hg.mozilla.org/mozilla-central

I found the problem. An error in the logs upload exited the process and prevented the mapping upload from running earlier. I've changed the script to proceed if there are errors from the s3 archive+upload processes (using set -e, so I've added || true on the uploads). So that problem will not repeat.

The following runs completed with this change, and without any problems updating the mapping (and log archives).

Great, thank you! set -e is definitely a double-edged sword!

I'll leave it up to :garbas or someone else from the component to decide how to resolve the bug.

Bug 1565259 was created earlier this year to set up the mappings in s3. The s3 work here completed that.

Depends on: 1565259

Based on commentary I'm wontfixing this.

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → WONTFIX
Product: Release Engineering → Release Engineering Graveyard
You need to log in before you can comment on or make changes to this bug.