Closed Bug 1446794 Opened 7 years ago Closed 7 years ago

statsum.taskcluster.net not resolving

Categories

(Taskcluster :: Operations and Service Requests, task)

Product:
Taskcluster
Component:
Operations and Service Requests
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: jonasfj)

References

Details

dustin@lamport ~ $ dig statsum.taskcluster.net ; <<>> DiG 9.10.3-P4-Ubuntu <<>> statsum.taskcluster.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36192 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;statsum.taskcluster.net. IN A ;; ANSWER SECTION: statsum.taskcluster.net. 57 IN CNAME statsum.43a9858c.svc.dockerapp.io. ;; AUTHORITY SECTION: dockerapp.io. 5 IN SOA ns1.dockerapp.io. support.docker.com. 2018031722 7200 3600 1209600 5
The service in docker cloud shows as stopped 19 minutes ago.
2018-03-18T16:54:15.599340556Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:15.599343533Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:15.599346580Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:15.599349446Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:15.599352304Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:15.599355201Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:15.599357784Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:15.599360330Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:15.599362821Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:15.599365701Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:15.599368429Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:15.599383448Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:15.599386430Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:15.599389348Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:20.599834523Z ----------- 2018-03-18T16:54:20.599864792Z Memory usage: 4 mb 2018-03-18T16:54:20.599874223Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:20.599877682Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:20.599881036Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:20.599884024Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:20.599886922Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:20.599889762Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:20.599892524Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:20.599895286Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:20.599897842Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:20.599900341Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:20.599902841Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:20.599905272Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:20.599907851Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:20.599910397Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:20.599913001Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:20.599915676Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:25.600330647Z ----------- 2018-03-18T16:54:25.600375714Z Memory usage: 4 mb 2018-03-18T16:54:25.600385783Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:25.600389621Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:25.600393556Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:25.600396830Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:25.600400245Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:25.600403765Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:25.600406710Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:25.600409613Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:25.600421732Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:25.600424586Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:25.600427075Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:25.600429531Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:25.600431996Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:25.600434491Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:25.600438018Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:25.600440533Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:30.600897838Z ----------- 2018-03-18T16:54:30.600927022Z Memory usage: 4 mb 2018-03-18T16:54:30.600935419Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:30.600938997Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:30.600942430Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:30.600945477Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:30.600948421Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:30.600951299Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:30.600954049Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:30.600956881Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:30.600959551Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:30.600962175Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:30.600964761Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:30.600967315Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:30.600969791Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:30.600972412Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:30.600974996Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:30.600977642Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:35.601314882Z ----------- 2018-03-18T16:54:35.601349634Z Memory usage: 4 mb 2018-03-18T16:54:35.601357776Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:35.601360964Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:35.601364100Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:35.601366953Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:35.601378502Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:35.601381491Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:35.601384179Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:35.601386861Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:35.601389418Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:35.601391961Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:35.601394443Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:35.601396893Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:35.601399332Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:35.601401768Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:35.601404283Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:35.601406901Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:40.602135966Z ----------- 2018-03-18T16:54:40.602163868Z Memory usage: 4 mb 2018-03-18T16:54:40.602171807Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:40.602175388Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:40.602178181Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:40.602181158Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:40.602184118Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:40.602186876Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:40.602189427Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:40.602191992Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:40.602194520Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:40.602197173Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:40.602199658Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:40.602202206Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:40.602204739Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:40.602213156Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:40.602215665Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:40.602218268Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:45.602378850Z ----------- 2018-03-18T16:54:45.602394397Z Memory usage: 4 mb 2018-03-18T16:54:45.602397863Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:45.602407257Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:45.602410434Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:45.602413225Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:45.602416055Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:45.602418828Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:45.602421404Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:45.602423958Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:45.602426499Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:45.602429054Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:45.602432509Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:45.602435114Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:45.602437792Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:45.602440334Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:45.602442996Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:45.602446283Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:50.602871421Z ----------- 2018-03-18T16:54:50.602896873Z Memory usage: 4 mb 2018-03-18T16:54:50.602907594Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:50.602916905Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:50.602920074Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:50.602923210Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:54:50.602926110Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:50.602928919Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:50.602931637Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:50.602934317Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:50.602936897Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:50.602939375Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:50.602941865Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:50.602944407Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:50.602957665Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:50.602962405Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:50.602966479Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:50.602970636Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:55.603310783Z ----------- 2018-03-18T16:54:55.603337690Z Memory usage: 4 mb 2018-03-18T16:54:55.603345502Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:54:55.603348922Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:54:55.603351687Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:54:55.603354506Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:54:55.603357262Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:54:55.603359831Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:54:55.603362381Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:54:55.603364966Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:54:55.603367426Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:54:55.603369957Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:54:55.603372409Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:54:55.603374846Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:54:55.603377329Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:54:55.603379775Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:54:55.603382201Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:54:55.603384832Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:55:00.603915885Z ----------- 2018-03-18T16:55:00.603947426Z Memory usage: 4 mb 2018-03-18T16:55:00.604746738Z Project: taskcluster-notify counter metrics: 0 value metrics: 4 2018-03-18T16:55:00.604760461Z Project: docker-worker counter metrics: 217 value metrics: 381 2018-03-18T16:55:00.604768807Z Project: taskcluster-auth counter metrics: 21 value metrics: 20 2018-03-18T16:55:00.604776444Z Project: taskcluster-index counter metrics: 9 value metrics: 11 2018-03-18T16:55:00.604784397Z Project: statsum counter metrics: 1 value metrics: 3 2018-03-18T16:55:00.604792020Z Project: taskcluster-queue counter metrics: 77 value metrics: 79 2018-03-18T16:55:00.604799899Z Project: tc-github-staging counter metrics: 0 value metrics: 6 2018-03-18T16:55:00.604821466Z Project: tc-treeherder counter metrics: 8 value metrics: 4 2018-03-18T16:55:00.604828376Z Project: taskcluster-login counter metrics: 3 value metrics: 5 2018-03-18T16:55:00.604835049Z Project: taskcluster-github counter metrics: 4 value metrics: 8 2018-03-18T16:55:00.604841869Z Project: aws-provisioner counter metrics: 0 value metrics: 2 2018-03-18T16:55:00.604850882Z Project: ec2-manager counter metrics: 32 value metrics: 26 2018-03-18T16:55:00.604857772Z Project: staging-aws counter metrics: 0 value metrics: 4 2018-03-18T16:55:00.604864535Z Project: taskcluster-secrets counter metrics: 4 value metrics: 8 2018-03-18T16:55:00.604871198Z Project: taskcluster-hooks counter metrics: 1 value metrics: 5 2018-03-18T16:55:00.604877888Z Project: purge-cache counter metrics: 4 value metrics: 6 2018-03-18T16:55:05.010364530Z panic: failed to marshal payload to JSON: json: unsupported value: NaN 2018-03-18T16:55:05.010421198Z 2018-03-18T16:55:05.010433217Z goroutine 4821 [running]: 2018-03-18T16:55:05.010466910Z github.com/taskcluster/statsum/server.(*signalfxDrain).Flush(0xc4201b4000, 0x30000f0182030005, 0x10182020a0182) 2018-03-18T16:55:05.010485709Z /go/src/github.com/taskcluster/statsum/server/signalfx.go:51 +0x3e5 2018-03-18T16:55:05.010505677Z github.com/taskcluster/statsum/server.(*StatSum).scheduleSubmission.func2(0xc42044a000, 0xc420157d48, 0xc420183b00, 0xa510e0, 0xc4201b4000) 2018-03-18T16:55:05.010510269Z /go/src/github.com/taskcluster/statsum/server/server.go:357 +0x49 2018-03-18T16:55:05.010514299Z created by github.com/taskcluster/statsum/server.(*StatSum).scheduleSubmission 2018-03-18T16:55:05.010518471Z /go/src/github.com/taskcluster/statsum/server/server.go:364 +0x3d6 2018-03-18T16:55:05.518429087Z + exec statsum server
Flags: needinfo?(jopsen)
Ideally this won't cause production issues. The service seems to start up OK sometimes, and fail others -- possibly something is sending a NaN that causes it to crash. I can't tell why that causes the hostname to not resolve, though. That may be a separate issue. Copying the hostname from the docker cloud UI: dustin@jemison ~ $ dig @ns1.dockerapp.io. statsum.43a9858c.svc.dockerapp.io ; <<>> DiG 9.10.5-P2-RedHat-9.10.5-2.P2.fc25 <<>> @ns1.dockerapp.io. statsum.43a9858c.svc.dockerapp.io ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 32799 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;statsum.43a9858c.svc.dockerapp.io. IN A ;; AUTHORITY SECTION: dockerapp.io. 5 IN SOA ns1.dockerapp.io. support.docker.com. 2018031809 7200 3600 1209600 5 ;; Query time: 124 msec ;; SERVER: 162.159.24.144#53(162.159.24.144) ;; WHEN: Sun Mar 18 13:01:28 EDT 2018 ;; MSG SIZE rcvd: 120 -- Note that stateless-dns.d85ca0db.svc.dockerapp.io still resolves -- that would likely cause a more subtantial outage.
This appears to not be causing an outage, so I'm going to start my travel and not worry about it.
Flags: needinfo?(jopsen)
Let's just deploy this to an instance in EC2. It will need to auto-restart.
Assignee: nobody → jopsen
Blocks: 1447585
This is in an EC2 instance now. I suspect we will soon kill statsum..
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: Operations → Operations and Service Requests
You need to log in before you can comment on or make changes to this bug.