Closed Bug 1692453 Opened 4 years ago Closed 3 years ago

[meta] identify ActiveData upstream and downstream consumers

Categories

(Testing Graveyard :: ActiveData, task)

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: mtabara, Assigned: jmaher)

References

Details

(Keywords: leave-open, meta)

Attachments

(1 file)

ActiveData is currently broken and it's unclear what its upstream pushers and downstream consumers are. Plan for now is to:
a) identify all of the above
b) if most of them have switched already to mozci, retire ActiveData and remove resources

This bug is to track all the individual pieces and should act as a meta bug tracker.

Summary: [meta] decouple ActiveData upstream and downstream consumers to assess usage → [meta] identify ActiveData upstream and downstream consumers

I have been looking into test-info. There are 4 jobs that run on mozilla-central and they are currently producing artifacts, which means they are using in-tree data, not activedata for their purposes. Consumers of this data are the fission team and searchfox, possibly others, but I am not sure.

In addition, ./mach test-info ... often calls out to activedata and I think we need to remove some options for the developer there. Overall it should be fine. I don't know who uses ./mach test-info locally, but I suspect a handful of developers do- either way, there are not active complaints on Matrix regarding the lack of activedata, so I will assume this is not causing pain or rarely used.

The last piece is writeruntimes script in-tree, this needs to be converted to use mozci, that is something :ahal can pick up when he returns. This is a script that our team runs manually once or twice a year to help rebalance manifest chunking by getting average runtimes for the manifests and keeping track of outliers so we can have better balanced chunks.

As we are probably needing to run this again, it makes sense to spend a couple days converting this and running it to update the runtimes.

remove activedata from test-info

Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/24db9230985e remove activedata from test-info. r=releng-reviewers,aki

this patch removed test-info, there is still writeruntimes.py and some work the fission team had been using activedata for monitoring web-platform-tests intermittent failures on fission tasks.

Keywords: leave-open

Backed out for lint failures

backout: https://hg.mozilla.org/integration/autoland/rev/988762becb0f6b7d9b569e31938a1932c6ddcadb

push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=I5oO_BGTSSy4xckNuQE9Iw.0&revision=24db9230985eb99160aaa467ff97e0ac372127af&searchStr=linting%2Copt%2Candroid%2Cgradle%2Ctests%2Csource-test-mozlint-android-lints%2Clints

failure log: https://treeherder.mozilla.org/logviewer?job_id=331114914&repo=autoland&lineNumber=794

[task 2021-02-25T00:05:03.651Z] Compilation with Kotlin compile daemon was not successful
[task 2021-02-25T00:05:03.651Z] java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
[task 2021-02-25T00:05:03.651Z] 	java.io.EOFException
[task 2021-02-25T00:05:03.651Z] 	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:254)
[task 2021-02-25T00:05:03.651Z] 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
[task 2021-02-25T00:05:03.651Z] 	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
[task 2021-02-25T00:05:03.651Z] 	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
[task 2021-02-25T00:05:03.651Z] 	at com.sun.proxy.$Proxy101.compile(Unknown Source)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.incrementalCompilationWithDaemon(GradleKotlinCompilerWork.kt:277)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemon(GradleKotlinCompilerWork.kt:190)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemonOrFallbackImpl(GradleKotlinCompilerWork.kt:133)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.run(GradleKotlinCompilerWork.kt:116)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runCompilerAsync(GradleKotlinCompilerRunner.kt:144)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runCompilerAsync(GradleKotlinCompilerRunner.kt:139)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runJvmCompilerAsync(GradleKotlinCompilerRunner.kt:79)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.gradle.tasks.KotlinCompile.callCompilerAsync$kotlin_gradle_plugin(Tasks.kt:449)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.gradle.tasks.KotlinCompile.callCompilerAsync$kotlin_gradle_plugin(Tasks.kt:359)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompile.executeImpl(Tasks.kt:316)
[task 2021-02-25T00:05:03.651Z] 	at org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompile.execute(Tasks.kt:288)
[task 2021-02-25T00:05:03.651Z] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[task 2021-02-25T00:05:03.651Z] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[task 2021-02-25T00:05:03.651Z] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[task 2021-02-25T00:05:03.651Z] 	at java.lang.reflect.Method.invoke(Method.java:498)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.project.taskfactory.IncrementalTaskAction.doExecute(IncrementalTaskAction.java:47)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.run(ExecuteActionsTaskExecuter.java:284)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:273)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:258)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$200(ExecuteActionsTaskExecuter.java:67)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:145)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.ExecuteStep.execute(ExecuteStep.java:49)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CancelExecutionStep.execute(CancelExecutionStep.java:34)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:69)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.TimeoutStep.execute(TimeoutStep.java:49)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CatchExceptionStep.execute(CatchExceptionStep.java:33)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:43)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:29)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CacheStep.executeWithoutCache(CacheStep.java:134)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CacheStep.lambda$execute$3(CacheStep.java:83)
[task 2021-02-25T00:05:03.651Z] 	at java.util.Optional.orElseGet(Optional.java:267)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:82)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:36)
[task 2021-02-25T00:05:03.651Z] 	at org.gradle.internal.execution.impl.steps.PrepareCachingStep.execute(PrepareCachingStep.java:33)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:38)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:23)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:96)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.lambda$execute$1(SkipUpToDateStep.java:91)
[task 2021-02-25T00:05:03.652Z] 	at java.util.Optional.orElseGet(Optional.java:267)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:91)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:36)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:34)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:91)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:119)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.ResolvePreviousStateExecuter.execute(ResolvePreviousStateExecuter.java:43)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:94)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:56)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:55)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:67)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:315)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:305)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:101)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:49)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
[task 2021-02-25T00:05:03.652Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[task 2021-02-25T00:05:03.652Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[task 2021-02-25T00:05:03.652Z] 	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
[task 2021-02-25T00:05:03.652Z] 	at java.lang.Thread.run(Thread.java:748)
[task 2021-02-25T00:05:03.652Z] Caused by: java.io.EOFException
[task 2021-02-25T00:05:03.652Z] 	at java.io.DataInputStream.readByte(DataInputStream.java:267)
[task 2021-02-25T00:05:03.652Z] 	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240)
[task 2021-02-25T00:05:03.652Z] 	... 91 more
[task 2021-02-25T00:05:03.652Z] Unable to clear jar cache after compilation, maybe daemon is already down: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
[task 2021-02-25T00:05:03.652Z] 	java.net.ConnectException: Connection refused (Connection refused)
[task 2021-02-25T00:05:03.652Z] Could not connect to kotlin daemon. Using fallback strategy.
[task 2021-02-25T00:05:03.652Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
[task 2021-02-25T00:05:03.652Z] OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f8380000, 105381888, 0) failed; error='Cannot allocate memory' (errno=12)
Flags: needinfo?(jmaher)
Flags: needinfo?(jmaher) → needinfo?(aryx.bugmail)
Pushed by ncsoregi@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/55b388a03c73 remove activedata from test-info. r=releng-reviewers,aki CLOSED TREE
Depends on: 1696065

Is sql.telemetry.mozilla.org ActiveData?

I use the data to inspect queues/workerTypes to see who's running jobs (used if there are a ton and we'd like to know who from):
https://sql.telemetry.mozilla.org/queries/78484/source
https://sql.telemetry.mozilla.org/queries/66371/source

I used to scan Treeherder and it was miserable.

(In reply to Andrew Erickson [:aerickson] from comment #9)

Is sql.telemetry.mozilla.org ActiveData?

I use the data to inspect queues/workerTypes to see who's running jobs (used if there are a ton and we'd like to know who from):
https://sql.telemetry.mozilla.org/queries/78484/source
https://sql.telemetry.mozilla.org/queries/66371/source

Active data is defunct. Those links you provided are from the redash dashboard - it gets data from the Treeherder's production replica db directly.

I used to scan Treeherder and it was miserable.
What do you mean? :)

I used to scan Treeherder and it was miserable.
What do you mean? :)

I have a tool that scrapes Treeherder (written before I knew about redash).

Update: It used the Treeherder API, but I had to paginate through a ton of records that weren't relevant to find jobs that I was interested in.

This component is no longer maintained so closing as WONTFIX
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: