[meta] identify ActiveData upstream and downstream consumers
Categories
(Testing Graveyard :: ActiveData, task)
Tracking
(Not tracked)
People
(Reporter: mtabara, Assigned: jmaher)
References
Details
(Keywords: leave-open, meta)
Attachments
(1 file)
ActiveData is currently broken and it's unclear what its upstream pushers and downstream consumers are. Plan for now is to:
a) identify all of the above
b) if most of them have switched already to mozci
, retire ActiveData and remove resources
This bug is to track all the individual pieces and should act as a meta bug tracker.
Reporter | ||
Updated•4 years ago
|
Assignee | ||
Comment 1•4 years ago
|
||
I have been looking into test-info. There are 4 jobs that run on mozilla-central and they are currently producing artifacts, which means they are using in-tree data, not activedata for their purposes. Consumers of this data are the fission team and searchfox, possibly others, but I am not sure.
In addition, ./mach test-info ...
often calls out to activedata and I think we need to remove some options for the developer there. Overall it should be fine. I don't know who uses ./mach test-info
locally, but I suspect a handful of developers do- either way, there are not active complaints on Matrix regarding the lack of activedata, so I will assume this is not causing pain or rarely used.
The last piece is writeruntimes script in-tree, this needs to be converted to use mozci, that is something :ahal can pick up when he returns. This is a script that our team runs manually once or twice a year to help rebalance manifest chunking by getting average runtimes for the manifests and keeping track of outliers so we can have better balanced chunks.
As we are probably needing to run this again, it makes sense to spend a couple days converting this and running it to update the runtimes.
Assignee | ||
Comment 2•4 years ago
|
||
remove activedata from test-info
Updated•4 years ago
|
Assignee | ||
Comment 4•4 years ago
|
||
this patch removed test-info, there is still writeruntimes.py and some work the fission team had been using activedata for monitoring web-platform-tests intermittent failures on fission tasks.
Comment 5•4 years ago
|
||
Backed out for lint failures
backout: https://hg.mozilla.org/integration/autoland/rev/988762becb0f6b7d9b569e31938a1932c6ddcadb
failure log: https://treeherder.mozilla.org/logviewer?job_id=331114914&repo=autoland&lineNumber=794
[task 2021-02-25T00:05:03.651Z] Compilation with Kotlin compile daemon was not successful
[task 2021-02-25T00:05:03.651Z] java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
[task 2021-02-25T00:05:03.651Z] java.io.EOFException
[task 2021-02-25T00:05:03.651Z] at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:254)
[task 2021-02-25T00:05:03.651Z] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
[task 2021-02-25T00:05:03.651Z] at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
[task 2021-02-25T00:05:03.651Z] at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
[task 2021-02-25T00:05:03.651Z] at com.sun.proxy.$Proxy101.compile(Unknown Source)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.incrementalCompilationWithDaemon(GradleKotlinCompilerWork.kt:277)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemon(GradleKotlinCompilerWork.kt:190)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemonOrFallbackImpl(GradleKotlinCompilerWork.kt:133)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.run(GradleKotlinCompilerWork.kt:116)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runCompilerAsync(GradleKotlinCompilerRunner.kt:144)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runCompilerAsync(GradleKotlinCompilerRunner.kt:139)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.compilerRunner.GradleCompilerRunner.runJvmCompilerAsync(GradleKotlinCompilerRunner.kt:79)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.gradle.tasks.KotlinCompile.callCompilerAsync$kotlin_gradle_plugin(Tasks.kt:449)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.gradle.tasks.KotlinCompile.callCompilerAsync$kotlin_gradle_plugin(Tasks.kt:359)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompile.executeImpl(Tasks.kt:316)
[task 2021-02-25T00:05:03.651Z] at org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompile.execute(Tasks.kt:288)
[task 2021-02-25T00:05:03.651Z] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[task 2021-02-25T00:05:03.651Z] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[task 2021-02-25T00:05:03.651Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[task 2021-02-25T00:05:03.651Z] at java.lang.reflect.Method.invoke(Method.java:498)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.project.taskfactory.IncrementalTaskAction.doExecute(IncrementalTaskAction.java:47)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.run(ExecuteActionsTaskExecuter.java:284)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:301)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:293)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:273)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:258)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$200(ExecuteActionsTaskExecuter.java:67)
[task 2021-02-25T00:05:03.651Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:145)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.ExecuteStep.execute(ExecuteStep.java:49)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CancelExecutionStep.execute(CancelExecutionStep.java:34)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:69)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.TimeoutStep.execute(TimeoutStep.java:49)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CatchExceptionStep.execute(CatchExceptionStep.java:33)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:43)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.SnapshotOutputStep.execute(SnapshotOutputStep.java:29)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CacheStep.executeWithoutCache(CacheStep.java:134)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CacheStep.lambda$execute$3(CacheStep.java:83)
[task 2021-02-25T00:05:03.651Z] at java.util.Optional.orElseGet(Optional.java:267)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:82)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.CacheStep.execute(CacheStep.java:36)
[task 2021-02-25T00:05:03.651Z] at org.gradle.internal.execution.impl.steps.PrepareCachingStep.execute(PrepareCachingStep.java:33)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:38)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.StoreSnapshotsStep.execute(StoreSnapshotsStep.java:23)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:96)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.lambda$execute$1(SkipUpToDateStep.java:91)
[task 2021-02-25T00:05:03.652Z] at java.util.Optional.orElseGet(Optional.java:267)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:91)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:36)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.execution.impl.DefaultWorkExecutor.execute(DefaultWorkExecutor.java:34)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:91)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:119)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.ResolvePreviousStateExecuter.execute(ResolvePreviousStateExecuter.java:43)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:94)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:56)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:55)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:67)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:315)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:305)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:175)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:101)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
[task 2021-02-25T00:05:03.652Z] at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:49)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
[task 2021-02-25T00:05:03.652Z] at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
[task 2021-02-25T00:05:03.652Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[task 2021-02-25T00:05:03.652Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[task 2021-02-25T00:05:03.652Z] at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
[task 2021-02-25T00:05:03.652Z] at java.lang.Thread.run(Thread.java:748)
[task 2021-02-25T00:05:03.652Z] Caused by: java.io.EOFException
[task 2021-02-25T00:05:03.652Z] at java.io.DataInputStream.readByte(DataInputStream.java:267)
[task 2021-02-25T00:05:03.652Z] at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240)
[task 2021-02-25T00:05:03.652Z] ... 91 more
[task 2021-02-25T00:05:03.652Z] Unable to clear jar cache after compilation, maybe daemon is already down: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
[task 2021-02-25T00:05:03.652Z] java.net.ConnectException: Connection refused (Connection refused)
[task 2021-02-25T00:05:03.652Z] Could not connect to kotlin daemon. Using fallback strategy.
[task 2021-02-25T00:05:03.652Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=utf-8
[task 2021-02-25T00:05:03.652Z] OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f8380000, 105381888, 0) failed; error='Cannot allocate memory' (errno=12)
Assignee | ||
Comment 6•4 years ago
|
||
these don't seem related to my push, retriggered 5 times, all green- seems like a random intermittent:
https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=I5oO_BGTSSy4xckNuQE9Iw.0&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=24db9230985eb99160aaa467ff97e0ac372127af&searchStr=lints
:aryx, can we reland this?
Updated•4 years ago
|
Comment 8•4 years ago
|
||
bugherder |
Comment 9•3 years ago
|
||
Is sql.telemetry.mozilla.org ActiveData?
I use the data to inspect queues/workerTypes to see who's running jobs (used if there are a ton and we'd like to know who from):
https://sql.telemetry.mozilla.org/queries/78484/source
https://sql.telemetry.mozilla.org/queries/66371/source
I used to scan Treeherder and it was miserable.
Comment 10•3 years ago
|
||
(In reply to Andrew Erickson [:aerickson] from comment #9)
Is sql.telemetry.mozilla.org ActiveData?
I use the data to inspect queues/workerTypes to see who's running jobs (used if there are a ton and we'd like to know who from):
https://sql.telemetry.mozilla.org/queries/78484/source
https://sql.telemetry.mozilla.org/queries/66371/source
Active data is defunct. Those links you provided are from the redash dashboard - it gets data from the Treeherder's production replica db directly.
I used to scan Treeherder and it was miserable.
What do you mean? :)
Comment 11•3 years ago
•
|
||
I used to scan Treeherder and it was miserable.
What do you mean? :)
I have a tool that scrapes Treeherder (written before I knew about redash).
Update: It used the Treeherder API, but I had to paginate through a ton of records that weren't relevant to find jobs that I was interested in.
Comment 12•3 years ago
|
||
Updated•3 years ago
|
Description
•