Closed Bug 1112777 Opened 10 years ago Closed 6 years ago

Generate up-to-date graph of builders relations

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: armenzg, Assigned: adusca, Mentored)

References

Details

Attachments

(1 file)

graph.json
826.96 KB, application/json
Details 
This will help generating the TryChooser UI and even validate it.
Anhad is interested in this bug but we need some better definition. As you know, "up to date" is a relative term and the "graph" is huge.

Some axioms we have are:
 - whatever is provided will be provided via an extension to the relengapi
 - it will (can) not be authoritative - it is always possible for a builder to be invalid by the time the build is requested. The client will need to expect and deal with that case.

Following XP conventions, we want to define the "minimal viable product" that will allow you to proceed.

Can you be more specific about the data values you're looking for, and how you intend to use them?
Assignee: nobody → ffledgling
Mentor: hwine
Flags: needinfo?(armenzg)
Hi hwine,
I have copy/pasted some information that I wrote on bug 983802 below.

Another approach and closer to the source of truth is to write a script that sets up try and test masters [1].
After those are setup we can probably call "python ~/.mozilla/releng/masters/try/config.py try" for each master to collect the latest data. We should probably an argument to allow us get a different output than the one generated [2]

If I do it by hand:
- ./setup-master.py master bm109-tests1-windows
- PYTHONPATH=`pwd`/master:$PYTHONPATH python master/config.py try

Remember there is also mobile_config.py and b2g_config.py under mozilla-tests.

Once you have the data from those masters you can saw them together.

I hope this helps!

[1] http://hg.mozilla.org/build/braindump/file/172091810231/community/create_community_slaves_and_masters.sh
[2] http://hg.mozilla.org/build/buildbot-configs/file/b545cad5b3cd/mozilla-tests/config.py#l2159


###############################################
The information can be extracted from here:
http://people.mozilla.org/~catlee/allthethings.json

The current code is in here:
http://hg.mozilla.org/build/tools/file/default/trychooser

The current web view is in here:
http://trychooser.pub.build.mozilla.org

For instance, we can determine the list of a available build jobs from [1] and [2].

Given one of the builders, e.g. "Android 4.2 x86 try build" we can find the platform that triggers it [3] which is: "platform": "android-x86","

We can then determine which test jobs listen to android-x86 to build the list associated to it.

I have in my mind a UI that will make things more beautiful once we have a graph of dependencies generated.

[1]
    "try-firefox": {
      "downstream": [
        "Android 2.2 Armv6 try build", 
        "Android 2.2 Debug try build", 
        "Android 2.2 try build", 
        "Android 4.2 x86 try build", 
        "Linux try build", 
        "Linux try leak test build", 
        "Linux try pgo-build", 
        "Linux x86-64 try asan build", 
        "Linux x86-64 try build", 
        "Linux x86-64 try debug asan build", 
        "Linux x86-64 try debug static analysis build", 
        "Linux x86-64 try leak test build", 
        "Linux x86-64 try pgo-build", 
        "Linux x86-64 try valgrind", 
        "OS X 10.7 64-bit try leak test build", 
        "OS X 10.7 try build", 
        "WINNT 5.2 try build", 
        "WINNT 5.2 try leak test build", 
        "WINNT 5.2 try pgo-build", 
        "WINNT 6.1 x86-64 try build", 
        "WINNT 6.1 x86-64 try leak test build", 
        "linux64-br-haz_try_dep", 
        "linux64-sh-haz_try_dep"
      ]
    },
[2]
    "b2g_try-b2g": {
      "downstream": [
        "b2g_try_emulator-debug_dep", 
        "b2g_try_emulator-jb-debug_dep", 
        "b2g_try_emulator-jb_dep", 
        "b2g_try_emulator-kk-debug_dep", 
        "b2g_try_emulator-kk_dep", 
        "b2g_try_emulator_dep", 
        "b2g_try_linux32_gecko build", 
        "b2g_try_linux64_gecko build", 
        "b2g_try_macosx64_gecko build", 
        "b2g_try_win32_gecko build"
      ]
    }, 
[3]
    "Android 4.2 x86 try build": {
      "properties": {
        "branch": "try", 
        "platform": "android-x86", 
        "product": "firefox", 
        "slavebuilddir": "try-and-x86-000000000000000000", 
        "stage_platform": "android-x86"
      }, 
      "shortname": "try-android-x86", 
      "slavebuilddir": "try-and-x86-000000000000000000", 
      "slavepool": "acfbcb424e5d81a16691f25875a7d77d949c9198"
    }, 

[4]     "Android 4.2 x86 Emulator try opt test androidx86-set-4": {
      "properties": {
        "branch": "try", 
        "platform": "android-x86", 
        "product": "mobile", 
        "repo_path": "try", 
        "script_repo_revision": "production", 
        "slavebuilddir": "test", 
        "stage_platform": "android-x86"
      }, 
      "shortname": "try_ubuntu64_hw_test-androidx86-set-4", 
      "slavebuilddir": "test", 
      "slavepool": "131acb84ca1c0b9db74a6df032310401fa91c53a"
    },
Flags: needinfo?(armenzg)
Armen - I am confused - per bug 983302 comment 3 (pasted above) you have everything you need already available to you (I assume you, even though that bug is not assigned to you).

I believe I misunderstood your needs initially, it now appears this is just part of implementing bug 983302. Whomever picks that up can also implement this piece following your guidelines.

The bug I _thought_ this was appears to be bug 1101548 -- where releng provides an api to the suite of try chooser assistants. Because I "miss sold" this bug to Anhad, pulling his assignment.

Can we dupe this to bug 1101548? If not, what's different about your use case?
Assignee: ffledgling → nobody
Flags: needinfo?(armenzg)
Sure; that works.
Flags: needinfo?(armenzg)
No longer blocks: 983802
This blocks https://github.com/armenzg/mozilla_ci_tools/issues/8.
I will be working around it with some mapping of the buildernames.
adusca is interested on fixing this.
I currently use platforms.py to help me trigger jobs and figure out which builds trigger a test job:
https://github.com/armenzg/mozilla_ci_tools/blob/master/mozci/platforms.py

FTR, today, in the "0.2.0" branch, I will have a fix for the b2g jobs.
Assignee: nobody → alicescarpa
This is a first step, but it still uses maps. I'm using allthethings to generate a map from platforms to prefixes. Currently this only works for builds in try-firefox, but I'd like some feedback on the general idea: pastebin.mozilla.org/8527121
Alice- I like this idea.  The code needs a s/item/build/ on line 16, but otherwise it parses it.  There are probably a lot of other job types to concern ourselves with in due time (nightly builds, etc.) which don't occur normally on try server.

the code you have written is easy to read and appears to be written that we could extend and scale it as needed!
adusca, I can tell that you're grasping what's going on.
At the final product I would like to get rid of the mapping altogether, however, I believe we have missing data.
I will file a bug, however, I believe we can use some mapping to help us for when we have all of the needed data.

Something that could be useful is for me to explain the whole idea.
Per $repo_name we generally have these two triggering schedulers (We are at first only going to focus on our 3 main products/jobs):
* ${repo_name}-firefox (This includes mobile and desktop on try)
* ${repo_name}-mobile
* b2g_${repo_name}-b2g

FYI, repo_name == branch. The term branch is incorrect depending on who you ask but it is a legacy term we live with.

Let me try to go through the whole process myself.

Cases
#####
* Given "Windows 7 32-bit mozilla-central opt test mochitest-1" we want to get "WINNT 5.2 mozilla-central build" [1]
* Given "Windows 7 32-bit mozilla-central pgo test mochitest-1" we want to get "WINNT 5.2 mozilla-central pgo-build" [2]
* Given "Windows 7 32-bit mozilla-central debug test mochitest-1" we want to get "WINNT 5.2 mozilla-central leak test build" [3]
* Given "Windows 7 32-bit mozilla-central talos chromez" we want to get "WINNT 5.2 mozilla-central build" [1]

NOTE: opt and talos jobs are triggered by the same build.

The only way that I can find a unique identifier for these builds [1][2][3] is if I look at them by using branch and stage_platform or their shortname.
      branch            stage_platform  shortname
leak  mozilla-central   win32           mozilla-central-win32
opt   mozilla-central   win32-pgo       mozilla-central-win32-pgo
pgo   mozilla-central   win32-debug     mozilla-central-win32-debug

Now, knowing how to associated a test builder to its build is what is complicated.

Let's think of "Windows 7 32-bit mozilla-central debug test mochitest-1" [4].
* Its scheduler is "tests-mozilla-central-win7-ix-debug-unittest" [6]
* These are the values from the test builder [4]
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "slavebuilddir": "test", 
        "stage_platform": "win32"
      "shortname": "mozilla-central_win7-ix-debug_test-mochitest-1", 

So we don't have "debug" or "mozilla-central-win32-debug" as I way to connect the test job to its build.

Also, if we compare the debug test job to an opt test job, there's no way of distinguishing them [5]:
* Its scheduler is "tests-mozilla-central-win7-ix-opt-unittest"
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "slavebuilddir": "test", 
        "stage_platform": "win32"
      }, 
      "shortname": "mozilla-central_win7-ix_test-mochitest-1", 

Perhaps we can cheat in this way (hack #1), we take all schedulers that end in "-unittest" and match the repo name (e.g. mozilla-central) [8]
Once we do that, we can split with "-" and grab the last string to determine if it is "debug", "opt" or "pgo":
    "tests-mozilla-central-win7-ix-debug-unittest": {
    "tests-mozilla-central-win7-ix-opt-unittest": {
    "tests-mozilla-central-win7-ix-pgo-unittest": {

Given this, I can take "Windows 7 32-bit mozilla-central debug test mochitest-1" and look for a builder that matches the shortname: "mozilla-central-win32-debug"

We will have to do something different for talos jobs [9] since the scheduler is named "tests-mozilla-central-win32-talos" [10].
Knowing the scheduler, we know that the build that triggered it is "mozilla-central-win32" (This is an opt build).

You also have to notice that not all jobs will actually trigger jobs (I added exclusion_state=all to show even hidden jobs):
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=6eb9c2bd39f6&exclusion_state=all
* Android armv7 API 9 mozilla-central debug build 
* b2g_mozilla-central_linux64-b2g-haz_dep
* etc

You also have to notice that the PGO naming in trees like mozilla-inbound (an integration repo), mozilla-{aurora,beta,release} (a release branch), try and b2g repos could be slightly different. We will know once we have unit tests.

[1]
    "WINNT 5.2 mozilla-central build": {
      "properties": {
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production"
      }, 
      "shortname": "mozilla-central-win32", 
      "slavebuilddir": "m-cen-w32-00000000000000000000", 
      "slavepool": "5d3ed72b7a0d89ae7c8055bfbe41fdc46da31d52"
    }, 
[2]
    "WINNT 5.2 mozilla-central pgo-build": {
      "properties": {
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production"
      }, 
      "shortname": "mozilla-central-win32-pgo", 
      "slavebuilddir": "m-cen-w32-pgo-0000000000000000", 
      "slavepool": "5d3ed72b7a0d89ae7c8055bfbe41fdc46da31d52"
    }, 

[3]
    "WINNT 5.2 mozilla-central leak test build": {
      "properties": {
        "branch": "mozilla-central", 
        "platform": "win32-debug", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production"
      }, 
      "shortname": "mozilla-central-win32-debug", 
      "slavebuilddir": "m-cen-w32-d-000000000000000000", 
      "slavepool": "5d3ed72b7a0d89ae7c8055bfbe41fdc46da31d52"
    }, 

[4]
    "Windows 7 32-bit mozilla-central debug test mochitest-1": {
      "properties": {
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production", 
        "slavebuilddir": "test", 
        "stage_platform": "win32"
      }, 
      "shortname": "mozilla-central_win7-ix-debug_test-mochitest-1", 
      "slavebuilddir": "test", 
      "slavepool": "951f92281e2d650bd5939b61de15d8c2089895f7"
    },
[5]
    "Windows 7 32-bit mozilla-central opt test mochitest-1": {
      "properties": {
        "branch": "mozilla-central", 
        "platform": "win32", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production", 
        "slavebuilddir": "test", 
        "stage_platform": "win32"
      }, 
      "shortname": "mozilla-central_win7-ix_test-mochitest-1", 
      "slavebuilddir": "test", 
      "slavepool": "951f92281e2d650bd5939b61de15d8c2089895f7"
    }, 

[6]
    "tests-mozilla-central-win7-ix-debug-unittest": {
      "downstream": [
        "Windows 7 32-bit mozilla-central debug test mochitest-1",
[7]
    "tests-mozilla-central-win7-ix-opt-unittest": {
      "downstream": [
        "Windows 7 32-bit mozilla-central opt test mochitest-1"

[8]
armenzg@armenzg-thinkpad:~/repos/bc.hg$ grep "mozilla\-central" allthethings.json | grep "\-unittest"
    "tests-mozilla-central-mountainlion-b2gdt-opt-unittest": {
    "tests-mozilla-central-mountainlion-debug-unittest": {
    "tests-mozilla-central-mountainlion-opt-unittest": {
    "tests-mozilla-central-panda_android-debug-unittest": {
    "tests-mozilla-central-panda_android-opt-unittest": {
    "tests-mozilla-central-snowleopard-debug-unittest": {
    "tests-mozilla-central-snowleopard-opt-unittest": {
    "tests-mozilla-central-ubuntu32_vm-b2gdt-opt-unittest": {
    "tests-mozilla-central-ubuntu32_vm-debug-unittest": {
    "tests-mozilla-central-ubuntu32_vm-opt-unittest": {
    "tests-mozilla-central-ubuntu32_vm-pgo-unittest": {
    "tests-mozilla-central-ubuntu64-asan_vm-opt-unittest": {
    "tests-mozilla-central-ubuntu64_hw-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-emulator-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-emulator-jb-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-emulator-kk-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-emulator-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-lg-emulator-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2g-lg-emulator-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2gdt-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm-b2gdt-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm-mulet-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm-pgo-unittest": {
    "tests-mozilla-central-ubuntu64_vm_armv6_large-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm_armv6_mobile-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm_large-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm_large-opt-unittest": {
    "tests-mozilla-central-ubuntu64_vm_mobile-debug-unittest": {
    "tests-mozilla-central-ubuntu64_vm_mobile-opt-unittest": {
    "tests-mozilla-central-win7-ix-debug-unittest": {
    "tests-mozilla-central-win7-ix-opt-unittest": {
    "tests-mozilla-central-win7-ix-pgo-unittest": {
    "tests-mozilla-central-win8_64-debug-unittest": {
    "tests-mozilla-central-win8_64-opt-unittest": {
    "tests-mozilla-central-win8_64-pgo-unittest": {
    "tests-mozilla-central-xp-ix-debug-unittest": {
    "tests-mozilla-central-xp-ix-opt-unittest": {
    "tests-mozilla-central-xp-ix-pgo-unittest": {

[9]
    "Windows 7 32-bit mozilla-central talos chromez": {
      "properties": {
        "branch": "mozilla-central", 
        "builddir": "mozilla-central_win7-ix_test-chromez", 
        "platform": "win7-ix", 
        "product": "firefox", 
        "repo_path": "mozilla-central", 
        "script_repo_revision": "production", 
        "slavebuilddir": "test", 
        "stage_platform": "win32"
      }, 
      "shortname": "mozilla-central_win7-ix_test-chromez", 
      "slavebuilddir": "test", 
      "slavepool": "951f92281e2d650bd5939b61de15d8c2089895f7"
    }, 

[10]
    "tests-mozilla-central-win32-talos": {
      "downstream": [
        "Windows 7 32-bit mozilla-central talos chromez", 

[11]
    "mozilla-central-firefox": {
      "downstream": [
        "Linux mozilla-central build", 
        "Linux mozilla-central leak test build", 
        "Linux x86-64 mozilla-central asan build", 
        "Linux x86-64 mozilla-central build", 
        "Linux x86-64 mozilla-central debug asan build", 
        "Linux x86-64 mozilla-central debug static analysis build", 
        "Linux x86-64 mozilla-central leak test build", 
        "Linux x86-64 mozilla-central valgrind", 
        "OS X 10.7 64-bit mozilla-central debug static analysis build", 
        "OS X 10.7 64-bit mozilla-central leak test build", 
        "OS X 10.7 mozilla-central build", 
        "WINNT 5.2 mozilla-central build", 
        "WINNT 5.2 mozilla-central leak test build", 
        "WINNT 6.1 x86-64 mozilla-central build", 
        "WINNT 6.1 x86-64 mozilla-central leak test build", 
        "linux64-br-haz_mozilla-central_dep"
      ]
    },
    "mozilla-central-mobile": {
      "downstream": [
        "Android 4.2 x86 mozilla-central build", 
        "Android armv7 API 11+ mozilla-central build", 
        "Android armv7 API 11+ mozilla-central debug build", 
        "Android armv7 API 9 mozilla-central build", 
        "Android armv7 API 9 mozilla-central debug build"
      ]
    }
    "b2g_mozilla-central-b2g": {
      "downstream": [
        "Linux x86-64 Mulet mozilla-central build", 
        "OS X Mulet mozilla-central build", 
        "Win32 Mulet mozilla-central build", 
        "b2g_mozilla-central_emulator-debug_dep", 
        "b2g_mozilla-central_emulator-jb-debug_dep", 
        "b2g_mozilla-central_emulator-jb_dep", 
        "b2g_mozilla-central_emulator_dep", 
        "b2g_mozilla-central_flame-kk_eng_dep", 
        "b2g_mozilla-central_hamachi_eng_dep", 
        "b2g_mozilla-central_linux32_gecko build", 
        "b2g_mozilla-central_linux64-b2g-haz_dep", 
        "b2g_mozilla-central_linux64_gecko build", 
        "b2g_mozilla-central_macosx64_gecko build"
      ]
    },
    "b2g_mozilla-central periodic": {
      "downstream": [
        "b2g_mozilla-central_dolphin_eng_periodic", 
        "b2g_mozilla-central_dolphin_periodic", 
        "b2g_mozilla-central_emulator-kk-debug_periodic", 
        "b2g_mozilla-central_emulator-kk_periodic", 
        "b2g_mozilla-central_flame-kk_eng-debug_periodic", 
        "b2g_mozilla-central_flame-kk_periodic", 
        "b2g_mozilla-central_hamachi_periodic", 
        "b2g_mozilla-central_linux32_gecko-debug build", 
        "b2g_mozilla-central_linux64_gecko-debug build", 
        "b2g_mozilla-central_macosx64_gecko-debug build", 
        "b2g_mozilla-central_nexus-4_eng_periodic", 
        "b2g_mozilla-central_nexus-4_periodic", 
        "b2g_mozilla-central_win32_gecko build", 
        "b2g_mozilla-central_win32_gecko-debug build"
      ]
    }, 
    "mozilla-central periodic": {
      "downstream": [
        "Linux mozilla-central pgo-build", 
        "Linux x86-64 mozilla-central pgo-build", 
        "WINNT 5.2 mozilla-central pgo-build", 
        "WINNT 6.1 x86-64 mozilla-central pgo-build"
      ]
    }, 
    "mozilla-central-android-api-11-l10n": {
      "downstream": [
        "Android armv7 API 11+ mozilla-central l10n nightly-1", 
        "Android armv7 API 11+ mozilla-central l10n nightly-2", 
        "Android armv7 API 11+ mozilla-central l10n nightly-3", 
        "Android armv7 API 11+ mozilla-central l10n nightly-4", 
        "Android armv7 API 11+ mozilla-central l10n nightly-5"
      ]
    }, 
    "weekly-mozilla-central": {
      "downstream": [
        "Linux x86-64 mozilla-central periodic file update", 
        "mozilla-central hg bundle"
      ]
    },
It's probably best to fix the data source:
http://hg.mozilla.org/build/braindump/file/961db9340928/buildbot-related/dump_allthethings.sh

I will check after lunch if there is any setup needed to run it.
I think the steps to run this script is:
hg clone https://hg.mozilla.org/build/braindump
cd community
./setup_buildbot_environment.sh
cd ~/.mozilla/releng/repos/buildbot-configs
source ../../venv/bin/activate
../braindump/buildbot-related/dump_allthethings.sh

allthethings.json will be generate under buildbot-configs.
Depends on: 1129594
adusca, you don't have to fix the data source if you don't want to go down that rabbit hole.
We can meanwhile fix it with the known hacks. We can file a separate bug to deal with the data source.

Let me know what works for you.

I filed the issue as bug 1129594.
adusca, are you fine going forward using some hacking rather than waiting on bug 1129594?

I think I'm going to have trouble figuring from where we can get the missing data to make the relations.
An alternative to using allthethings.json would be using this:
https://github.com/bhearsum/buildbot-scheduler-graph

adusca, we can file a bug specifically to making platforms.py "less mapped" and writing tests for it.
That way we don't depend on this bug.
Attached file graph.json — 
This graph is created using the version of allthethings.json generated by [1]. Since we are giving up on that for now this is a not a feasible way of generating up-to-date graphs of builders relations.

[1]https://bug1129594.bugzilla.mozilla.org/attachment.cgi?id=8562079
Blocks: 983802
See also bug 1123390.

We (releng/a-team/sheriffs/whoever) will continue to have problems with stale data and divergence unless we align on the same tool/source.
See Also: → 1123390
We want to generate the data from allthethings as well so we won't fall out of date.

For bug 1123390, would you mind using mozci to deal with allthethings?
We also learned how to improve the data generated in allthethings.json if you're interested:
https://wiki.mozilla.org/ReleaseEngineering/How_To/allthethings.json
We have a proof-of-concept:
In order to generate the data all you have to do is:
git clone https://github.com/armenzg/mozilla_ci_tools.git
cd mozilla_ci_tools
python setup.py develop
python scripts/misc/write_tests_per_platform_graph.py

That's it! You will then have a graphs.json dictionary with some of the data needed. Once we have an idea on how to generate the UI and what we're missing we can modify this script.

Here's some of the output:
{
    "android": [
        "cppunit",
        "crashtest",
        "crashtest-1",
        "crashtest-2",
        "jsreftest-1",
        "jsreftest-2",
...

Here are the remaining keys:
[u'android', u'android-api-11', u'android-api-9', u'android-armv6', u'android-x86', u'emulator', u'emulator-jb', u'emulator-kk', u'linux', u'linux-pgo', u'linux32_gecko', u'linux64', u'linux64-asan', u'linux64-cc', u'linux64-mulet', u'linux64-pgo', u'linux64_gecko', u'macosx64', u'win32', u'win32-pgo', u'win64', u'win64-pgo']

This is currently missing debug job keys.
Component: Tools → General
buildbot is dying!
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: