1255738 - Add select method to Dataset API to extract a subset of fields from pings.

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Roberto Agostino Vitillo (:rvitillo)]

See http://goessner.net/articles/JsonPath/.

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

See also http://jmespath.org/

Comment 3

[Katie Parlante]

Assigning to Sunah to take when she has an opportunity

Comment 4

[GitHub Autolander Bot]

Attachment: [python_moztelemetry] mozilla:bug-1255738-dataset-select > mozilla:master

Comment 5

[Mauro Doglio [:mdoglio]]

:sunahsuh I took a stab at this, I hope you don't mind :)

Comment 6

[William Lachance (:wlach)]

Going to investigate why Mauro's patch isn't working in production.

Comment 7

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #6)
> Going to investigate why Mauro's patch isn't working in production.

It's these instructions that are broken: https://wiki.mozilla.org/Telemetry/Custom_analysis_with_spark#How_do_I_load_an_external_library_into_the_cluster.3F

If I hand install it by copying the files into /mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/ everything works great (and seems like a nice improvement). I'll look into filing a bug and figuring out why that doesn't work tomorrow, if someone else doesn't get to it first.

Comment 8

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #7)
> (In reply to William Lachance (:wlach) (use needinfo!) from comment #6)
> > Going to investigate why Mauro's patch isn't working in production.
> 
> It's these instructions that are broken:
> https://wiki.mozilla.org/Telemetry/
> Custom_analysis_with_spark#How_do_I_load_an_external_library_into_the_cluster
> .3F

Actually, I was wrong: these steps work fine too. I think I just made some kind of error on my side that made loading the egg not work. 

Apparently this feature works for mdoglio too. I think we're clear to land?

Comment 9

[Georg Fritzsche [:gfritzsche]]

Mauro, it looks like the docs were not updated with this method?

http://python-moztelemetry.readthedocs.io/en/latest/userguide.html#module-moztelemetry.dataset

Comment 10

[William Lachance (:wlach)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #9)
> Mauro, it looks like the docs were not updated with this method?
> 
> http://python-moztelemetry.readthedocs.io/en/latest/userguide.html#module-
> moztelemetry.dataset

It does appear under the "stable" docs:

http://python-moztelemetry.readthedocs.io/en/stable/userguide.html#moztelemetry.dataset.Dataset.select

I think the problem is that 'latest' points to a weird tag:

http://readthedocs.org/projects/python-moztelemetry/versions/

... we should probably reset that to master or something.

Roberto, can you add me to the list of owners for this repository so I can fix this?

Comment 11

[Mauro Doglio [:mdoglio]]

:wlach I added you

Comment 12

[William Lachance (:wlach)]

I believe the doc issue is fixed. Mauro, shall we resolve this?

Comment 13

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #12)
> I believe the doc issue is fixed. Mauro, shall we resolve this?

Oops, didn't mean to actually resolve. Oh well, someone can always reopen.