Last Comment Bug 768456 - API call for getting the first n docs to appear for a search term
: API call for getting the first n docs to appear for a search term
Status: RESOLVED WONTFIX
:
Product: Mozilla Developer Network
Classification: Other
Component: API (show other bugs)
: unspecified
: All All
P2 normal with 12 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
:
Mentors:
: 671893 (view as bug list)
Depends on: 794525 839214
Blocks: 768469 965197 989063
  Show dependency treegraph
 
Reported: 2012-06-26 07:23 PDT by Paul Rouget [:paul]
Modified: 2016-01-08 07:22 PST (History)
20 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
screenshot of the current approach (292.63 KB, image/png)
2012-06-26 07:23 PDT, Paul Rouget [:paul]
no flags Details

Description User image Paul Rouget [:paul] 2012-06-26 07:23:38 PDT
Created attachment 636702 [details]
screenshot of the current approach

We want to introduce a "mdn" command in the Firefox Command Line. This command will let the user request for the documentation for any CSS, HTML and JavaScript elements.

> mdn transform-origin

It is a little trick to do that today because we don't know:
- what is the language of the user
- what is the nature of its request (CSS, HTML or JavaScript)

In an ideal world, we would just make a request to:
http://developer.mozilla.org/imfeelingluck?q=foobar
…and that would return a page with the correct locale, and with the best doc match.

Current approach (CSS only): https://gist.github.com/1d943933ffb43a72946d
Comment 1 User image John Karahalis [:openjck] 2012-06-26 07:35:32 PDT
This is a wonderful idea. Thank you for filing this, Paul.

The team is currently heads-down preparing for a new release of the MDN this summer, but I will be sure to bring this to the attention of the development team soon after that.
Comment 2 User image Eric Shepherd [:sheppy] 2012-06-26 07:48:59 PDT
Yes, I asked Paul to file this because it would be awesome for dev tools and IDEs.
Comment 3 User image Paul Rouget [:paul] 2012-08-09 03:26:44 PDT
It would be great to also be able (but not mandatory) to specify "type" and "lang":

http://developer.mozilla.org/imfeelingluck?q=foobar&type=css&lang=fr_FR
Comment 4 User image Eric Shepherd [:sheppy] 2012-08-09 05:46:15 PDT
Yes, this would be very good to have, and is a good explanation of a feature I've been trying to figure out how to describe for months. Thanks, Paul!
Comment 5 User image Les Orchard [:lorchard] 2012-08-09 13:21:08 PDT
Just for the heck of it, I tried hacking on a variant of this CSS command using our new Kuma resources:

https://gist.github.com/3307756

It's a bit simpler, though absolutely terrible because I don't know much about xpcom or the new CLI stuff :)
Comment 6 User image Paul Rouget [:paul] 2012-09-04 23:49:32 PDT
Do we have anything plan yet? How can I help?
Comment 7 User image Luke Crouch [:groovecoder] 2012-09-05 06:44:06 PDT
Les's command puts the "lucky" in "I'm feeling lucky." Can you just use that? Do most of the css properties live right under docs/CSS like that?
Comment 8 User image Paul Rouget [:paul] 2012-09-05 07:40:02 PDT
(In reply to Luke Crouch [:groovecoder] from comment #7)
> Les's command puts the "lucky" in "I'm feeling lucky." Can you just use
> that? Do most of the css properties live right under docs/CSS like that?

We don't want to target only CSS> We want to be able to request the documentation for any language (CSS, JavaScript and HTML) without knowing at first what the language is.
Comment 9 User image Les Orchard [:lorchard] 2012-09-05 08:15:25 PDT
Sadly, we don't have a decent self-hosted search feature yet, so I think that's a dependency for implementing something like this. Currently, we just rely on Google site search
Comment 10 User image Les Orchard [:lorchard] 2013-02-08 10:00:13 PST
This is blocked by actually having a search feature.
Comment 11 User image John Karahalis [:openjck] 2013-03-13 10:52:53 PDT
:basta is interested in this as well, and with Elastic Search coming very soon it should be doable.
Comment 12 User image John Karahalis [:openjck] 2013-06-11 12:56:30 PDT
*** Bug 671893 has been marked as a duplicate of this bug. ***
Comment 13 User image John Karahalis [:openjck] 2013-06-11 12:57:29 PDT
Hey all. We are now using Bugzilla votes to gauge priorities. Please vote this up if it is one of the 25 MDN bugs you care most about.
Comment 14 User image Florian Scholz [:fscholz] (MDN) 2013-12-19 02:44:46 PST
This is an awesome idea, Paul!

How much does our new Elastic Search help you?

> mdn transform-origin
https://developer.mozilla.org/en-US/search.json?q=transform-origin

You will get better results if you provide the topic in some cases:

>mdn js string.contains
https://developer.mozilla.org/en-US/search.json?q=string.contains?topic=js

More information about search parameters (locale= is available, too):
https://developer.mozilla.org/en-US/docs/Project:MDN/Contributing/Advanced_search

What can we improve to make this happen?
Comment 15 User image Paul Rouget [:paul] 2013-12-19 05:06:04 PST
(In reply to Florian Scholz [:fscholz] (elchi3) from comment #14)

This sounds perfect. We should totally try integrate that.

Should something like that:

> doc string.contains

… returns a list of results, or just the first result summary?

The syntax could be:

> doc <keyworkd> [<hint>]

where hint is the topic or tags (which one should we use?)
Comment 16 User image Luke Crouch [:groovecoder] 2013-12-19 07:49:27 PST
needinfo? and +1 for :jezdez who implemented the search json API.

:jezdez, can you help :paul integrate MDN search results into firefox command line?
Comment 17 User image Jannis Leidel [:jezdez] 2013-12-19 08:37:08 PST
Absolutely, I'm stoked we can offer that API now. There are a few things missing that may be required to make the integration solid, e.g. API throttling and/or caching. That said, ZEUS our HTTP frontend proxy should cache responses for a minute or so.

:paul right now you can basically take *any* search URL and append a ".json" to the URL path (or send a proper "Accept: application/json" request header) to get the search results formatted as json.

There are a few parameters you can set:

- page -- the search result page (1-indexed)
- per_page -- the number of results per page, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&per_page=5 (default is 10, max is 100)
- topic -- one or more additive topic filters, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js (the full list of filters is also transfered as part of the response data in the "filters" item)
- q -- the full text search term

Here is an overview of the returned search result data:

    {
        "count": number of results,
        "next": url of next result page or null,
        "previous": url of previous result page or null,
        "query": value of fulltext search parameter,
        "page": current search result page number,
        "pages": total number of search result pages,
        "start": start index of search result on current page,
        "end": end index of search result on current page,
        "filters": list of topic filters available on current page (see below),
        "documents": list of search results (see below)
    }

The list of "filters" is actually a list of filter groups that each have a list of assigned filters. that's mostly a result of the frontend UI but should also allow you to hide uninteresting filters if you want to offer them to users through the "doc" command. Anyways, here's the schema for each of those filters:

    {
        "name": filter name,
        "slug": filter slug as used in the topic query parameter,
        "count": number of documents matching that topic on the current search result page,
        "active": boolean describing whether this filter is active on the current search result page,
        "urls": {
            "active": absolute URL path for activating this filter,
            "inactive": absolute URL path for deactivating this filter
        }
    }

Here is the schema for each document as listed in the "documents" item above:

    {
        "title": title of wiki document,
        "slug": slug of wiki document as used in URL path,
        "locale": locale code of wiki document as used in URL path,
        "excerpt": short except of wiki document,
        "url": absolute URL of wiki document,
        "edit_url": absolute URL of wiki document for the edit interface,
        "tags": list of tags,
    }
Comment 18 User image Jannis Leidel [:jezdez] 2014-01-27 12:13:40 PST
:paul Anything you need further to implement this on your side (assuming it hasn't happened yet)?
Comment 19 User image Paul Rouget [:paul] 2014-01-28 07:45:23 PST
I'm not working on that anymore, but I think :jwalker or :pbrosset can help.

Joe, Patrick, this API sounds ideal to integrate documentation in the devtools. Maybe via the command line, or even via a popup. What do you guys think?

Jannis, I think we can safely close the bug. The API is great. Let's just make sure the appropriate bugs are filed to use this API at the devtools level (maybe bug 768469 is enough).
Comment 20 User image Patrick Brosset <:pbro> (PTO, back on Feb 27th) 2014-01-29 03:25:29 PST
Cool API!
That should come in pretty handy for a number of features in the devtools.
jwalker just filed bug 965197 which I think could be pretty useful.
Comment 21 User image Joe Walker [:jwalker] (needinfo me or ping on irc) 2014-01-29 03:26:44 PST
(In reply to Paul Rouget [:paul] from comment #19)
> Joe, Patrick, this API sounds ideal to integrate documentation in the
> devtools. Maybe via the command line, or even via a popup. What do you guys
> think?

I think that bug 768469 (gcli command) is a good place to start, but there are a bunch of things like Right-click on property name in inspector (just raised bug 965197) are other obvious integration points.

:jezdez I think we should probably pass you some traceability info so you know where the traffic is coming from, do you have an suggestions for how to do that?
Comment 22 User image Jannis Leidel [:jezdez] 2014-01-29 08:49:01 PST
:jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you suffix the user-agent somehow? Alternatively and maybe a better idea is to simply send along an additional query parameter (e.g. /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start tracking this on the server side if you're okay with that.

:groovecoder Do you think we can start using GA to track search usage via clients?
Comment 23 User image Joe Walker [:jwalker] (needinfo me or ping on irc) 2014-01-29 10:14:37 PST
(In reply to Jannis Leidel [:jezdez] from comment #22)
> :jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you
> suffix the user-agent somehow? Alternatively and maybe a better idea is to
> simply send along an additional query parameter (e.g.
> /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start
> tracking this on the server side if you're okay with that.

An additional query parameter would be easiest.

Gerv: There are a few places in Firefox developer tools where we're thinking of providing links to MDN, and considering adding a query parameter so we know where the links came from.

I think that telling MDN that the click comes from inside developer tools (and even where inside developer tools) is OK because this is similar to a referrer.

I think it's OK to tell MDN what firefox version we're using because that's discoverable from the user-agent anyway.
Comment 24 User image Gervase Markham [:gerv] 2014-01-29 10:19:39 PST
That seems fine to me, but maybe I'm not the right person to know our policies on such things. NEEDINFO alina.

Gerv
Comment 25 User image Luke Crouch [:groovecoder] 2014-01-30 07:43:41 PST
If we can, it would be great to add tracking params to the MDN links from devtools like so:

https://developer.mozilla.org/<locale>/docs/<article-slug>?utm_campaign=SearchAPI&utm_source=Firefox&utm_medium=DevTools[utm_content=<specific-tool>]

Using GA to track the the search API traffic sounds non-trivial. We would need to add GA Measurement Protocol[1] code to the ES python code.

[1] https://developers.google.com/analytics/devguides/collection/protocol/v1/
Comment 26 User image Jannis Leidel [:jezdez] 2014-01-30 08:08:31 PST
:jwalker I think we misunderstood each other, I meant the URLs of the calls against the search API should contain a parameter, not only the links to MDN documents. That would allow us to distinguish the traffic generated against the search from webtools from the common traffic.

:groovecoder Calling the GA API is actually rather trivial (e.g. https://github.com/mirumee/google-measurement-protocol) can you elaborate what makes you think it's not?

I definitely think that only tracking the traffic to the documents and not requests against the search itself would leave us with only partial data on what users are searching from the webtools. Clicking on a link in the webtools is AFAIU the second action a user would take, after being offered that link via running a CLI command or hovering over a HTML element of the page.
Comment 27 User image Luke Crouch [:groovecoder] 2014-01-30 08:52:03 PST
:jezdez - primarily, we need to upgrade to GA Universal Analytics to enable the Measurement Protocol. I'll work with :garethc and :cmore on that but it might take a while.
Comment 28 User image Luke Crouch [:groovecoder] 2014-01-30 10:01:53 PST
Update: We are planning to upgrade to UA when it comes out of beta sometime in the next couple months.
Comment 29 User image Eric Shepherd [:sheppy] 2014-02-28 23:09:34 PST
The SEO summary for each article needs to be exposed here; it's a more useful textual overview of the article's contents than the "excerpt" being reported here, which seems to be a semi-randomly selected piece of text rather than the summary that Kuma selects or uses if one is specifically established with the summary class.
Comment 30 User image Joe Walker [:jwalker] (needinfo me or ping on irc) 2014-03-17 05:58:20 PDT
(In reply to Jannis Leidel [:jezdez] from comment #17)
> - topic -- one or more additive topic filters, e.g.
> https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js

I'm currently getting a server error from this URL. Is there anything up with MDN? Do I need to configure anything to be able to access it. From what I recall it worked a few weeks ago.
Comment 31 User image Jean-Yves Perrier [:teoli] 2014-03-17 06:46:31 PDT
I just clicked on the link right now and it worked. Maybe a transient problem?
Comment 32 User image David Bruant 2014-03-17 07:00:21 PDT
(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked.
Same here. (btw, I use the JSONView addon and wonder why this isn't the default JSON view in Firefox. anyway...)
Comment 33 User image Joe Walker [:jwalker] (needinfo me or ping on irc) 2014-03-17 08:55:02 PDT
(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked. Maybe a transient
> problem?

Yup. Working for me again.
It looks like it was transient.
Comment 34 User image Gervase Markham [:gerv] 2014-10-02 02:22:18 PDT
ahua: ping?

Gerv
Comment 35 User image Gervase Markham [:gerv] 2015-04-30 10:04:52 PDT
:paul/:openjck: is this still something we want to do?

Gerv
Comment 36 User image Gervase Markham [:gerv] 2015-05-21 09:48:14 PDT
No response -> WONTFIX.

Gerv
Comment 37 User image Eric Shepherd [:sheppy] 2015-05-21 13:56:26 PDT
We almost certainly do in fact still want this. Let me needinfo the right people -- if they agree, they can reopen this. I'm fairly sure that either Paul or Will can tell us if this should remain open.
Comment 38 User image Will Bamberg [:wbamberg] 2015-05-21 14:11:13 PDT
My 2c: we don't need this.

The CSS help in the devtools feature[1] gets a specific page, based on the URL structure, rather than a set of search results, and that's much more likely to be useful. 

What would be great, would be structured data in MDN that we can query using a JSON API (so we can ask for, say, the syntax + an example for Array.prototype.slice()). But that's not this bug, and it's a lot of work in both code- and content-side.

[1] https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#Get_help_for_CSS_properties

Note You need to log in before you can comment on or make changes to this bug.