768456 - API call for getting the first n docs to appear for a search term

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: API, defect, P2)

Description

[Paul Rouget [:paul]]

Attachment: screenshot of the current approach

We want to introduce a "mdn" command in the Firefox Command Line. This command will let the user request for the documentation for any CSS, HTML and JavaScript elements.

> mdn transform-origin

It is a little trick to do that today because we don't know:
- what is the language of the user
- what is the nature of its request (CSS, HTML or JavaScript)

In an ideal world, we would just make a request to:
http://developer.mozilla.org/imfeelingluck?q=foobar
…and that would return a page with the correct locale, and with the best doc match.

Current approach (CSS only): https://gist.github.com/1d943933ffb43a72946d

Comment 1

[John Karahalis [:openjck]]

This is a wonderful idea. Thank you for filing this, Paul.

The team is currently heads-down preparing for a new release of the MDN this summer, but I will be sure to bring this to the attention of the development team soon after that.

Comment 2

[Eric Shepherd [:sheppy]]

Yes, I asked Paul to file this because it would be awesome for dev tools and IDEs.

Comment 3

[Paul Rouget [:paul]]

It would be great to also be able (but not mandatory) to specify "type" and "lang":

http://developer.mozilla.org/imfeelingluck?q=foobar&type=css&lang=fr_FR

Comment 4

[Eric Shepherd [:sheppy]]

Yes, this would be very good to have, and is a good explanation of a feature I've been trying to figure out how to describe for months. Thanks, Paul!

Comment 5

[Les Orchard [:lorchard]]

Just for the heck of it, I tried hacking on a variant of this CSS command using our new Kuma resources:

https://gist.github.com/3307756

It's a bit simpler, though absolutely terrible because I don't know much about xpcom or the new CLI stuff :)

Comment 6

[Paul Rouget [:paul]]

Do we have anything plan yet? How can I help?

Comment 7

[Luke Crouch [:groovecoder]]

Les's command puts the "lucky" in "I'm feeling lucky." Can you just use that? Do most of the css properties live right under docs/CSS like that?

Comment 8

[Paul Rouget [:paul]]

(In reply to Luke Crouch [:groovecoder] from comment #7)
> Les's command puts the "lucky" in "I'm feeling lucky." Can you just use
> that? Do most of the css properties live right under docs/CSS like that?

We don't want to target only CSS> We want to be able to request the documentation for any language (CSS, JavaScript and HTML) without knowing at first what the language is.

Comment 9

[Les Orchard [:lorchard]]

Sadly, we don't have a decent self-hosted search feature yet, so I think that's a dependency for implementing something like this. Currently, we just rely on Google site search

Comment 10

[Les Orchard [:lorchard]]

This is blocked by actually having a search feature.

Comment 11

[John Karahalis [:openjck]]

:basta is interested in this as well, and with Elastic Search coming very soon it should be doable.

Comment 13

[John Karahalis [:openjck]]

Hey all. We are now using Bugzilla votes to gauge priorities. Please vote this up if it is one of the 25 MDN bugs you care most about.

Comment 14

[Florian Scholz (Open Web Docs)]

This is an awesome idea, Paul!

How much does our new Elastic Search help you?

> mdn transform-origin
https://developer.mozilla.org/en-US/search.json?q=transform-origin

You will get better results if you provide the topic in some cases:

>mdn js string.contains
https://developer.mozilla.org/en-US/search.json?q=string.contains?topic=js

More information about search parameters (locale= is available, too):
https://developer.mozilla.org/en-US/docs/Project:MDN/Contributing/Advanced_search

What can we improve to make this happen?

Comment 15

[Paul Rouget [:paul]]

(In reply to Florian Scholz [:fscholz] (elchi3) from comment #14)

This sounds perfect. We should totally try integrate that.

Should something like that:

> doc string.contains

… returns a list of results, or just the first result summary?

The syntax could be:

> doc <keyworkd> [<hint>]

where hint is the topic or tags (which one should we use?)

Comment 16

[Luke Crouch [:groovecoder]]

needinfo? and +1 for :jezdez who implemented the search json API.

:jezdez, can you help :paul integrate MDN search results into firefox command line?

Comment 17

[Jannis Leidel [:jezdez]]

Absolutely, I'm stoked we can offer that API now. There are a few things missing that may be required to make the integration solid, e.g. API throttling and/or caching. That said, ZEUS our HTTP frontend proxy should cache responses for a minute or so.

:paul right now you can basically take *any* search URL and append a ".json" to the URL path (or send a proper "Accept: application/json" request header) to get the search results formatted as json.

There are a few parameters you can set:

- page -- the search result page (1-indexed)
- per_page -- the number of results per page, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&per_page=5 (default is 10, max is 100)
- topic -- one or more additive topic filters, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js (the full list of filters is also transfered as part of the response data in the "filters" item)
- q -- the full text search term

Here is an overview of the returned search result data:

    {
        "count": number of results,
        "next": url of next result page or null,
        "previous": url of previous result page or null,
        "query": value of fulltext search parameter,
        "page": current search result page number,
        "pages": total number of search result pages,
        "start": start index of search result on current page,
        "end": end index of search result on current page,
        "filters": list of topic filters available on current page (see below),
        "documents": list of search results (see below)
    }

The list of "filters" is actually a list of filter groups that each have a list of assigned filters. that's mostly a result of the frontend UI but should also allow you to hide uninteresting filters if you want to offer them to users through the "doc" command. Anyways, here's the schema for each of those filters:

    {
        "name": filter name,
        "slug": filter slug as used in the topic query parameter,
        "count": number of documents matching that topic on the current search result page,
        "active": boolean describing whether this filter is active on the current search result page,
        "urls": {
            "active": absolute URL path for activating this filter,
            "inactive": absolute URL path for deactivating this filter
        }
    }

Here is the schema for each document as listed in the "documents" item above:

    {
        "title": title of wiki document,
        "slug": slug of wiki document as used in URL path,
        "locale": locale code of wiki document as used in URL path,
        "excerpt": short except of wiki document,
        "url": absolute URL of wiki document,
        "edit_url": absolute URL of wiki document for the edit interface,
        "tags": list of tags,
    }

Comment 18

[Jannis Leidel [:jezdez]]

:paul Anything you need further to implement this on your side (assuming it hasn't happened yet)?

Comment 19

[Paul Rouget [:paul]]

I'm not working on that anymore, but I think :jwalker or :pbrosset can help.

Joe, Patrick, this API sounds ideal to integrate documentation in the devtools. Maybe via the command line, or even via a popup. What do you guys think?

Jannis, I think we can safely close the bug. The API is great. Let's just make sure the appropriate bugs are filed to use this API at the devtools level (maybe bug 768469 is enough).

Comment 20

[Patrick Brosset <:pbro>]

Cool API!
That should come in pretty handy for a number of features in the devtools.
jwalker just filed bug 965197 which I think could be pretty useful.

Comment 21

[Joe Walker [:jwalker]]

(In reply to Paul Rouget [:paul] from comment #19)
> Joe, Patrick, this API sounds ideal to integrate documentation in the
> devtools. Maybe via the command line, or even via a popup. What do you guys
> think?

I think that bug 768469 (gcli command) is a good place to start, but there are a bunch of things like Right-click on property name in inspector (just raised bug 965197) are other obvious integration points.

:jezdez I think we should probably pass you some traceability info so you know where the traffic is coming from, do you have an suggestions for how to do that?

Comment 22

[Jannis Leidel [:jezdez]]

:jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you suffix the user-agent somehow? Alternatively and maybe a better idea is to simply send along an additional query parameter (e.g. /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start tracking this on the server side if you're okay with that.

:groovecoder Do you think we can start using GA to track search usage via clients?

Comment 23

[Joe Walker [:jwalker]]

(In reply to Jannis Leidel [:jezdez] from comment #22)
> :jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you
> suffix the user-agent somehow? Alternatively and maybe a better idea is to
> simply send along an additional query parameter (e.g.
> /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start
> tracking this on the server side if you're okay with that.

An additional query parameter would be easiest.

Gerv: There are a few places in Firefox developer tools where we're thinking of providing links to MDN, and considering adding a query parameter so we know where the links came from.

I think that telling MDN that the click comes from inside developer tools (and even where inside developer tools) is OK because this is similar to a referrer.

I think it's OK to tell MDN what firefox version we're using because that's discoverable from the user-agent anyway.

Comment 24

[Gervase Markham [:gerv]]

That seems fine to me, but maybe I'm not the right person to know our policies on such things. NEEDINFO alina.

Gerv

Comment 25

[Luke Crouch [:groovecoder]]

If we can, it would be great to add tracking params to the MDN links from devtools like so:

https://developer.mozilla.org/<locale>/docs/<article-slug>?utm_campaign=SearchAPI&utm_source=Firefox&utm_medium=DevTools[utm_content=<specific-tool>]

Using GA to track the the search API traffic sounds non-trivial. We would need to add GA Measurement Protocol[1] code to the ES python code.

[1] https://developers.google.com/analytics/devguides/collection/protocol/v1/

Comment 26

[Jannis Leidel [:jezdez]]

:jwalker I think we misunderstood each other, I meant the URLs of the calls against the search API should contain a parameter, not only the links to MDN documents. That would allow us to distinguish the traffic generated against the search from webtools from the common traffic.

:groovecoder Calling the GA API is actually rather trivial (e.g. https://github.com/mirumee/google-measurement-protocol) can you elaborate what makes you think it's not?

I definitely think that only tracking the traffic to the documents and not requests against the search itself would leave us with only partial data on what users are searching from the webtools. Clicking on a link in the webtools is AFAIU the second action a user would take, after being offered that link via running a CLI command or hovering over a HTML element of the page.

Comment 27

[Luke Crouch [:groovecoder]]

:jezdez - primarily, we need to upgrade to GA Universal Analytics to enable the Measurement Protocol. I'll work with :garethc and :cmore on that but it might take a while.

Comment 28

[Luke Crouch [:groovecoder]]

Update: We are planning to upgrade to UA when it comes out of beta sometime in the next couple months.

Comment 29

[Eric Shepherd [:sheppy]]

The SEO summary for each article needs to be exposed here; it's a more useful textual overview of the article's contents than the "excerpt" being reported here, which seems to be a semi-randomly selected piece of text rather than the summary that Kuma selects or uses if one is specifically established with the summary class.

Comment 30

[Joe Walker [:jwalker]]

(In reply to Jannis Leidel [:jezdez] from comment #17)
> - topic -- one or more additive topic filters, e.g.
> https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js

I'm currently getting a server error from this URL. Is there anything up with MDN? Do I need to configure anything to be able to access it. From what I recall it worked a few weeks ago.

Comment 31

[Jean-Yves Perrier [:teoli]]

I just clicked on the link right now and it worked. Maybe a transient problem?

Comment 32

[David Bruant]

(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked.
Same here. (btw, I use the JSONView addon and wonder why this isn't the default JSON view in Firefox. anyway...)

Comment 33

[Joe Walker [:jwalker]]

(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked. Maybe a transient
> problem?

Yup. Working for me again.
It looks like it was transient.

Comment 34

[Gervase Markham [:gerv]]

ahua: ping?

Gerv

Comment 35

[Gervase Markham [:gerv]]

:paul/:openjck: is this still something we want to do?

Gerv

Comment 36

[Gervase Markham [:gerv]]

No response -> WONTFIX.

Gerv

Comment 37

[Eric Shepherd [:sheppy]]

We almost certainly do in fact still want this. Let me needinfo the right people -- if they agree, they can reopen this. I'm fairly sure that either Paul or Will can tell us if this should remain open.

Comment 38

[Will Bamberg [:wbamberg]]

My 2c: we don't need this.

The CSS help in the devtools feature[1] gets a specific page, based on the URL structure, rather than a set of search results, and that's much more likely to be useful. 

What would be great, would be structured data in MDN that we can query using a JSON API (so we can ask for, say, the syntax + an example for Array.prototype.slice()). But that's not this bug, and it's a lot of work in both code- and content-side.

[1] https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#Get_help_for_CSS_properties