API call for getting the first n docs to appear for a search term

RESOLVED WONTFIX

Status

Mozilla Developer Network
API
P2
normal
RESOLVED WONTFIX
5 years ago
a year ago

People

(Reporter: paul, Unassigned)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
Created attachment 636702 [details]
screenshot of the current approach

We want to introduce a "mdn" command in the Firefox Command Line. This command will let the user request for the documentation for any CSS, HTML and JavaScript elements.

> mdn transform-origin

It is a little trick to do that today because we don't know:
- what is the language of the user
- what is the nature of its request (CSS, HTML or JavaScript)

In an ideal world, we would just make a request to:
http://developer.mozilla.org/imfeelingluck?q=foobar
…and that would return a page with the correct locale, and with the best doc match.

Current approach (CSS only): https://gist.github.com/1d943933ffb43a72946d
This is a wonderful idea. Thank you for filing this, Paul.

The team is currently heads-down preparing for a new release of the MDN this summer, but I will be sure to bring this to the attention of the development team soon after that.
Yes, I asked Paul to file this because it would be awesome for dev tools and IDEs.
Version: unspecified → Kuma
(Reporter)

Updated

5 years ago
Blocks: 768469
(Reporter)

Comment 3

5 years ago
It would be great to also be able (but not mandatory) to specify "type" and "lang":

http://developer.mozilla.org/imfeelingluck?q=foobar&type=css&lang=fr_FR
Yes, this would be very good to have, and is a good explanation of a feature I've been trying to figure out how to describe for months. Thanks, Paul!
Just for the heck of it, I tried hacking on a variant of this CSS command using our new Kuma resources:

https://gist.github.com/3307756

It's a bit simpler, though absolutely terrible because I don't know much about xpcom or the new CLI stuff :)
(Reporter)

Comment 6

5 years ago
Do we have anything plan yet? How can I help?
Les's command puts the "lucky" in "I'm feeling lucky." Can you just use that? Do most of the css properties live right under docs/CSS like that?
(Reporter)

Comment 8

5 years ago
(In reply to Luke Crouch [:groovecoder] from comment #7)
> Les's command puts the "lucky" in "I'm feeling lucky." Can you just use
> that? Do most of the css properties live right under docs/CSS like that?

We don't want to target only CSS> We want to be able to request the documentation for any language (CSS, JavaScript and HTML) without knowing at first what the language is.
Sadly, we don't have a decent self-hosted search feature yet, so I think that's a dependency for implementing something like this. Currently, we just rely on Google site search
(Assignee)

Updated

5 years ago
Version: Kuma → unspecified
(Assignee)

Updated

5 years ago
Component: Website → Landing pages
Product: Mozilla Developer Network → Mozilla Developer Network
Component: Landing pages → API
Priority: -- → P1
Summary: Implement a "I'm feeling lucky" mechanism → API call for getting the first doc to appear for a search term
Whiteboard: c=API
Priority: P1 → P2
Priority: P2 → P1
This is blocked by actually having a search feature.
Depends on: 794525
:basta is interested in this as well, and with Elastic Search coming very soon it should be doable.
OS: Mac OS X → All
Priority: P1 → P2
Hardware: x86 → All
Summary: API call for getting the first doc to appear for a search term → API call for getting the first n docs to appear for a search term
Whiteboard: c=API
Depends on: 839214
Duplicate of this bug: 671893
Hey all. We are now using Bugzilla votes to gauge priorities. Please vote this up if it is one of the 25 MDN bugs you care most about.
This is an awesome idea, Paul!

How much does our new Elastic Search help you?

> mdn transform-origin
https://developer.mozilla.org/en-US/search.json?q=transform-origin

You will get better results if you provide the topic in some cases:

>mdn js string.contains
https://developer.mozilla.org/en-US/search.json?q=string.contains?topic=js

More information about search parameters (locale= is available, too):
https://developer.mozilla.org/en-US/docs/Project:MDN/Contributing/Advanced_search

What can we improve to make this happen?
Flags: needinfo?(paul)
(Reporter)

Comment 15

4 years ago
(In reply to Florian Scholz [:fscholz] (elchi3) from comment #14)

This sounds perfect. We should totally try integrate that.

Should something like that:

> doc string.contains

… returns a list of results, or just the first result summary?

The syntax could be:

> doc <keyworkd> [<hint>]

where hint is the topic or tags (which one should we use?)
Flags: needinfo?(paul)
needinfo? and +1 for :jezdez who implemented the search json API.

:jezdez, can you help :paul integrate MDN search results into firefox command line?
Flags: needinfo?(jezdez)
Absolutely, I'm stoked we can offer that API now. There are a few things missing that may be required to make the integration solid, e.g. API throttling and/or caching. That said, ZEUS our HTTP frontend proxy should cache responses for a minute or so.

:paul right now you can basically take *any* search URL and append a ".json" to the URL path (or send a proper "Accept: application/json" request header) to get the search results formatted as json.

There are a few parameters you can set:

- page -- the search result page (1-indexed)
- per_page -- the number of results per page, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&per_page=5 (default is 10, max is 100)
- topic -- one or more additive topic filters, e.g. https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js (the full list of filters is also transfered as part of the response data in the "filters" item)
- q -- the full text search term

Here is an overview of the returned search result data:

    {
        "count": number of results,
        "next": url of next result page or null,
        "previous": url of previous result page or null,
        "query": value of fulltext search parameter,
        "page": current search result page number,
        "pages": total number of search result pages,
        "start": start index of search result on current page,
        "end": end index of search result on current page,
        "filters": list of topic filters available on current page (see below),
        "documents": list of search results (see below)
    }

The list of "filters" is actually a list of filter groups that each have a list of assigned filters. that's mostly a result of the frontend UI but should also allow you to hide uninteresting filters if you want to offer them to users through the "doc" command. Anyways, here's the schema for each of those filters:

    {
        "name": filter name,
        "slug": filter slug as used in the topic query parameter,
        "count": number of documents matching that topic on the current search result page,
        "active": boolean describing whether this filter is active on the current search result page,
        "urls": {
            "active": absolute URL path for activating this filter,
            "inactive": absolute URL path for deactivating this filter
        }
    }

Here is the schema for each document as listed in the "documents" item above:

    {
        "title": title of wiki document,
        "slug": slug of wiki document as used in URL path,
        "locale": locale code of wiki document as used in URL path,
        "excerpt": short except of wiki document,
        "url": absolute URL of wiki document,
        "edit_url": absolute URL of wiki document for the edit interface,
        "tags": list of tags,
    }
Flags: needinfo?(jezdez)
:paul Anything you need further to implement this on your side (assuming it hasn't happened yet)?
Flags: needinfo?(paul)
(Reporter)

Comment 19

3 years ago
I'm not working on that anymore, but I think :jwalker or :pbrosset can help.

Joe, Patrick, this API sounds ideal to integrate documentation in the devtools. Maybe via the command line, or even via a popup. What do you guys think?

Jannis, I think we can safely close the bug. The API is great. Let's just make sure the appropriate bugs are filed to use this API at the devtools level (maybe bug 768469 is enough).
Flags: needinfo?(pbrosset)
Flags: needinfo?(paul)
Flags: needinfo?(jwalker)
Blocks: 965197
Cool API!
That should come in pretty handy for a number of features in the devtools.
jwalker just filed bug 965197 which I think could be pretty useful.
Flags: needinfo?(pbrosset)
(In reply to Paul Rouget [:paul] from comment #19)
> Joe, Patrick, this API sounds ideal to integrate documentation in the
> devtools. Maybe via the command line, or even via a popup. What do you guys
> think?

I think that bug 768469 (gcli command) is a good place to start, but there are a bunch of things like Right-click on property name in inspector (just raised bug 965197) are other obvious integration points.

:jezdez I think we should probably pass you some traceability info so you know where the traffic is coming from, do you have an suggestions for how to do that?
Flags: needinfo?(jwalker)
:jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you suffix the user-agent somehow? Alternatively and maybe a better idea is to simply send along an additional query parameter (e.g. /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start tracking this on the server side if you're okay with that.

:groovecoder Do you think we can start using GA to track search usage via clients?
(In reply to Jannis Leidel [:jezdez] from comment #22)
> :jwalker Yeah, that'd be great, I'm not sure if that's possible, but can you
> suffix the user-agent somehow? Alternatively and maybe a better idea is to
> simply send along an additional query parameter (e.g.
> /en-US/search.json?q=whatever&client=gcli-28.0) or similar? We could start
> tracking this on the server side if you're okay with that.

An additional query parameter would be easiest.

Gerv: There are a few places in Firefox developer tools where we're thinking of providing links to MDN, and considering adding a query parameter so we know where the links came from.

I think that telling MDN that the click comes from inside developer tools (and even where inside developer tools) is OK because this is similar to a referrer.

I think it's OK to tell MDN what firefox version we're using because that's discoverable from the user-agent anyway.
Flags: needinfo?(gerv)
That seems fine to me, but maybe I'm not the right person to know our policies on such things. NEEDINFO alina.

Gerv
Flags: needinfo?(gerv) → needinfo?(ahua)
If we can, it would be great to add tracking params to the MDN links from devtools like so:

https://developer.mozilla.org/<locale>/docs/<article-slug>?utm_campaign=SearchAPI&utm_source=Firefox&utm_medium=DevTools[utm_content=<specific-tool>]

Using GA to track the the search API traffic sounds non-trivial. We would need to add GA Measurement Protocol[1] code to the ES python code.

[1] https://developers.google.com/analytics/devguides/collection/protocol/v1/
:jwalker I think we misunderstood each other, I meant the URLs of the calls against the search API should contain a parameter, not only the links to MDN documents. That would allow us to distinguish the traffic generated against the search from webtools from the common traffic.

:groovecoder Calling the GA API is actually rather trivial (e.g. https://github.com/mirumee/google-measurement-protocol) can you elaborate what makes you think it's not?

I definitely think that only tracking the traffic to the documents and not requests against the search itself would leave us with only partial data on what users are searching from the webtools. Clicking on a link in the webtools is AFAIU the second action a user would take, after being offered that link via running a CLI command or hovering over a HTML element of the page.
:jezdez - primarily, we need to upgrade to GA Universal Analytics to enable the Measurement Protocol. I'll work with :garethc and :cmore on that but it might take a while.
Update: We are planning to upgrade to UA when it comes out of beta sometime in the next couple months.
The SEO summary for each article needs to be exposed here; it's a more useful textual overview of the article's contents than the "excerpt" being reported here, which seems to be a semi-randomly selected piece of text rather than the summary that Kuma selects or uses if one is specifically established with the summary class.
(In reply to Jannis Leidel [:jezdez] from comment #17)
> - topic -- one or more additive topic filters, e.g.
> https://developer.mozilla.org/en-US/search.json?q=string.contains&topic=js

I'm currently getting a server error from this URL. Is there anything up with MDN? Do I need to configure anything to be able to access it. From what I recall it worked a few weeks ago.
I just clicked on the link right now and it worked. Maybe a transient problem?

Comment 32

3 years ago
(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked.
Same here. (btw, I use the JSONView addon and wonder why this isn't the default JSON view in Firefox. anyway...)
(In reply to Jean-Yves Perrier [:teoli] from comment #31)
> I just clicked on the link right now and it worked. Maybe a transient
> problem?

Yup. Working for me again.
It looks like it was transient.
Blocks: 989063
ahua: ping?

Gerv
:paul/:openjck: is this still something we want to do?

Gerv
No response -> WONTFIX.

Gerv
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
We almost certainly do in fact still want this. Let me needinfo the right people -- if they agree, they can reopen this. I'm fairly sure that either Paul or Will can tell us if this should remain open.
Flags: needinfo?(wbamberg)
Flags: needinfo?(paul)
My 2c: we don't need this.

The CSS help in the devtools feature[1] gets a specific page, based on the URL structure, rather than a set of search results, and that's much more likely to be useful. 

What would be great, would be structured data in MDN that we can query using a JSON API (so we can ask for, say, the syntax + an example for Array.prototype.slice()). But that's not this bug, and it's a lot of work in both code- and content-side.

[1] https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#Get_help_for_CSS_properties
Flags: needinfo?(wbamberg)
(Reporter)

Updated

2 years ago
Flags: needinfo?(paul)
Flags: needinfo?(ahua)
You need to log in before you can comment on or make changes to this bug.