Closed Bug 1640832 Opened 6 years ago Closed 6 years ago

Support Multiple level sourcemap

Categories

(DevTools :: Console, enhancement)

Product:
DevTools
Component:
Console
Version:
78 Branch
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: descent-tact0m, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.13 Safari/537.36

Steps to reproduce:

  1. Open devtools
  2. Run:

eval(globalThis. console.log(1) //# sourceMappingURL=data:application/json;charset=utf-8;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoiY29udmVydGVkLmpzLm1hcCIsInNvdXJjZXMiOlsic291cmNlLmpzIl0sInNvdXJjZXNDb250ZW50IjpbInByaW50KDEpIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiJBQUFBLFdBQUsifQ== //# sourceMappingURL=data:application/json;charset=utf-8;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoiYnVuZGxlLmpzIiwic291cmNlcyI6WyJmb28uanMiLCJiYXIuanMiXSwic291cmNlc0NvbnRlbnQiOlsiZ2xvYmFsVGhpcy4iLCJjb25zb2xlLmxvZygxKVxuLy8jIHNvdXJjZU1hcHBpbmdVUkw9ZGF0YTphcHBsaWNhdGlvbi9qc29uO2NoYXJzZXQ9dXRmLTg7YmFzZTY0LGV5SjJaWEp6YVc5dUlqb3pMQ0ptYVd4bElqb2lZMjl1ZG1WeWRHVmtMbXB6TG0xaGNDSXNJbk52ZFhKalpYTWlPbHNpYzI5MWNtTmxMbXB6SWwwc0luTnZkWEpqWlhORGIyNTBaVzUwSWpwYkluQnlhVzUwS0RFcElsMHNJbTVoYldWeklqcGJYU3dpYldGd2NHbHVaM01pT2lKQlFVRkJMRmRCUVVzaWZRPT0iXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6IkFBQUEsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQztBQ0FWLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQztBQUNkLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMsQ0FBQyxDQUFDLENBQUMifQ==)
3. Inspect the source

Actual results:

Firefox warns, invalid source map. (Partly works in Chrome).

Related link: https://bugs.chromium.org/p/chromium/issues/detail?id=1086380

Expected results:

Should able to view the original & middle source

Original file: "print(1)"
Middle source: "console.log(1)"
Final source: "globalThis.\nconsole.log(1)"

Hi jackworks!
Please could you add more exact steps to test this and which is more exactly the expected result?

Flags: needinfo?(jackworks)

I think my example is very clear. When a js file with sourcemap get transformed again, the sourcemap will be multiple level. The Firefox can't handle this well. I have given my multiple levels sourcemap in the question.

Transformer1(source) => intermediate+sourcemap1
Transformer2(intermediate+sourcemap1)=>final(final code+souurcemap1)+sourcemap2

Flags: needinfo?(jackworks)

I was try on latest Nightly version 79.0a1 (2020-06-18)(64-bit) on Windows 10 following these steps:
open firefox
open Devtools console
run command: .....
actual result: nothing happens
I'll add this to the Devtools - Console component in the hope someone with more expertise can take a look at this and advice.

Component: Untriaged → Console
Product: Firefox → DevTools

Logan, is it something we're aware of?

Flags: needinfo?(loganfsmyth)

This is not something that is expected to work. If you want to transform a file multiple times, that transform process should parse the original sourcemap, create a new sourcemap, and replace whatever sourceMappingURL comment was originally present. Appending two separate sourcemap comments to a file is not some something we can support.

Status: UNCONFIRMED → RESOLVED
Closed: 6 years ago
Flags: needinfo?(loganfsmyth)
Resolution: --- → WONTFIX

No, it's not resolved. There is use case in the real world, and I encountered. This use case also mentioned in the sourcemap specification so I expect it's a supported feature.

Could you elaborate more on what this feature would improve? Perhaps that would help clarify your goal. What is the primary benefit of showing the intermediate output of the transformations in devtools? If you hit a breakpoint, which version would you expect to open? When would looking at the intermediate file be more helpful than the original file or the final file?

Flags: needinfo?(jackworks)

In my use case, I have a webpack bundle that has a source map. I have to wrap it in runtime before execute it. But that will break the source map cause the source map doesn't get updated. The only choice I have is a multiple level source map but there is no browser supporting it. So I have to give up the whole source map.

Please notice, Firefox even doesn't handle my example as a single-level source map. You can compare the result with Chrome, Chrome treated it as a single-level source map.

When would looking at the intermediate file be more helpful than the original file or the final file?

When the intermediate toolchain has a bug or I want to debug the internal code of the intermediate execution process but not the final result.

If you hit a breakpoint, which version would you expect to open?

The default is the original file, but the developer can select on the intermediate level if something isn't right with the toolchain.

What is the primary benefit of showing the intermediate output of the transformations in dev tools?

When the middle tool has a bug and I have to debug it. For example, I want to debug the code after the Vue single file webpack loader but before the compressor.

Finally, supporting multiple level source map enables non-linear toolchain. In my example, it's a little toolchain modify the code after the webpack process.

Flags: needinfo?(jackworks)

In my use case, I have a webpack bundle that has a source map. I have to wrap it in runtime before execute it. But that will break the source map cause the source map doesn't get updated. The only choice I have is a multiple level source map but there is no browser supporting it. So I have to give up the whole source map.

Could you elaborate more on what you mean by wrap? I would usually expect that a step like this would parse the sourcemap, and then adjust the mappings to fit the new file post-wrapping, so if you add 2 lines before the webpack bundle, you'd have to parse the map, move all the mappings 2 lines, and then re-serialize the map. Is that something you've attempted and been unable to do?

When the middle tool has a bug and I have to debug it. For example, I want to debug the code after the Vue single file webpack loader but before the compressor.

I think part of the issue here is that you may be expecting more from what sourcemaps are than is realistic. Sourcemaps are just point to point location mappings from one file to another. If you have 2 maps, we could maybe show both sets of mappings, though I think that would be super complex. If you expect one map to build on the mappings of the previous one, I honestly don't know that it's possible to do that in the way you think it is.

Please notice, Firefox even doesn't handle my example as a single-level source map. You can compare the result with Chrome, Chrome treated it as a single-level source map.

As soon as there are two comments, I honestly don't know which one Firefox will try to use, so if there is a difference, most likely Chrome and Firefox are just choosing a different comment from the other to use.

so if you add 2 lines before the webpack bundle, you'd have to parse the map, move all the mappings 2 lines, and then re-serialize the map. Is that something you've attempted and been unable to do?

Yes, it is possible but I think it is too heavy to parse an MB-level file and re-serialize it in runtime. The better choice is to add a more simple source map to reflect the change of "adding 2 lines" before the whole file.

If you expect one map to building on the mappings of the previous one, I honestly don't know that it's possible to do that in the way you think it is.

TypeScript eats .ts files, emit .js files with source map; Rollup eats .js files and emit a single .js file with source map; Uglify eats a .js file and emit another compressed js file with source map.

Example:
"out.js" maps to the un-compressed version of "out.js" (source map provided from uglifying). Then the uncompressed version of "out.js" maps to 12 JS files (map provided by rollup). Finally, the JS file can be mapped into the original TS file.

I know it is not how to source map works today. Today the source map goes through a "pipe" and drops all middle transform information. But it is useful to keep internal info for debugging.

As soon as there are two comments, I honestly don't know which one Firefox will try to use, so if there is a difference, most likely Chrome and Firefox are just choosing a different comment from the other to use.

Yes, Firefox seems to use the prior one and the Chrome is using the latter one which is correct.
My example is a file with source map, then wrap it again and appen new source map after it.

Yes, it is possible but I think it is too heavy to parse an MB-level file and re-serialize it in runtime. The better choice is to add a more simple source map to reflect the change of "adding 2 lines" before the whole file.

How do you propose that would be represented in the sourcemap format?

Have you considered using Webpack's eval-source-map format? Then you wouldn't need to do this because Webpack itself will produce an output where each file has an individual map. Have you considered using eval for your own wrapping in order to keep the existing map output by Webpack functional?

I'm afraid it is not possible to use eval-source-map. Our project are limited by the CSP

Got it. Could you explore using the "sections" source-map format to insert and offset before the original map? https://sourcemaps.info/spec.html

e.g.

var orignalMap = { version: 3 /* ... whatever your current map has */ };
var newMap = {
  version: 3,
  sections: [
    { offset: { line: 2, column: 9 }, map: originalMap },
  ]
};

That would allow you to avoid some of the process of reprocessing the original map, if all you need is to wrap a map with some additional changes.

Hmm yes it's possible, but not a good solution though. Happy to see multiple level sourcemap get developed.

not a good solution though

Could you elaborate on why you feel that way?

Happy to see multiple level sourcemap get developed.

I still don't feel that it's clear how you see this all working. The sourcemap format doesn't support what you're asking for if I understand what you want, and it's not clear to me how it could be changed to address your goal. I'm open to continuing the discussion, but I think any solution would require broad agreement on functionality for implementers, and it is unlikely that the existence of this bug on its own would move the discussion forward. If it's a discussion you'll be driving it along, we could potentially leave this open, but there's not much reason to leave this open if it doesn't have any clear direction forward.

You need to log in before you can comment on or make changes to this bug.