Open Bug 1544234 Opened 5 years ago Updated 2 years ago

MediaStreamTrack of first video track recorded by MediaRecorder using RTCRtpSender.replaceTrack() is missing seconds and up to last second of audio of last video is muted

Categories

(Core :: Audio/Video: Recording, defect, P2)

Product:
Core
Component:
Audio/Video: Recording
Version:
68 Branch
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: guest271314, Unassigned)

Details

Attachments

(2 files)

mediaFragmentRecorderWebRTCFirefox.html
10.76 KB, text/html
Details
firefox_first_video_not_recorded_in_entirety.png
93.64 KB, image/png
Details

User Agent: Mozilla/5.0 (X11; Linux i686; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

  1. Include two <video> elements at HTML
  2. Create two RTCPeerConnection instances (used for RTCRtpSender.replaceTrack())
  3. Create one MediaStreamTrack (kind "audio") and one MediaStreamTrack (kind "video")
  4. Create one MediaStream instance
  5. Add audio and video MediaStreamTrack to RTCPeerConnection with addTrack() and second parameter set to MediaStream at 4.
  6. For each of an array of URLs set src of one of <video> elements to URL
  7. Call mozCaptureStream() on <video> element
  8. If MediaRecorder is not defined, a) execute RTCRtpreplaceTrack() with current audio and video MediaStreamTrack catured at 7., b) create a new MediaStream with MediaStreamTracks returned from RTCPeerConnection.getSenders(), start MediaRecorder, else repeat 8.a)
  9. At pause event of second <video> element, remove() element from DOM, repeat 8
  10. When all URLs in array have been played execute MediaRecorder.stop()

Actual results:

  1. The playback at each <video> element appeared to have a reduced playback rate
  2. Only the first video MediaTrackTrack is recorded, with audio being "clipped", and video playback rate appearing to be reduce, although the duration of the webm file is the duration of all played media resources (MediaStreamTracks) that were recorded using MediaRecorder (in this case 41-42 seconds)

Expected results:

  1. The playback of media at <video> elements set to Blob URL and MediaStream, respectively, should not have reduced playback rate
  2. The video and audio MediaStreamTracks of each of the media resources played should be recorded without reduced playback rate, and the resulting webm file (Blob) at event.data of dataavailable event of MediaRecorder should playback all of the MediaStreamTracks replaced within the MediaStream passed to MediaRecorder
  3. Chromium 73 outputs expected result using RTCRtpSender.replaceTrack() and MediaRecorder (https://raw.githubusercontent.com/guest271314/MediaFragmentRecorder/webrtc-replacetrack/MediaFragmentRecorder.html)
Component: Untriaged → Audio/Video: Recording
Product: Firefox → Core

When existing MediaStream is passed to MediaRecorder() the same error is thrown that is described in the previous bug report:

SecurityError: The operation is insecure.

after the media resource has sent packet following mozCaptureStream() called on HTMLVideoElement and replaceTrack(withTrack /* current live MediaStreamTrack */).

Is the expected output not currently possible at Firefox?

Created a version which does record the second through last MediaStreamTracks though does not record the first video when replaceTrack() is used. SecurityError: The operation is insecure. is caught once at the first video. MediaRecorder captures the first frame (one image) of the first video then captures the remainder of videos to a single webm file.

The issue with reduced playback was apparently due to testing to code repeatedly at same browser tab at plnkr. When refreshed session the issue decreased until tested code repeatedly again.

mozCaptureStream() does not appear to return a MediaStream with two MediaStreamTrack instances when called more than once on the same HTMLVideoElement. Created a new <video> element and attached events for each Blob URL in array.

Resolved the issue at #3 for Firefox current implementation by copying and unshifting the first Blob URL in array, playing 0.2 seconds of of the copied media resource where SecurityError: The operation is insecure. is handled when MediaRecorder.start() is called, then playing the full media fragment.

<!DOCTYPE html>
<html>

<head>
<title>Record media fragments to single webm video using RTCPeerConnection(), RTCRtpSender.replaceTrack(), MediaRecorder()</title>
</head>

<body>
<h1 id="click">click</h1>
<video id="video" src="" controls="true" autoplay="true"></video>
<video id="playlist" src="" controls="true" autoplay="true" muted="true"></video>
<script>
const captureStream = mediaElement =>
!!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

const createMediaStreamTracks = _ => {
  const connection = new RTCPeerConnection();
  const tracks = ["video", "audio"].map(kind => {
    return connection.addTransceiver(kind).receiver.track;
  });
  connection.close();
  return tracks;
}

const width = 320;
const height = 240;
const videoConstraints = {
  frameRate: 30,
  resizeMode: "crop-and-scale",
  width,
  height
};
const blobURLS = [];
const urls = Promise.all([{
  src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
  from: 0,
  to: 4
}, {
  from: 10,
  to: 20,
  src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
}, {
  from: 55,
  to: 60,
  src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
  from: 0,
  to: 6,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
}].map(async({
  from,
  to,
  src
}) => {
  try {
    const request = await fetch(src);
    const blob = await request.blob();
    const blobURL = URL.createObjectURL(blob);
    blobURLS.push(blobURL);
    const url = new URL(src);
    console.log(url.hash);
    return blobURL + (url.hash || `#t=${from},${to}`);
  } catch (e) {
    throw e;;
  }
}));

let playlist = document.getElementById("playlist");
playlist.width = width;
playlist.height = height;

const video = document.getElementById("video");
video.width = width;
video.height = height;

let recorder;
let resolveResult;
const promiseResult = new Promise(resolve => resolveResult = resolve);

document.getElementById("click")
  .addEventListener("click", async e => {
    try {
      // create audio and video MediaStreamTrack
      const mediaStreamTracks = createMediaStreamTracks();

      const [videoTrack, audioTrack] = mediaStreamTracks;

      try {
        await videoTrack.applyConstraints(videoConstraints);
      } catch (e) {

        console.error(e, e.name === "OverconstrainedError");
      }

      let mediaStream = new MediaStream([videoTrack, audioTrack]);
      console.log("initial MediaStream, audio and video MediaStreamTracks", mediaStream, mediaStream.getTracks());

      let tracks = 0;
      console.log(mediaStream.getTracks().map(({
        id
      }) => id));

      const fromLocalPeerConnection = new RTCPeerConnection();
      const toLocalPeerConnection = new RTCPeerConnection();

      const fromConnection = new Promise(resolve => fromLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("from", e);
        try {
          await toLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      const toConnection = new Promise(resolve => toLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("to", e);
        try {
          await fromLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      fromLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      toLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      const mediaStreamTrackPromise = new Promise(resolve => {
        toLocalPeerConnection.addEventListener("track", track => {
          console.log("track event", track);

          console.log(tracks);
          // Wait for both "track" events
          if (typeof tracks === "number" && ++tracks === 2) {
            const {
              streams: [stream]
            } = track;
            console.log(stream);
            // Reassign stream to initial MediaStream reference

            mediaStream.getTracks().forEach(track => track.stop);

            mediaStream = stream;
            // set video srcObject to reassigned MediaStream
            video.srcObject = mediaStream;
            tracks = void 0;
            let result;
            recorder = new MediaRecorder(mediaStream, {
              mimeType: "video/webm;codecs=vp8,opus",
              audioBitsPerSecond: 128000,
              videoBitsPerSecond: 2500000
            });
            recorder.addEventListener("start", e => {
              console.log(e);
            });
            recorder.addEventListener("stop", e => {
              console.log(e);
              resolveResult(result);
            });
            recorder.addEventListener("dataavailable", e => {
              console.log(e);
              result = e.data;
            });
            recorder.addEventListener("error", e => {
              console.log(e);
            });
            resolve();
          }
        });
      });
      // Add initial audio and video MediaStreamTrack to PeerConnection, pass initial MediaStream
      const audioSender = fromLocalPeerConnection.addTrack(audioTrack, mediaStream);
      const videoSender = fromLocalPeerConnection.addTrack(videoTrack, mediaStream);
      const offer = await fromLocalPeerConnection.createOffer();
      await toLocalPeerConnection.setRemoteDescription(offer);
      await fromLocalPeerConnection.setLocalDescription(toLocalPeerConnection.remoteDescription);
      const answer = await toLocalPeerConnection.createAnswer();
      await fromLocalPeerConnection.setRemoteDescription(answer);
      await toLocalPeerConnection.setLocalDescription(fromLocalPeerConnection.remoteDescription);
      let media = await urls;
      await mediaStreamTrackPromise;
      await fromConnection;
      await toConnection;
      // Firefox 68 Only first frame of first fragment is recorded 
      // copy, unshift and play 0.2 second media fragment of first video in media array
      // following SecurityError: The operation is insecure. at MediaRecorder.start()
      // play full media fragment at index 1 of media array (copied to index 0)
      // adds 1-2 seconds to resulting webm file
      if (!!playlist.mozCaptureStream) {
        media.unshift(media[0].replace(/#.+$/, "#t=0,0.2"));
        console.log(media);
      }
      console.log(audioSender, videoSender, mediaStream);

      for (const blobURL of media) {
        await new Promise(async resolve => {
          playlist.addEventListener("playing", async e => {
            console.log(e, e.target.readyState);
            const stream = captureStream(playlist);
            const [playlistVideoTrack] = stream.getVideoTracks();
            const [playlistAudioTrack] = stream.getAudioTracks();
            // Apply same constraints on each video MediaStreamTrack
            // deviceId of MediaStreamTrack changes
            try {
              await playlistVideoTrack.applyConstraints(videoConstraints);
            } catch (e) {
              console.error(e, e.name === "OverconstrainedError");
            }
            console.log(playlistVideoTrack.getSettings());
            // Replace audio and video MediaStreamTrack with a new media resource
            // id of the initial MediaStreamTrack does not change
            // replaceTrack() 6.4.3 https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack
            // "If sending is true, and withTrack is not null, 
            // have the sender switch seamlessly to transmitting withTrack 
            // instead of the sender's existing track." 
            await videoSender.replaceTrack(playlistVideoTrack);
            await audioSender.replaceTrack(playlistAudioTrack);

            if (recorder.state === "inactive") {
              console.log(playlist.readyState);
              try {
                recorder.start();
              } catch (e) {
                console.error(e);
              }
            }
            console.log(recorder.state, recorder.stream.getTracks());
          }, {
            once: true
          });

          playlist.addEventListener("pause", e => {
            playlist.remove();
            playlist = document.createElement("video");
            playlist.width = width;
            playlist.height = height;
            playlist.autoplay = true;
            document.body.appendChild(playlist);
            resolve();
          }, {
            once: true
          });
          playlist.src = blobURL;
        });
      }
      recorder.stop();
      blobURLS.forEach(blobURL => URL.revokeObjectURL(blobURL));
      [audioTrack, videoTrack, ...mediaStream.getTracks()]
      .forEach(track => {
        track.stop();
        track.enabled = false;
        console.log(track);
      });

      fromLocalPeerConnection.close();
      toLocalPeerConnection.close();

      let blob = await promiseResult;
      console.log(blob);

      video.remove();
      playlist.remove();
      const videoStream = document.createElement("video");
      videoStream.width = width;
      videoStream.height = height;
      videoStream.controls = true;
      document.body.appendChild(videoStream);
      videoStream.src = URL.createObjectURL(blob);
    } catch (e) {
      console.error(e);
      console.trace();
    };
  }, {
    once: true
  });

</script>
</body>

</html>

Rank: 19
Priority: -- → P2

(In reply to guest271314 from comment #4)

The issue with reduced playback was apparently due to testing to code repeatedly at same browser tab at plnkr. When refreshed session the issue decreased until tested code repeatedly again.

mozCaptureStream() does not appear to return a MediaStream with two MediaStreamTrack instances when called more than once on the same HTMLVideoElement. Created a new <video> element and attached events for each Blob URL in array.

Resolved the issue at #3 for Firefox current implementation by copying and unshifting the first Blob URL in array, playing 0.2 seconds of of the copied media resource where SecurityError: The operation is insecure. is handled when MediaRecorder.start() is called, then playing the full media fragment.

<!DOCTYPE html>
<html>

<head>
<title>Record media fragments to single webm video using RTCPeerConnection(), RTCRtpSender.replaceTrack(), MediaRecorder()</title>
</head>

<body>
<h1 id="click">click</h1>
<video id="video" src="" controls="true" autoplay="true"></video>
<video id="playlist" src="" controls="true" autoplay="true" muted="true"></video>
<script>
const captureStream = mediaElement =>
!!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

const createMediaStreamTracks = _ => {
  const connection = new RTCPeerConnection();
  const tracks = ["video", "audio"].map(kind => {
    return connection.addTransceiver(kind).receiver.track;
  });
  connection.close();
  return tracks;
}

const width = 320;
const height = 240;
const videoConstraints = {
  frameRate: 30,
  resizeMode: "crop-and-scale",
  width,
  height
};
const blobURLS = [];
const urls = Promise.all([{
  src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
  from: 0,
  to: 4
}, {
  from: 10,
  to: 20,
  src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
}, {
  from: 55,
  to: 60,
  src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
  from: 0,
  to: 6,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
}].map(async({
  from,
  to,
  src
}) => {
  try {
    const request = await fetch(src);
    const blob = await request.blob();
    const blobURL = URL.createObjectURL(blob);
    blobURLS.push(blobURL);
    const url = new URL(src);
    console.log(url.hash);
    return blobURL + (url.hash || `#t=${from},${to}`);
  } catch (e) {
    throw e;;
  }
}));

let playlist = document.getElementById("playlist");
playlist.width = width;
playlist.height = height;

const video = document.getElementById("video");
video.width = width;
video.height = height;

let recorder;
let resolveResult;
const promiseResult = new Promise(resolve => resolveResult = resolve);

document.getElementById("click")
  .addEventListener("click", async e => {
    try {
      // create audio and video MediaStreamTrack
      const mediaStreamTracks = createMediaStreamTracks();

      const [videoTrack, audioTrack] = mediaStreamTracks;

      try {
        await videoTrack.applyConstraints(videoConstraints);
      } catch (e) {

        console.error(e, e.name === "OverconstrainedError");
      }

      let mediaStream = new MediaStream([videoTrack, audioTrack]);
      console.log("initial MediaStream, audio and video MediaStreamTracks", mediaStream, mediaStream.getTracks());

      let tracks = 0;
      console.log(mediaStream.getTracks().map(({
        id
      }) => id));

      const fromLocalPeerConnection = new RTCPeerConnection();
      const toLocalPeerConnection = new RTCPeerConnection();

      const fromConnection = new Promise(resolve => fromLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("from", e);
        try {
          await toLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      const toConnection = new Promise(resolve => toLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("to", e);
        try {
          await fromLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      fromLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      toLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      const mediaStreamTrackPromise = new Promise(resolve => {
        toLocalPeerConnection.addEventListener("track", track => {
          console.log("track event", track);

          console.log(tracks);
          // Wait for both "track" events
          if (typeof tracks === "number" && ++tracks === 2) {
            const {
              streams: [stream]
            } = track;
            console.log(stream);
            // Reassign stream to initial MediaStream reference

            mediaStream.getTracks().forEach(track => track.stop);

            mediaStream = stream;
            // set video srcObject to reassigned MediaStream
            video.srcObject = mediaStream;
            tracks = void 0;
            let result;
            recorder = new MediaRecorder(mediaStream, {
              mimeType: "video/webm;codecs=vp8,opus",
              audioBitsPerSecond: 128000,
              videoBitsPerSecond: 2500000
            });
            recorder.addEventListener("start", e => {
              console.log(e);
            });
            recorder.addEventListener("stop", e => {
              console.log(e);
              resolveResult(result);
            });
            recorder.addEventListener("dataavailable", e => {
              console.log(e);
              result = e.data;
            });
            recorder.addEventListener("error", e => {
              console.log(e);
            });
            resolve();
          }
        });
      });
      // Add initial audio and video MediaStreamTrack to PeerConnection, pass initial MediaStream
      const audioSender = fromLocalPeerConnection.addTrack(audioTrack, mediaStream);
      const videoSender = fromLocalPeerConnection.addTrack(videoTrack, mediaStream);
      const offer = await fromLocalPeerConnection.createOffer();
      await toLocalPeerConnection.setRemoteDescription(offer);
      await fromLocalPeerConnection.setLocalDescription(toLocalPeerConnection.remoteDescription);
      const answer = await toLocalPeerConnection.createAnswer();
      await fromLocalPeerConnection.setRemoteDescription(answer);
      await toLocalPeerConnection.setLocalDescription(fromLocalPeerConnection.remoteDescription);
      let media = await urls;
      await mediaStreamTrackPromise;
      await fromConnection;
      await toConnection;
      // Firefox 68 Only first frame of first fragment is recorded 
      // copy, unshift and play 0.2 second media fragment of first video in media array
      // following SecurityError: The operation is insecure. at MediaRecorder.start()
      // play full media fragment at index 1 of media array (copied to index 0)
      // adds 1-2 seconds to resulting webm file
      if (!!playlist.mozCaptureStream) {
        media.unshift(media[0].replace(/#.+$/, "#t=0,0.2"));
        console.log(media);
      }
      console.log(audioSender, videoSender, mediaStream);

      for (const blobURL of media) {
        await new Promise(async resolve => {
          playlist.addEventListener("playing", async e => {
            console.log(e, e.target.readyState);
            const stream = captureStream(playlist);
            const [playlistVideoTrack] = stream.getVideoTracks();
            const [playlistAudioTrack] = stream.getAudioTracks();
            // Apply same constraints on each video MediaStreamTrack
            // deviceId of MediaStreamTrack changes
            try {
              await playlistVideoTrack.applyConstraints(videoConstraints);
            } catch (e) {
              console.error(e, e.name === "OverconstrainedError");
            }
            console.log(playlistVideoTrack.getSettings());
            // Replace audio and video MediaStreamTrack with a new media resource
            // id of the initial MediaStreamTrack does not change
            // replaceTrack() 6.4.3 https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack
            // "If sending is true, and withTrack is not null, 
            // have the sender switch seamlessly to transmitting withTrack 
            // instead of the sender's existing track." 
            await videoSender.replaceTrack(playlistVideoTrack);
            await audioSender.replaceTrack(playlistAudioTrack);

            if (recorder.state === "inactive") {
              console.log(playlist.readyState);
              try {
                recorder.start();
              } catch (e) {
                console.error(e);
              }
            }
            console.log(recorder.state, recorder.stream.getTracks());
          }, {
            once: true
          });

          playlist.addEventListener("pause", e => {
            playlist.remove();
            playlist = document.createElement("video");
            playlist.width = width;
            playlist.height = height;
            playlist.autoplay = true;
            document.body.appendChild(playlist);
            resolve();
          }, {
            once: true
          });
          playlist.src = blobURL;
        });
      }
      recorder.stop();
      blobURLS.forEach(blobURL => URL.revokeObjectURL(blobURL));
      [audioTrack, videoTrack, ...mediaStream.getTracks()]
      .forEach(track => {
        track.stop();
        track.enabled = false;
        console.log(track);
      });

      fromLocalPeerConnection.close();
      toLocalPeerConnection.close();

      let blob = await promiseResult;
      console.log(blob);

      video.remove();
      playlist.remove();
      const videoStream = document.createElement("video");
      videoStream.width = width;
      videoStream.height = height;
      videoStream.controls = true;
      document.body.appendChild(videoStream);
      videoStream.src = URL.createObjectURL(blob);
    } catch (e) {
      console.error(e);
      console.trace();
    };
  }, {
    once: true
  });

</script>
</body>

</html>

Created a plunker for this working example.
https://next.plnkr.co/edit/yGkA08y4fGQnBmj8?preview

(In reply to larspaenij from comment #5)

(In reply to guest271314 from comment #4)

The issue with reduced playback was apparently due to testing to code repeatedly at same browser tab at plnkr. When refreshed session the issue decreased until tested code repeatedly again.

mozCaptureStream() does not appear to return a MediaStream with two MediaStreamTrack instances when called more than once on the same HTMLVideoElement. Created a new <video> element and attached events for each Blob URL in array.

Resolved the issue at #3 for Firefox current implementation by copying and unshifting the first Blob URL in array, playing 0.2 seconds of of the copied media resource where SecurityError: The operation is insecure. is handled when MediaRecorder.start() is called, then playing the full media fragment.

<!DOCTYPE html>
<html>

<head>
<title>Record media fragments to single webm video using RTCPeerConnection(), RTCRtpSender.replaceTrack(), MediaRecorder()</title>
</head>

<body>
<h1 id="click">click</h1>
<video id="video" src="" controls="true" autoplay="true"></video>
<video id="playlist" src="" controls="true" autoplay="true" muted="true"></video>
<script>
const captureStream = mediaElement =>
!!mediaElement.mozCaptureStream ? mediaElement.mozCaptureStream() : mediaElement.captureStream();

const createMediaStreamTracks = _ => {
  const connection = new RTCPeerConnection();
  const tracks = ["video", "audio"].map(kind => {
    return connection.addTransceiver(kind).receiver.track;
  });
  connection.close();
  return tracks;
}

const width = 320;
const height = 240;
const videoConstraints = {
  frameRate: 30,
  resizeMode: "crop-and-scale",
  width,
  height
};
const blobURLS = [];
const urls = Promise.all([{
  src: "https://upload.wikimedia.org/wikipedia/commons/a/a4/Xacti-AC8EX-Sample_video-001.ogv",
  from: 0,
  to: 4
}, {
  from: 10,
  to: 20,
  src: "https://mirrors.creativecommons.org/movingimages/webm/ScienceCommonsJesseDylan_240p.webm#t=10,20"
}, {
  from: 55,
  to: 60,
  src: "https://nickdesaulniers.github.io/netfix/demo/frag_bunny.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://raw.githubusercontent.com/w3c/web-platform-tests/master/media-source/mp4/test.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4"
}, {
  from: 0,
  to: 5,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerJoyrides.mp4"
}, {
  from: 0,
  to: 6,
  src: "https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerMeltdowns.mp4#t=0,6"
}].map(async({
  from,
  to,
  src
}) => {
  try {
    const request = await fetch(src);
    const blob = await request.blob();
    const blobURL = URL.createObjectURL(blob);
    blobURLS.push(blobURL);
    const url = new URL(src);
    console.log(url.hash);
    return blobURL + (url.hash || `#t=${from},${to}`);
  } catch (e) {
    throw e;;
  }
}));

let playlist = document.getElementById("playlist");
playlist.width = width;
playlist.height = height;

const video = document.getElementById("video");
video.width = width;
video.height = height;

let recorder;
let resolveResult;
const promiseResult = new Promise(resolve => resolveResult = resolve);

document.getElementById("click")
  .addEventListener("click", async e => {
    try {
      // create audio and video MediaStreamTrack
      const mediaStreamTracks = createMediaStreamTracks();

      const [videoTrack, audioTrack] = mediaStreamTracks;

      try {
        await videoTrack.applyConstraints(videoConstraints);
      } catch (e) {

        console.error(e, e.name === "OverconstrainedError");
      }

      let mediaStream = new MediaStream([videoTrack, audioTrack]);
      console.log("initial MediaStream, audio and video MediaStreamTracks", mediaStream, mediaStream.getTracks());

      let tracks = 0;
      console.log(mediaStream.getTracks().map(({
        id
      }) => id));

      const fromLocalPeerConnection = new RTCPeerConnection();
      const toLocalPeerConnection = new RTCPeerConnection();

      const fromConnection = new Promise(resolve => fromLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("from", e);
        try {
          await toLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      const toConnection = new Promise(resolve => toLocalPeerConnection.addEventListener("icecandidate", async e => {
        console.log("to", e);
        try {
          await fromLocalPeerConnection.addIceCandidate(e.candidate ? e.candidate : null);
          resolve();
        } catch (e) {
          console.error(e);
        }
      }, {
        once: true
      }));

      fromLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      toLocalPeerConnection.addEventListener("negotiationneeded", e => {
        console.log(e);
      });
      const mediaStreamTrackPromise = new Promise(resolve => {
        toLocalPeerConnection.addEventListener("track", track => {
          console.log("track event", track);

          console.log(tracks);
          // Wait for both "track" events
          if (typeof tracks === "number" && ++tracks === 2) {
            const {
              streams: [stream]
            } = track;
            console.log(stream);
            // Reassign stream to initial MediaStream reference

            mediaStream.getTracks().forEach(track => track.stop);

            mediaStream = stream;
            // set video srcObject to reassigned MediaStream
            video.srcObject = mediaStream;
            tracks = void 0;
            let result;
            recorder = new MediaRecorder(mediaStream, {
              mimeType: "video/webm;codecs=vp8,opus",
              audioBitsPerSecond: 128000,
              videoBitsPerSecond: 2500000
            });
            recorder.addEventListener("start", e => {
              console.log(e);
            });
            recorder.addEventListener("stop", e => {
              console.log(e);
              resolveResult(result);
            });
            recorder.addEventListener("dataavailable", e => {
              console.log(e);
              result = e.data;
            });
            recorder.addEventListener("error", e => {
              console.log(e);
            });
            resolve();
          }
        });
      });
      // Add initial audio and video MediaStreamTrack to PeerConnection, pass initial MediaStream
      const audioSender = fromLocalPeerConnection.addTrack(audioTrack, mediaStream);
      const videoSender = fromLocalPeerConnection.addTrack(videoTrack, mediaStream);
      const offer = await fromLocalPeerConnection.createOffer();
      await toLocalPeerConnection.setRemoteDescription(offer);
      await fromLocalPeerConnection.setLocalDescription(toLocalPeerConnection.remoteDescription);
      const answer = await toLocalPeerConnection.createAnswer();
      await fromLocalPeerConnection.setRemoteDescription(answer);
      await toLocalPeerConnection.setLocalDescription(fromLocalPeerConnection.remoteDescription);
      let media = await urls;
      await mediaStreamTrackPromise;
      await fromConnection;
      await toConnection;
      // Firefox 68 Only first frame of first fragment is recorded 
      // copy, unshift and play 0.2 second media fragment of first video in media array
      // following SecurityError: The operation is insecure. at MediaRecorder.start()
      // play full media fragment at index 1 of media array (copied to index 0)
      // adds 1-2 seconds to resulting webm file
      if (!!playlist.mozCaptureStream) {
        media.unshift(media[0].replace(/#.+$/, "#t=0,0.2"));
        console.log(media);
      }
      console.log(audioSender, videoSender, mediaStream);

      for (const blobURL of media) {
        await new Promise(async resolve => {
          playlist.addEventListener("playing", async e => {
            console.log(e, e.target.readyState);
            const stream = captureStream(playlist);
            const [playlistVideoTrack] = stream.getVideoTracks();
            const [playlistAudioTrack] = stream.getAudioTracks();
            // Apply same constraints on each video MediaStreamTrack
            // deviceId of MediaStreamTrack changes
            try {
              await playlistVideoTrack.applyConstraints(videoConstraints);
            } catch (e) {
              console.error(e, e.name === "OverconstrainedError");
            }
            console.log(playlistVideoTrack.getSettings());
            // Replace audio and video MediaStreamTrack with a new media resource
            // id of the initial MediaStreamTrack does not change
            // replaceTrack() 6.4.3 https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-replacetrack
            // "If sending is true, and withTrack is not null, 
            // have the sender switch seamlessly to transmitting withTrack 
            // instead of the sender's existing track." 
            await videoSender.replaceTrack(playlistVideoTrack);
            await audioSender.replaceTrack(playlistAudioTrack);

            if (recorder.state === "inactive") {
              console.log(playlist.readyState);
              try {
                recorder.start();
              } catch (e) {
                console.error(e);
              }
            }
            console.log(recorder.state, recorder.stream.getTracks());
          }, {
            once: true
          });

          playlist.addEventListener("pause", e => {
            playlist.remove();
            playlist = document.createElement("video");
            playlist.width = width;
            playlist.height = height;
            playlist.autoplay = true;
            document.body.appendChild(playlist);
            resolve();
          }, {
            once: true
          });
          playlist.src = blobURL;
        });
      }
      recorder.stop();
      blobURLS.forEach(blobURL => URL.revokeObjectURL(blobURL));
      [audioTrack, videoTrack, ...mediaStream.getTracks()]
      .forEach(track => {
        track.stop();
        track.enabled = false;
        console.log(track);
      });

      fromLocalPeerConnection.close();
      toLocalPeerConnection.close();

      let blob = await promiseResult;
      console.log(blob);

      video.remove();
      playlist.remove();
      const videoStream = document.createElement("video");
      videoStream.width = width;
      videoStream.height = height;
      videoStream.controls = true;
      document.body.appendChild(videoStream);
      videoStream.src = URL.createObjectURL(blob);
    } catch (e) {
      console.error(e);
      console.trace();
    };
  }, {
    once: true
  });

</script>
</body>

</html>

Created a plunker for this working example.
https://next.plnkr.co/edit/yGkA08y4fGQnBmj8?preview

The code still has the issue of only 1-2 seconds or none of the first video being recorded, and audio from the last video being recorded. The total duration of the recorded video should be 41-42 seconds. The code at the linked plnkr can have output of a resulting video with duration of 37 seconds, part of the missing seconds are from the first video. Adjusting the code at line 210 to

media.unshift(media[0].replace(/#.+$/, "#t=0,0.4"));

increasing the data and duration of the first audio and video streams to .4 seconds appears to resolve the first video issue, with the artifact of the noticeable effect of a clipped media preceding actual media being recorded.

There is minimal though noticeable missing fraction of a second up to a second of the audio at the last video.

Summary: Reduced playback rate of MediaStreamTrack only first video track recorded by MediaRecorder when using RTCRtpSender.replaceTrack() → MediaStreamTrack of first video track recorded by MediaRecorder using RTCRtpSender.replaceTrack() is missing seconds and up to last second of audio of last video is muted
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: