Reference to Video element using Prebuilt

Trying to capture still image from the video feed, but unable to programmatically access the video element. Triggered from a custom button callback event. Code below returns null.

var video = document.getElementsByTagName('video')[0] // document.querySelector("video");

«1

Answers

  • mark_at_daily
    mark_at_daily Community Manager, Dailynista admin

    Daily Prebuilt is often embedded as an iframe into an application. When this is the case, it's not possible to capture the video across the iframe boundary.

    Can you share more detail on what you're trying to accomplish in case we can find a workaround for you? Which user or users streams are you looking to capture? Are you looking to capture a frame from the camera stream?

  • I have custom button within iFramed Prebuilt that I want to grab camera video frame and copy to canvas object for me to display and save. We typically have 1:1 video chats and would like to grab the feed of the other party. If it needs to be a composite image with other feeds inset, that would be ok.

  • Here is code excerpt—most of code is taken from the Prebuilt example code.

        callFrame.on('left-meeting', handleLeftMeeting);
    callFrame.on("custom-button-click", (ev) => {
    // Save camera image captured from video feed
    if (ev.button_id === "capture") {
    var video = document.getElementsByTagName('video')[0];
    //captureImage();
    }

  • I hoped to capture the video within the iFrame and then make ajax call to upload encoded image data.

  • christian
    christian Dailynista

    Hey @lmeyers,

    you can't access the video element inside of the iframe, due to browser security limitations.

    Also important to note: the event callback for the custom-button-click runs in the JavaScript context of your application, not the iframe.

    However, you can render your own video outside of the iframe and capture images from that. Since you already have the callFrame, you're pretty much setup to do this:

    // sessionId is the id of the participant, you want to render the video for.
    // If it's the local participant, it should be "local"
    const videoTrack = callFrame.participants()[sessionId].tracks.video.persistentTrack;
    const video = document.createElement('video');
    // autoplay, playsinline & muted make sure the video can autoplay without issues
    video.autoplay = true;
    video.playsinline = true;
    video.muted = true;
    video.srcObject = new MediaStream([videoTrack]);
    // Apply styles or CSS class names
    // video.style = '';
    document.body.appendChild(video);
    

    You can then access the video and capture the frames to your canvas.

    In case you're building with React, you can also leverage the DailyVideo component that does all the setup for you. It would work something like this:

    <DailyProvider callObject={callFrame}>
      <VideoCapture />
    </DailyProvider>
    

    and then the component code:

    export const VideoCapture = () ⇒ {
      const remoteIds = useParticipantIds({ filter: 'remote' });
      const id = remoteIds?.[0];
      const videoRef = useRef(null);
        
      const handleCapture = () ⇒ {
        const video = videoRef.current;
        // your capture image code
      }
    
      return (
        <div>
          <DailyVideo
            key={`capture-${id}`}
            ref={videoRef}
            sessionId={id}
            style={{}}
            type="video"
          />
          <button onClick={handleCapture}>Capture</button>
        </div>
      );
    }
    

    Hope this helps in building a solution!

  • Hi @christian Thank you so much for the detailed response. I will give it a try…

  • Your code recommendations are of great help and I am getting close to the solution. Unfortunately I'm not able to get reference to video track from participant-joined event. Hoping you can take quick look at this code shot and spot the problem.

  • christian
    christian Dailynista

    Hey @lmeyers

    I'd recommend using persistentTrack, not videoTrack. persistentTrack is always available (including on participant-joined), whereas videoTrack only when the camera is turned on.

    With that you should get it over the finish line ☺️

  • Hi @christian

    Thanks again for the help and we are oh so close. The persistentTrack (and videoTrack) always came up undefined. (see img). I'm testing this .Net app on localHost with Chrome incognito, if that makes any difference. I can see the local daily video track playing at the time of error.

  • christian
    christian Dailynista

    Hey @lmeyers,

    I'm only now realizing that media tracks captured in a secured environment (like an iframe), can't be accessed from the outside, similar to accessing content from an iframe, due to browser security.

    That explains why you're seeing undefined for all tracks outside of the frame.

    Apologies for sending you down that path! 🙇‍♂️

    I think you can still achieve what you are trying, but you'd have to call getUserMedia() yourself to access the video track for the local participant, which is something we usually don't recommend, due to the intricacies of dealing with custom video tracks. The DailyVideo component doesn't accept a custom media track, but setting up a custom <video> element works as I described earlier.

    You would want to react to situations where the video track is re-initialized (e.g. when the user switches to another camera) and re-gUM() with the updated device id. You can leverage the selected-devices-updated event to achieve this.

    Let me draft up some sample code:

    const updateVideoTrack = async (deviceId) ⇒ {
      const stream = await navigator.mediaDevices.getUserMedia({
        video: {
          deviceId: {
            ideal: deviceId,
          },
        },
      });
      const videoEl = document.getElementById('video-capture');
      videoEl.srcObject = new MediaStream(stream.getVideoTracks());
    }
    
    callFrame.on('selected-devices-updated', (ev) ⇒ {
      updateVideoTrack(ev.devices.camera.deviceId);
    });
    

    You should definitely take this with a grain of salt, as there might be unexpected behaviors when multiple JavaScripts contexts on the same page try to access media devices at roughly the same time.

    A different solution would include Daily Python to capture and store the video frames on a Python server, but depending on your setup this might not be feasible.

    The main blocker here is browser security features, disallowing certain data and media to be passed between frames with different origins. You wouldn't have this issue if you've built your own custom call object app, but that is arguably a bigger lift.

    Again, I'm sorry for the previous misguidance. I simply forgot about this limitation on the media tracks when embedding Daily Prebuilt.