Streaming video in multiple Electron windows

Hey there,

I need a way to stream videos in multiple Electron windows. In the main window, user joins the room and enables video / audio.

The second window is Picture-in-picture where user can see the video participants in a small window.

I can't see how I can implement PIP window without joining the same room twice. Does Daily.co's React API provide a way to stream a participant video without requiring user to re-join the room in the second window?

Tagged:

Comments

  • Tamara
    Tamara Moderator, Dailynista admin

    Hi there, if you're looking to use PiP while streaming, using React, you can pass it under mode in your startLiveStreaming() instance - see https://docs.daily.co/reference/rn-daily-js/instance-methods/start-live-streaming#mode. This should allow you to stream and use PiP in the same room without having to join twice.

    Hope that helps, let me know if you have any additional questions we can help with.

  • @Tamara Thanks for the answer, I don't use the startLiveStreaming method though. I followed the official documentation and examples, and used DailyProvider:

    https://docs.daily.co/reference/daily-react/daily-provider


    DailyProvider requires a CallObject. This is what breaks the PiP window. Should I refactor the way users join calls in my app?

  • kwindla
    kwindla Community Manager, Dailynista admin

    Hi!

    Just to make sure I understand, in the main window you some (or all) of the video tracks, and then in the PiP window you are playing some/all of the tracks a second time?

    I have done this, but long enough ago that we didn't have the daily-react library at that time. The basic idea, though, is that you can share the MediaStreamTrack object between the two windows and use it in a <video> element in both windows. My memory is that I used the Electron built-in remote/getGlobal functionality for this.

    https://github.com/electron/remote

    You can get the MediaStreamTrack object for a participant using the useMediaTrack daily-react hook. Then you can share the track between the two windows. And attach the track to a <video> object manually, rather than using the <DailyVideo> helper.

    Does that make sense?

  • elephantshadow
    edited May 2023

    @kwindla Thanks for the answer - do you have an example of copying the MediaStream object ? I can't really wrap my head around copying it; does it copy the video contents as well? Or will the video re-streamed from the PiP window ?

    Also, it's recommended to use IPC over the remote module. If we're copying the binary data and sending it to the PiP window constantly, what would be the performance implication?

    I have my own simple video element which takes a daily session id and calls useMediaTrack to stream. Unfortunately, this approach requires me to re-join the same room from PiP window.


    interface Props {
      participantId: string
    }
    
    
    export function PictureInPictureVideo(props: Props) {
      const track = useMediaTrack(props.participantId, 'video')
      const el = useRef<HTMLVideoElement | null>(null)
    
    
      useEffect(() => {
        const playerEl = el.current
    
    
        if (!playerEl || !track?.persistentTrack) return
    
        log.info('Set track', track.persistentTrack)
        playerEl.srcObject = new MediaStream([track?.persistentTrack])
      }, [track?.persistentTrack?.id])
    
    
      const videoEl = useMemo(() => {
        log.info('Re-render element')
    
        return <video autoPlay muted playsInline ref={el} />
      }, [track.persistentTrack])
    
    
      return videoEl
    }
    
  • kwindla
    kwindla Community Manager, Dailynista admin

    You're right, that copying all the video stream data in real time wouldn't work very well at the Electron level. but the MediaStreamTrack object is a javascript handle to an opaque and complicated structure down in the lower-level browser/libwebrtc code.

    So if you can copy the MediaStreamTrack object between window/renderer contexts, all you're doing is copying that lightweight handle that lets you access the actual data.

    My memory is that the IPC approach doesn't allow you to copy the MediaStreamTrack, but that using remote/globals does. But I can't remember for sure. I'll see if we can dig up some sample code.

  • elephantshadow
    edited May 2023

    @kwindla Yeah, that's precisely what I need to accomplish PiP. However, IPC did not allow me to copy MediaStreamTrack. It complains about inability to clone the media stream object. I wish Daily had an API which would allow me serialize/deserialize this handle.

    Error I'm getting from IPC:


  • kwindla
    kwindla Community Manager, Dailynista admin

    The handle can't be serialized/deserialized because of how it's implemented by the browser. But I do think the remote/global approach might work.

  • elephantshadow
    edited May 2023

    It seems to be considered as a harmful practice though. Why can't we just stream video by something like session id?

  • Does Daily would charge double if I get the user re-join in the Picture-in-Picture room?

  • kwindla
    kwindla Community Manager, Dailynista admin

    Yes, if you join a room twice, even from the same application, that's considered a second client and double the number of participant minutes.