How do I control GPU?

rrroy
edited August 2022 in Q&A 🤔

We are playing around with filters and backgrounds, but whenever we enable in testing, our gpu skyrockets. Is there a way around this?

Tagged:

Best Answer

  • kwindla
    kwindla Dailynista
    Answer ✓

    In general, real-time processing of video or audio tracks in a web browser uses a lot of CPU, GPU, or both. There isn't always a way around this.

    On Chrome, it's now possible to do processing of audio and video in a worker thread. This can help, because modern machines are generally multi-core, and getting CPU-intensive work off of the main Javascript thread is often a big win.

    Here's a WebRTC sample showing how to process video data in a worker thread.

    I do not believe this is available yet in Safari.

Answers

  • RaviAtDaily
    RaviAtDaily Dailynista
    edited July 2022

    +1 to everything Kwindla said 👆️

    Unfortunately, additional CPU + GPU load is inevitable as each video frame must be run through the segmentation model (in the case of backgrounds) for inference. Various factors can influence this load, including:

    • Size of machine learning model
    • Backend of the chosen library/framework for inference. Some might optimize for running on the CPU, some on the GPU
    • Video resolution of the camera stream that is being processed
    • Video frame rate

    ------

    While additional CPU+GPU load is unavoidable, there are some approaches one can use to optimize this. At Daily, we did a lot of testing to optimize this. Obviously reducing video frame size is one way to improve the load, however that does compromise the end user's experience. Reducing video frame rate is another similar optimization technique

    We found that - one of the best approaches was to enable different frame rates on the video and the ML inference. So that one can optimize how frequently a new inference is calculated on the video without compromising the video frame rate as visible to the end user.

    Hope this is useful.

    -----