[Support]: High CPU usage for ffmpeg process #6461

poulti · 2023-05-10T21:41:58Z

Describe the problem you are having

CPU usage of the ffmpeg process is high, and slighlty higher (!) with hwaccel turned on. I'd like to optimise it as I can't scale down the resolution further on some cameras (G4s) and CPU usage is still high.

I'm running on a Raspberry Pi CM4 with 8GB RAM, and I've set GPU memory to 256MB.
I'm using the cameras named "Tapo2" and "G42" just to watch the stream in Home Assistant. No detection, no recording. And most of the time, little change there. The camera named "G41" is in the middle of the house, with lots of movements, and detection enabled (processed by the Coral USB).

Tests done (G41 and G42 have very similar CPU usage despite different configs) - see pictures below

(1) Tapo2 on 640x360 stream (lowest quality), no hwaccel --> ffpmeg CPU process around 10% (HAPPY)
(2) Tapo2 on 640x360 stream (lowest quality), with hwaccel --> ffpmeg CPU process around 14% (still HAPPY, but surprising)
(1) G41 on 1280x720 stream (lowest quality!), no hwaccel --> ffpmeg CPU process around 68% (SAD)
(2) G41 on 1280x720 stream (lowest quality!), with hwaccel --> ffpmeg CPU process around 74% (SAD)
(3) Tapo2 on 1920x1080 stream (high quality), no hwaccel --> ffpmeg CPU process around 77% (SAD)

What I don't understand:

Between Tapo2 low resolution and a G4 there are 4x more pixels (and twice more fps?) --> CPU usage is way higher (10% vs 68%)
Between a G4 and Tapo2 high resolution there are 2x more pixels --> CPU usage is slightly higher (68% vs 77%)
Why is hwaccel CPU usage for ffmpeg worse?
Is there any unwanted operations happening in ffmpeg (resizing?) I could deactivate?

Is my configuration incorrect in some ways? Should I add other arguments to ffmpeg input/output config? Any other suggestion?
(some links from #3860 are now dead, and couldn't find another idea to try from other "high CPU usage" issues)

Thanks a lot for the guidance

Version

0.12.0-DA3E197

Frigate config file (Test number 2: low res, with hwaccel)

# yaml-language-server: $schema=http://localhost:5555/api/config/schema
mqtt:
  host: mosquitto

## Hardware acceleration for raspberry 4 (64bit OS)
ffmpeg:
  hwaccel_args: preset-rpi-64-h264
# ffmpeg:
#   hwaccel_args:
#     - -c:v
#     - h264_v4l2m2m

cameras:

  camera_tapo2: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:xxx@xxx:554/stream2 # Low 640x360 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection in kids room
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 360

  camera_g41: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: true
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_g42: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection for FFA
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_iegeek1: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:554/2 # Low 640x352 stream
          roles:
            - detect
    motion:
      mask:
        #- 0,0,0,981,523,142,1212,105,1920,326,1920,0 # Mask for High quality stream
        - 0,0,0,352,139,32,344,31,640,260,640,0 # Mask for Low quality stream
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: True
      width: 640 # <---- update for your camera's resolution
      height: 352 # <---- update for your camera's resolution
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.8
    record:
      enabled: true
      retain:
        days: 0
        mode: motion
      events:
        retain:
          default: 2
          mode: active_objects
        pre_capture: 5
        post_capture: 5
        objects:
          - person
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 352

snapshots:
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: True


detectors:
  coral:
    type: edgetpu
    device: usb

Relevant log output

Nothing very relevant to me here...

2023-05-10 20:13:39.176710078  [INFO] Starting Frigate...
2023-05-10 20:13:43.706241317  [2023-05-10 20:13:43] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-05-10 20:13:43.847165359  [2023-05-10 20:13:43] peewee_migrate                 INFO    : Starting migrations
2023-05-10 20:13:43.917734907  [2023-05-10 20:13:43] peewee_migrate                 INFO    : There is nothing to migrate
2023-05-10 20:13:44.019192115  [2023-05-10 20:13:44] detector.coral                 INFO    : Starting detection process: 292
2023-05-10 20:13:44.029202701  [2023-05-10 20:13:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-05-10 20:13:44.036203392  [2023-05-10 20:13:44] frigate.app                    INFO    : Output process started: 294
2023-05-10 20:13:44.080914675  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_tapo2: 300
2023-05-10 20:13:44.117291725  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g41: 302
2023-05-10 20:13:44.138320408  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g42: 303
2023-05-10 20:13:44.182987895  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_iegeek1: 305
2023-05-10 20:13:44.215882201  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_tapo2: 307
2023-05-10 20:13:44.237082641  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g41: 308
2023-05-10 20:13:44.276377793  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g42: 312
2023-05-10 20:13:44.329073553  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_iegeek1: 317
2023-05-10 20:13:46.818909676  [2023-05-10 20:13:46] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

FFprobe output from your cameras

Tapo2:
  Metadata:
    title           : Session streamed by "TP-LINK RTSP Server"
    comment         : stream2
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 640x360, 15 fps, 28.58 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

G41:
  Metadata:
    title           : D021F99547EB_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

G42:
  Metadata:
    title           : D021F995415F_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

Frigate stats

See screen shots below

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

1x Tapo c200, 2x Ubiquiti G4, 1x ieGeek whatever

Any other information that may be helpful

Test (1) - low res, no hwaccel
Test (2) - low res, with hwaccel
Test (3) - tapo high res, no hwaccel

blakeblackshear · 2023-05-10T22:33:22Z

Can you give this a try?

ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m

poulti · 2023-05-11T07:06:45Z

Thank you for the swift reply. It looks better for a hwaccel setting, ie at the level of the non hwaccel (at least it's not worse!) but not significantly better than non hwaccel.

Tapo Low res + NEW hwaccel settings
Tapo High res + NEW hwaccel settings

rovingclimber · 2023-05-12T10:12:35Z

Hi!

Glad this has been raised actually - it's on my backlog to look at - I have 6 TP-link TAPO cameras (various models) which work really well with Frigate but definitely seeing higher CPU usage that I'd like (Intel J5040 / Coral PCIE). I've had to use the 1080p stream for detect on 3 of them because I'm looking specifically to detect our cats coming and going, and on the low-res streams a cat just don't give enough pixels :)

I've been using preset-intel-qsv-h264 so far, will try the setting above and see if it improves.

blakeblackshear · 2023-05-12T11:30:04Z

@rovingclimber this issue is specific to raspberry pi and it's hardware acceleration. I would suggest making your own issue as this issue isn't really relevant to your hardware.

blacknell · 2023-05-18T13:10:00Z

I'm running into a similar problem. I have an RPi4 8GB with the follow CPU utilisations. Two cameras same model.

Acceleration off. 26% and 29%
Acceleration preset-rpi-64-h264 76% and 72%
Acceleration -c:v:1 and h264_v4l2m2m 29% and 26%

I'm also not getting any usage stats from GPUs (I am not running the addon - Frigate is standalone). GPU memory set to 1024MB

NickM-27 · 2023-05-18T13:24:29Z

The RPi does not support usage stats so that is expected

blacknell · 2023-05-18T16:34:40Z

I should have left that point out as it's not my primary question. I'm most interested, and confused, about the CPU load given hardware acceleration settings.

ferrarimarco · 2023-05-18T19:31:56Z

I'm experiencing the same issue. My setup similar (Raspberry Pi 4 8GB).

The CPU usage drops from about 45% to 13% after switching from preset-rpi-64-h264 to -c:v:1 h264_v4l2m2m with a single camera (2560x1920, resized to defaults for detection). Changing the configuration to avoid resizing makes the CPU load to jump to even higher values.

If I read the code correctly, that preset corresponds to:

frigate/frigate/ffmpeg_presets.py

Line 57 in 6d0c2ec

"preset-rpi-64-h264": ["-c:v", "h264_v4l2m2m"],

According to the ffmpeg docs, stream_type[:additional_stream_specifier] is defined as:

stream_type is one of following: ’v’ or ’V’ for video, ’a’ for audio, ’s’ for subtitle, ’d’ for data, and ’t’ for attachments. ’v’ matches all video streams, ’V’ only matches video streams which are not attached pictures, video thumbnails or cover arts. If additional_stream_specifier is used, then it matches streams which both have this type and match the additional_stream_specifier. Otherwise, it matches all streams of the specified type.

So, if I get the v:1 selector correctly, it matches a specific stream (1), while v matches all video streams to set the -c h264_v4l2m2m option.

I'm not sure about the reason behind this behavior :(

My GPU memory is set to 76 MB, which is the max recommended for Raspberry Pi 4:

$ vcgencmd get_mem gpu
gpu=76M

PS: I just found #6385 that seems to be about this.

poulti · 2023-05-19T07:36:49Z

PS: I just found #6385 that seems to be about this.

Indeed, it looks like it would include the ":1" selector to all users of the rpi preset.

I'm not sure about the reason behind this behavior :(

That's a good question!
And the overall CPU usage remains high even when "doing nothing" with the stream. Obviously the app (ffmpeg in particular) is not "doing nothing" but I don't know if/what I could change to reduce the load.

Is there any other optimisation somebody think off?

ZdenekM · 2023-06-06T13:36:31Z

Can you give this a try?

ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m

In my case, the load went from 80 % to about 30 %. Nice!

poulti · 2023-06-12T07:22:20Z

Can you give this a try?
ffmpeg:
  hwaccel_args:
    - -c:v:1
    - h264_v4l2m2m
In my case, the load went from 80 % to about 30 %. Nice!

That's encouraging! Could you post a bit more details on your camera settings? (stream size, frigate config for those etc.)
Thanks

blakeblackshear · 2023-06-12T10:04:39Z

This change is included in the preset as of 0.12.1

blacknell · 2023-06-12T10:14:30Z

Off topic perhaps but I wanted this change too.

Don't understand why the new docker build isnt pulled automatically when I have image: ghcr.io/blakeblackshear/frigate:stable in my docker compose

blakeblackshear · 2023-06-12T10:41:14Z

Did you run docker compose pull?

blacknell · 2023-06-12T10:42:49Z

No. Didn’t realise as thought the latest tag would do this next time compose was started. Thanks Regards Paul

…

On 12 Jun 2023, at 11:41, Blake Blackshear ***@***.***> wrote: Did you run docker compose pull? — Reply to this email directly, view it on GitHub <#6461 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDT75B565SR4AWJ5XVZKA3XK3W5LANCNFSM6AAAAAAX5IPN4M>. You are receiving this because you commented.

blakeblackshear · 2023-06-12T11:07:32Z

Compose doesnt pull new images on startup

Lockie85 · 2023-06-21T14:42:54Z

I believe I'm also having the same issue so thought I'd join in sharing as much info as I can to help.

Hardware
Raspberry Pi 4 (rpi4-64) 4GB RAM
Tapo C110
Tapo C200

Setup
Home Assistant 2023.6.2
Supervisor 2023.06.2
Operating System 10.3
Frontend 20230608.0 - latest

Frigate Version
v0.12.1-367d724

Frigate config file

mqtt:
  enabled: False

detectors: 
  coral:
    type: edgetpu 
    device: usb

go2rtc:  
  streams:  
    driveway:  
      - rtsp://user:[email protected]:554/stream1
  office:
      - rtsp://user:[email protected]:554/stream1

cameras:
  Driveway:
    ffmpeg:
      hwaccel_args: preset-rpi-64-h264
      inputs:
        - path: rtsp://user:[email protected]:554/stream1
          roles:
            - detect
            - record
    detect:
      enabled: True
      width: 1280
      height: 720
      fps: 5
    record: 
      enabled: True

  Garden:
    ffmpeg:
      hwaccel_args: preset-rpi-64-h264
      inputs:
        - path: rtsp://user:[email protected]:554/stream1
          roles:
            - detect
            - record
    detect:
      enabled: True
      width: 1280
      height: 720
      fps: 5
    record: 
      enabled: True

Frigate Log On Startup

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
2023-06-21 14:31:27.174186204  [INFO] Preparing go2rtc config...
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-06-21 14:31:27.208175951  [INFO] Starting NGINX...
2023-06-21 14:31:27.227518293  [INFO] Starting Frigate...
s6-rc: info: service legacy-services successfully started
2023-06-21 14:31:27.367359919  [INFO] Got IP address from supervisor: 192.168.5.4
2023-06-21 14:31:27.560599410  [INFO] Got WebRTC port from supervisor: 8555
2023-06-21 14:31:28.695611775  [INFO] Starting go2rtc...
2023-06-21 14:31:28.940308055  15:31:28.940 INF go2rtc version 1.2.0 linux/arm64
2023-06-21 14:31:28.941822029  15:31:28.941 INF [api] listen addr=:1984
2023-06-21 14:31:28.943361298  15:31:28.943 INF [rtsp] listen addr=:8554
2023-06-21 14:31:28.944765143  15:31:28.944 INF [srtp] listen addr=:8443
2023-06-21 14:31:28.945795958  15:31:28.945 INF [webrtc] listen addr=:8555
2023-06-21 14:31:32.815050709  [2023-06-21 15:31:32] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-06-21 14:31:32.894624073  [2023-06-21 15:31:32] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-06-21 14:31:32.903393120  [2023-06-21 15:31:32] peewee_migrate                 INFO    : Starting migrations
2023-06-21 14:31:32.920107953  [2023-06-21 15:31:32] peewee_migrate                 INFO    : There is nothing to migrate
2023-06-21 14:31:32.983297774  [2023-06-21 15:31:32] detector.coral                 INFO    : Starting detection process: 299
2023-06-21 14:31:32.989949934  [2023-06-21 15:31:32] frigate.app                    INFO    : Output process started: 301
2023-06-21 14:31:32.995724998  [2023-06-21 15:31:32] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-06-21 14:31:33.019675980  [2023-06-21 15:31:33] frigate.app                    INFO    : Camera processor started for Driveway: 305
2023-06-21 14:31:33.041041452  [2023-06-21 15:31:33] frigate.app                    INFO    : Camera processor started for Garden: 308
2023-06-21 14:31:33.061334573  [2023-06-21 15:31:33] frigate.app                    INFO    : Capture process started for Driveway: 310
2023-06-21 14:31:33.083712193  [2023-06-21 15:31:33] frigate.app                    INFO    : Capture process started for Garden: 311
2023-06-21 14:31:35.741484032  [2023-06-21 15:31:35] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-06-21 14:31:37.205268812  [INFO] Starting go2rtc healthcheck service...

System Info

Additional Notes
After only recently purchasing the Coral USB Accelerator, I'm unsure it's made any difference to my Raspberry Pi CPU usage by Frigate. Perhaps there is something wrong with my setup. Or perhaps it relates to this bug/issue?

github-actions · 2023-07-22T00:09:17Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

poulti · 2023-07-22T12:49:04Z

Any hope of using less CPU on raspberry, or should I upgrade the hardware?
Thanks

github-actions · 2023-08-23T00:08:31Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

poulti added support triage labels May 10, 2023

github-actions bot added the stale label Jul 22, 2023

github-actions bot removed the stale label Jul 23, 2023

github-actions bot added the stale label Aug 23, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Support]: High CPU usage for ffmpeg process #6461

[Support]: High CPU usage for ffmpeg process #6461

poulti commented May 10, 2023 •

edited

Loading

blakeblackshear commented May 10, 2023

poulti commented May 11, 2023 •

edited

Loading

rovingclimber commented May 12, 2023

blakeblackshear commented May 12, 2023

blacknell commented May 18, 2023

NickM-27 commented May 18, 2023

blacknell commented May 18, 2023

ferrarimarco commented May 18, 2023 •

edited

Loading

poulti commented May 19, 2023

ZdenekM commented Jun 6, 2023

poulti commented Jun 12, 2023

blakeblackshear commented Jun 12, 2023

blacknell commented Jun 12, 2023

blakeblackshear commented Jun 12, 2023

blacknell commented Jun 12, 2023 via email

blakeblackshear commented Jun 12, 2023

Lockie85 commented Jun 21, 2023

github-actions bot commented Jul 22, 2023

poulti commented Jul 22, 2023

github-actions bot commented Aug 23, 2023

[Support]: High CPU usage for ffmpeg process #6461

[Support]: High CPU usage for ffmpeg process #6461

Comments

poulti commented May 10, 2023 • edited Loading

Describe the problem you are having

Version

Frigate config file (Test number 2: low res, with hwaccel)

Relevant log output

FFprobe output from your cameras

Frigate stats

Operating system

Install method

Coral version

Network connection

Camera make and model

Any other information that may be helpful

blakeblackshear commented May 10, 2023

poulti commented May 11, 2023 • edited Loading

rovingclimber commented May 12, 2023

blakeblackshear commented May 12, 2023

blacknell commented May 18, 2023

NickM-27 commented May 18, 2023

blacknell commented May 18, 2023

ferrarimarco commented May 18, 2023 • edited Loading

poulti commented May 19, 2023

ZdenekM commented Jun 6, 2023

poulti commented Jun 12, 2023

blakeblackshear commented Jun 12, 2023

blacknell commented Jun 12, 2023

blakeblackshear commented Jun 12, 2023

blacknell commented Jun 12, 2023 via email

blakeblackshear commented Jun 12, 2023

Lockie85 commented Jun 21, 2023

github-actions bot commented Jul 22, 2023

poulti commented Jul 22, 2023

github-actions bot commented Aug 23, 2023

poulti commented May 10, 2023 •

edited

Loading

poulti commented May 11, 2023 •

edited

Loading

ferrarimarco commented May 18, 2023 •

edited

Loading