Help with recording 24/7 in ver 3.0.0b11

[Chapshan]

Hi,

I'm new to viseron and this is my first time to try it.
I have created a docker container with version 3.0.0b11 and worked my way to figure out how I should write the config file, but for some reason I'm still not able to make a 24/7 recording for my test camera.

My need is quite basic. I don't need any AI or any motion detection. only recording for 24/7.
This is how I wrote my config:

ffmpeg:
  camera:
    camera_1: 
      name: test
      host: !secret host_a
      port: !secret port_a
      path: !secret path_a
      username: !secret username_a
      password: !secret password_a
      fps: 60

storage:
  recorder:
    tiers:
      - path: /
        continuous:
          max_size:
            gb: 2

nvr:
  camera_1:

This is the logs I have at the beginning (not sure what is the meaning of the WARNING message):

[2024-11-17 20:49:21] [INFO    ] [viseron.core] - -------------------------------------------
[2024-11-17 20:49:21] [INFO    ] [viseron.core] - Initializing Viseron 3.0.0b11
[2024-11-17 20:49:21] [INFO    ] [viseron.components] - Setting up component data_stream
[2024-11-17 20:49:21] [INFO    ] [viseron.components] - Setup of component data_stream took 0.0 seconds
[2024-11-17 20:49:21] [INFO    ] [viseron.components] - Setting up component storage
[2024-11-17 20:49:21] [INFO    ] [alembic.runtime.migration] - Context impl PostgresqlImpl.
[2024-11-17 20:49:21] [INFO    ] [alembic.runtime.migration] - Will assume transactional DDL.
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setup of component storage took 0.1 seconds
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setting up component webserver
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setup of component webserver took 0.0 seconds
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setting up component nvr
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setting up component ffmpeg
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setup of component nvr took 0.0 seconds
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setup of component ffmpeg took 0.0 seconds
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setting up domain camera for component ffmpeg with identifier camera_1
[2024-11-17 20:49:22] [INFO    ] [viseron.components] - Setting up domain nvr for component nvr with identifier camera_1
[2024-11-17 20:49:22] [WARNING ] [tornado.access] - 404 GET /api/v1/hls/camera_1/index.m3u8?start_timestamp=1731869002&daily=true (192.168.1.238) 255.89ms
[2024-11-17 20:49:24] [WARNING ] [tornado.access] - 404 GET /api/v1/hls/camera_1/index.m3u8?start_timestamp=1731869002&daily=true (192.168.1.238) 2.96ms
[2024-11-17 20:49:24] [WARNING ] [tornado.access] - 404 GET /api/v1/hls/camera_1/index.m3u8?start_timestamp=1731869002&daily=true (192.168.1.238) 3.00ms
[2024-11-17 20:49:25] [WARNING ] [tornado.access] - 404 GET /api/v1/hls/camera_1/index.m3u8?start_timestamp=1731869002&daily=true (192.168.1.238) 4.10ms
[2024-11-17 20:49:26] [INFO    ] [viseron.components] - Setup of domain camera for component ffmpeg with identifier camera_1 took 3.4 seconds
[2024-11-17 20:49:26] [INFO    ] [viseron.components.nvr.nvr.camera_1] - Motion detector is disabled
[2024-11-17 20:49:26] [INFO    ] [viseron.components.nvr.nvr.camera_1] - Object detector is disabled
[2024-11-17 20:49:26] [INFO    ] [viseron.components.nvr.nvr.camera_1] - NVR for camera test initialized
[2024-11-17 20:49:26] [INFO    ] [viseron.components] - Setup of domain nvr for component nvr with identifier camera_1 took 0.0 seconds
[2024-11-17 20:49:26] [INFO    ] [viseron.core] - Viseron initialized in 4.2 seconds

My recording path is actually a docker-volume (but I have tried it also with a bind, and ended up with the same results). so when I go to that path, I see the following:

$ ls -R ./
./:
camera_1

./camera_1:
$

so looks like the segment for the camera is being created correctly, but nothing can be found under that path.

as for the configuration - I can tell that my "secrets" are good, as I do get live feed for the camera.

any one know what is the problem?

[roflcoopter]

I think it might be a problem with the volumes and bind mounts overshadowing each other.
I declare the volumes in the Dockerfile which i realize now might be a mistake.

Try to stop Viseron, delete the volumes and then start the container again.
Hopefully that should work.

I will remove the volumes from the Docker container for the next release

[Chapshan]

Hi @roflcoopter
thanks for the quick answer.

I think I have found the missing recordings in another volume based on your reply, but I'm not sure what you meant by deleting the volumes so I didn't do a thing yet about it to "fix" my environment and make it record on the desired volume.

sorry for being ignorant about it, but did you mean that I need to delete the volumes I have created or those that were created by the dockerfile? or maybe you meant for both? or maybe you meant that I need to try and tinker with the dockerfile somehow?

also, one small suggestion which can be very useful:
maybe worth saving the recorded files in a directory hierarchy that represents their date and time. something like:

2024
|__11
|____16
|______22_00
|______23_00
|____17
|______00_00
|______01_00
|______02_00
...
...
...
2025
|__01
|____01
|______00_00
|______01_00
...
...

also, one more question:
how can I change the format of the recorded files? I have tried watching them with VLC with no luck. I believe I need to change the codec or something like that, but have very limited knowledge about it. can you suggest on something that would work for me?

[roflcoopter]

Hi @roflcoopter thanks for the quick answer.

I think I have found the missing recordings in another volume based on your reply, but I'm not sure what you meant by deleting the volumes so I didn't do a thing yet about it to "fix" my environment and make it record on the desired volume.

sorry for being ignorant about it, but did you mean that I need to delete the volumes I have created or those that were created by the dockerfile? or maybe you meant for both? or maybe you meant that I need to try and tinker with the dockerfile somehow?

See this comment of another user with a similar problem: #721 (reply in thread)

But lets take a step back, in v3 the streams are stored in short 5s segments in the /segments folder by default.
The /recordings folder will only contain MP4 files if the config option create_event_clip: true is set.

also, one small suggestion which can be very useful:
maybe worth saving the recorded files in a directory hierarchy that represents their date and time. something like:

Which parent folder are you thinking about here, segments or recordings?
The segments are not really meant for manual browsing.

also, one more question:
how can I change the format of the recorded files? I have tried watching them with VLC with no luck. I believe I need to change the codec or something like that, but have very limited knowledge about it. can you suggest on something that would work for me?

The files are in HLS format (.m4s) to support 24/7 recordings. To play it in VLC you need an m3u8 playlist.
You can retrieve a playlist for a specified time period using the API if you are interested in that

[Chapshan]

Thanks @roflcoopter again for the quick reply.

I assumed that the recordings folder will be the directory in which I can find my continuous /recording, since this is the path we can configure for the docker. I wasn't aware of the existence of the /segments folder before my original message and only now (I think) I understand its purpose.

to answer your question, then my suggestion for the folder structure/hierarchy was regarding the /segments folder, as this is the only folder I had with files when I was searching for the recorded files (also, from what I could understand from reading around, the /recordings folder already have such a structure/hierarchy, but I didn't see it as I only use the 24/7 continuous recording).

so, from what I can see and understand now, and according to my configuration, I only have files under /segments folder for the continuous recording. this is great for what I need, however I still have few questions and maybe some few requests:

1. I'm new to .m4s files and not sure I understand how I can use them. I tried to read around a bit on how to create the m3u8 playlist as you suggested. the thing is, I'm not sure I understand how I should use it once I have the m3u8 playlist ready (should I place the m3u8 under the /segments folder and access it remotely? if so - how should I allow such access? or maybe I should copy the .m4s files locally and place the m3u8 in that local folder? maybe something else?).
2. you have mentioned using the API for creating a playlist. can you elaborate a bit about that? I'm not sure I understand which API you are referring to and how should I use it. is there somewhere I can read about it?
3. a request: is it possible to have the /segment folder be configured also in the docker?
4. a request: my current understanding is that there is no (existing) option of creating a continuous mp4 files. is it possible to add such support? the way I can think of how it will work is by having a configurable length for each such mp4 file (and with a default of, let's say, 60 seconds) so that every X seconds will be created a new mp4 file. those video clips could be stored under the a new folder in a sub-folder structure/hierarchy that is similar to what I have suggested above or like the one under /recordings folder (my initial thought was to place those continuous video clips under /recordings folder, but then I realized that for someone who wants to have a mix of continuous recording and also events, this will make things more complicated. so probably better to place them under a new folder to avoid confusion).
5. a request: alternatively to the previous suggestion (or maybe in addition to it, if possible) - maybe add an option in the web-UI (probably under the "timeline" page) to set from and to fields, and a "generate" button that will create a single video-clip of the custom selected period. maybe even to add in addition (but not instead, so that time-period that starts on one day and ends on another one will still be possible) an option to drag and select the desired time-period from the timeline itself, which will "feed" the from and to fields.
6. a request: I noticed that if a camera is disconnected, then in its timeline page we can not use the playback. is it possible to allow watching playback even if the camera is disconnected? since I believe that the source files for the playback are taken from the '/segments` folder which to my understanding is available even when the camera is disconnected from power.

again, many thanks for the help and for the quick answers.
hopefully, my requests are doable and not complex to implement.
appreciate all your hard work 🙏

[roflcoopter]

1. I'm new to .m4s files and not sure I understand how I can use them. I tried to read around a bit on how to create the m3u8 playlist as you suggested. the thing is, I'm not sure I understand how I should use it once I have the m3u8 playlist ready (should I place the m3u8 under the /segments folder and access it remotely? if so - how should I allow such access? or maybe I should copy the .m4s files locally and place the m3u8 in that local folder? maybe something else?).

That depends on how you want to consume it.
If you observe the devtools when viewing the timeline/events/recordings you see calls to Viserons API and the HLS endpoint.
It will look something like this: http://host:port/api/v1/hls/garage_front/<recording_id>/index.m3u8
You can input that directly in VLC to view it.

If you want to get more familiar with how the streaming works you can read up on the HLS protocol (the m3u8 playlist) which Viseron is using.

2. you have mentioned using the API for creating a playlist. can you elaborate a bit about that? I'm not sure I understand which API you are referring to and how should I use it. is there somewhere I can read about it?

Viserons API does it for the frontend, but again it depends on how you want to consume it.
The API is not documented yet sadly.

3. a request: is it possible to have the /segment folder be configured also in the docker?

Do you mean changing what path the segments are stored in inside the container?
That can be done by changing the path for your tier in the storage config

4. a request: my current understanding is that there is no (existing) option of creating a continuous mp4 files. is it possible to add such support? the way I can think of how it will work is by having a configurable length for each such mp4 file (and with a default of, let's say, 60 seconds) so that every X seconds will be created a new mp4 file. those video clips could be stored under the a new folder in a sub-folder structure/hierarchy that is similar to what I have suggested above or like the one under /recordings folder (my initial thought was to place those continuous video clips under /recordings folder, but then I realized that for someone who wants to have a mix of continuous recording and also events, this will make things more complicated. so probably better to place them under a new folder to avoid confusion).

What is your usecase for that?
It could definitely be added but that means that the m4s files and mp4 files would exist for the same timespans using twice as much disk space.

5. a request: alternatively to the previous suggestion (or maybe in addition to it, if possible) - maybe add an option in the web-UI (probably under the "timeline" page) to set from and to fields, and a "generate" button that will create a single video-clip of the custom selected period. maybe even to add in addition (but not instead, so that time-period that starts on one day and ends on another one will still be possible) an option to drag and select the desired time-period from the timeline itself, which will "feed" the from and to fields.

Yes an export feature is on the todo list before v3 is release: #716

6. a request: I noticed that if a camera is disconnected, then in its timeline page we can not use the playback. is it possible to allow watching playback even if the camera is disconnected? since I believe that the source files for the playback are taken from the '/segments` folder which to my understanding is available even when the camera is disconnected from power.

Will look into that!