Frigate+ Base Model 2024.3 Update #14762
Replies: 64 comments 178 replies
-
|
fox! Brilliant - this is one of my main ID goals (so it means that soon I can differentiate between dog/cat and fox and, hopefully eventually, badger). |
Beta Was this translation helpful? Give feedback.
-
|
+1 |
Beta Was this translation helpful? Give feedback.
-
|
+1 for badger! Fox, bird and squirrel are very welcome additions though. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
It seems that the new model. even without adjusting thresholds etc. in the config, is somehow seeing far more foxes even allowing for previous misidentifications. Prior to this latest model, I would "see" perhaps a dozen foxes a night, mainly showing up on two particular cameras (but labelled as cats or dogs). |
Beta Was this translation helpful? Give feedback.
-
|
Im running two AI nvrs for a little while to catch all the things that the current Frigate NVR doesnt. Fox was one of them on mine that wasnt even registering as an object and am submitting a bunch of non-object detects already. |
Beta Was this translation helpful? Give feedback.
-
|
Out of interest, does anyone have an issue with ducks? I have some ducks and the model constantly thinks they are cats, dogs or people. Maybe this new model might help |
Beta Was this translation helpful? Give feedback.
-
|
That's because the model doesn't have a duck label, so it finds the closest approximation instead. Just submit them as false positives and relabel them as birds (same with chickens). |
Beta Was this translation helpful? Give feedback.
-
|
That's exactly what I am having to do. The number of false positives is very high for my ducks, it is constantly flipping between bird, cat, dog, person. I am submitting loads of images to hopefully improve the labelling as bird. I'm guessing the model is well-trained on "normal" birds, but not so much on water fowl. Maybe it's confused as they are not flying, on the ground and around the same size as a cat or small dog. |
Beta Was this translation helpful? Give feedback.
-
|
I have decoy ducks in my lake that float around and I have been ignoring labelling as I didnt want to label it as a bird. I have 'bird' as an alert for more interesting creatures (heron, etc) I dont go out of my way to register it as a false positive knowing it is for the 'greater good' though. Wonder how Blake is going to be discriminating the user submissions in the base model. In the end it wouldnt be a big deal if the decoys started appearing as birds but am wondering how to filter it as they move around on a long line. Guess I would have to draw some very careful zones around them hoping a more interesting bird doesnt get in front of them. 
|
Beta Was this translation helpful? Give feedback.
-
|
Blake van you tell you everything about the user submissions... Based on the picture I can't tell you anything about what I'm looking at, so AI will probably have the same problem. But if it looks like a duck, for AI it probably is a duck. Or you have to feed the model tons of decoy ducks and real ducks to make sure the model can tell you the difference. |
Beta Was this translation helpful? Give feedback.
-
|
When including images in the base model, I label them based on what I think a typical person would be able to recognize. I'm not sure I would have even noticed those decoys. |
Beta Was this translation helpful? Give feedback.
-
|
Great work! Was the base model trained with the existent submissions? If so, can we expect better results? |
Beta Was this translation helpful? Give feedback.
-
|
I incorporated examples for all the new labels from existing submissions. Some have more than others, but they should all work reasonably well. Performance for the new labels should get a lot better in the next base model with all the new examples. This also included new examples for existing labels, so I would expect some improvement there too. |
Beta Was this translation helpful? Give feedback.
-
|
Nice!! Awesome work sir. So question then - I have maybe 2k images with 3 or 4 of those new unlabeled things. Would you recommend working through those existing images immediately and then pull the model? Or pulling the model, just submit and label another round of images as they occur focusing on those new objects? Trying to make the best of this with a good balance of invested time and quick results 😊😁 |
Beta Was this translation helpful? Give feedback.
-
|
The BBQ label seems odd to me. I can't think of the scenario where suddenly there's a grill in my yard that I want to be notified about. Are there people out there who have a big problem with their neighbors suddenly placing a grill on their driveway and running away? It's kind of like having a couch label. You know it's there. It's not going anywhere. Why do you want to save images of it sitting there all day? |
Beta Was this translation helpful? Give feedback.
-
|
Some of the labels were chosen because they are a common source for false positives. Adding them as an actual label is more helpful than continuing to mark them as false positives. I don't expect anyone to actually track their bbq grill. |
Beta Was this translation helpful? Give feedback.
-
I was wondering this about the bbq grill inclusion as well, but then I started going through my reference pics yesterday and noticed how many false positives there were on the grill so that totally makes sense |
Beta Was this translation helpful? Give feedback.
-
Can we get a baby stroller added? I have been getting a bunch of false positives for bike for my neighbor's stroller especially when only 2 wheels are easily visible. Tracking when my garage camera sees a bicycle is extremely important to me as the main reason I got the garage camera in the first place was in response to 2 of mine getting stolen so in a similar way I would love to reduce the false positives caused by my neighbor walking by with the stroller. |
Beta Was this translation helpful? Give feedback.
-
|
Go ahead and report them as false positives. I have reported a few of them myself. This may be sufficient without adding a label, but I will add it to my list anyway. |
Beta Was this translation helpful? Give feedback.
-
|
Great work! It seem like all my tags are gone ? Do I need to tag 1035 pictures again? 
It says 2089 labels So maybe it is a bug while showing the thumbnails ? |
Beta Was this translation helpful? Give feedback.
-
|
They aren't gone. This just means the image hasn't been checked for some of the new labels you added to the camera. The existing labels will still be there when you look at them again. |
Beta Was this translation helpful? Give feedback.
-
|
Ahh - this was the big thing I wanted to understand. So the work I have done already drawing boxes, choosing labels, applying "difficult" is all still there for whatever I did before - it just now will "unverify" those images and I'll go through them to add any new label I activate. So most of the work (assuming that the new label isn't in every image) is to just look to see if the new label is there, box it, and verify? |
Beta Was this translation helpful? Give feedback.
-
|
. So most of the work (assuming that the new label isn't in every image) is to just look to see if the new label is there, box it, and verify? yes |
Beta Was this translation helpful? Give feedback.
-
|
I was waiting for the motorcycle! Thank you very much! Can't wait to test it! |
Beta Was this translation helpful? Give feedback.
-
|
Love the new labels and thanks to the update realized I can change labels on a per camera basis which makes tagging much easier! I'm curious if there's a reason why trucks and vans aren't separated from cars in the label taxonomy? Also, being that I'm more rural, would be awesome to have a 'tractor' tag for vehicles as well (mainly so I can NOT alert when driving mine through) |
Beta Was this translation helpful? Give feedback.
-
The default COCO models really struggle to differentiate here, so I just kept that in place. I will probably pull off very specific vehicle types in the future such as school bus, garbage truck, etc. |
Beta Was this translation helpful? Give feedback.
-
|
+1 to the "tractor" label (though not sure if we'd want a separate one for riding lawn mowers or zero turns). Similarly ATV would be nice (unless we want to group that with motorcycle) |
Beta Was this translation helpful? Give feedback.
-
|
why are the new labels not appearing? Do i need a new base model first? 
|
Beta Was this translation helpful? Give feedback.
-
|
I figured it out. You have to edit the existing cameras in frigate+ to enable the new labels. |
Beta Was this translation helpful? Give feedback.
-
What do you mean? I just uploaded a few pics with the new frigate plus model, I got to the plus page and I am trying to edit the new pics and also the previous ones I had but I still don't see the new options. |
Beta Was this translation helpful? Give feedback.
-
|
So is there a way to export out all my images and labels? I would much rather write a script to quickly go through and relabel my dog labels (which I had been labeling foxes as). The current UI is nice in general but has no way to drill in and re-label specific things. |
Beta Was this translation helpful? Give feedback.
-
|
There isn't a way to export, and this would require an import feature as well. When I roll out the label specific review process, it will be easier to go back through images and adjust. I could also write a script for you if you reach out via email. |
Beta Was this translation helpful? Give feedback.
-
|
The most basic script would just be something to let me re-verify all the images with a specific label. But honestly, with how much time I have invested in uploading to frigate+ then verifying (and potentially re-verifying), it really seems important to be able to export and import images and labels. If anything goes wrong with an update or something I could lose hours of effort. |
Beta Was this translation helpful? Give feedback.
-
|
I keep coming back to a desire to be able to export my tags. I have processed over 2000 images and would like to experiment with different types of models or even be able to fix tagging mistakes (which currently is really tedious to go back and find the last thing I verified). I'd want to export my set, run more compute intensive analysis of my images, find mis-tagged items (such as only tagging one deer and missing that there was a second or third behind it). Right now I suspect those errors are messing up my training significantly. Can we start with an export and figure out import later on? I could at least train my own model on the set that I have spent many hours tagging at this point. |
Beta Was this translation helpful? Give feedback.
-
|
How's this for a bird label? lol
|
Beta Was this translation helpful? Give feedback.
-
|
How'd you get a bird label? |
Beta Was this translation helpful? Give feedback.
-
Read the post that started this thread. It’s now in the Frigate+ models, along with a bunch more new stuff! |
Beta Was this translation helpful? Give feedback.
-
|
No I know that. I looked in frigate+ to label some birds, and it wasn't there, but some of the other new labels were. |
Beta Was this translation helpful? Give feedback.
-
In frigate+ go to cameras and edit the camera there is a list of check boxes there, check bird. |
Beta Was this translation helpful? Give feedback.
-
|
Got it thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Very excited to see new labels being added (particularly USPS), though it's a bit of a bummer I requested new models earlier this week instead of waiting lol. @blakeblackshear I've got three questions:
|
Beta Was this translation helpful? Give feedback.
-
|
For what it's worth, I just went back and labeled all my chickens as birds lol... we'll see!! |
Beta Was this translation helpful? Give feedback.
-
|
Yeah that may make sense - I'll probably do that as well. That said, they look quite distinct from other birds in my area (so it seems like it would be easy for a model to differentiate between them) and I very rarely care about other common wild birds like robins, jays and sparrows that are all over the place in my cameras. I would care about hawks, owls, and eagles (again because they are predators) but I don't have a ton of examples of them to use as training data (maybe adding "bird" will help with that though). |
Beta Was this translation helpful? Give feedback.
-
|
Chickens seem fairly common, so I will probably add them in the future. Not sure about elk. Deer might be a better option. There isn't a way to add labels that the model doesn't support yet, but that's definitely on my short list. It would help seed data for expanding the model. |
Beta Was this translation helpful? Give feedback.
-
|
Indeed, chickens, ducks and any other animals one might commonly get in a garden would be awesome |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear Actually, if we add chickens, adding "egg" too would be amazing! Being able to have my camera inside the coop tell me how many eggs I have waiting would be so cool and helpful |
Beta Was this translation helpful? Give feedback.
-
|
I added some new labels to one camera, started going through the images and then accidentally clicked out. Is there an easy way to start at the next unreviewed image for a given camera? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, this does not work. Even when I go through unreviewed unreviewed , sometime (more and more often), he jumps out back to the overview page instead to the next unreviewed image. I have 18k images, maybe I does not scale for such a high number of images. |
Beta Was this translation helpful? Give feedback.
-
|
It should be looking within the next 1000 images. I will see if I can improve this. |
Beta Was this translation helpful? Give feedback.
-
|
The new camera/unverified filters are amazing! I was seeing the same behavior as blaess. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
I'm also seeing some similarly unusual behaviour. In the F+ images page, when I click "hide unverified images" I get a subset with the grey rather than green icon backgrounds. I then look at the image, tag it if needs be and press space to verify and move to the next one. If I come out of that process and go aback in, I see that the image I believe I had verified by pressing the space bar (or using the mouse to select the verify and save button in the top RHS) has not made the verification "stick". This happens if I revisit the images later having F5 reloaded the page or stop/started Frigate (this is 0.13 with yesterday's model added). |
Beta Was this translation helpful? Give feedback.
-
|
This should be working better as of a few minutes ago. Note that its not a "hide unverified images", but rather a "hide verified" checkmark. |
Beta Was this translation helpful? Give feedback.
-
|
Do I really have to go back through EVERY image and verify them after adding available tags??? |
Beta Was this translation helpful? Give feedback.
-
|
It literally erased ALL the work I've done labeling my 5k images... |
Beta Was this translation helpful? Give feedback.
-
|
It didn't erase anything. All your existing labels are still there. All those images are still used for training even if you don't go back through them. Your new model just may not detect the new labels if you don't add them to your images. It won't impact the performance of your existing labels. You should probably just uncheck the new labels from your cameras and wait for the suggestion feature to be ready. |
Beta Was this translation helpful? Give feedback.
-
|
How about adding skunk to the label list? That should pretty easy to identify and train on. This way I can write an automation to notify, before my dog goes outside... |
Beta Was this translation helpful? Give feedback.
-
|
For those with a lot of images (I have 4k), it would help to have a "only show unverified images" on Frigate+. Until then, here's a snippet you can use to hide all of the verified images. Just open the inspector tools, go to elements, click 'body' on the left, then the + (new rule) button on the right. Then add the css shown above: Then you can just click Load More over and over and only the unverified images will render. |
Beta Was this translation helpful? Give feedback.
-
|
The suggestions are great for me so far. The only issue I have, as some above have said, is the creation of suggested boxes within boxes that I've already marked like in the attached image. Unrelated to suggestions, but two additional comments:
|
Beta Was this translation helpful? Give feedback.
-
|
I tagged an image in frigate to be sent to frigate plus. It had a person tagged but no other objects. I would have expect Plus to have made suggestions - there are several bins and cars in frame plus (arguably) a face What am I missing? 
|
Beta Was this translation helpful? Give feedback.
-
|
You are missing the fact that Blake is running suggestions manually for those who request to test it out. New suggestions are not processed automatically yet. |
Beta Was this translation helpful? Give feedback.
-
|
What be a cool thing to have is a feature that can import my zone config into the annotation tool and show statistics for all labels and zones. This would really help to define zone specific min/max areas and ratios in order to minimize false positives. |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear - just reviewing latest suggestions... Would I expect sub labels to be suggested? for example, it found waste bins and a package but not the face of the person carrying it. plus 2 cars were omitted. Also the blue boxes shown when suggesting are now white when I reviewed images again (having verified). I would have thought these remain blue (white being what I have outlined) 
|
Beta Was this translation helpful? Give feedback.
-
|
When you verify an image, all the suggested labels get saved as official labels. That's why they are no longer blue. It should suggest faces and delivery logos as long as they score high enough. Just like anything else, it's a balancing act between true and false positives. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for letting me try out suggestions @blakeblackshear Worked brilliantly and really streamlines the process of labelling new images. I worked through it so quickly I didn’t manage to take any screen prints but the suggestions were impressively accurate. The odd object missed, otherwise labels were accurate & bounding boxes spot on. Was it just a one off manual run of the suggestions, or will newly uploaded images have them too? |
Beta Was this translation helpful? Give feedback.
-
|
It was a manual run for now. Working on having it run automatically. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks - look forward to that feature! |
Beta Was this translation helpful? Give feedback.
-
|
it would be great if the suggestions could auto-mark boxes as "difficult" based on confidence score. would also help me to understand what should or shouldnt be marked as difficult moving forward |
Beta Was this translation helpful? Give feedback.
-
|
also, I imagine right now you are utilizing the compute power of the Frigate+ backend to generate the suggestions. it would be great to be able to be able to re-run the suggestions feature on single images or even cameras (like if you add or change available labels), but I imagine with enough users this might be a non-negligible amount of resources. is the plan to keep this free and just part of the Frigate+ subscription fee (maybe with a cooldown timer on suggestions generated) ? or will this feature eventually make its way to the main Frigate app such that it uses a users local resources? i guess that brings up another question which is that since its running on the server end, users configurations with respect to min_score and threshold are not being considered? probably not a big deal since the suggestions are just that--suggestions and to be reviewed anyways. |
Beta Was this translation helpful? Give feedback.
-
|
I'm planning to run it on new image uploads for all users with an active subscription automatically. It will also run when/if new labels are added to a camera. It's a much larger model than what runs locally, so your thresholds and min_score values aren't applicable. I will probably look at ways to use local Frigate detectors in future versions. Not sure if it will work as well. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, first of all thank you for the great work. I'm a frigate user for almost 2 years now and it's great! Since about 1.5 weeks I'm also a subscriber to frigate+. But here I encountered quite some issues... But what I have at the moment can be best described as some kind of "car mania". The model sees cars in almost anything (like tree branches, cabins, ...). In between my submissions I requested new models. Now I'm at 4 and it's only sightly getting better. I feel like I'll be ending up consuming all of my 12 requests/year in just a few weeks. Who can help me out here? Thanks a lot in advance. |
Beta Was this translation helpful? Give feedback.
-
|
Here are my label stats: |
Beta Was this translation helpful? Give feedback.
-
|
it would be best if you create a support discussion so we can see an example of your config and related information |
Beta Was this translation helpful? Give feedback.
-
|
Created #15383 |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear First things first, thank you for all the time and effort you've put into this project. Two quick questions:
|
Beta Was this translation helpful? Give feedback.
-
|
There are a little over 30k images so far. I'm not sure what the ideal number is in this case. COCO is an incredibly diverse set of images including many scenes that will never be applicable to a security camera. I just keep pulling in images that are unique and try to level out the counts so that person and car aren't too over represented. It doesn't matter how many labels there are per image since the data augmentation generates multiple different crops of the images anyway. You end up with similar amounts of training data. |
Beta Was this translation helpful? Give feedback.
-
|
I just subscribed to Frigate+. I am very impressed with the suggestion tool--I wish I had known this tool gets unlocked with a subscription. I had already tagged over 500 images from scratch before I subscribed. It wasn't clear to me based on the documentation. Does the suggestion tool use the 2024.3 base model to create the blue boxes? It seems to do an excellent job with cars and people. I hope I'll see similar performance with my custom model. |
Beta Was this translation helpful? Give feedback.
-
|
It's brand new, so I haven't had time to update the documentation yet. It uses a different model architecture than the base model. Because suggestions don't have to be run in realtime, I can use a slower, larger model to make multiple passes over an image. It is trained on the same training set as the base model. |
Beta Was this translation helpful? Give feedback.
-
|
Super excited for this Blake, I'm going to have to hit you up to pay for more trainings soon :) I just went through and confirmed tags on probably 50 images with only a handful of cases where the suggested tags were wrong. Going to save SO much time on busy / road facing cameras. |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear The suggestion tool is really going to speed things up. Thank you so much for creating it! |
Beta Was this translation helpful? Give feedback.
-
|
Suggestions is AWESOME. Wanted to call this one out: That's a pretty tight overlap between the runtime model and the suggestion model and surprised it wasn't flagged as dupe |
Beta Was this translation helpful? Give feedback.
-
|
That is surprising. I can definitely improve this. |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear already pointed out by some people more, but my first impression of the suggest function is really good! Thank you! Just wondering, are you capable of running some sort of stats after we verified your suggested images? I already had a couple of images where my own model annotated a bicycle much better than your model did. I can imagine those images can be useful to train the base model with? |
Beta Was this translation helpful? Give feedback.
-
|
I definitely look for true positives that the base model can't reproduce as well as false positives that it does reproduce. These are some key targets. The suggestion model is actually separate from the base model. The base model is optimized for speed so it can run fast locally. The suggestion model is optimized for accuracy and runs about a dozen times on each image. |
Beta Was this translation helpful? Give feedback.
-
|
Another question... Sometimes the suggested box overlaps with the detected box. Does it make any difference for you which box we delete (either the detected or suggested one)? Or should we keep the one that suits best? |
Beta Was this translation helpful? Give feedback.
-
|
Just leave the most accurate box. |
Beta Was this translation helpful? Give feedback.
-
|
Under the "camera" menu, if I click on "edit" I see that there are several fields that Frigate+ has for the camera, including height, scene, area, etc. How important are these fields in creating the custom model? Also, for height, what should I enter if the view is slowing downward? For example, several of my cameras fixed to the front of my house, but my house sits up a hill from the street so even though the cameras are about 7 feet above the ground where they are located, they are more like 15 feet above the sidewalk. What height should I use? |
Beta Was this translation helpful? Give feedback.
-
|
I use them as a way to try and select diverse images for the base model training. In the future, I plan to try and cluster users with similar camera configurations and have multiple versions of the base model. Use the height where they are located even though it's on a downward slope. |
Beta Was this translation helpful? Give feedback.
-
|
Just came to say that the suggestions feature is great. I have been uploading more images in the background as I see issues but I haven't been on to verify for a while as it has been working so well. I was very pleased to see the suggestions feature and it will likely lead to me uploading even more now that I don't have as much work to do on the sea of cars etc in the frame. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I apologize if this question has been asked, but I am not seeing anything on this issue. In Frigate 0.14.1, when I go to the |
Beta Was this translation helpful? Give feedback.
-
|
In 0.14 you just click on the items on the plus page and then choose is / is not that object type. in 0.15 the explore tab you can use filters to filter on events that have snapshots, then just use the UI in exactly the same way |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @NickM-27 I realize that I haven't enabled snapshots in my |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 and @blakeblackshear -- I think I've found a bug in the Frigate+ functionality within Frigate 0.14.1. I am using Microsoft Edge browser to access my Frigate instance. When I go to the |
Beta Was this translation helpful? Give feedback.
-
|
yes, this is known, fixed in 0.15 |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 Excellent! |
Beta Was this translation helpful? Give feedback.
-
|
@blakeblackshear Has the zoom function in the annotation tool changed recently? Zooming with my laptop's touchpad is so slow it's barely usable. |
Beta Was this translation helpful? Give feedback.
-
|
It hasn't changed. It works better with a mouse wheel. Not sure why it's slow with a touchpad. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW I find it scrolls quickly on a touchpad. Firefox on Apple M2 if it matters. Wonder if it's browser specific @tpjanssen ? |
Beta Was this translation helpful? Give feedback.
and others
-
The base model in Frigate+ has now been updated to 2024.3. All model requests going forward will use this version. In order to get a new model, you will need to submit a model request.
The 2024.3 base model update was focused on expanding the supported labels with the following:
New delivery logos will not populate the objects sub label until 0.15. I made a lot of changes under the hood to make it easier to add more labels in the future, so this is just the beginning.
Many of you will now have unlabeled objects in your verified images. For those with a large number of images missing the new labels, your model will probably unlearn these new labels during fine tuning until enough of the labels are added.
These new labels are not added automatically to your cameras. When you add new labels to your cameras, your existing images will no longer be verified for that camera and you will be able to go back through them. I suggest adding the new labels 1 or 2 at a time so you can focus on looking for a specific label. Note that it will skip to the next image within the same camera when verifying, so you will review one camera at a time.
I fully appreciate that this is time consuming. I delayed this update until I had implemented a portion of the changes to support suggested labels, so that is coming next. In addition, camera/label specific workflows and static suggestions are still in the works.
edit: I have rolled out some changes to the way verified images are indicated as well as some filtering options in the images view
Beta Was this translation helpful? Give feedback.
All reactions