Understanding Object Detection in Videos with Monitoring and Alerting

Introduction

In a previous post, we demonstrated that EyesOnIt supports object detection in individual images and object detection in videos with monitoring and alerting. We discussed that both modes are supported by the configuration UI and by the REST API. The previous post covered object detection in individual images through the configuration UI. This post will cover object detection in videos with monitoring and alerting, also in the context of the configuration UI. Future posts will cover the use of the REST API for image processing and video monitoring.

The EyesOnIt configuration UI provides a quick and easy way to set up video monitoring and alerting without writing any code. Like all EyesOnIt usage scenarios, video monitoring and alerting are based on text descriptions for object detection. The UI also allows users to configure alerting confidence thresholds, alert trigger times, alert reset times, and SMS notification options.

To access the video configuration UI, launch EyesOnIt in your web browser. If you started the EyesOnIt Docker container with the default port (port 8000), then you can open a browser on that same computer and navigate to http://localhost:8000/dashboard. If you changed the port or if the browser is running on a different computer, you will have to modify that URL slightly. Please contact us if you need help.

EyesOnIt Dashboard Home Page

On the EyesOnIt Dashboard, you’ll see these choices:

Choose the Video option to see the Video configuration UI.

EyesOnIt Dashboard Video Configuration UI

When you first open the video configuration UI, you’ll see an empty stream list with the option to add a stream:

Adding a Stream

Click the green button to add a stream. You will then see the input UI for the basic stream properties.

For the RTSP URL, enter the RTSP URL for your video stream. You may need to consult the documentation for your camera or video management system to find this URL. Sometimes this URL contains credentials. EyesOnIt needs those credentials to access the video stream. Again, if you need help figuring out your RTSP URL, please contact us.

For the stream friendly name, just enter something to help you remember which video stream you’re looking at. Each video streams should have a unique friendly name.

After you enter these inputs, hit the blue arrow button to get the latest frame from the video. Your page should look something like this:

The video configuration UI allows you to tile and mask the input stream just like you can with an individual image. You can use the scrollbars to create tiles in rows and columns, and then you can select the tiles for your mask. We’re going to detect vehicles on the driveway when they stop in front of the sidewalk, so we’ll create our tile mask as shown here:

Now we’re ready to add object descriptions and other inputs to configure monitoring and alerting. Video monitoring and alerting starts with the same text descriptions that we used to detect objects in individual images. These text descriptions describe the objects you want to detect. You can also optionally describe the background if there are periods in your video when none of the objects you want to detect will be present.

To configure alerting, you need to provide the following additional pieces of information:

1. Alerting confidence levels for each object description

2. Alert trigger and reset time intervals

3. SMS notification options

For our scenario of detecting different types of vehicles on the driveway, we’re going to use these settings:

Let’s look at each of these settings in detail.

Alerting Confidence Levels for Each Object Description

For each object description, you will specify the alerting threshold. This is the minimum confidence level that will trigger an alert. You can read more about confidence levels here. When you specify your alert threshold, choose a value that is high enough to avoid false alerts but low enough to include correct detections. We’ll show you a tool that will help you choose this threshold later in this post.

Alert Trigger and Reset Intervals

EyesOnIt lets you configure the amount of time that you want the confidence level for your object to remain above the alert threshold before the alert is actually sent. This can help avoid false alerts caused by momentary conditions that don’t meet the criteria for an alert. It also gives you options for scenarios where you don’t want an alert until the condition has persisted for an extended period. The value for the “Seconds Above Threshold” setting can be between 0.1 seconds and 3600 seconds (one hour). Keep in mind that while this setting will help reduce false alerts, it will also introduce latency into your alerts. Make sure you keep this value below the maximum latency that you can tolerate for your scenario.

You can also tell EyesOnIt to wait for an amount of time before resetting the alarm state. This setting can help reduce nuisance alerts, which are technically correct but unnecessary because you have already been alerted of the situation. For example, if you are using EyesOnIt to detect a trespasser, you will likely investigate when you get an alert. Maybe you expect your investigation to take 3 minutes. You could set the “Seconds Below Threshold” setting to 180 seconds to ensure that you wouldn’t get more alerts while you are investigating the first one. EyesOnIt would then wait until the confidence level for your object description has remained below the alert threshold that you specified for the Seconds Below Threshold time before it considers sending additional alerts. Just as with the Seconds Above Threshold time, the Seconds Below Threshold time can be between 0.1 and 3600 seconds.

SMS notification options

EyesOnIt reports alerts in the dashboard UI. It can also report alerts through SMS. Future releases of EyesOnIt will send alerts through REST APIs for improved integration with common video management systems. To receive an SMS alert from EyesOnIt, enter your phone number in the form of +11234567890. In other words, you need to include the country prefix (like +1 for the US and Canada) with the rest of your number.

If you want to receive an image with your text message, check the box in the UI.

Please note that there are costs associated with receiving SMS alerts. Please see the EyesOnIt pricing page for these costs.

The previous image shows our input settings to detect a mail truck, a UPS truck, or an Amazon delivery truck. We set the alert threshold to 95% for each of these object types. We indicated that we wanted to observe the confidence level above 95% for at least one second before sending an alert and that we wanted to observe the confidence below 95% for at least one second before considering sending more alerts. The phone number entered for alerts is obviously just an example.

EyesOnIt Dashboard Video Stream List

When we hit submit, EyesOnIt will validate our inputs and provide suggestions if needed. Once the inputs have been accepted, we’ll see that our video stream has been added to the stream list.

For each video stream in the stream list, there are several actions that we can perform:

Start monitoring the stream.

When a stream is being monitored, this button changes to pause monitoring.

Discard the stream settings.

Edit the stream settings.

View the current frame from the video.

View a chart showing the confidence levels for each object description.

After we add or change the monitoring and alerting settings, EyesOnIt waits to start monitoring the stream until we specifically tell it to do so by hitting the “Start” button.

The confidence chart can only be shown while the stream is being monitored. When you open the confidence chart, you’ll see something like this:

The confidence chart shows confidence levels for each object description over the last minute. This chart is a helpful tool to determine what confidence levels to use for reliable alerts. As you can see, the confidence level for each vehicle is around 99% when that vehicle is present. When any vehicle is present, the confidence levels for all the vehicle types is greater than zero, because each vehicle shares some similar visual attributes.

From the chart, you can see that an alerting threshold of 50% would generate a false alert for the blue Amazan delivery van when the brown United Parcel Service truck is actually present at the -19 second mark. It appears that an alerting threshold of 80% would accurately identify each vehicle type. You will likely need to fine-tune your alert thresholds as you test your monitoring inputs. Different lighting conditions may require different object descriptions. Using the REST API, you could easily detect the lighting conditions and change your prompts programmatically.

EyesOnIt Dashboard Video Alerts

The last topic we’ll cover is the alerts that appear within the dashboard UI. Each time the alert conditions are met, EyesOnIt will show the alert details in the Alerts list. The details include the time of the alert, the object description that triggered the alert, the confidence level for that object description when the alert was triggered, and a picture from the video stream taken at the time the alert was triggered.

Conclusion

EyesOnIt monitoring and alerting is not only extremely powerful, but it can be configured quickly and easily without any programming. This post has provided some ideas about how to get started. We’d love to hear about how you are using EyesOnIt. Please send us an email if you have ideas that you would like to share or discuss. Or, if you need further help, please see our user guide, or contact us.