Using Video Analytics with Pan-Tilt-Zoom Cameras

(An earlier version of this article first appeared in Remote Magazine)

Although available for many years, it is not until recently that video analytics has been more widely deployed for use on in-motion PTZ cameras

Although available for many years, it is not until recently that video analytics has been more widely deployed for use on in-motion PTZ cameras

A common misunderstanding in today’s security world is that video analytics is restricted to fixed cameras, when in fact; the ability to utilize video analytics on in-motion pan-tilt-zoom (PTZ) cameras has been available for many years.  It is not until recently, however, that the capability to perform video detection while a PTZ camera is in full motion has become more readily available to security integrators and dealers.   This capability, also referred to as motion detection while scanning, has been slow to commercialization partly due to the complexity of the solution, and the limited number of manufacturers that have addressed the technical hurdles.  Today, there are several commercially viable solutions on the market, opening a new era of automated video surveillance and detection.

Video Analytics is the ability to analyze a live or recorded video feed and intelligently extract useful data concerning what is happening in the scene.  This is a widely used capability for the protection of critical facilities and assets.  Types of video analytics vary greatly, from simple detection, object classification, counting, monitoring for removed objects to loitering detection.  The ability to utilize these algorithms to automatically provide insight to security personnel has proven to be a valuable tool.  Although there are many types of video analytics available, most algorithms used in today’s security environment have been limited to fixed cameras and fixed scenes.   This is unfortunate for security personnel as the PTZ camera continues to remain one of the most powerful tools in the security arsenal.  This is partly due to the camera’s ability to point in any direction and at many zoom levels, allowing the security person to cover a large area without the need for many additional fixed sensors.  This is an important capability, as a vast amount of any facility is typically not covered with full time video surveillance.  This continued reliance on the PTZ camera has driven an increased demand to utilize video analytics on in-motion PTZ cameras.  Although many different solutions are available, in-motion video analytics remain a complex task with several technical hurdles.

To insure high levels of accuracy, video analytics require some knowledge of the video scene.

To insure high levels of accuracy, video analytics require some knowledge of the video scene.

Understanding Frame of Reference: At its most basic premise, detection using video analytics is about understanding changes in pixels.  However, to avoid alarming on pixel changes due to lighting, shadows or falling leaves, video analytic algorithms need to understand some details of the scene.  Understanding that a pixel, or several pixels, represents a very small object (a leaf) or a large object (a vehicle), is critical to understanding what is a real item of interest, versus nuisance motion.  Knowledge of the horizon, or geospatial data, is also extremely useful.  The amount of scene information required varies greatly over the spectrum of video analytic solutions, but in all cases, some details of the scene must be communicated to the algorithm during setup.

The technical challenge occurs when the scene changes.  When utilizing a PTZ camera, every change in pan, tilt or zoom is the equivalent to a scene change.  This requires new scene data to be provided to the video analytics algorithm.  When in motion, this equates to an almost continuous change in scenes.  In order for the video analytics algorithm to be intelligent when detecting potential targets, details of the constantly changing scene must be automatically provided in real time, without user intervention.

Background Modeling: Another challenge which needed to be addressed with in-motion video analytics, was a change in background modeling.  Video Analytics can range from very simple pixel change detection to sophisticated facial ID and long range object classification.  For security type applications, video analytics typically employ the concept of background modeling or learning.  This is the ability to understand the overall nature of the scene; the trees, the vegetation, the lighting, and how these change over time.  Background modeling results in lower false alarm rates and higher levels of target discrimination, such as movement, size and object classification.  When a camera is moving continuously, the background is also constantly changing, and as a result, it is difficult to continuously learn the new background. This means more advanced algorithms needed to be deployed to deal with constantly changing backgrounds.

Camera Auto Tracking automatically adjusts the pan, tilt, and zoom of a PTZ camera to keep the subject centered in the video frame

Camera Auto Tracking automatically adjusts the pan, tilt, and zoom of a PTZ camera to keep the subject centered in the video frame

Higher performing CPUs and the creation of more efficient algorithms can both be attributed to making in-motion video analytics more commercially available.  As a result, there are many in-motion video analytics solutions available from several different suppliers.  Some of these capabilities are cited below.

Camera Auto Tracking: When presented with a video scene, a camera auto tracking video analytics will detect one or multiple targets and continually adjust the pan, tilt, and often the zoom, of the PTZ camera to keep the subjects centered in the video frame.  The ability to control the camera automatically allows the security person to perform other tasks relating to the event.   The expansion of this capability is the ability to place a camera on a mobile platform, such as a drone, and allow the camera to remain fixed on a target while the vehicle is maneuvering.

"Scan to Target" searches for a target while the PTZ camera is scanning an area of interest

“Scan to Target” searches for a target while the PTZ camera is scanning an area of interest

Scan to Target: In many security systems, PTZs can be manually or automatically steered to a location of a potential target.  In many cases, once the PTZ has steered to the new scene, the target may no longer be visible in the field of view.  This may be due to a variety of reasons: a fast moving target, slight errors in the pointing coordinate or differences in mounting precision.  “Scan to Target” is a video analytic capability that steers the camera in a dynamically determined search pattern, and simultaneously attempts to detect the target while the camera is being moved.  Once the target is detected, it will terminate the search and remain locked on the target using camera auto tracking.

Motion Detection while Scanning – In much the same way a spot light scans a prison wall at night, security professionals desire the ability to scan a PTZ and have it simultaneously search for potential targets.  This is not to be confused with a “PTZ Guard Tour”, whereby the PTZ goes to several pre-defined scenes and analyzes each scene before systematically moving to the next.  In a true random search, the PTZ can be commanded to any pan, tilt and zoom combination and it will perform accurate motion detection within the scene.  As with its fixed camera counterpart, this is done in such a manner to ignore moving items like leaves, birds and lighting effects.   This capability is already used in more sophisticated implementations like border surveillance and open water situations, but it continues to become more affordable and available for all types of security installations.

The advancement of technology related to live and recorded video continues to increase at a rapid pace.  This is evident in the security industry and in the use of video analytics to help automate surveillance tasks.  In the same way video analytics has become a valuable tool when implemented on fixed cameras, the ability to utilize these capabilities on PTZ cameras offers a new set of tools to aid in the protection of critical assets and facilities.

This entry was posted in Geospatial, Perimeter Surveillance, Security Sensor Integration, Video Analytics, VMS and tagged , , , , , , , , , , , , , , , , , , , , . Bookmark the permalink.