Real Words or Buzzwords?: Computer Vision - RBCS - Ray Bernard Consulting Services

This is the 54th article in the award-winning “Real Words or Buzzwords?” series about how real words become empty words and stifle technology progress, also published on SecurityInfoWatch.com.

By Ray Bernard, PSP, CHS-III

Why a good understanding of the computer vision concept is important for evaluating today’s security video analytics products.

All-in-one RWOB

#1 Proof of the buzzword that killed tech advances in the security industry—but not other industries.

#2 Next Generation (NextGen): A sure way to tell hype from reality.

#3 Customer Centric: Why all security industry companies aren't customer centric.

#4 Best of Breed: What it should mean to companies and their customers.

#5 Open: An openness scale to rate platforms and systems

#6 Network-friendly: It's much more than network connectivity.

#7 Mobile first: Not what it sounds like.

#8 Enterprise Class (Part One): To qualify as Enterprise Class system today is world's beyond what it was yesterday.

#9 Enterprise Class (Part Two): Enterprise Class must be more than just a top-level label.

#10 Enterprise Class (Part Three): Enterprise Class must be 21st century technology.

#11 Intuitive: It’s about time that we had a real-world testable definition for “intuitive”.

#12 State of the Art: A perspective for right-setting our own thinking about technologies.

#13 True Cloud (Part One): Fully evaluating cloud product offerings.

#14 True Cloud (Part Two): Examining the characteristics of 'native-cloud' applications.

#15 True Cloud (Part Three): Due diligence in testing cloud systems.

#16 IP-based, IP-enabled, IP-capable, or IP-connectable?: A perspective for right-setting our own thinking about technologies.

#17 Five Nines: Many people equate high availability with good user experience, yet many more factors are critically important.

#18 Robust: Words like “robust” must be followed by design specifics to be meaningful.

#19 Serverless Computing – Part 1: Why "serverless computing" is critical for some cloud offerings.

#20 Serverless Computing – Part 2: Why full virtualization is the future of cloud computing.

#21 Situational Awareness – Part 1: What products provide situational awareness?

#22 Situational Awareness – Part 2: Why system designs are incomplete without situational awareness?

#23 Situational Awareness – Part 3: How mobile devices change the situational awareness landscape?

#24 Situational Awareness – Part 4: Why situational awareness is a must for security system maintenance and acceptable uptime.

#25 Situational Awareness – Part 5: We are now entering the era of smart buildings and facilities. We must design integrated security systems that are much smarter than those we have designed in the past.

#26 Situational Awareness – Part 6: Developing modern day situational awareness solutions requires moving beyond 20th century thinking.

#27 Situational Awareness – Part 7: Modern day incident response deserves the help that modern technology can provide but doesn’t yet. Filling this void is one of the great security industry opportunities of our time.

#28 Unicity: Security solutions providers can spur innovation by envisioning how the Unicity concept can extend and strengthen physical access into real-time presence management.

#29 The API Economy: Why The API Economy will have a significant impact on the physical security industry moving forward.

#31 The Built Environment: In the 21st century, “the built environment” means so much more than it did just two decades ago.

#32 Hyper-Converged Infrastructure: Hyper-Converged Infrastructure has been a hot phrase in IT for several years, but do its promises hold true for the physical security industry?

#33 Software-Defined: Cloud-computing technology, with its many software-defined elements, is bringing self-scaling real-time performance capabilities to physical security system technology.

#34 High-Performance: How the right use of "high-performance" can accelerate the adoption of truly high-performing emerging technologies.

#35 Erasure Coding: Why RAID drive arrays don’t work anymore for video storage, and why Erasure Coding does.

#36 Presence Control: Anyone responsible for access control management or smart building experience must understand and apply presence control.

#37 Internet+: The Internet has evolved into much more than the information superhighway it was originally conceived to be.

#38 Digital Twin: Though few in physical security are familiar with the concept, it holds enormous potential for the industry.

#39 Fog Computing: Though commonly misunderstood, the concept of fog computing has become critically important to physical security systems.

#40 Scale - Part 1: Although many security-industry thought leaders have advocated that we should be “learning from IT,” there is still insufficient emphasis on learning about IT practices, especially for large-scale deployments.

#41 Scale - Part 2: Why the industry has yet to fully grasp what the ‘Internet of Things’ means for scaling physical security devices and systems.

#42 Cyberspace - Part 1: Thought to be an outdated term by some, understanding ‘Cyberspace’ and how it differs from ‘Cyber’ is paramount for security practitioners.

#43 Cyber-Physical Systems - Part 1: We must understand what it means that electronic physical security systems are cyber-physical systems.

#44 Cyberspace - Part 2: Thought to be an outdated term by some, understanding ‘Cyberspace’ and how it differs from ‘Cyber’ is paramount for security practitioners.

#45 Artificial Intelligence, Machine Learning and Deep Learning: Examining the differences in these technologies and their respective benefits for the security industry.

#46 VDI – Virtual Desktop Infrastructure: At first glance, VDI doesn’t seem to have much application to a SOC deployment. But a closer look reveals why it is actually of critical importance.

#47 Hybrid Cloud: The definition of hybrid cloud has evolved, and it’s important to understand the implications for physical security system deployments.

#48 Legacy: How you define ‘legacy technology’ may determine whether you get to update or replace critical systems.

#49 H.264 - Part 1: Examining the terms involved in camera stream configuration settings and why they are important.

#50 H.264 - Part 2: A look at the different H.264 video frame types and how they relate to intended uses of video.

#51 H.264 - Part 3: Once seen as just a marketing term, ‘smart codecs’ have revolutionized video compression.

#52 Presence Technologies: The proliferation of IoT sensors and devices, plus the current impacts of the COVID-19 pandemic, have elevated the capabilities and the importance of presence technologies.

#53 Anonymization, Encryption and Governance: The exponential advance of information technologies requires an exponential advance in the application of data protection.

#54 Computer Vision: Why a good understanding of the computer vision concept is important for evaluating today’s security video analytics products.

#55 Exponential Technology Advancement: The next 10 years of security technology will bring more change than in the entire history of the industry to now.

#56 IoT and IoT Native: The next 10 years of security technology will bring more change than in the entire history of the industry to now.

#57 Cloud Native IoT: The next 10 years of security technology will bring more change than in the entire history of the industry to now.

#58 Bluetooth vs. Bluetooth LE: The next 10 years of security technology will bring more change than in the entire history of the industry to now.

More to come about every other week.

Computer Vision

Just a few weeks ago, Amazon Web Services (AWS) announced their release of a new technology for video cameras called Panorama, which surprisingly has nothing to do with panoramic cameras (i.e. 360 degree cameras). AWS refers to Panorama as computer vision (CV) technology, and right away I asked myself: Is there actually a technical basis for calling a security video analytics technology “computer vision?” Or, is this just another marketing hype term that is going to add more confusion into the world of security video analytics?

About a year ago, in a Real Words or Buzzwords? article titled, Artificial Intelligence, Machine Learning and Deep Learning, I said that future articles will discuss more aspects of AI, ML and DL as products using them continue to emerge. This is one such article. In that article I mentioned that the history of AI research goes like this:

Artificial Intelligence – 1950s to 1980s and still ongoing
– –> Machine Learning – 1980s to 2010s and still ongoing
– –> – –> Deep Learning – 2010s to now and still ongoing

As Wikipedia explains: In the late 1960s, computer vision began at universities which were pioneering artificial intelligence. It was meant to mimic the human visual system, as a stepping stone to endowing robots with intelligent behavior. In 1966, it was believed that this could be achieved through a summer project, by attaching a camera to a computer and having it “describe what it saw”. What distinguished computer vision from the prevalent field of digital image processing at that time was a desire to extract three-dimensional structure from images with the goal of achieving full scene understanding.

“Full scene understanding” is another way of saying “situational awareness”, an important and also broad topic covered in a series of seven Real Words or Buzzwords? articles (Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, and Part 7). Situational awareness applies in a great many fields, including sports, for which there is sport-specific situational awareness training. It’s critically important to autonomous driving, including for anti-collision and other vehicle safety features, for which a combination of sensors, including radar and video analytics are applied.

Security Video Analytics Hype Factor

It is the recent advances in deep learning that have attracted billions of dollars of annual investment in video analytics and has resulted in very significant improvements in video analytics capabilities, which has been good news for the security video industry. However, advances in video analytics have historically been a good-news/bad-news situation for the security industry, which has already experienced two over-hyped phases of video analytics, with a third new phase already under way.

These three phases evolved from the three phases of computer vision research listed above: early artificial intelligence, machine learning, and now deep learning. The previous two phases of security video analytics were both over-hyped, and so security industry end user customers, as well as security service providers, are now doubly cautious about current security video analytics claims.

From Hype to Reality

There are several major differences today from the earlier decades of security video analytics, including the fact that there is now very good reason for referring to video analytics applications as computer vision applications, as AWS is now doing. First, security video in general is benefiting from the related simultaneous IT megatrends of continuing technology cost reduction and continuing exponential increase in technology capabilities. These are explained in the short but meaty article by Pivot 3 in this month’s Security Business magazine, that highlights the modern technologies that are simplifying the design, deployment and scaling of intelligent video analytics deployments.

Second – as explained in another Security Business article, AI and Specialty Analytics are Changing Video Surveillance – various categories of specialty analytics now make offer tremendous value both for security and safety situational awareness and for business operations purposes. The emergence of specialty analytics is made possible by advances in computing and data storage technology for devices, on-site equipment, and in the cloud.

Third, advances in free open source and licensed proprietary deep learning software have enabled the development of a wide range of specialty analytics – and this is what AWS, Microsoft Azure, Bosch and others now have a strong focus on.

Computer Vision and Video Analytics

As explained in the white paper that I wrote for the Security Industry Association titled, The State of Video Security Analytics (PDF), the basic definition of video analytics, also known as video content analysis (VCA) in the academic world, is the extraction of meaningful and relevant information from digital video. Figure 1 below shows how the world of academics – where most of the AI/ML/DL research is done – views computer vision and video content analysis (what we call video analytics). This is how areas of research are labeled, not products.

Figure 1. Artificial Intelligence and Computer Vision

When it comes to products, it’s easy to understand why the video security industry used the “video analytics” label as opposed to “computer vision” for its products. At the time that security video analytics were introduced, machine vision’s widely in use in manufacturing, where automated production lines were able to run at high speed through the use of computer vision. Initially simple light sensors were used, for example, to detect the presence or absence of boxes and other objects on a production line. Computer vision provided optical character recognition whereby part and product labels could be read automatically, and computers could manage production line activity and log the tracked results.

These and other manufacturing applications were in the general public’s thinking when it came to computer vision. Thus, it made sense to call video security’s use of computer vision “video analytics.” At that time, simple pattern recognition was used for video analytics, and machine learning and deep learning were not involved.

As AI advanced and evolved to the state shown in Figure 1, the term video analytics continued to apply to the security video industry products and there was no reason to change it. However, AWS is very strongly involved in providing software tools and cloud computing platforms for machine learning, deep learning and the broader field of computer vision, and so has very good reasons to label its Panorama technology in the computer vision category, as it is narrowly focused use of the broader AI tools and computing platforms it offers.

Additionally, Panorama technology consist of much more than what security video analytics offerings have provided in the past. Video analytics are a part of the larger Panorama technology offering. And AWS’s purpose for Panorama is not to provide video analytics products per se, but to provide the capabilities for developing them and the technology that enables running them on devices (cameras), computers on site and in the cloud.

This is a similar purpose to Security & Safety Things (S&ST) initiative from Bosch, with a significantly greater cloud technology aspect, as the Panorama appliance contains hardware elements specifically designed to run deep learning applications, and its software has built-in connectivity to the AWS and Azure cloud environments. More will be said about such specialized hardware capabilities in an upcoming Real Words or Buzzwords? article.

What Is AWS Panorama?

The Panorama technology consists of firmware, software, a software development kit (SDK) and an appliance for running what AWS calls “computer vision applications”, and is intended to enable the running of Panorama-based machine learning (ML) video analytics on cameras that have the Panorama technology embedded in them, or on the Panorama appliance on site – close to where the cameras, or in the cloud hosted on AWS in Microsoft Azure.

Since Panorama’s purpose is to foster the development of high-value niche video applications and their deployment, it’s perfectly fine to categorize the technology category as “video analytics.” But at the same time, it’s important to realize that Panorama provides much more than what we have seen in the past from security video industry video analytics vendors.

For more details about the AWS Panorama technology, see the Security Business article titled, Is AWS Panorama a Game-Changer for Video Analytics? [Joel: link needed here.]

An Expanding Category of End-User Applications

An important final point is that today’s deep learning based analytics are model-based. It’s not the old days where applications were simply software code. Most of the AI software code already exists, and the important work is in gathering and selecting the data for training, and in selecting the right training models.

Students in the field have a wealth of open source resources available to them that give them knowledge and experience directly applicable to real world applications. Many students work on real works applications prior to graduation. Although much of the academic work available online is steeped in the AI domain’s vocabulary, the technology is very understandable, especially at a high level.

Application development involves selecting the right machine learning and deep learning models, and then selecting and/or developing the right training sets so that the initial deep learning training enables the analytics applications to continue leaning in the directions intended.

More information about this is available on the Machine Learning Mastery website, which has a section titled, Start Machine Learning. It contains a wealth of information of the kind formerly never available to interested learners. One article is titled, How to Go From Working in a Bank To Hired as Senior Data Scientist at Target. Another is, 14 Different Types of Learning in Machine Learning – which is a roughly 7,000-word introduction to the subject with references to more advanced information.

I’m not suggesting that readers dive right into the technical details. The point is that AI is a field of study that is growing exponentially in popularity and in its wealth of available online knowledge and tools. Students skilled in deep learning application development are a new kind of resource that is now available to security integrators and to large end-user companies who have an internal technical team.

For example, Amazon Protective Services (high-level corporate security group) has their own software development team that has been working with AWS AI technology for years. For the earlier in-person AWS re:Invent event, the group developed vehicle tracking analytics that used people and vehicle counting, and GPS information, to determine how many seats were still available on which shuttles and buses. This enable their event’s mobile app to help attendees at their choose the best mode of transportation to local venues.

AWS Panorama and S&ST platforms and technology, coupled with a new generation of AI technologists, provide integrators with a way to address customer-specific important analytics use cases. These are the kinds of high-value analytics that provide continuing benefits to end-user customers who are happy to subscribe to them as an incremental expansion of their security system capabilities.

Ray Bernard, PSP CHS-III, is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities (www.go-rbcs.com). He is the author of the Elsevier book Security Technology Convergence Insights available on Amazon. Mr. Bernard is a Subject Matter Expert Faculty of the Security Executive Council (SEC) and an active member of ASIS International.