This is the 40th article in the award-winning “Real Words or Buzzwords?” series about how real words become empty words and stifle technology progress.
By Ray Bernard, PSP, CHS-III
Although many security-industry thought leaders have advocated that we should be “learning from IT,” there is still insufficient emphasis on learning about IT practices, especially for large-scale deployments.
★ ★ ★ GET NOTIFIED! ★ ★ ★
SIGN UP to be notified by email the day a new Real Words or Buzzwords? article is posted!
Real Words or Buzzwords?
The Award-Winning Article Series
#1 Proof of the buzzword that killed tech advances in the security industry—but not other industries.
#2 Next Generation (NextGen): A sure way to tell hype from reality.
#3 Customer Centric: Why all security industry companies aren't customer centric.
#4 Best of Breed: What it should mean to companies and their customers.
#5 Open: An openness scale to rate platforms and systems
#6 Network-friendly: It's much more than network connectivity.
#7 Mobile first: Not what it sounds like.
#8 Enterprise Class (Part One): To qualify as Enterprise Class system today is world's beyond what it was yesterday.
#9 Enterprise Class (Part Two): Enterprise Class must be more than just a top-level label.
#10 Enterprise Class (Part Three): Enterprise Class must be 21st century technology.
#11 Intuitive: It’s about time that we had a real-world testable definition for “intuitive”.
#12 State of the Art: A perspective for right-setting our own thinking about technologies.
#13 True Cloud (Part One): Fully evaluating cloud product offerings.
#14 True Cloud (Part Two): Examining the characteristics of 'native-cloud' applications.
#15 True Cloud (Part Three): Due diligence in testing cloud systems.
#16 IP-based, IP-enabled, IP-capable, or IP-connectable?: A perspective for right-setting our own thinking about technologies.
#17 Five Nines: Many people equate high availability with good user experience, yet many more factors are critically important.
#18 Robust: Words like “robust” must be followed by design specifics to be meaningful.
#19 Serverless Computing – Part 1: Why "serverless computing" is critical for some cloud offerings.
#20 Serverless Computing – Part 2: Why full virtualization is the future of cloud computing.
#21 Situational Awareness – Part 1: What products provide situational awareness?
#22 Situational Awareness – Part 2: Why system designs are incomplete without situational awareness?
#23 Situational Awareness – Part 3: How mobile devices change the situational awareness landscape?
#24 Situational Awareness – Part 4: Why situational awareness is a must for security system maintenance and acceptable uptime.
#25 Situational Awareness – Part 5: We are now entering the era of smart buildings and facilities. We must design integrated security systems that are much smarter than those we have designed in the past.
#26 Situational Awareness – Part 6: Developing modern day situational awareness solutions requires moving beyond 20th century thinking.
#27 Situational Awareness – Part 7: Modern day incident response deserves the help that modern technology can provide but doesn’t yet. Filling this void is one of the great security industry opportunities of our time.
#28 Unicity: Security solutions providers can spur innovation by envisioning how the Unicity concept can extend and strengthen physical access into real-time presence management.
#29 The API Economy: Why The API Economy will have a significant impact on the physical security industry moving forward.
#31 The Built Environment: In the 21st century, “the built environment” means so much more than it did just two decades ago.
#32 Hyper-Converged Infrastructure: Hyper-Converged Infrastructure has been a hot phrase in IT for several years, but do its promises hold true for the physical security industry?
#33 Software-Defined: Cloud-computing technology, with its many software-defined elements, is bringing self-scaling real-time performance capabilities to physical security system technology.
#34 High-Performance: How the right use of "high-performance" can accelerate the adoption of truly high-performing emerging technologies.
#35 Erasure Coding: Why RAID drive arrays don’t work anymore for video storage, and why Erasure Coding does.
#36 Presence Control: Anyone responsible for access control management or smart building experience must understand and apply presence control.
#37 Internet+: The Internet has evolved into much more than the information superhighway it was originally conceived to be.
#38 Digital Twin: Though few in physical security are familiar with the concept, it holds enormous potential for the industry.
#39 Fog Computing: Though commonly misunderstood, the concept of fog computing has become critically important to physical security systems.
#40 Scale - Part 1: Although many security-industry thought leaders have advocated that we should be “learning from IT,” there is still insufficient emphasis on learning about IT practices, especially for large-scale deployments.
#41 Scale - Part 2: Why the industry has yet to fully grasp what the ‘Internet of Things’ means for scaling physical security devices and systems.
#42 Cyberspace - Part 1: Thought to be an outdated term by some, understanding ‘Cyberspace’ and how it differs from ‘Cyber’ is paramount for security practitioners.
#43 Cyber-Physical Systems - Part 1: We must understand what it means that electronic physical security systems are cyber-physical systems.
#44 Cyberspace - Part 2: Thought to be an outdated term by some, understanding ‘Cyberspace’ and how it differs from ‘Cyber’ is paramount for security practitioners.
#45 Artificial Intelligence, Machine Learning and Deep Learning: Examining the differences in these technologies and their respective benefits for the security industry.
#46 VDI – Virtual Desktop Infrastructure: At first glance, VDI doesn’t seem to have much application to a SOC deployment. But a closer look reveals why it is actually of critical importance.
#47 Hybrid Cloud: The definition of hybrid cloud has evolved, and it’s important to understand the implications for physical security system deployments.
#48 LegacyHow you define ‘legacy technology’ may determine whether you get to update or replace critical systems.
#49 H.264 - Part 1Examining the terms involved in camera stream configuration settings and why they are important.
#50 H.264 - Part 2A look at the different H.264 video frame types and how they relate to intended uses of video.
More to come about every other week.
This article covers the seldom-spoken-of conflict of interest issues, large-scale pain points, and the misleading statements still to be found in product A&E Specifications that relate to device manageability for large-scale security system deployments.
Conflicts of Interests
There is also a conflict-of-interest issue involved that could be considered a holdover from the 1980s and early 1990s – when the amount of a systems integrator’s revenue and employee headcount was dependent in part upon how many cable home runs were needed for a security system installation. This was part of the initial reluctance to adopt network-based physical security systems, especially those that utilized a customer’s existing IT network. That kind of deployment was an employee job threat for the integrators who depended on the money earned by cable installation work.
That same conflict of interest shadow discouraged some integrators from adopting IT tools and remote system access in favor of maintaining truck rolls for all service calls. Obviously that approach was not in the best interest of customers, and so now the use of diagnostic tools and remote service tools is becoming the prevailing model.
Because manageability at scale has not been a major concern of security industry manufacturers, the integrator revenue from any deployment project scaled linearly with the amount of equipment being installed, especially for video system deployments. Typically, video cameras have required a significant amount of manual data entry for each camera.
Management-at-scale features change the total cost of ownership (TCO) picture for end user customers, as well as the Total Cost to Serve (TCS) picture for security service providers. Customers get more value for their money, and service providers can scale deployment size using force-multiplier technology that increases their profits. For more on this aspect see these articles: Total Cost of Ownership and Are Integrators Overlooking Total Cost to Serve?.
Large Scale Pain Points
The support for multiple device configuration varies across access control and other systems. There are systems that, for example, allow spreadsheet configuration data import. I’m not trying to provide a detailed overview of the product landscape – I’m merely pointing out by example that large-scale device management has not been given enough attention in the industry. However, you can find an overview of the IP camera firmware update landscape in a freely-available IPVM article that provide a directory of the status of firmware upgrades for 15 major brands of cameras. The article documents the firmware installation times for each brand.
Intelligent network video cameras are the worst-case example of device management pain.
Most customers with high-camera-count video deployments don’t update the camera firmware, due to the manual labor cost previously involved in updating firmware (15 to 30 minutes per camera) and the problems that could occur during firmware updates, some of which resulted in bricking the camera (making the camera inoperable). Furthermore, many camera manufacturers – at some point – had firmware updates that required returning the camera to factory default settings, meaning that each camera had to be reconfigured manually after the firmware upgrade.
And it’s still the case that there is a version-dependency issue for cameras that provide a file download of the camera’s configuration settings. The configuration file can only be uploaded to the same version of firmware from which it was downloaded. Camera configuration backup doesn’t survive a version upgrade. When I brought this to the attention of Geoffrey Bauer, the manager of the A&E Program at Axis Communications, he told me that the configuration backups should survive across most if not all of the minor version upgrades, but currently not across major firmware upgrades. Download the Axis firmware upgrade program document here, which does not yet include any information on configuration backup compatibility. Reps from a few other camera vendors I asked were not aware of this issue.
I’m currently lobbying for camera manufacturers to implement camera configuration backups in a standard way that would enable configuration restoration to survive firmware changes. It’s long overdue.
In a recent SIA Cyber Office Hours webinar, Rodney Thayer asked what integrators and end users would do if the FBI called and said, “We have evidence of a credible to your electronic physical security system infrastructure. Update the firmware and change the passwords on your networked security devices and physical security systems IT equipment.” That’s a scary thought, as it presents a scenario that most critical infrastructure security system owners and service providers aren’t prepared to respond to.
To say that such a scenario wouldn’t be likely ignores the insider threat possibilities, which include accidental configuration errors and convenience-based password mishandling (which is common) as well as intentional acts.
The Human Error Factor
I know integrators who typically use spreadsheets, database tools, and configuration scripts to manage many aspects of large-scale deployments. Some integrators follow the typical IT practice of performing network configuration using command-line scripts to configure network switches and routers. This has many benefits: speed of configuration, elimination of manual data entry mistakes, auditability of configurations, and data backup for use in restoring all or part of the network configuration.
Typically repetitive manual data entry results in errors. Where manual data entry is required, keeping a record of the entry data provides a basis for checking and correcting such errors.
But where are the third-party tools and built-in support for large-scale system management?
Lights at the End of the Tunnel
Signaling a potential sea change regarding large-scale deployment device manageability are Viakoo and The Boring Lab, who have products that are born out of deployment experience in both the IT and physical security technology sectors.
Viakoo provides camera firmware update management (not just automated task performance) across multiple brands of cameras, and goes well-beyond the level of firmware update capabilities of VMS systems. Via integration with Milestone’s XProtect VMS, Viakoo will halt camera streaming just before performing the camera firmware update and restore streaming after the update or reversion to the original firmware in case of an update error.
Viakoo also has a camera password manager whose full list of features will be revealed at the upcoming ISC West 2019 event. It manages camera passwords not just in the cameras, but also in the VMS systems to which it integrates.
The Boring Lab intends to make large-scale deployments of the Milestone XProtect VMS as boring as for small systems. Its Boring Toolbox product is intended to reduce the time spent managing Milestone XProtect deployments by up to 97%. It does this through automation of as many aspects as possible of initial setup and device management, including firmware updates and device passwords. Both end users and integrators benefit from the capabilities the Boring Toolbox provides.
A&E Spec Issue
In recent years, in the System Capacities sections of product A&E specifications documents, I see “unlimited” device counts. For on-premise systems, I see the phrase “limited only by hardware constraints.” For a VMS deployment that actually means “limited only by the number of recording servers installed” – which is what the specification should say. But does it mean that device counts in the high thousands are truly manageable at that scale? No, it doesn’t. The Boring Toolbox is evidence of that fact, and the Rodney Thayer question about “fast updates” – an important cybersecurity capability – confirms it.
Right now, it doesn’t matter much because all A&E specs are about equal on this point. However, expect product specs to start highlighting large-scale management features this year, as VMs makers are going to begin highlighting them, and I expect Lenel and others to start doing the same. For example, OnGuard’s Policies feature enables auditable management of access level definitions based on access policy rules.
Integrated Controls Technology (ICT) provides an impressive access-video-intrusion unified platform for a one-time purchase price with free software upgrades thereafter. I was surprised to see the high caliber of their software and the many system management features, including some for large-scale deployment support, and I expect that aspect of the product to keep improving.
This is a coming trend that should be reflected in A&E specs once the companies and consultants catch on to the value of management-at-scale features.
Learning from IT
Although many security-industry thought leaders have advocated that we should be “learning from IT”, there is still insufficient emphasis on learning about IT practices, especially for large-scale deployments. We’ll explore that more in the next article in this series.
Scale – Part Two
The next article in this series will also address the difference between horizontal scaling (adding more hardware instances) and vertical scaling (such as adding more CPU power and RAM) and how this should be considered in the system design phase – given current technology trends including fog computing.
Ray Bernard, PSP CHS-III, is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities (www.go-rbcs.com). In 2018 IFSEC Global listed Ray as #12 in the world’s Top 30 Security Thought Leaders. He is the author of the Elsevier book Security Technology Convergence Insights available on Amazon. Mr. Bernard is a Subject Matter Expert Faculty of the Security Executive Council (SEC) and an active member of the ASIS International member councils for Physical Security and IT Security. Follow Ray on Twitter: @RayBernardRBCS.
© 2019 RBCS