Real Words or Buzzwords?: Scale (Part 1)

“Management at scale” is a concept that seems to have avoided the general physical security industry mindset, even though a key case study book about it was published in 1992, and the topic of managing large-scale infrastructure has been covered in articles about IT infrastructure management for at least the past four years. I guess that should be no surprise as the physical security industry in general seems to lag IT progress and practice by 5 to 15 years.

This article covers the seldom-spoken-of conflict of interest issues, large-scale pain points, and the misleading statements still to be found in product A&E Specifications that relate to device manageability for large-scale security system deployments.

Conflicts of Interests

There is also a conflict-of-interest issue involved that could be considered a holdover from the 1980s and early 1990s – when the amount of a systems integrator’s revenue and employee headcount was dependent in part upon how many cable home runs were needed for a security system installation. This was part of the initial reluctance to adopt network-based physical security systems, especially those that utilized a customer’s existing IT network. That kind of deployment was an employee job threat for the integrators who depended on the money earned by cable installation work.

That same conflict of interest shadow discouraged some integrators from adopting IT tools and remote system access in favor of maintaining truck rolls for all service calls. Obviously that approach was not in the best interest of customers, and so now the use of diagnostic tools and remote service tools is becoming the prevailing model.

Because manageability at scale has not been a major concern of security industry manufacturers, the integrator revenue from any deployment project scaled linearly with the amount of equipment being installed, especially for video system deployments. Typically, video cameras have required a significant amount of manual data entry for each camera.

Management-at-scale features change the total cost of ownership (TCO) picture for end user customers, as well as the Total Cost to Serve (TCS) picture for security service providers. Customers get more value for their money, and service providers can scale deployment size using force-multiplier technology that increases their profits. For more on this aspect see these articles: Total Cost of Ownership and Are Integrators Overlooking Total Cost to Serve?.

Large Scale Pain Points

The support for multiple device configuration varies across access control and other systems. There are systems that, for example, allow spreadsheet configuration data import. I’m not trying to provide a detailed overview of the product landscape – I’m merely pointing out by example that large-scale device management has not been given enough attention in the industry. However, you can find an overview of the IP camera firmware update landscape in a freely-available IPVM article that provide a directory of the status of firmware upgrades for 15 major brands of cameras. The article documents the firmware installation times for each brand.

Intelligent network video cameras are the worst-case example of device management pain.

Most customers with high-camera-count video deployments don’t update the camera firmware, due to the manual labor cost previously involved in updating firmware (15 to 30 minutes per camera) and the problems that could occur during firmware updates, some of which resulted in bricking the camera (making the camera inoperable). Furthermore, many camera manufacturers – at some point – had firmware updates that required returning the camera to factory default settings, meaning that each camera had to be reconfigured manually after the firmware upgrade.

And it’s still the case that there is a version-dependency issue for cameras that provide a file download of the camera’s configuration settings. The configuration file can only be uploaded to the same version of firmware from which it was downloaded. Camera configuration backup doesn’t survive a version upgrade. When I brought this to the attention of Geoffrey Bauer, the manager of the A&E Program at Axis Communications, he told me that the configuration backups should survive across most if not all of the minor version upgrades, but currently not across major firmware upgrades. Download the Axis firmware upgrade program document here, which does not yet include any information on configuration backup compatibility. Reps from a few other camera vendors I asked were not aware of this issue.

I’m currently lobbying for camera manufacturers to implement camera configuration backups in a standard way that would enable configuration restoration to survive firmware changes. It’s long overdue.

In a recent SIA Cyber Office Hours webinar, Rodney Thayer asked what integrators and end users would do if the FBI called and said, “We have evidence of a credible to your electronic physical security system infrastructure. Update the firmware and change the passwords on your networked security devices and physical security systems IT equipment.” That’s a scary thought, as it presents a scenario that most critical infrastructure security system owners and service providers aren’t prepared to respond to.

To say that such a scenario wouldn’t be likely ignores the insider threat possibilities, which include accidental configuration errors and convenience-based password mishandling (which is common) as well as intentional acts.

The Human Error Factor

I know integrators who typically use spreadsheets, database tools, and configuration scripts to manage many aspects of large-scale deployments. Some integrators follow the typical IT practice of performing network configuration using command-line scripts to configure network switches and routers. This has many benefits: speed of configuration, elimination of manual data entry mistakes, auditability of configurations, and data backup for use in restoring all or part of the network configuration.

Typically repetitive manual data entry results in errors. Where manual data entry is required, keeping a record of the entry data provides a basis for checking and correcting such errors.

But where are the third-party tools and built-in support for large-scale system management?

Lights at the End of the Tunnel

Signaling a potential sea change regarding large-scale deployment device manageability are Viakoo and The Boring Lab, who have products that are born out of deployment experience in both the IT and physical security technology sectors.

Viakoo provides camera firmware update management (not just automated task performance) across multiple brands of cameras, and goes well-beyond the level of firmware update capabilities of VMS systems. Via integration with Milestone’s XProtect VMS, Viakoo will halt camera streaming just before performing the camera firmware update and restore streaming after the update or reversion to the original firmware in case of an update error.

Viakoo also has a camera password manager whose full list of features will be revealed at the upcoming ISC West 2019 event. It manages camera passwords not just in the cameras, but also in the VMS systems to which it integrates.

The Boring Lab intends to make large-scale deployments of the Milestone XProtect VMS as boring as for small systems. Its Boring Toolbox product is intended to reduce the time spent managing Milestone XProtect deployments by up to 97%. It does this through automation of as many aspects as possible of initial setup and device management, including firmware updates and device passwords. Both end users and integrators benefit from the capabilities the Boring Toolbox provides.

A&E Spec Issue

In recent years, in the System Capacities sections of product A&E specifications documents, I see “unlimited” device counts. For on-premise systems, I see the phrase “limited only by hardware constraints.” For a VMS deployment that actually means “limited only by the number of recording servers installed” – which is what the specification should say. But does it mean that device counts in the high thousands are truly manageable at that scale? No, it doesn’t. The Boring Toolbox is evidence of that fact, and the Rodney Thayer question about “fast updates” – an important cybersecurity capability – confirms it.

Right now, it doesn’t matter much because all A&E specs are about equal on this point. However, expect product specs to start highlighting large-scale management features this year, as VMs makers are going to begin highlighting them, and I expect Lenel and others to start doing the same. For example, OnGuard’s Policies feature enables auditable management of access level definitions based on access policy rules.

Integrated Controls Technology (ICT) provides an impressive access-video-intrusion unified platform for a one-time purchase price with free software upgrades thereafter. I was surprised to see the high caliber of their software and the many system management features, including some for large-scale deployment support, and I expect that aspect of the product to keep improving.

This is a coming trend that should be reflected in A&E specs once the companies and consultants catch on to the value of management-at-scale features.

Learning from IT

Although many security-industry thought leaders have advocated that we should be “learning from IT”, there is still insufficient emphasis on learning about IT practices, especially for large-scale deployments. We’ll explore that more in the next article in this series.

Scale – Part Two

The next article in this series will also address the difference between horizontal scaling (adding more hardware instances) and vertical scaling (such as adding more CPU power and RAM) and how this should be considered in the system design phase – given current technology trends including fog computing.

 

Ray Bernard, PSP CHS-III, is the principal consultant for Ray Bernard Consulting Services (RBCS), a firm that provides security consulting services for public and private facilities (www.go-rbcs.com). In 2018 IFSEC Global listed Ray as #12 in the world’s Top 30 Security Thought Leaders. He is the author of the Elsevier book Security Technology Convergence Insights available on Amazon. Mr. Bernard is a Subject Matter Expert Faculty of the Security Executive Council (SEC) and an active member of the ASIS International member councils for Physical Security and IT Security. Follow Ray on Twitter: @RayBernardRBCS.