Security system maintenance done right: effective problem management

Security system downtime is a major worry for the majority of businesses, as it can cause major disruptions to business operations. Security system malfunctions are one of the greatest sources of stress for security and facility managers.


Even more distressing is when a system goes down repeatedly, even after an issue appears to have been solved. This often happens because a company’s security maintenance service providers fail to find the root cause of a system issue.


For this reason, a truly effective system maintenance service is based on the theory of “effective problem management”. This approach was pioneered most famously by car manufacturer Toyota. Applied to security maintenance, this approach helps companies to avoid constant “firefighting”, which would otherwise waste time, lower productivity and affect business performance.


What does “effective problem management” mean?

When dealing with system malfunctions, many service providers mistakenly believe that simply helping a client to resume normal business operations as soon as possible is enough to solve a problem. However, this is not effective problem management, just incident management. While incident management deals with the short term effects of a system malfunction, problem management is a way of identifying the root cause of a system problem and taking steps to make sure it does not occur again.


In other words, true problem management deals with the cause of the problem, while incident management only treats the symptom.


Toyota is the most famous practitioner of problem management theory. Toyota encouraged managers constantly to ask why when trying to find the source of a problem using a “five why’s” approach, shown in the example below:


Scenario: My car will not start…


Why #1  ...why does my car not start? The battery is dead.
Why #2 ... why is the battery dead? The alternator is not working.
Why #3 ... why is the alternator not working?

The alternator belt was broken.

Why #4 ... why was the belt broken? 

It has never been replaced

Why #5 … why was the belt never replaced? I was not maintaining my car according to the service schedule


Therefore, the solution is to ensure that the car is properly maintained according to the desired schedule to ensure that the problem does not occur again.


Similarly, with security system maintenance and troubleshooting, service providers can only solve system issues if they constantly ask questions and explore every possible source of a problem. Here is an example of how this might work:


Scenario: The door at the main entrance to the office does not lock properly…


Why # 1 …why doesn’t the door lock properly?

The door connectors do not connect to each other correctly.

Why #2 …why don’t the connectors connect correctly?

The door itself was not fitted properly.

Why #3…why wasn’t the door fitted properly?

The decorating company did not measure the dimensions of the door and the door frame accurately.

Why #4…why aren't the measurements right?

The engineer was not careful enough in their work

Why #5…why wasn’t the engineer careful enough?

The decoration company does not conduct comprehensive installation quality inspections.


In this way, we can see that the root cause of the security flaw is that the decoration company does not have sufficiently rigorous quality control measures. The solution may therefore be either to request that the vendor change their work procedure, or to replace them with a more reliable vendor that delivers higher installation quality.


Another crucial component of effective problem management is the Plan-Do-Check-Act (PDCA) cycle. This method allows us to identify problems and test solutions in the most systematic way possible.


By constantly testing solutions and assessing their performance, PDCA encourages trouble shooters to dig deeper when solutions are found to be ineffective and enables them to find the true cause of the problem. In this way, it is possible eventually to reach the most effective solution to any problem using PDCA.

How we implement this approach: security maintenance case studies

Case study 1: A client’s employees’ access cards cannot be read easily

A global insurance company working with ICD required that their card readers be concealed within the walls next to their office entrance for aesthetic reasons. Yet when the security system installation was complete, they discovered that employees found it difficult to swipe their access cards successfully. This was because the card reader was not picking up the signal easily. ICD’s service team first requested that the decoration company that helped the client reduced the thickness of the wall by the concealed card reader, but this solution was found to be ineffective.


The source of the problem was uncovered when our service team borrowed a programming card from the card reader manufacturer. Using the programing card, it was discovered that the card reader was converting the signal from the card into a proprietary format rather than reading the proximity format of the card directly.


ICD used the programming card to reprogram the card reader concealed in the wall to receive signals directly in proximity format. Uncovering this problem allowed ICD’s service team to make recommendations on how to configure card readers correctly for future system extensions.


Case study 2: A client’s card readers stop working

A multinational chemicals company encountered a serious disruption to their operations in one office when each card reader at the site stopped working. Not only could employees not enter the building, but one associate was almost trapped inside the office overnight. The card reader control pane was found to be functioning correctly, but the problem still persisted. Eventually, the source of the problem was found to be a result of the client’s network programming structure, which was set to a “Client” rather than a “Transparent” configuration, meaning that the network did not interface properly with the card controller.


The final solution was therefore to ask the client’s network engineer to change the configuration. By establishing the correct source of the problem, we were able to deal not only with the symptom that was the malfunctioning card controller; we were also able to help the client prevent future disruptions due to card controllers.


Case study 3: A client’s cameras go offline in multiple sites

A global IT corporation discovered that IP cameras for 4 of their China sites had stopped transmitting video footage to the system. Tests using multiple camera brands showed that the issue was not caused the type of camera used, but was instead related to the network system. Multiple solutions were tested and implemented, including reducing video frame rates, adjusting the number of cameras assigned to each video recorder, degrading the firmware and reinstalling the NVR software.


Eventually, it was established that the loss of communication between the cameras and the system was due to an issue related to the client’s internal corporate network. After the issue was resolved, we provided a detailed report of the security system network configuration to both the client’s security team and the security management system vendor to ensure that similar incidents did not occur again.


The final word

As we can see, a rigorous approach to problem management allows us to develop genuinely effective system maintenance solutions for our clients.


Another crucial benefit of this approach is that it clearly and objectively demonstrates who is responsible for taking action to solve a problem. As we have seen from the case studies, many security issues can only be solved if the client or other vendors take action; not all system maintenance issues can be handled by the security services provider alone.


We therefore hope that our clients and partners join us in embracing the philosophy of effective problem management to ensure that security end users are supported by highly robust security systems.


For more tips on security, follow ICD on our LinkedIn Company Page.