Agenda item

To update Panel Members on the investigation led by PSPS ICT into the ICT network issues that impacted on service delivery during the outage, and the business continuity de-brief that was carried out subsequently (report of the Assistant Director - Regulatory and the Assistant Director – Corporate enclosed).

Consideration was given to the report of the Assistant Director – Regulatory and the Assistant Director – Corporate to update Panel Members on the investigation led by PSPS ICT into the ICT network issues that impacted on service delivery during the outage, and the business continuity de-brief that was carried out subsequently.

The Head of Public Protection introduced the report which outlined:

• the network outage issue; and
• impact upon services and subsequent action taken/recommended. This had included a business continuity debrief which recommended that service managers review their respective business continuity plans.

The Head of ICT and Digital (PSPS) outlined the IT investigation;

• the supplier (Nasstar) had conducted a Major Incident Review which had failed to identify the root cause with certainty, but had suggested that the issue was caused by a circuit fault;
• one of the BT Openreach connections had been identified as the failed element of the service which was resolved after a test and circuit reset; 
• PSPS had conducted a Major Incident Review which recommended several non-technical actions, including the review of Business Continuity plans and documentation of any workarounds; and 
• a single internet link had been used and detailed resiliency options were being investigated; recommendations would come forward in due course.

Members considered the report and made the following comments:

• Members asked whether the outage had related to a problem with a physical link.
  • The Head of ICT and Digital (PSPS) confirmed that the problem had applied to a physical internet link.

• Members stated that resiliency investigations needed to be as thorough as possible; and to ensure that the recommendation stated at point 2.2 of the report was implemented. The issue needed to be seriously regarded.

• Members had been made aware that schools had been affected by the issue and suggested that the Council sought/shared solutions with schools.

o   The Head of ICT and Digital responded that public sector institutions across the East Midlands utilised the same Public Services Network (PSN) connection and had therefore been affected by the same issue.

• Members sought clarity as to whether the cause was a fault with a defective BT line.
  • The Head of ICT and Digital confirmed this was the case however there had been other failings which were highlighted in the Major Incident Review: this had included an uptime of some equipment of seven years which was much longer than industry standard expectations; as a result, PSPS were holding the provider to account and the provider was working through recommendations from its own Major Incident Review.

• The Assistant Director – Regulatory stated that:

o   as a Partnership, with PSPS, the three Councils were in a stronger position and had worked together to deal with the incident both as it occurred and during the debrief phase;

o   learning would be taken forward;

o   risks around IT were acknowledged and mitigations had been in place however technological mitigations were only part of the solution; and

o   mitigations included both IT solutions and effective service level Business Continuity Plans.

• Members referred to the modern-day dependency on ICT and stated that the risk should have been more carefully considered. Members requested that the pre- and post-incident risk scores, which related to the reliability on a single network line, be circulated to the Panel.

AGREED:

a)       That after consideration of the report and findings the comments of the Panel be noted;

b)      That the Panel supported the conclusions stated at point 2 of the report; and

c)       That when appropriate, a report come forward to the Performance Monitoring Panel detailing the results of the investigation to develop options and the subsequent action taken.