Core SA&O Platform Capability: Metrics & Reporting

This article is a part of a series describing the essential criteria of a Security Automation and Orchestration platform.

Two key benefits from your automation efforts should include increased productivity and increased quality. Metrics that demonstrate these increases are critical to measuring the overall effectiveness of a Security Automation and Orchestration (SA&O) platform. Metrics also allow you to identify where further automation improvements can be made that will increase your overall Return on Investment (ROI).

Below are some additional metrics and reporting criteria that you should consider when evaluating SA&O platforms.

Flexible Dashboards

Metrics are often specific to organizations and individuals. Because of this, many users want the ability to define and organize metrics in a way that makes the most sense for them. An SA&O platform, in turn, should support this flexibility and enable highly-configurable metrics and dashboard views. Users should be able to configure which metrics should be displayed and the time period that is used to calculate the metrics.

Performance Reporting

It is critical to understand the SA&O platform’s contribution to performance gains and resource savings. This performance information should be readily available on the platform and regularly used to evaluate automation performance over time.

Examples of key performance metrics available from an SA&O platform should include:

  • Mean Time to Resolve (MTTR)
  • Mean Dwell Time (MDT), which is defined here as the period of time between a compromise (by a threat actor) and taking an appropriate response
  • Analyst hours saved through automated execution
  • Number of Full Time Equivalents (FTEs) gained through automated execution
  • Money saved (FTE-cost x FTEs-gained)

Security Effectiveness Reporting

Automation is also deployed to increase the security effectiveness and posture of your organization. Understanding the total number of security alerts managed, along with the pace at which they are being managed, are critical to understanding the effectiveness that automation is providing you.

Examples of key security effectiveness metrics that the platform should provide:

  • MTTR and MDT (introduced above)
  • Total number of open alerts
  • Alerts opened per day (hour, week, or month also appropriate)
  • Alerts closed per day (hour, week, or month also appropriate)
  • Performance against Service Level Agreements (SLAs)

App Integration and Playbook Performance

Understanding the most frequently invoked playbooks can help shed light on the places where further automation investments can be made. Ideally, playbook design should strive for the automated closure of false positive or high-confidence true positive alerts. In cases where automation is not closing the alert triage gap, playbook revisions may be necessary.

To identify gaps in automation, as well as the effectiveness of tool integrations, the following example metrics should be provided by the automation platform:

  • Alerts closed through automation (per hour, day, week, month, or other time window)
  • Most active app integrations
  • Most active actions (manual and automated)
  • Most active automated playbooks
  • Playbook execution time
  • Action execution time

Human Workload

While automation is intended to close the human resource gap, there are still cases where humans need to be involved in the day-to-day activity of an SA&O platform. These cases include where manual triage and other actions are required on an alert, or when human approvals are inserted into the playbook to achieve “supervised automation.” Understanding human workload can also help identify areas where further automation and tuning may be needed.

The following example metrics should be provided by the automation platform to understand the human workload involved in the automation process:

  • Alerts assigned to an individual
  • Alerts closed by an individual

Next Steps

Metrics and Reporting are essential for being able to measure and demonstrate your automation performance. Be sure to thoroughly evaluate these capabilities during any evaluation of an SA&O platform. To see the Phantom 2.1 dashboard in action, visit our YouTube channel.

Other articles in this series can be found here.  You can also download the entire Phantom Buyer’s Guide here. Access the free Community Edition when you join the Phantom Community.