Meltdown & Spectre: Current Results from Neo4j Performance Testing

As mentioned in our earlier blog post about Meltdown and Spectre, we have been running tests to discover the impact on Neo4j of the patches and workarounds for the Meltdown and Spectre vulnerabilities.

The Neo4j engineering team has now completed a series of tests – in various environments and under a wide range of workloads – and we’d like to share the results.

Server Types

To try to understand the performance impact across different Neo4j deployment configurations, we have tested in three different environments, as follows:

1. AWS Ubuntu instances
   Instance type: M3 Large
   Meltdown patch tested with: Linux, 4.4.0-1047-aws
2. Dedicated hardware, low end servers
   Instance type: 1 x 4-core Xeon Skylake-DT – 64GB RAM – SSD
   Meltdown patch tested with: Linux, 4.4.0-109
3. Dedicated hardware, medium level servers
   Instance type: 1 x 22-core Xeon Gold 6152 (22C/44T) @ 2.1Ghz – 512GB RAM – SSD
   Meltdown patch tested with: Linux, 4.4.0-109

Workloads

We tested with several performance suites, each generating a distinctly different set of loads:

1. Internal performance testing workloads based on the LDBC SNB Interactive benchmark.

• Store sizes varying between 1-200GB
• Tested with up to 64 concurrent clients

Realistic read and write workloads against real (customer-provided) database stores

Targeted micro-benchmarks, designed to stress the major internal components of Neo4j

Very large data imports, using datasets ranging from 100 million entities to 100 billion entities

Neo4j Versions

We ran the workloads against the latest patch release of all the supported versions of Neo4j:

• 3.0.12
• 3.1.7
• 3.2.9
• 3.3.2

Results

The combination of server types, workloads and Neo4j versions creates a large matrix of tested scenarios. For every scenario we captured results from both before and after applying the relevant Meltdown patch.

Our analysis of the results shows only negligible performance impacts. Any change is within the range of variance we normally observe. These charts show results for two of the scenarios in the context of normal variation:





The results suggest that Neo4j users using similar servers should not experience any significant performance degradation. However, we are continuing our testing and will report back if we find scenarios where there is a measurable impact.

For advice about your specific deployment, please contact Neo4j Support.

Unrelated to Meltdown, the second chart shows some substantial improvements in Neo4j performance from versions 3.1 → 3.2 and from 3.2 → 3.3. This is as expected and reflects ongoing efforts to improve the product across multiple dimensions.

Future Work

We expect further OS patches and firmware patches to become available over the coming weeks and months. Especially, we are waiting for stable patches from Intel. We’ll continue our testing to evaluate these patches as the become available.