With the growth of the graph category, the transition to cloud and the rise of graph data science, we are seeing more and more people and projects using the Neo4j database, developer tools, and GDS products.
Historically, we have focused on the developer and adoption of the use of the graph, with Aura Free and Professional, the new Data Importer, familiar Browser, and, for the analysts, Bloom. For our data science customers, we have the Graph Data Science product, with the recently released AuraDS for cloud managed Graph Data Science.
Within the Enterprise, with managed cloud and self managed databases increasing in number, scale, and type, we find our Database Administrators (or SREs, DevOps Engineers) having to cope with a plethora of systems and increasing pressure from the business to deliver.
For a number of years, the Neo4j database has provided an industry leading graph database, rich in features and capabilities. But for the ops team, having to pick up this new technology when the business uses it for a project – with unfamiliar terminology, syntax, and complex command line or third party tool integration. Some labs projects like Halin have addressed this partially in the past, but even so the Admin experience has been “suboptimal.”
Today, two things changed. We launched a tool to assist our next group of users, the Administrators of Neo4j in the Enterprise, in the form of Neo4j Ops Manager. This is a tool with functions to simplify the monitoring, administration, and operation of Neo4j Enterprise databases, and provide the administrators with an enjoyable experience. And secondly, we make this available as an official product.
Objectives
When we set out to build Neo4j Ops Manager (this was not the first attempt over the last five years to provide this capability), we identified some objectives for the product. We will discuss these briefly, and then look at how version 1 responds to these objectives.
Any Type, Any How, Any Where
Neo4j Ops Manager will manage Enterprise Edition Neo4j DBMS instances (version 4.4 or later) regardless of the type, the deployment, or the location of the instance.
- Support single instances and clusters
- Support all distributions used by self-managed customers
- Support managed cloud or self managed (cloud or on-premises)
Estate View
Neo4j Ops Manager can provide a view of all the managed instances in a summary view of the Neo4j estate, be this one single instance or a large collection of DBMSs
Any Time
Neo4j Ops Manager can manage the instances asynchronously, allowing monitoring of the managed instances whilst the server is running – ideally 24×7. This also allows a history of metrics and other elements to be collected in Neo4j Ops Manager and viewed retrospectively ≠ for example, as part of incident or root cause analysis.
Four Pillars
Management of a (set of) Neo4j instance(s) falls into four categories. Neo4j Ops Manager can provide some native capabilities in all of these, but is not intended to offer the full functions of dedicated solutions, like alert managers. These categories are:
- Monitoring: review metrics, logs, and other key information about the state of the instance(s) and generate alerts based on metadata defined rules.
- Administration: make it easy to manage security, configuration, databases, clusters, and other elements.
- Operations: simplify and/or automate complex or time-consuming tasks through a simple user interface, such as upgrades, backups, or regular jobs.
- Integration: simplify the integration of data and metadata with external tooling, such as alert events into a central IT management system.
Secure and Protected
As Neo4j Ops Manager will have privileged access to one or more instances of Neo4j, it is imperative that it is as secure as possible and protects the customer data.
- Neo4j Ops Manager will never store any customer data – only metadata.
- There is no direct channel from the UI to a managed system.
- All APIs are secured with authentication, and data in transit is encrypted.
- Credentials are provided via environment variables.
Loose Coupled Architecture
Neo4j Ops Manager will implement separate tiers in the architecture connected by APIs.
Lightweight, Efficient User Interface
Neo4j Ops Manager will provide a user interface which is:
- Lightweight and run using commodity web browsers.
- Designed to enable the user to achieve their goal as simply as possible.
- A common look and feel with other Neo4j products.
Extensible and Explainable
This tool was designed to be built incrementally over time:
- Core architecture to allow features to be added.
- Prioritization based on best value for the user.
Neo4j Ops Manager will be built with explainability:
- Actions will be able to display the commands which will be run.
- Recommendations will explain why they have been made.
Architecture
Neo4j Ops Manager is implemented as a three tier architecture. While the installation and configuration is a bit more complex than previous options like Halin, this brings additional benefits to match the objectives.
The administrator accesses Neo4j Ops Manager through a user interface which is implemented as a web application and accessed using a standard web browser. This is secured with TLS and customer’s SSL certificates.
The user interface communicates with the server via GraphQL APIs, which are token authorized for additional security. These will be exposed in the future as a public API for integration with other customer tooling, if required.
The Neo4j Ops Manager server is a Java application, which manages the main logic, and also hosts the UI and agent APIs. It requires Java17 to run, and leverages Neo4j Spring integration and SDN to persist the metadata into a Neo4j database (persistence). Ideally this is a dedicated single instance database (for which a limited use, resource limited license is included).
The server communicates with one or more agents using gRPC. This is also TLS encrypted, and has been chosen for its HTTP/2 features such as streaming, along with cross platform support.
Each host (server, VM, container) of a Neo4j instance to be managed will run an agent process, which acts as a proxy for the server on the host and interacts with both the O/S and Neo4j instance(s). This is written in Go, giving support for all platforms, a smaller memory footprint, and smaller, static binaries with no dependencies. This helps limit the agent’s impact on the managed instance(s).
Where possible, the agent will collect the information with the minimum impact on the DBMS. For example, when collecting metrics, Neo4j Ops Manager will use the prometheus endpoint so that metrics can be collected without writing and parsing files, or loading the database with cypher queries, and thus cluttering the query log.
Features
Neo4j Ops Manager 1.0 release provides a significant portion of the target architecture along with a set of initial features in Monitoring, Administration, and Operations.
Core product features include support for single instance and cluster, agent management, password management, DBMS rename, and supporting frameworks.
Monitoring Features
In monitoring, we start the application on the Home Page which gives us an estate wide view of the managed DBMSs and their current status. There is also an alert tab with details of any issues detected in the topology. This enables the administrator to see at a glance which system needs any immediate attention.
The administrator can select a DBMS to work with from the header, and view a more detailed topology of that DBMS in the Status Panel. This provides details of the DBMS, instance(s), and databases deployed in that DBMS, along with the status of each element in either graphical or tabular layout.
The heart of the monitoring in Neo4j Ops Manager 1.0 comes with the Dashboard, which provides charts of metrics over time for host (operating system), instance (including JVM) and database metrics. This allows the administrator to see DBMS-wide metrics for current or historical data at a glance.
Future plans include adding contextual value and threshold alerting, as well as the ability for the administrator to define which metrics to add to the dashboard.
Administration Features
In Administration, we have Security Manager which provides a user interface over Role-Based Access Control and fine-grained security in the managed DBMS. This includes user and role creation, modification, and deletion, the assignment of users to roles and granting, denying, or revoking of privileges from roles.
Future plans include supporting the new features coming in the next Neo4j release, along with support for SSO, LDAP, and user perspectives of privileges.
Operations Features
In operations, we have the first part of the Upgrade Manager, which will be fully available for the next major release of the Neo4j database.
The initial offering provides details of the current version and support date, along with current upgrade options, release date, support date, and links to release notes and download.
Future plans include providing a step-by-step guided plan for upgrading or migration of a DBMS to a later version.
Go Get It
Neo4j Ops Manager is available in the download center.
License Requirements
The license is linked to a valid Enterprise Edition license for the Neo4j database.
- There is no additional license cost for Enterprise Edition customers.
- Neo4j Ops Manager may be evaluated with a Trial License of Enterprise Edition Neo4j database.
Managed DBMS Support
The currently supported DBMSs which Neo4j Ops Manager can manage are:
- Neo4j Enterprise Edition version 4.4.0 or later
- Linux tarball installations or Windows zip file installations
- Self managed deployments (on-premises or cloud)
