Cluster server discovery

In order to join a running cluster, any new member must know the addresses of at least some of the other servers in the cluster. This information is necessary to connect to the servers, run the discovery protocol, and obtain all the information about the cluster.

Neo4j provides several mechanisms for cluster members to discover each other and form a cluster based on the configuration and the environment in which the cluster is running, as well as the version of Neo4j being used.

In Neo4j 5.23, a new discovery service was introduced. You must move to the new version before you start using Neo4j 2025.01.

See Cluster server discovery in Operations Manual version 5 for more information.

Methods for server discovery

Depending on the type of dbms.cluster.discovery.resolver_type currently in use, the discovery service can use a list of server addresses, DNS records, or Kubernetes services to discover other servers in the cluster. The discovery configuration is used for initial discovery and to continuously exchange information about changes to the topology of the cluster.

Regardless of the method used to resolve the list of server addresses, ensure that the endpoint for each server hosting the system database in primary mode is included.

Discovery using a list of server addresses

If the addresses of the other cluster members are known upfront, they can be listed explicitly. However, this method has limitations, such as:

If servers are replaced and the new members have different addresses, the list becomes outdated. An outdated list can be avoided by ensuring that the new members can be reached via the same address as the old members, but this is not always practical.
Under some circumstances the addresses are unknown when configuring the cluster. This can be the case, for example, when using container orchestration to deploy a cluster.

To use this method, set dbms.cluster.discovery.resolver_type=LIST and hard code the addresses in the configuration of each server. For example:

dbms.cluster.discovery.resolver_type=LIST

server.cluster.advertised_address=server01.example.com:6000
dbms.cluster.endpoints=server01.example.com:6000,server02.example.com:6000,server03.example.com:6000

An example of using this method is illustrated by Configure a cluster with three servers.

Discovery using DNS with multiple records

Where it is not practical or possible to explicitly list the addresses of cluster members to discover, you can use DNS-based mechanisms. In such cases, a DNS record lookup is performed when a server starts up based on configuration settings. Once a server has joined a cluster, further topology changes are communicated amongst the servers in the cluster as part of the discovery service.

The following DNS-based mechanisms can be used to get the addresses of other servers in the cluster for discovery:

dbms.cluster.discovery.resolver_type=DNS

With this configuration, the initial discovery members are resolved from DNS A records to find the IP addresses to contact. For example:

dbms.cluster.discovery.resolver_type=DNS

server.cluster.advertised_address=server01.example.com:6000
dbms.cluster.endpoints=cluster01.example.com:6000

When a DNS lookup is performed, the domain name returns an A record for every server in the cluster, where each A record contains the IP address of the server. The configured server uses all the IP addresses from the A records to join or form a cluster.

The discovery port must be the same on all servers when using this configuration. If this is not possible, consider using the discovery type SRV.

dbms.cluster.discovery.resolver_type=SRV

With this configuration, the initial discovery members are resolved from DNS SRV records to find the IP addresses/hostnames and discovery service ports (v1) or cluster advertised ports (v2) to contact.

The value of dbms.cluster.endpoints must be set to a single domain name and the port set to 0. The domain name returns a single SRV record when a DNS lookup is performed. For example:

dbms.cluster.discovery.resolver_type=SRV

server.cluster.advertised_address=server01.example.com:6000
dbms.cluster.endpoints=cluster01.example.com:0

The SRV record returned by DNS should contain the IP address or hostname, and the cluster port for the servers to be discovered. The configured server uses all the addresses from the SRV record to join or form a cluster.

Discovery in Kubernetes

A special case is when a cluster is running in Kubernetes and each server is running as a Kubernetes service. Then, the addresses of the other servers can be obtained using the List Service API, as described in the Kubernetes API documentation.

The following settings are used to configure for this scenario:

Set dbms.cluster.discovery.resolver_type=K8S.
Set dbms.kubernetes.label_selector to the label selector for the cluster services. For more information, see the Kubernetes official documentation.
Set dbms.kubernetes.discovery.service_port_name to the name of the service port used in the Kubernetes service definition for the Core’s discovery port. For more information, see the Kubernetes official documentation.

With this configuration, dbms.cluster.endpoints is not used and any value assigned to it is ignored.

The pod running Neo4j must use a service account that has permission to list services. For further information, see the Kubernetes documentation on RBAC authorization or ABAC authorization.
The configured server.cluster.advertised_address must exactly match the Kubernetes-internal DNS name, which is of the form <service-name>.<namespace>.svc.cluster.local.

The discovery configuration is used for initial discovery and to continuously exchange information about changes to the topology of the cluster.

Metrics

You can use the following discovery metrics to monitor the discovery service.