Aggregate a database backup chain

Command

The aggregate command turns a a chain of backup artifacts into a single full backup artifact.

backup chain aggregation

The benefits of aggregating a backup chain are notably:

Reduces the size of backup artifacts in a given backup folder.
Keeps the recovery time objective (RTO) low by generating a single backup artifact ready to be restored. As part of the aggregation, transactions contained in the differential backups are applied to the store contained in the full backup artifact. This operation is called recovery and can be costly.
Reduces the risk of losing chain’s links.

Syntax

neo4j-admin database aggregate-backup [-h] [--expand-commands]
                                      [--verbose] [--keep-old-backup[=true|false]]
                                      [--parallel-recovery[=true|false]]
                                      [--additional-config=<file>]
                                      --from-path=<path> [<database>]

Description

Aggregates a chain of backup artifacts into a single artifact.

Parameters

Table 1. `neo4j-admin database aggregate-backup` parameters
Parameter	Description
`[<database>]`	Name of the database for which to aggregate the artifacts. Can contain `*` and `?` for globbing.

Options

Table 2. `neo4j-admin database aggregate-backup` options
Option	Description	Default
`--additional-config=<file>`	Configuration file with additional configuration.
`--expand-commands`	Allow command expansion in config value evaluation.
`--from-path=<path>`	Accepts either a path to a single artifact file or a folder containing backup artifacts. When a file is supplied, the <database> parameter should be omitted. The option to supply a file is only available from Neo4j 5.2 onwards.
`-h, --help`	Show this help message and exit.
`--keep-old-backup[=true\|false]`	If set to true, the old backup chain is not removed.	`false`
`--parallel-recovery[=true\|false]`	Allow multiple threads to apply pulled transactions to a backup in parallel. For some databases and workloads, this may reduce aggregate times significantly. Note: this is an EXPERIMENTAL option. Consult Neo4j support before use.	`false`
`--verbose`	Enable verbose output.

From Neo4j 5.19, the --from-path=<path> option can also load backup artifacts from AWS S3 URIs. For more information, see Aggregating a backup chain located in an AWS S3 storage.

Examples

Aggregating a backup chain located in a given folder

The following is an example of how to perform aggregation of a set of backups located in a given folder for the neo4j database:

bin/neo4j-admin database aggregate-backup --from-path=/mnt/backups/ neo4j

The command first looks inside the /mnt/backups/ directory for a backup chain for the database neo4j. If found, it is then aggregated into a single backup artifact.

Aggregating a backup chain identified using a given backup file

The following is an example of how to perform aggregation of a set of backups identified using a given backup file for the neo4j database:

bin/neo4j-admin database aggregate-backup --from-path=/mnt/backups/neo4j-2022-10-18T13-00-07.backup

The command checks the /mnt/backups/ directory for a backup chain including the file neo4j-2022-10-18T13-00-07.backup, for the database neo4j. If found, it is then aggregated into a single backup artifact. This option is only available in Neo4j 5.2 and later.

Aggregating a backup chain located in an AWS S3 storage

The following is an example of how to perform aggregation of a set of backups located in a given folder in an AWS S3 storage.

Neo4j uses the AWS SDK v2 to call the APIs on AWS using AWS URLs. Alternatively, you can override the endpoints so that the AWS SDK can communicate with alternative storage systems, such as Ceph, Minio, or LocalStack, using the system variables aws.endpointUrls3, aws.endpointUrlS3, or aws.endpointUrl, or the environments variables AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL.

Ensure that the AWS CLI is installed and configured with the necessary credentials.
1. Install the AWS CLI by following the instructions in the AWS official documentation — Install the AWS CLI version 2.
2. Use aws configure command to set your aws_access_key_id and aws_secret_access_key from AWS.
3. Create an S3 bucket and a directory to store the backup files using the AWS CLI:
  aws s3 mb --region=us-east-1 s3://myBucket aws s3api put-object --bucket myBucket --key myDirectory/
  For more information on how to create a bucket and use the AWS CLI, see the AWS official documentation — Use Amazon S3 with the AWS CLI and Use high-level (s3) commands with the AWS CLI.
4. Verify that the ~/.aws/config is correct by running the following command:
  cat ~/.aws/config
  The output should look like this:
  [default] region=eu-north-1
5. Verify that the ~/.aws/credentials is correct:
  cat ~/.aws/credentials
  The output should look like this:
  [default] aws_access_key_id=this.is.secret aws_secret_access_key=this.is.super.secret
Then, use the following command to aggregate the backup chain located in the given folder in the AWS S3 storage:
```
bin/neo4j-admin database aggregate-backup --from-path=s3://myBucket/myDirectory/myBackup.backup neo4j
```