We’ve finally added the feature to produce your backups from your running sandboxes and continue the fun locally or in Aura.
To try it out, just request a backup from the “backups” panel.
When you request a backup, our backup server processes the request. You have to wait for the backup to finish. Depending on your database size and how many people are currently doing backups. It can take anywhere from 30 seconds to 10 minutes.
Once finished, your backup will pop up in the same tab. When you request a download link, a pre-signed URL will be made available for 30 days, and you will be able to download your database dump.
How to Use Your Dump File
Depending on what you want to accomplish. The dump file can be used in multiple ways. For example, Import your dump file to;
- Aura / Aura Free (push-to-cloud or Aura Console)
- Neo4j Desktop (neo4j-admin load)
- A Neo4j Docker image
- Or other Neo4j instances **
** Neo4j Sandbox will not support dump file imports.
How Does It Work?
Sandbox is a demo environment where you can experiment with your Neo4j database at no cost and explore different datasets and concepts of a graph database.
However, sandboxes have a limited lifespan of 3 days that can be extended by another seven days.
But when you have done your initial tinkering and learning, the following steps might be for you to migrate to Aura, Neo4j’s managed cloud solution, for production loads.
The good thing about Aura is that you can now get an instance for free. That’s free forever, meaning you can run light loads for as long as it takes for you to finish your POC or whatever project you are working on currently.
On the backend of things, Neo4j hosts your sandboxes in the cloud on AWS, and in the cloud, there is persistent storage and ephemeral storage. Sandboxes use ephemeral storage, as part of their Fargate instances that run the Sandbox Docker images, so once you terminate that instance, your data is gone. This is not the case with Aura.
So recently, we added the feature to backup your sandboxes by providing the user with a dump file of the current database. And in order to do so, we had to implement some new infrastructure.
How does it do that?
Neo4j-Admin provides the tools needed to make backups from a database instance, but with some limitations e.g. those are not dump files. To produce a dump from a database, we need an offline neo4j database (not server just database). The problem is that we can’t take the sandbox server instance offline without losing our data since our storage is ephemeral.
To make things more secure, tty/ssh access is denied to running tasks in our infrastructure unless you circumvent good practice. This means we can‘t stop a running database without some ugly scripting baked into our images (and you wouldn’t want us to stop your live database either).
So producing a direct dump from the sandbox infrastructure is a no-go scenario.
Instead, we use the following procedure.
- Make an Online Backup neo4j-admin backup –from <sandbox> –backup-dir <backup_dir>
- Restore the database locally neo4j-admin restore –from <backup_dir + “/neo4j> –database <backup_id>
- Make a dump from the local database neo4j-admin dump –database <sandbox_id> –to <backup_id>.dump
- Upload it to s3.
Now you might think to yourself, that’s quite the chunk of computing. No way will the user wait for a synchronous request to do all that. And you’ll be right, so that’s why we leverage Amazon SQS and a Celery backup service hosted in our Fargate cluster to make it all asynchronous.
Amazon SQS is a queueing service to schedule celery tasks when a user requests a backup from our REST service.
Our backup service then picks up the task and runs the procedure until completion.
How to Load Your Dump File into Aura
Loading your data into Aura can be done in a couple of ways, but if you need to get it done by a dump file.
That, you can get straight from sandbox or use the dump files in the data folders of the public examples hosted here. (You might need “git lfs” for some of the examples).
- Login to https://console.neo4j.io/
- Create a database (Read more about this here).
- Select the name of the database you want to import the data.
- Select the IMPORT DATABASE tab.
- Drag and drop your .dump file into the provided window or select Select a .dump file and select your file.
- Select Upload.
Now that you know a little more about backups in Neo4j Sandbox have fun exploring the readily available use-cases in a sandbox, save your results, and tag #neo4j with your findings on Twitter.
New Feature: Neo4j Sandbox Backups was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
