GraphSAGE

If you are embedding a graph that has an isolated node, the aggregation step in GraphSAGE can only draw information from the node itself. When all the properties of that node are 0.0, and the activation function is ReLU, this leads to an all-zero vector for that node. However, since GraphSAGE normalizes node embeddings using the L2-norm, and a zero vector cannot be normalized, we assign all-zero embeddings to such nodes under these special circumstances. In scenarios where you generate all-zero embeddings for orphan nodes, that may have impacts on downstream tasks such as nearest neighbor or other similarity algorithms. It may be more appropriate to filter out these disconnected nodes prior to running GraphSAGE.

When doing memory estimation of the training, the feature dimension is computed as if each feature property is scalar.

Since training a GraphSAGE model may take a lot of time and memory on large graphs, it can be helpful to sample a smaller subgraph prior to training, and then training on that subgraph. The trained model can still be applied to predict embeddings on the full graph (or other graphs) since GraphSAGE is inductive. To sample a structurally representative subgraph, see Random walk with restarts sampling.

It may be useful to generate node embeddings with GraphSAGE as a node property step in a machine learning pipeline (like Link prediction pipelines and Node property prediction). It is not supported to train the GraphSAGE model inside the pipeline, but rather one must first train the model outside the pipeline. Once the model is trained, it is possible to add GraphSAGE as a node property step to a pipeline using gds.beta.graphSage or the shorthand beta.graphSage as the procedureName procedure parameter, and referencing the trained model in the procedure configuration map as one would with the predict mutate mode.

The size of the node embedding as well as its hidden layer. A large embedding size captures more information, but increases the required memory and computation time. A small embedding size is faster, but can cause the input features and graph topology to be insufficiently encoded in the embedding.

An aggregator defines how to combine a node’s embedding and the sampled neighbor embeddings from the previous layer. GDS supports the Mean and Pool aggregators.

Mean is simpler, requires less memory and is faster to compute. Pool is more complex and can encode a richer neighbourhood.

The activation function is used to convert the input of a neuron in the neural network. We support Sigmoid and leaky ReLu .

Each sample size represents a hidden layer with an output of size equal to the embedding dimension. The layer uses the given aggregator and activation function. More layers result in more distant neighbors being considered for a node’s embedding. Layer N uses the sampled neighbor embeddings of distance <\= N at Layer N -1. The more layers the higher memory and computation time.

A sample size n means we try to sample at most n neighbors from a node. Higher sample sizes also require more memory and computation time.

This parameter defines how many training examples are grouped in a single batch. For each training example, we will also sample a positive and a negative example. The gradients are computed concurrently on the batches using concurrency many threads.

The batch size does not affect the model quality, but can be used to tune for training speed. A larger batch size increases the memory consumption of the computation.

This parameter defines the maximum number of epochs for the training. Before each epoch, the new neighbors are sampled for each layer as specified in Sample sizes. Independent of the model’s quality, the training will terminate after these many epochs. Note, that the training can also stop earlier if an epoch converged if the loss converged (see Tolerance).

Setting this parameter can be useful to limit the training time for a model. Restricting the computational budget can serve the purpose of regularization and mitigate overfitting, which becomes a risk with a large number of epochs.

Because each epoch resamples neighbors, multiple epochs avoid overfitting on specific neighborhoods.

This parameter defines the maximum number of iterations run for a single epoch. Each iteration uses the gradients of randomly sampled batches, which are summed and scaled before updating the weights. The number of sampled batches is defined via Batch sampling ratio. Also, it is verified if the loss converged (see Tolerance).

A high number of iterations can lead to overfitting for a specific sample of neighbors.

This parameter defines the number of batches to sample for a single iteration.

The more batches are sampled, the more accurate the gradient computation will be. However, more batches also increase the runtime of each single iteration.

In general, it is recommended to make sure to use at least the same number of batches as the defined concurrency.

This parameter defines the maximum depth of the random walks which sample positive examples for each node in a batch.

How close similar nodes are depends on your dataset and use case.

This parameter defines the weight of the negative samples compared to the positive samples in the loss computation. Higher values increase the impact of negative samples in the loss and decreases the impact of the positive samples.

This parameter defines the influence of the regularization term on the loss function. The l2 penalty term is computed over all the weights from the layers defined based on the Aggregator and Sample sizes.

While the regularization can avoid overfitting, a high value can even lead to underfitting. The minimal value is zero, where the regularization term has no effect at all.

When updating the weights, we move in the direction dictated by the Adam optimizer based on the loss function’s gradients. The learning rate parameter dictates how much to update the weights after each iteration.

This parameter defines the convergence criteria of an epoch. An epoch converges if the loss of the current iteration and the loss of the previous iteration differ by less than the tolerance.

A lower tolerance results in more sensitive training with a higher probability to train longer. A high tolerance means a less sensitive training and hence resulting in earlier convergence.

This parameter is only relevant if you want to distinguish between multiple node labels.

Before we are able to generate node embeddings we need to train a model and store it in the model catalog. Below is an example of how to do that.

Looking at the results we can draw the following conclusions, the training converged after a single epoch, the losses are almost identical. Tuning the algorithm parameters, such as trying out different sampleSizes, searchDepth, embeddingDimension or batchSize can improve the losses. For different datasets, GraphSAGE may require different train parameters for producing good models.

The trained model is automatically registered in the model catalog.

In this section we describe how to train on a graph with multiple labels. The different labels may have different sets of properties. To run on such a graph, GraphSAGE is run in multi-label mode, in which the feature properties are projected into a common feature space. Therefore, all nodes have feature vectors of the same dimension after the projection.

The projection for a label is linear and given by a matrix of weights. The weights for each label are learned jointly with the other weights of the GraphSAGE model.

In the multi-label mode, the following is applied prior to the usual aggregation layers:

The projected feature dimension is configured with projectedFeatureDimension, and specifying it enables the multi-label mode.

The feature properties used for a label are those present in the featureProperties configuration parameter which exist in the graph for that label. In the multi-label mode, it is no longer required that all labels have all the specified properties.

The GraphSAGE implementation supports training using relationship weights. Greater relationship weight between nodes signifies that the nodes should have more similar embedding values.

In the case when you have a graph that does not have node properties we recommend to use existing algorithm in mutate mode to create node properties. Good candidates are Centrality algorithms or Community algorithms.

The following example illustrates calling Degree Centrality in mutate mode and then using the mutated property as feature of GraphSAGE training. For the purpose of this example we are going to use the Persons graph, but we will not load any properties to the in-memory graph.

gds.degree.mutate will create a new node property degree for each of the nodes in the in-memory graph, which then can be used as featureProperty in the GraphSAGE.train mode.

To generate embeddings and stream them back to the client we can use the stream mode. We must first train a model, which we do using the gds.beta.graphSage.train procedure.

Once we have trained a model (named 'graphSage') we can use it to generate and stream the embeddings.

The model trained as part of the stream example can be reused to write the results to the in-memory graph using the mutate mode of the procedure. Below is an example of how to achieve this.

The model trained as part of the stream example can be reused to write the results to Neo4j. Below is an example of how to achieve this.