Changelog

This page lists changes to the Neo4j Connector for Kafka.

Version 5.1.5

New and Updated Features

Feature Details

Functionality New

Added neo4j.payload-mode configuration property for source connector.

Introduced the neo4j.payload-mode option to define the structure of change messages. Available values are COMPACT and EXTENDED. COMPACT provides simpler messages but faces schema compatibility issues with property type changes, while EXTENDED includes type information to avoid such issues. Default is EXTENDED.

Bug Fixed

Prevented exception caused by adding duplicate fields in schema generation for CDC source events.

Resolved an issue in the ChangeEvent schema generation process where duplicate fields were causing an exception (org.apache.kafka.connect.errors.SchemaBuilderException: Cannot create field because of field name duplication id). When building the schema for key array elements, if different maps contained the same field name, the field was being added multiple times, leading to this exception. Now, duplicate fields are handled appropriately to avoid this issue.

Version 5.1.4

New and updated features

Feature Details

Bug Fixed

Fixed a bug in the CDC schema sink connector where relationship update and delete events without any start or end node keys incorrectly generated Cypher statements.

Resolved an issue where messages referring to relationship update and delete events, which do not contain any start or end node keys, generate Cypher statements that update or delete wider entities on the target database. These messages now generate a failure which should be handled through usual error handling mechanisms.

Version 5.1.3

New and updated features

Feature Details

Bug Fixed

Fixed a bug in the Sink connector where messages without valid constraints would not be processed.

Resolved an issue where messages created from the neo4j-streams 4.1 source plugin, which include unique constraints without a corresponding value in the entity, result in a Cypher query containing null property filters to match on. Depending on the operation type (UPDATE/DELETE/CREATE), this can cause unexpected behavior as the query may not match the intended nodes.

Version 5.1.2

New and updated features

Feature Details

Fix

Support JSON strings with JsonConverter when schemas are not enabled

Resolved an issue with the org.apache.kafka.connect.json.JsonConverter where it did not support JSON strings when schemas.enable=false. The connector now correctly handles incoming JSON strings without requiring schema definitions.

Version 5.1.1

New and updated features

Feature Details

Fix

Updated the Neo4j driver to mitigate Netty’s CVE-2023-34462.

The Neo4j driver has been updated to the latest 4.4.x version (4.4.18), which includes a fix for Netty’s CVE-2023-34462.

Version 5.1.0

New and updated features

Feature Details

Feature New

Added value serialisation strategies for CDC source messages.

We’ve introduced an optional property neo4j.cdc.topic.$topic.value-strategy, which specifies how the values of CDC source messages will be serialised. Available options as CHANGE_EVENT, ENTITY_EVENT.

Version 5.1.0-rc02

New and updated features

Feature Details

Functionality Breaking

Schema shape for change events generated by the Source connector is changed in a way that’s not compatible with earlier versions.

In earlier versions of Neo4j Connector for Kafka, the Source connector derived Kafka Connect structs to express entity properties, which had limitations around schema compatibility over property type changes. Schema derivation logic is now updated to encode type information into a dedicated struct so that this limitation no longer applies.

Functionality New

Pattern sink strategy now checks for matching key constraint for properties marked as key.

Added a new check into pattern sink strategy handlers, which will now verify if there is a matching constraint for properties marked as key in provided patterns. If no matching constraint is found, it is logged as a message into Kafka Connect logs.

Configuration Changed

Renamed following sink configuration properties:

  • neo4j.pattern.node.topic.{TOPIC_NAME} to neo4j.pattern.topic.{TOPIC_NAME}

  • neo4j.pattern.relationship.topic.{TOPIC_NAME} to neo4j.pattern.topic.{TOPIC_NAME}

  • neo4j.pattern.node.merge-properties to neo4j.pattern.merge-node-properties

  • neo4j.pattern.relationship.merge-properties to neo4j.pattern.merge-relationship.properties

Improved Pattern sink configuration so that there is no need to specify pattern type as part of the configuration property name.

Version 5.1.0-rc01

New and updated features

Feature Details

Feature New

Added support for error handling in sink connector.

The Sink connector now has proper error handling in-place so that it can make use of Kafka Connect error handling configuration and direct failing messages into a DLQ.

Configuration Removed

Removed following source properties:

  • neo4j.enforce-schema

Removed following sink properties:

  • neo4j.batch-parallelize

The new version of source connector always generates messages with schema support. To disable schemas, configure your key and value converters accordingly.

The new version of sink connector does not implement any parallelization logic, and processes changes in the order they appear in the topic.

Configuration Changed

Renamed the following configuration properties:

  • topic to neo4j.query.topic

The topic configuration for Query source strategy has been renamed to neo4j.query.topic for consistency.

Feature Configuration New

Added new sink properties:

  • neo4j.cypher.bind-timestamp-as

  • neo4j.pattern.bind-timestamp-as

  • neo4j.pattern.bind-header-as

  • neo4j.pattern.bind-key-as

  • neo4j.pattern.bind-value-as

  • neo4j.pattern.bind-value-as-event

Added new source properties:

  • neo4j.query.poll-duration

Cypher strategy now also allows binding of message timestamp in user provided Cypher queries.

Node and relationship pattern strategies now allow binding of message timestamp, headers, key and value as separate and configurable variables in user provided patterns.

Query source strategy is re-implemented so that a poll cycle can be configured to wait for a configured amount of duration in which the configured query is executed multiple times until a change is discovered.

Feature New

Revamped pattern strategies in sink connector.

Node and relationship pattern sink strategies have been re-implemented with several improvements.

  • Message order is now preserved while applying changes into the database.

  • Patterns now allow mapping of properties from source message to a custom property name as defined in the pattern.

  • Patterns can include references to message timestamp, headers, key and value when defining properties.

Version 5.1.0-beta02

New and updated features

Feature Details

Feature New

Added new sink properties:

  • neo4j.cypher.bind-header-as

  • neo4j.cypher.bind-key-as

  • neo4j.cypher.bind-value-as

  • neo4j.cypher.bind-value-as-event

Cypher strategy now allows binding of message headers, key and value as separate and configurable variables to be used in user provided Cypher queries.

Fix Breaking

Schema conversion of Cypher types

A problem in conversion of Cypher values from Connect values, which are defined as custom Connect types (like Date, Time, Point, etc.), has been fixed. In addition to String based temporal types, we have added Struct based temporal types that will work better with all converters. Also a new dimension field is added for point type, which improves it’s compatibility with all converters. These new types will be used for temporal types generated by the connector.

This had been causing problems when using Protobuf converters.

This change might break existing schema compatibility for schemas generated by earlier versions of this connector. Please consider relaxing schema compatibility mode for those topics or start a new topic that would pick the updated schemas automatically.

Version 5.1.0-beta01

New and updated features

Feature Details

Feature New

CDC message support for sink connector

Both the new CDC message format available in Neo4j Aura 5 and latest Neo4j 5.x Enterprise Edition, and old streams change message format are now supported in CDC Source ID and Schema sink strategies.

Configuration Changed

Some sink properties are renamed.

In order to provide consistent and more intuitive naming we have renamed some sink properties. This affects the following properties (old → now):

  • neo4j.cdc.sourceId.topic.$topic → neo4j.cdc.source-id.topic.$topic

  • neo4j.cdc.sourceId.labelName → neo4j.cdc.source-id.label-name

  • neo4j.cdc.sourceId.propertyName → neo4j.cdc.source-id.property-name

Version 5.1.0-alpha03

New and updated features

Feature Details

Configuration Changed

Some sink properties are renamed.

In order to provide consistent and more intuitive naming we have renamed some sink properties. This affects the following properties (old → now):

  • neo4j.topic.cypher.$topic → neo4j.cypher.topic.$topic

  • neo4j.topic.cdc.sourceId → neo4j.cdc.sourceId.topic.$topic

  • neo4j.topic.cdc.sourceId.labelName → neo4j.cdc.sourceId.labelName

  • neo4j.topic.cdc.sourceId.idName → neo4j.cdc.sourceId.propertyName

  • neo4j.topic.cdc.schema → neo4j.cdc.schema.topics

  • neo4j.topic.pattern.node.$topic → neo4j.pattern.node.topic.$topic

  • noe4j.topic.pattern.merge-node-properties → neo4j.pattern.merge-node-properties

  • neo4j.topic.pattern.relationship.$topic → neo4j.pattern.relationship.topic.$topic

  • neo4j.topic.pattern.merge-relationship-properties → neo4j.pattern.merge-relationship-properties

  • neo4j.topic.cud → neo4j.cud.topics

Functionality Changed

Schema for CDC events is updated.

We have updated a schema to be compatible for CREATE, UPDATE, and DELETE operations, so it would be possible to publish them to the one Kafka topic with backward compatibility mode.

Bug Fixed

Fixed a bug with how CDC index pattern configuration is parsed

Previously it would be possible to receive an error if multiple continuous indexes were specified

Version 5.1.0-alpha02

New and updated features

Feature Details

Feature New

Added key serialisation strategies for CDC source messages.

We’ve introduced an optional property neo4j.cdc.topic.$topic.key-strategy, which specifies keys of CDC source messages will be serialised. Available options as SKIP, ELEMENT_ID, ENTITY_KEYS, WHOLE_VALUE.

Functionality Changed

Supports new keys fields introduced in change events with Neo4j 5.15

Neo4j 5.15 introduced a breaking change on change event schema, where the structure of the keys field in node and relationship change events is updated to handle multiple key constraints. This release supports both the old and the new structure for compatibility.