Skip to main content

Integrating Amazon MSK with ClickHouse

Prerequisites​

We assume you are familiar with the Amazon MSK and Confluent Platform, specifically Kafka Connect. We recommend the Amazon MSK Getting Started guide and MSK Connect guide.

The official Kafka connector from ClickHouse with Amazon MSK​

Steps​

  1. Create an MSK instance.
  2. Create and assign IAM role.
  3. Download a jar file from ClickHouse Connect Sink Release page.
  4. Install the downloaded jar file on Custom plugin page of Amazon MSK console.
  5. If Connector communicates with a public ClickHouse instance, enable internet access.
  6. Provide a topic name, ClickHouse instance hostname, and password in config.
connector.class=com.clickhouse.kafka.connect.ClickHouseSinkConnector
tasks.max=1
topics=<topic_name>
ssl=true
security.protocol=SSL
hostname=<hostname>
database=<database_name>
password=<password>
ssl.truststore.location=/tmp/kafka.client.truststore.jks
port=8443
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
exactlyOnce=true
username=default
schemas.enable=false