SSL Encryption for Kafka Pipelines

Question

Any [SingleStore] users out there?

For enabling SSL encryption (only) for Kafka pipelines, we created the pipeline like below:

CREATE PIPELINE `kafka_ssl`
AS LOAD DATA KAFKA '<kafka-host>:<kafka-port>/test'
CONFIG '{"security.protocol": "ssl"
"ssl.ca.location": "<path-to-the-ca-certificate>"}''
INTO table <t>;

from https://docs.memsql.com/v7.0/concepts/pipelines/kafka-kerberos-ssl/ 22

Please note that we’re not looking for Client authentication (mutual TLS connection). This requires us to manually copy the CA Cert into memsql nodes and provide that path. This works fine for standalone Kafka cluster.

What is the recommended approach if we’re using AWS MSK?

Following documentation points to the client configuration for SSL encryption for MSK clients. They use client truststore location as a property which is not an identified property for memsql pipeline config json

https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.html 10 https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html

See others having the same issue in the SingleStore forums. Unable to create a pipeline from Singlestore in a box to AWS MSK.

Answer 1

Let's suppose,

This is your (== customer’s) service name: com.amazonaws.vpce.ap-southeast-2.vpce-svc-05738b36bf477df03

This is mine (== ss’s) dns name for your endpoint: com.amazonaws.vpce.ap-southeast-2.vpce-svc-05738b36bf477df03 vpce-07e59dd12b36925ee-q2lyxq0y.vpce-svc-05738b36bf477df03.ap-southeast-2.vpce.amazonaws.com vpce-svc-06f99678d75724074.ap-southeast-2.vpce.amazonaws.com

DNS == vpce-07e59dd12b36925ee-q2lyxq0y.vpce-svc-05738b36bf477df03.ap-southeast-2.vpce.amazonaws.com Which you use for your __spoof.dns; vpce-svc-06f99678d75724074.ap-southeast-2.vpce..com com.amazonaws.vpce.ap-southeast-2.vpce-svc-06f99678d75724074

CREATE OR REPLACE PIPELINE kafka_insert_thing_event_v0 AS LOAD DATA KAFKA 'DNS:6000/prod_common_v0_event_wrapper_thing_event' CONFIG '{ "__spoof.dns": { "DNS:6000": "DNS:6000", "b-1.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6001", "b-2.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6002", "b-3.events.eg3qf0.c2.kafka.ap-southeast-2.amazonaws.com:9092": "DNS:6003" } }' INTO TABLE t ;

I have also attached MSK documentation for integrating with SingleStore for the reference. I hope it will help you.

https://docs.google.com/document/d/17nl9S__oxIo4J30OM1WHTkkqrCSYFqJ44sUcJxKM0QA/edit?usp=sharing&urp=gmail_link