Show HN: C++ AWS MSK IAM Auth Implementation – Goodbye Kafka Passwords
github.comIn 2023, AWS announced[1] IAM authentication for MSK Kafka clusters with support for "all programming languages"… except C++. While Java[2], Python[3], Go[4], and others got official SDKs, C++ developers/vendors were stuck hardcoding SCRAM-SHA credentials in code/configs or relying on heavier Java-based tools like Kafka Connect or Apache Flink.
Later, community projects added Rust[5] and Ruby[6] support. Why no C++? Rust might be the new favorite, but C++ is still king for high-performance data systems: minimal dependencies, lean resource use, and raw speed.
At Timeplus, we needed IAM auth for our C++ streaming engine, Proton, so we built it ourselves. Today, we’re open-sourcing our code for AWS MSK IAM authentication. It’s live in Timeplus Proton 1.6.12
Just attach an IAM role to your EC2 instance or EKS pod, then put the Timeplus Proton single binary inside, start the server, then run the following SQL to read or write MSK:
CREATE EXTERNAL STREAM msk_stream(column_defs) SETTINGS type='kafka',topic='topic2', brokers='prefix.kafka.us-west-2.amazonaws.com:9098', security_protocol='SASL_SSL', sasl_mechanism='AWS_MSK_IAM';
The core logic is just two files under 200 lines and you can reuse the code anywhere. https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka... https://github.com/timeplus-io/proton/blob/develop/src/IO/Ka...
We’d love to get your feedback and work together to make this a standalone library—or even get it into ClickHouse or AWS SDK for C++.
For those curious about Timeplus Proton: it’s an open-source streaming engine we built in C++ (think “FlinkSQL in C++” meets ClickHouse’s columnar storage). Later this month, we will also open-source our C++ code for Apache Iceberg read&write. Stay tuned.
Links:
[1] https://aws.amazon.com/blogs/big-data/amazon-msk-iam-authent... [2] https://github.com/aws/aws-msk-iam-auth [3] https://github.com/aws/aws-msk-iam-sasl-signer-python [4] https://github.com/aws/aws-msk-iam-sasl-signer-go [5] https://docs.rs/aws-msk-iam-sasl-signer [6] https://rubygems.org/gems/aws-msk-iam-sasl-signer/
Congratulations, and how shameful for AWS not to meet developers where they are
MSK IAM support has long mystified me. I think they only supported Java for the first 9 months or so. Even then they still don't have GO or PHP support. It's not a ton of work, they're reusing request signer code anyways.
According to my teammate who actually wrote the C++ code for this, there are lack of documentations of how the AWS_MSK_IAM is supposed to work. He has to check the Java/Python implementation line by line to avoid those guesswork
Well, there's precedent for that since the $(aws eks get-token) is just a base64 pre-signed GetCallerIdentity URL but I don't think that's documented anywhere, either, but can be spotted by squinting at aws-iam-authenticator source
My suspicion is that if they didn't want to bother to write a C++ client, they for sure wouldn't have the empathy(?) to document how anyone else could, too. I said empathy but I kind of wonder if by publishing how something works they're committing to it, versus they're currently only one commit away from changing it in their clients, without having to notify anyone
That’s how it works when writing CDK outside of TypeScript. You have to review the TS docs to get anywhere.
Exactly, once figured out how it works, the implementation is quite straightforward.
It's interesting that AWS chose(?) not to support this. Can you think of any reason why they wouldn't have? Perhaps it's harder to predict use cases in C++?
I don't know exactly why AWS didn't include C++ SDK when they announced all languages are supported. Maybe it's more common today to use Java/Rust/Python to build data products, but C++ is still such a solid case. Anyway, I will try to reach out AWS team to see whether they are open to put this into the AWS SDK for C++
It's great that you're contributing back like that
surprised since aws has sdk for c++ https://github.com/aws/aws-sdk-cpp
I tried to find related code with iam and msk in the file names and failed. To get authenticated with MSK, need to get proper signed and pass to the MSK endpoints. Maybe AWS SDK team are just too busy or not get enough customer ask(escalations?) for using C++ to access MSK cluster with IAM only auth.
Having a SDK and supporting IAM auth for MSK are different stories though. I mean, for supporting IAM auth for MAK for Java, AWS has a separate library instead of putting the code in their Java AWS SDK: https://github.com/aws/aws-msk-iam-auth.
yeah what i meant is that it's weird that they consider cpp is first class supported language but not having iam auth for msk - cpp.
[dead]
[dead]
[dead]
[flagged]