Differences

This shows you the differences between two versions of the page.

--- kb:kafka [2019/02/21 11:01] – created yehuda
+++ kb:kafka [2022/01/03 16:03] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Kafka ======
-http://bytecontinnum.com/2016/06/exactly-kafka-storm-integration/
+===== hard dedup/exacly once =====
+  * https://stackoverflow.com/questions/29647656/effective-strategy-to-avoid-duplicate-messages-in-apache-kafka-consumer
+  * http://bytecontinnum.com/2016/06/exactly-kafka-storm-integration/
+  *
+==== Flink ====
+According [[https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html|this]] :
+   Starting with Flink 1.4.0, both the Pravega and Kafka 0.11 producers provide exactly-once semantics;
+   Kafka introduced transactions for the first time in Kafka 0.11, which is what made the
+   Kafka exactly-once producer possible in Flink.
+it means: that only from Kafka 0.11 and Flink 1.4.0 and Pravega - Exactly once supported
+===== How to implement =====
+  * [[https://blog.bluesoftglobal.com/dyi-data-deduplication-filter-in-10-minutes/|dedup in flink and kafka 0.11]]

kb/kafka.1550746878.txt.gz · Last modified: 2022/01/03 16:03 (external edit)

Back to top