====== Kafka ======

===== hard dedup/exacly once =====
  * https://stackoverflow.com/questions/29647656/effective-strategy-to-avoid-duplicate-messages-in-apache-kafka-consumer
  * http://bytecontinnum.com/2016/06/exactly-kafka-storm-integration/
  * 


==== Flink ====

According [[https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html|this]] :


   Starting with Flink 1.4.0, both the Pravega and Kafka 0.11 producers provide exactly-once semantics; 
   Kafka introduced transactions for the first time in Kafka 0.11, which is what made the 
   Kafka exactly-once producer possible in Flink.
 
it means: that only from Kafka 0.11 and Flink 1.4.0 and Pravega - Exactly once supported

===== How to implement =====

  * [[https://blog.bluesoftglobal.com/dyi-data-deduplication-filter-in-10-minutes/|dedup in flink and kafka 0.11]]