Differences

This shows you the differences between two versions of the page.

--- kb:bigdata:file_types [2018/11/21 14:23] – yehuda
+++ kb:bigdata:file_types [2022/01/03 16:03] (current) – external edit 127.0.0.1
@@ Line 2: / Line 2: @@
 [[https://www.youtube.com/watch?v=_0Wpwj_gvzg|Spark + Parquet In Depth]]
+[[https://www.youtube.com/watch?v=2vOfh064uUM|File Format Benchmark Avro JSON ORC and Parquet]]
+[[https://www.youtube.com/watch?v=aIcxFIyL6xo|Berlin buzzwords18: Owen O'Malley – Fast Access To Your Complex Data - Avro, JSON, ORC, and Parquet]]
 ===== Parquet =====
   * Better Column selecting
@@ Line 9: / Line 12: @@
   * Binary format
   * Encoded & Compressed
-  *
+  * Support schema evolution - Format supports
+Limitation:
+  * Pushdown filters dont works on String / Binary ([[https://www.youtube.com/watch?v=_0Wpwj_gvzg|source]])
+  * Write speed tradeoff
+Walkaround(s):
+  * Immutability
+    * Write using partitioning
+    * Combine with a database (i.e. Cassandara) - after a while spilt out parquets
+    * Write mode append, that added embedded schema
+=== vs ORC ===
+  * indexed
+  * dont handles nested data
+  *
 ===== ORC =====

kb/bigdata/file_types.1542810191.txt.gz · Last modified: 2022/01/03 16:03 (external edit)

Back to top