https://wiki.korotkin.co.il/ 2026-04-30T00:02:43+00:00 https://wiki.korotkin.co.il/ https://wiki.korotkin.co.il/_media/wiki/logo.png text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/apache_phoenix?rev=1641225830&do=diff Apache Phoenix Update hbase table CREATE VIEW "users_data" ( ROWKEY VARCHAR PRIMARY KEY, "personal_info"."firstName" VARCHAR, "personal_info"."lastName" VARCHAR, "personal_info"."gender" VARCHAR, "contact_info"."mail" VARCHAR, "contact_info"."mobile" VARCHAR ) ; text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/cassandra?rev=1641225830&do=diff Cassandra DBeaver GUI tool text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/eco-system?rev=1641225830&do=diff BigData ecoSystem and tools etc * Apache Ozone text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/file_types?rev=1641225830&do=diff File types Spark + Parquet In Depth File Format Benchmark Avro JSON ORC and Parquet Berlin buzzwords18: Owen O'Malley – Fast Access To Your Complex Data - Avro, JSON, ORC, and Parquet Parquet * Better Column selecting * Columnar format * Binary format * Encoded & Compressed * Support schema evolution - Format supports Limitation: * Pushdown filters dont works on String / Binary (source) * Write speed tradeoff text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/hive?rev=1641225830&do=diff Quick code set hive.execution.engine=mr; set hive.execution.engine=tez; hive --hiveconf hive.root.logger=DEBUG,console ADDING UDFs grant all on uri 'file:///opt/local/hive/lib/tex*********t.jar' to role etl_ops drop function etl_db.test_yehuda_hash create function etl_db.test_yehuda_hash as 'com.*******HashMultiParamsUDF' using jar 'hdfs:///data_lake/udfs/tex***.jar'; select et2l_db.test_yehuda_hash("test","a") text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/hortonworks?rev=1641225830&do=diff Hortonworks hdp passwords text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/knox?rev=1641225830&do=diff Knox * Add a service * HDP 2.6.5 Knox documentation text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/nifi?rev=1641225830&do=diff Apache NiFi Nice presentations: * <https://www.slideshare.net/Hadoop_Summit/best-practices-and-lessons-learnt-from-running-apache-nifi-at-renault> text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/spark?rev=1641225830&do=diff Spark Snippits * Import Json * ML Save and Load model * Spark Scala and Maven pom.xml * PySpark and HBase Spark ML * spark-mllib-pipelines-persistence * ML Pipeline text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/sqoop?rev=1641225830&do=diff Sqoop TeraData sqoop Sqoop from teradata import from teradata text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/kb/bigdata/yarn?rev=1641225830&do=diff Yarn Snippets Show stdout logs # located /usr/bin/yarn yarn logs --applicationId application_1513381650251_42086