https://wiki.korotkin.co.il/ 2026-04-09T12:39:00+00:00 https://wiki.korotkin.co.il/ https://wiki.korotkin.co.il/_media/wiki/logo.png text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/ambari?rev=1641225830&do=diff ambari Install accross nodes and monitor text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/avro?rev=1641225830&do=diff Avro Data exchage. Generice for hadoop text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/cassandra?rev=1641225830&do=diff Cassandra NoSQL database text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/chukwa?rev=1641225830&do=diff chukwa RealTime log proccesing tools text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/crunch?rev=1641225830&do=diff Crunch Runing MapReduce Pipelines, Join and aggregate data text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/data_block?rev=1641225830&do=diff Data Block Data (ex. file) is splitted to Data blocks. text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/data_blocks?rev=1641225830&do=diff Data blocks * In Hadoop default is to store 64MB block size * Block placment will put block of single file near to each other by define the distance with bandwidth. text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/data_node?rev=1641225830&do=diff Data Nodes * Block OPs * Replications * Communicate directly with clients * Read and writes data to hard disk * Data node Send heart-bit every 3 secounds to Name node text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/drill?rev=1641225830&do=diff Drill Nasted Data investigation text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/edit_log?rev=1641225830&do=diff Edit log (aka Journal) Track any client changes of information text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/flume?rev=1641225830&do=diff flume RealTime log proccesing tools text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/fsimage?rev=1641225830&do=diff fsImage see Name node and Secondary name node fsImage is file that can loaded to RAM that contains mapping of Data blocks on Data nodes Structure * Snapshot of entire file system * Tracks all the files * Tracks Replication value - can be configured by Replication factor text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hadoop?rev=1641225830&do=diff Hadoop Hadoop is framework that has set of tools to distrebute and proccess data over clasters. * Scalable * Flexible * Fault-tolerant * Intelligent Main tools are HDFS (HaDoop File System), MapReduce and YARN Installing Hadoop Single node Cluser * Standalon mode - all hadoop components run under single text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hama?rev=1641225830&do=diff Hama Bulk sync processing text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hbase?rev=1641225830&do=diff HBase Tools * <https://github.com/NiceSystems/hrider/wiki> Speed-up HBase # Disable WAL LOAD HBASE DATA INPATH 'somewhere/data.txt' DELIMITED FIELDS TERMINATED BY '|' INTO TABLE ORDERS DISABLE WAL # Inc. write buffer set hbase.client.write.buffer = 8388600 text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hcatalog?rev=1641225830&do=diff HCatalog Meta table - shared schemas text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hdfs?rev=1641225830&do=diff HDFS (HaDoop File System) HDFS is one of utils that included at Hadoop framework. * HDFS knows to handle large file (Ex. 1ptb) and split it to Data blocks and distrebute it over Cluster. * HDFS has Fault tolerance - HDFS replicate same Data block on X Data nodes. X is Replication factor * HDFS Master / Slaves architecture text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hdt?rev=1641225830&do=diff HDT Hadoop development tool integrate with Eclipse text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/hive?rev=1641225830&do=diff Hive HiveQL - simmilar to SQL. text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/ibm_biginsights?rev=1641225830&do=diff # Generate root key ssh-keygen -t rsa #copy key ssh-copy-id root@bi # should be installed yum -y install expect mksh.x86_64 libaio compat-libstdc++-33 pam.i686 ksh rpm-build ntp chkconfig ntpd on ntpdate 0.asia.pool.ntp.org echo "0.000" > /var/lib/ntp/drift service ntpd start # should be password less chkconfig iptables off service iptables stop chkconfig ip6tables off service ip6tables stop iptables -F #disable selinux vi /etc/sysconfig/selinux setenforce 0 # see the text /etc/security/li… text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/install_hadoop_eco_system_single_mode?rev=1641225830&do=diff Install Hadoop eco-system singlemode install openjdk debian sudo apt-get install openjdk-7-jdk ssh rsync sudo addgroup hadoop sudo adduser --ingroup hadoop hduser su - hduser ssh-keygen -t rsa -P "" cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/job_client?rev=1641225830&do=diff Job Client Client who submit a job to Cluster text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/job_tracker?rev=1641225830&do=diff Job Tracker Master the jobs. should define execution plan and timing when to execute accross the Task trackers Similar to Name node text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/konx?rev=1641225830&do=diff Knox Cenralized security aginst clusters text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/lucene?rev=1641225830&do=diff Lucene Full text text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/mahout?rev=1641225830&do=diff mahout Machine larninge - Recommendation engin Recumnedation, Clastring, Classification text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/mapreduce?rev=1641225830&do=diff Map Reduce Map reduce work on all data inside a cluster. Job client. Master: Job tracker, Task tracker Map Used to Map the data that we want to work with (Eq. to “Where” sql) Reduce Used to aggregate the selected data from Map text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/name_node?rev=1641225830&do=diff Name node see also Secondary name node * :!: MOST IMPORATANT FOR CLUSTER! * in-memory location of every Data block in cluster / rack * Controlls DataNodes * FS Ops * Block Mapping * Send cordinates to clients where to get the Data blocks and witch Data node * Name node retrives every 3 secounds text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/oozie?rev=1641225830&do=diff Oozie Workflow tool text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/pig?rev=1641225830&do=diff Pig High level data flow. * Pig latin - Scripting language * Pig Runtime - converts Pig latin to MapReduce and execute it text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/replication_factor?rev=1641225830&do=diff Replication Factor In HDFS default RF is 3. that means each Data block will replicate 3 times on diffrent Data nodes text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/secondary_name_node?rev=1641225830&do=diff Secondary name node See Name node. NOT HIGH AVAILBILITY :!: Uses * “System restore” for Name node * Resopnsibole of keeping less dataloss if the Name node is failed How it works * Structure * fsImage * Edit Logs * Every hour copy Edit log and fsImage from Name node * If Name node failed - marges text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/spark?rev=1641225830&do=diff Spark In memory data - should do it faster text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/sqoop?rev=1641225830&do=diff sqoop Integration with relational system text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/task_tracker?rev=1641225830&do=diff Task tracker * Resposibole to execute map & reduce * Report thir progress to Job tracker * text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/thrift?rev=1641225830&do=diff Thrift Language nuralization form text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/yarn?rev=1641225830&do=diff YARN aka.MapReduce V2 text/html 2022-01-03T16:03:50+00:00 Anonymous (anonymous@undisclosed.example.com) https://wiki.korotkin.co.il/learn/bigdata/zookeeper?rev=1641225830&do=diff zookeeper Cenralized managment point, that keeps clusters and services sync