Showing posts with label NoSql. Show all posts
Showing posts with label NoSql. Show all posts

Wednesday, 24 June 2015

Spark-Part2

What is Apache Spark?
Why it is a hot topic in Big Data forums?
Is Apache Spark going to replace hadoop?
If you are into BigData analytics business then, should you really care about Spark?
I hope this blog post will help to answer some of your questions which might have coming to your mind these days.

Introduction to Apache Spark

It is a framework for performing general data analytics on distributed computing cluster like Hadoop.It provides in memory computations for increase speed and data process over mapreduce.It runs on top of existing hadoop cluster and access hadoop data store (HDFS), can also process structured data in Hive and Streaming data from HDFS,Flume,Kafka,Twitter
Spark Architecture

Is Apache Spark going to replace Hadoop?

Hadoop is parallel data processing framework that has traditionally been used to run map/reduce jobs. These are long running jobs that take minutes or hours to complete. Spark has designed to run on top of Hadoop and it is an alternative to the traditional batch map/reduce model that can be used for real-time stream data processing and fast interactive queries that finish within seconds. So, Hadoop supports both traditional map/reduce and Spark.
We should look at Hadoop as a general purpose Framework that supports multiple models and We should look at Spark as an alternative to Hadoop MapReduce rather than a replacement to Hadoop.
Hadoop Echo System

Hadoop MapReduce vs. Spark –Which One to Choose?

Spark uses more RAM instead of network and disk I/O its relatively fast as compared to hadoop. But as it uses large RAM it needs a dedicated high end physical machine for producing effective results
It all depends and the variables on which this decision depends keep on changing dynamically with time.


Difference between Hadoop Mapreduce and Apache Spark

Spark stores data in-memory whereas Hadoop stores data on disk. Hadoop uses replication to achieve fault tolerance whereas Spark uses different data storage model, resilient distributed datasets (RDD), uses a clever way of guaranteeing fault tolerance that minimizes network I/O.For details see the UC Berkeley's link Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing.
From the Spark academic paper: "RDDs achieve fault tolerance through a notion of lineage: if a partition of an RDD is lost, the RDD has enough information to rebuild just that partition." This removes the need for replication to achieve fault tolerance.

Do I need to learn Hadoop first to learn Apache Spark?

No, you don't need to learn Hadoop to learn Spark. Spark was an independent project . But after YARN and Hadoop 2.0, Spark became popular because Spark can run on top of HDFS along with other Hadoop components. Spark has become another data processing engine in Hadoop ecosystem and which is good for all businesses and community as it provides more capability to Hadoop stack.
For developers, there is almost no overlap between the two. Hadoop is a framework in which you write MapReduce job by inheriting Java classes. Spark is a library that enables parallel computation via function calls.
For operators to running a cluster, there is an overlap in general skills, such as monitoring configuration, and code deployment.

Apache Spark's features

Lets go through some of Spark's features which are really highlighting it in the Big Data world!
From http://spark.apache.org/:


i) Speed:

Spark enables applications in Hadoop clusters to run up to 100x faster in memory, and 10x faster even when running on disk. Spark makes it possible by reducing number of read/write to disc. It stores this intermediate processing data in-memory. It uses the concept of an Resilient Distributed Dataset (RDD), which allows it to transparently store data on memory and persist it to disc only it’s needed. This helps to reduce most of the disc read and write – the main time consuming factors – of data processing.
Logistic regression in Hadoop and Spark

ii) Ease of Use:

Spark lets you quickly write applications in Java, Scala, or Python. This helps developers to create and run their applications on their familiar programming languages and easy to build parallel apps. It comes with a built-in set of over 80 high-level operators.We can use it interactively to query data within the shell too.
Word count in Spark's Python API
datafile = spark.textFile("hdfs://...")
datafile.flatMap(lambda line: line.split())
        .map(lambda word: (word, 1))
        .reduceByKey(lambda x, y: x+y)

iii) Combines SQL, streaming, and complex analytics.

In addition to simple “map” and “reduce” operations, Spark supports SQL queries, streaming data, and complex analytics such as machine learning and graph algorithms out-of-the-box. Not only that, users can combine all these capabilities seamlessly in a single workflow.

iv) Runs Everywhere

Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, S3.

Spark’s major use cases over Hadoop

  • Iterative Algorithms in Machine Learning
  • Interactive Data Mining and Data Processing
  • Spark is a fully Apache Hive-compatible data warehousing system that can run 100x faster than Hive.
  • Stream processing: Log processing and Fraud detection in live streams for alerts, aggregates and analysis
  • Sensor data processing: Where data is fetched and joined from multiple sources, in-memory dataset really helpful as they are easy and fast to process.
Note : Spark is still working out bugs as it matures.

Your Turn: Go Get Started

Spark is very easy to get started writing powerful Big Data applications. Your existing Hadoop and/or programming skills will have you productively interacting with your data in minutes. Go get started today:

Wednesday, 13 May 2015

Hadoop Daemons Generated Logs

Apache Hadoop’s jobtracker, namenode, secondary namenode, datanode, and tasktracker all generate logs. That includes logs from each of the daemons under normal operation, as well as configuration logs, statistics, standard error, standard out, and internal diagnostic information. Many  users aren’t entirely sure what the differences are among these logs, how to analyze them, or even how to handle simple administrative tasks like log rotation.  This blog post describes each category of log, and then details where they can be found for each Hadoop component.
The log categories are:
·         Hadoop Daemon Logs
These logs are created by the Hadoop daemons, and exist on all machines running at least one Hadoop daemon. Some of the files end with .log, and others end with .out. The .out files are only written to when daemons are starting. After daemons have started successfully, the .out files are truncated. By contrasts, all log messages can be found in the .log files, including the daemon start-up messages that are sent to the .out files. There is a .log and .out file for each daemon running on a machine. When the namenode, jobtracker, and secondary namenode are running on the same machine, then there are six daemon log files: a .log and .out for the each of the three daemons.
The .log and .out file names are constructed as follows:
hadoop-<user-running-hadoop>-<daemon>-<hostname>.log
where <user-running-hadoop> is the user running the Hadoop daemons (this is always ‘hadoop’ with Cloudera’s distribution), <daemon> is the daemon these logs are associated (for example, namenode or jobtracker), and <hostname> is the hostname of the machine on which the daemons are running.
For example:
hadoop-hadoop-datanode-ip-10-251-30-53.log
By default, the .log files are rotated daily by log4j. This is configurable with /etc/hadoop/conf/log4j.properties. Administrators of a Hadoop cluster should review these logs regularly to look for cluster-specific errors and warnings that might have to do with daemons running incorrectly.  Note that the namenode and secondarynamenode logs should not be deleted more frequently than fs.checkpoint.period, so in the event of a secondarynamenode edits log compaction failure, logs from the namenode and secondarynamenode will be available for diagnostics.
These logs grow slowly when the cluster is idle. When jobs are running, they grow very rapidly. Some problems create considerably more log entries, but some problems only create a few infrequent messages. For example, if the jobtracker can’t connect to the namenode, the jobtracker daemon logs explode with the same error (something like “Retrying connecting to namenode [..]“). Lots of log entries here does not necessarily mean that there is a problem: you have to search through these logs to look for a problem.
·         Job Configuration XML
The job configuration XML logs are created by the jobtracker. The jobtracker creates a .xml file for every job that runs on the cluster. These logs are stored in two places:/var/log/hadoop and /var/log/hadoop/history. The XML file describes the job configuration.
The /hadoop file names are constructed as follows:
job_<job_ID>_conf.xml
For example:
job_200908190029_0001_conf.xml
The /hadoop/history file names are constructed as follows:
<hostname>_<epoch-of-jobtracker-start>_<job-id>_conf.xml
where <hostname> is the hostname of the machine on which the jobtracker is running, <epoch-of-jobtracker-start> is the number of milliseconds that had elapsed since Unix Epoch when the jobtracker daemon was started, and <job-id> is the job ID. For example:
ec2-72-44-61-184.compute-1.amazonaws.com_1250641772616_job_200908190029_0001_conf.xml
These logs are not rotated. These files may be more interesting to developers than system administrators, because their contents are job-specific. You can clear these logs periodically without affecting Hadoop. However, consider archiving the logs if they are of interest in the job-development process. Make sure you do not move or delete a file that is currently being written to by a running job.
Individual job configuration logs are created for each job that is submitted to the cluster. Each log file will be more or less the same size for each job.
·         Job Statistics
These logs are created by the jobtracker. The jobtracker runtime statistics from jobs to these files. Those statistics include task attempts, time spent shuffling, input splits given to task attempts, start times of tasks attempts and other information.
The statistics files are named:
<hostname>_<epoch-of-jobtracker-start>_<job-id>_<job-name>
where <hostname> is the hostname of the machine creating these logs, <epoch-of-jobtracker-start> is the number of milliseconds that had elapsed since Unix Epoch when the jobtracker daemon was started, <job-id> is the job ID, and <job-name> is the name of the job.
For example:
ec2-72-44-61-184.compute-1.amazonaws.com_1250641772616_job_200908190029_0002_hadoop_test-mini-mr
These logs are not rotated.  You can clear these logs periodically without affecting Hadoop. However, consider archiving the logs if they are of interest in the job development process. Make sure you do not move or delete a file that is being written to by a running job.
Individual statistics logs are created for each job that is submitted to the cluster. The size of each log file varies. Jobs with more tasks produce larger files.
·         Standard Error for a particular task attempt
These logs are created by each tasktracker. They contain information written to standard error (stderr) captured when a task attempt is run. These logs can be used for debugging.  For example, a developer can include System.err.println(“some useful information”) calls in the job code. The output will appear in the standard error files.
The parent directory name for these logs is constructed as follows:
/var/log/hadoop/userlogs/attempt_<job-id>_<map-or-reduce>_<attempt-id>
where <job-id> is the ID of the job that this attempt is doing work for, <map-or-reduce> is either “m” if the task attempt was a mapper, or “r” if the task attempt was a reducer, and <attempt-id> is the ID of the task attempt.
For example:
/var/log/hadoop/userlogs/attempt_200908190029_0001_m_000001_0
These logs are rotated according to the mapred.userlog.retain.hours property. You can clear these logs periodically without affecting Hadoop. However, consider archiving the logs if they are of interest in the job development process. Make sure you do not move or delete a file that is being written to by a running job.
The size of these log files depends on the amount of stderr calls used in the job code.
·         Standard Out for a particular task attempt
These logs are very similar to the standard error logs, except they capture stdout instead of stderr. File size depends entirely on how much data the task writes to stdout.
·         log4j informational messages from within the task process
These logs contains anything that log4j writes from within the task process. This includes some Hadoop internal diagnostic info. If the job’s mapper or reducer implementations include call such as LOG.info(), then that output also get written here. Messages can include information about the task, such as how big its record buffer was or how many reduce tasks there are. The size of these log files depends on the number of log4j calls used in the job code.
These logs are rotated according to the mapred.userlog.retain.hours property. You can clear these logs periodically without affecting Hadoop. However, consider archiving the logs if they are of interest in the job development process. Make sure you do not move or delete a file that is being written to by a running job.
The Hadoop components produce the following logs when installing Hadoop from CDH RPM or DEB packages:
·         On the jobtracker:
1
2
3
4
5
6
/var/log/hadoop
              /hadoop-* =&gt; daemon logs
              /job_*.xml =&gt; job configuration XML logs
              /history
                     /*_conf.xml =&gt; job configuration logs
                     &lt; everything else &gt; =&gt; job statistics logs
·         On the namenode:
1
2
/var/log/hadoop
              /hadoop-* =&gt; daemon logs
·         On the secondary namenode:
1
2
/var/log/hadoop
              /hadoop-* =&gt; daemon logs
·         On the datanode:
1
2
/var/log/hadoop
              /hadoop-* =&gt; daemon logs
·         On the tasktracker:
1
2
3
4
5
6
7
/var/log/hadoop
              /hadoop-* =&gt; daemon logs
              /userlogs
                      /attempt_*
                               /stderr =&gt; standard error logs
                               /stdout =&gt; standard out logs
                               /syslog =&gt; log4j logs

It’s probably clear now that Hadoop generates plenty of logs, each with a different purpose and audience.  I hope this post will help you debug your MapReduce jobs and Hadoop cluster.

WordCountProgram with Counters


import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCountWithCounter {

        static enum WordsNature { STARTS_WITH_DIGIT, STARTS_WITH_LETTER, ALL }
        /**
         * The map class of WordCount.
         */
        public static class TokenCounterMapper
        extends Mapper<Object, Text, Text, IntWritable> {

               private final static IntWritable one = new IntWritable(1);
               private Text word = new Text();

               public void map(Object key, Text value, Context context)
               throws IOException, InterruptedException {
                       StringTokenizer itr = new StringTokenizer(value.toString());
                       while (itr.hasMoreTokens()) {
                               word.set(itr.nextToken());
                               context.write(word, one);
                       }
               }
        }
        /**
         * The reducer class of WordCount
         */
        public static class TokenCounterReducer
        extends Reducer<Text, IntWritable, Text, IntWritable> {
               public void reduce(Text key, Iterable<IntWritable> values, Context context)
               throws IOException, InterruptedException {
                       int sum = 0;
                      
                       String token = key.toString();
                       if( StringUtils.startsWithDigit(token) ){
                               context.getCounter(WordsNature.STARTS_WITH_DIGIT).increment(1);
                       }
                       else if( StringUtils.startsWithLetter(token) ){
                               context.getCounter(WordsNature.STARTS_WITH_LETTER).increment(1);
                       }
                       context.getCounter(WordsNature.ALL).increment(1);
                      
                       for (IntWritable value : values) {
                               sum += value.get();
                       }
                       context.write(key, new IntWritable(sum));
               }
        }
        /**
         * The main entry point.
         */
        public static void main(String[] args) throws Exception {
               Configuration conf = new Configuration();
               Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
               job.setJarByClass(WordCountOld.class);
               job.setMapperClass(TokenCounterMapper.class);
               job.setReducerClass(TokenCounterReducer.class);
               job.setOutputKeyClass(Text.class);
               job.setOutputValueClass(IntWritable.class);
               FileInputFormat.addInputPath(job, new Path(args[0]));
               FileOutputFormat.setOutputPath(job, new Path(args[1]));
               System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
} 


Tuesday, 12 May 2015

NoSql

Need for NoSQL:
In today’s time data is becoming easier to access and capture through third parties such as Facebook, Google+ and others. Personal user information, social graphs, geo location data, user-generated content and machine logging data are just a few examples where the data has been increasing exponentially. To avail the above service properly, it is required to process huge amount of data. Which SQL databases were never designed. The evolution of NoSql databases is to handle these huge data properly.



What is NoSQL:
NoSQL is a non-relational database management systems, different from traditional relational database management systems in some significant ways. It is designed for distributed data stores where very large scale of data storing needs (for example Google or Facebook which collects terabits of data every day for their users). These type of data storing may not require fixed schema, avoid join operations and typically scale horizontally.
Characteristics of NoSQL:
- Stands for Not Only SQL
- No declarative query language
- No predefined schema
 
- Key-Value pair storage, Column Store, Document Store, Graph databases
- Eventual consistency rather ACID property
 
- Unstructured and unpredictable data
- CAP Theorem
 
- Prioritizes high performance, high availability and scalability
-open source
- Came out of 21st century web culture
Even though NoSQL don’t have any schema, it has Implicit schema . If you say order[price],it expects to have a price column.

DataModels:
NoSQL databases use different data models.
Companies have developed different datamodels to cater their own requirements.For example, Google developed Bigtable and amzon developed Dynamo which are their own datastorage systems.
There are four data models :
1.key/value
2.document based
3.Column family based
4.Graph based
An aggregate is a collection of data that we interact with as a unit. Aggregates form the boundaries for ACID operations with the database. The first three i.e,Key-value, document, and column-family databases can all be seen as forms of aggregate-oriented database. Aggregates make it easier for the database to manage data storage over clusters.
Instead of splattering data across a number of tables, in these datamodel, it can be stored in one go and the database what you aggregate boundaries are .



NoSQL databases from Different Companies:


1.Key / Value databases:
The simplest one is key/value pair. Database does n’t know what the value is, the value can be a image or a document. The model is reduced to a simple hash table which consists of key / value pairs. It is often easily distributed across multiple servers. The most famous products of this group include Redis, Dynamo and Riak.
Example application: Consider a forum software where you have a home profile page that gives the user's statistics (messages posted, etc) and the last ten messages by them. The page reads from a key that is based on the user's id and retrieves a string of JSON that represents all the relevant information. A background process recalculates the information every 15 minutes and writes to the store independently.


2.Document databases:
The second data model which is common is document data model. This data model consists of document collections where single document can have multiple fields, without necessarily defining a schema. Each doc can be a complex data structure ,usually that datastructure is represented in forms of JSON,it can be done in xml also.
Portions of documents can be retrieved using some rows and columns. Difference with key/value is ,its opaque and the document is transparent.
Example application: Consider a software that creates profiles of refugee children with the aim of reuniting them with their families. The details you need to record for each child vary tremendously with circumstances of the event and they are built up piecemeal, for example a young child may know their first name and you can take a picture of them but they may not know their parent's first names. Later a local may claim to recognise the child and provide you with additional information that you definitely want to record but until you can verify the information you have to treat it sceptically.
The best known and used are MongoDB, CouchDB and ArangoDB;

3.Column family:
The third type of data model is column family datamodel.It says it has a single key called row key with in that multiple column family where each column family is a combination of columns that can fit together.Data is stored in sections of columns which offers more flexibility and easy aggregation. Facebook's Cassandra, BigTable from Google or Amazon's SimpleDB are the examples which belongs to this group.Its slightly complex datamodel,its easy to pull individual columns.



4.Graph:
Graph databases are not aggregated oriented at all. It is basically of node and graph datamodel.Its easy to handle relationships and its optimized to handle those.
This domain model consists of vertices interconnected by edges which creates a rich graph structure.
Examples of this group are for example OrientDB, Neo4J, Sones and InfinityGraph.
Example application: Any application that requires social networking is best suited to a graph database. These same principles can be extended to any application where you need to understand what people are doing, buying or enjoying so that you can recommend further things for them to do, buy or like. Any time you need to answer the question along the lines of "What restaurants do the sisters of people who are over-40, enjoy skiing and have visited Kenya dislike?" a graph database will usually help.


Aggregate oriented takes a lot of data which is splattered and put them into bigger lump .where as graph break into smaller units and play with smaller units carefully.


CAP Theorem (Brewer’s Theorem)
CAP theorem states that there are three basic requirements which exist in a special relation when designing applications for a distributed architecture.
Consistency - This means that the data in the database remains consistent after the execution of an operation. For example after an update operation all clients see the same data.
Availability - This means that the system is always on (service guarantee availability), no downtime.
Partition Tolerance - This means that the system continues to function even the communication among the servers is unreliable, i.e. the servers may be partitioned into multiple groups that cannot communicate with one another.
In theoretically it is impossible to fulfill all 3 requirements. CAP provides the basic requirements for a distributed system to follow 2 of the 3 requirements. Therefore all the current NoSQL database follow the different combinations of the C, A, P from the CAP theorem. Here is the brief description of three combinations CA, CP, AP :
CA - Single site cluster, therefore all nodes are always in contact. When a partition occurs, the system blocks. 
CP - Some data may not be accessible, but the rest is still consistent/accurate. 
AP - System is still available under partitioning, but some of the data returned may be inaccurate.


NoSQL pros/cons
Advantages 
- High scalability
- Distributed Computing
- Lower cost
 
- Schema flexibility, semi-structure data
- No complicated Relationships

Disadvantages 
- No standardization
- Limited query capabilities (so far)
- Eventual consistent is not intuitive to program for


Summary:
Category Description Name of the database
Document Oriented Data is stored as documents. An example format may be like - FirstName="Arun", Address="St. Xavier's Road", Spouse=[{Name:"Kiran"}], Children=[{Name:"Rihit", Age:8}] CouchDB, Jackrabbit, MongoDB, OrientDB, SimpleDB,Terrastore etc.
XML database Data is stored in XML format BaseX, eXist, MarkLogic Server etc.
Graph databases Data is stored as a collection of nodes, where nodes are analogous to objects in a programming language. Nodes are connected using edges. AllegroGraph, DEX, Neo4j, FlockDB, Sones GraphDB etc.
Key-value store In Key-value-store category of NoSQL database, an user can store data in schema-less way. A key may be strings, hashes, lists, sets, sorted sets and values are stored against these keys. Cassandra, Riak, Redis, memcached, BigTable etc.