Showing posts with label MapReduce. Show all posts
Showing posts with label MapReduce. Show all posts

Friday, 4 September 2015

Hadoop Interview Questions V2

1. What is Big Data?
Any data that cannot be stored into traditional RDBMS is termed as Big Data. As we know most of the data that we use today has been generated in the past 20 years. And this data is mostly unstructured or semi structured in nature. More than the volume of the data – it is the nature of the data that defines whether it is considered as Big Data or not.
2. What do the four V’s of Big Data denote?
IBM has a nice, simple explanation for the four critical features of big data:
a) Volume –Scale of data
b) Velocity –Different forms of data
c) Variety –Analysis of streaming data
d) Veracity –Uncertainty of data
IBM has a nice, simple explanation for the four critical features of big data:
a) Volume –Scale of data
b) Velocity –Different forms of data
c) Variety –Analysis of streaming data
d) Veracity –Uncertainty of data
For more the Basic questions and answers click here
2) Hadoop HDFS Interview Questions
1. What is a block and block scanner in HDFS?
Block - The minimum amount of data that can be read or written is generally referred to as a “block” in HDFS. The default size of a block in HDFS is 64MB.
Block Scanner - Block Scanner tracks the list of blocks present on a DataNode and verifies them to find any kind of checksum errors. Block Scanners use a throttling mechanism to reserve disk bandwidth on the datanode.
2. Explain the difference between NameNode, Backup Node and Checkpoint NameNode.
NameNode: NameNode is at the heart of the HDFS file system which manages the metadata i.e. the data of the files is not stored on the NameNode but rather it has the directory tree of all the files present in the HDFS file system on a hadoop cluster. NameNode uses two files for the namespace-
fsimage file- It keeps track of the latest checkpoint of the namespace.
edits file-It is a log of changes that have been made to the namespace since checkpoint.
Checkpoint Node-
Checkpoint Node keeps track of the latest checkpoint in a directory that has same structure as that of NameNode’s directory. Checkpoint node creates checkpoints for the namespace at regular intervals by downloading the edits and fsimage file from the NameNode and merging it locally. The new image is then again updated back to the active NameNode.
BackupNode:
Backup Node also provides check pointing functionality like that of the checkpoint node but it also maintains its up-to-date in-memory copy of the file system namespace that is in sync with the active NameNode.
For more the Hadoop HDFS Interview Questions click here
3) MapReduce Interview Questions
1. Explain the usage of Context Object.
Context Object is used to help the mapper interact with other Hadoop systems. Context Object can be used for updating counters, to report the progress and to provide any application level status updates. ContextObject has the configuration details for the job and also interfaces, that helps it to generating the output.
2. What are the core methods of a Reducer?
The 3 core methods of a reducer are –
1)setup () – This method of the reducer is used for configuring various parameters like the input data size, distributed cache, heap size, etc.
Function Definition- public void setup (context)
2)reduce () it is heart of the reducer which is called once per key with the associated reduce task.
Function Definition -public void reduce (Key,Value,context)
3)cleanup () - This method is called only once at the end of reduce task for clearing all the temporary files.
Function Definition -public void cleanup (context)
For more the MapReduce Interview Questions click here
4) Hadoop HBase Interview Questions
1. When should you use HBase and what are the key components of HBase?
HBase should be used when the big data application has –
1)A variable schema
2)When data is stored in the form of collections
3)If the application demands key based access to data while retrieving.
Key components of HBase are –
Region- This component contains memory data store and Hfile.
Region Server-This monitors the Region.
HBase Master-It is responsible for monitoring the region server.
Zookeeper- It takes care of the coordination between the HBase Master component and the client.
Catalog Tables-The two important catalog tables are ROOT and META.ROOT table tracks where the META table is and META table stores all the regions in the system.
For more the Hadoop HBase Interview Questions click here
5) Hadoop Sqoop Interview Questions
1. Explain about some important Sqoop commands other than import and export.
Create Job (--create)
Here we are creating a job with the name my job, which can import the table data from RDBMS table to HDFS. The following command is used to create a job that is importing data from the employee table in the db database to the HDFS file.
$ Sqoop job --create myjob \
--import \
--connect jdbc:mysql://localhost/db \
--username root \
--table employee --m 1
Verify Job (--list)
‘--list’ argument is used to verify the saved jobs. The following command is used to verify the list of saved Sqoop jobs.
$ Sqoop job --list
Inspect Job (--show)
‘--show’ argument is used to inspect or verify particular jobs and their details. The following command and sample output is used to verify a job called myjob.
$ Sqoop job --show myjob
Execute Job (--exec)
‘--exec’ option is used to execute a saved job. The following command is used to execute a saved job called myjob.
$ Sqoop job --exec myjob
For moreHadoop Sqoop Interview Questions click here
6) Hadoop Flume Interview Questions
1. Explain about the core components of Flume.
The core components of Flume are –
Event- The single log entry or unit of data that is transported.
Source- This is the component through which data enters Flume workflows.
Sink-It is responsible for transporting data to the desired destination.
Channel- it is the duct between the Sink and Source.
Agent- Any JVM that runs Flume.
Client- The component that transmits event to the source that operates with the agent.
For more Hadoop Flume Interview Questions click here
7) Hadoop Zookeeper Interview Questions
1. Can Apache Kafka be used without Zookeeper?
It is not possible to use Apache Kafka without Zookeeper because if the Zookeeper is down Kafka cannot serve client request.
2. Name a few companies that use Zookeeper.
Yahoo, Solr, Helprace, Neo4j, Rackspace
For more Hadoop Zookeeper Interview Questions click here
8) Pig Interview Questions
1. What do you mean by a bag in Pig?
Collection of tuples is referred as a bag in Apache Pig
2. Does Pig support multi-line commands?
Yes
For more Pig Interview Questions click here
9) Hive Interview Questions
1. What is a Hive Metastore?
Hive Metastore is a central repository that stores metadata in external database.
2. Are multiline comments supported in Hive?
No
For more Hive Interview Questions click here
10) Hadoop YARN Interview Questions
1. What are the stable versions of Hadoop?
Release 2.7.1 (stable)
Release 2.4.1
Release 1.2.1 (stable)
2. What is Apache Hadoop YARN?
YARN is a powerful and efficient feature rolled out as a part of Hadoop 2.0.YARN is a large scale distributed system for running big data applications.

Tuesday, 25 August 2015

Hadoop Interview Questions – HDFS

Q1. What is Hadoop?
Ans. Hadoop is free java based programming framework. It supports big data processing with distributed computing environment. 
Q2. Who is the provider of Hadoop?
Ans. Hadoop forms part of Apache project provided by Apache Software Foundation. 
Q3. What is the use of Hadoop?
Ans. With Hadoop the user can run applications on the systems that have thousands of nodes spreading through innumerable terabytes. Rapid data processing and transfer among nodes helps uninterrupted operation even when a node fails preventing system failure. 
Q4. What are the operating systems on which Hadoop works?
Ans. Windows and Linux are the preferred operating system though Hadoop can work on OS x and BSD. 
Q5. What is meant by Big Data?
Ans. Big Data refers to assortment of huge amount of data which is difficult capturing, storing, processing or reprieving. Traditional database management tools cannot handle them but Hadoop can. 
Q6. Can you indicate Big Data examples?
Ans. Facebook alone generates more than 500 terabytes of data daily whereas many other organizations like Jet Air and Stock Exchange Market generates 1+ terabytes of data every hour. These are Big Data. 
Q7. What are major characteristics of Big Data?
Ans. The three characteristics of Big Data are volume, velocity, and veracity. Earlier it was assessed in megabytes and gigabytes but now the assessment is made in terabytes. 
Q8. What is the use of Big Data Analysis for an enterprise?
Ans.  Analysis of Big Data identifies the problem and focus points in an enterprise. It can prevent big losses and make profits helping the entrepreneurs take informed decision. 
Q9. What are the characteristics of data scientists?
Ans. Data scientists analyze data and provide solutions for business problems. They are gradually replacing business and data analysts. 
Q10. What are the basic characteristics of Hadoop?
Ans. Written in Java, Hadoop framework has the capability of solving issues involving Big Data analysis. Its programming model is based on Google Mapreduce and infrastructure is based on Google’s Big Data and distributed file systems. Hadoop is scalable and more nodes can be added to it. 
Q11. Which are the major players on the web that uses Hadoop?
Ans. Introduce in 2002 by Doug Cutting, Hadoop was used in Google MapReduce and HDFS project in 2004 and 2006. Yahoo and Facebook adopted it in 2008 and 2009 respectively. Major commercial enterprises using Hadoop include EMC, Hortonworks, Cloudera, MaOR, Twitter, EBay, and Amazon among others. 
Q12. How is Hadoop different from traditional RDBMS?
Ans. RDBMS can be useful for single files and short data whereas Hadoop is useful for handling Big Data in one shot. 
Q13. What are the main components of Hadoop?
Ans. Main components of Hadoop are HDFS used to store large databases and MapReduce used to analyze them. 
Q.14. What is HDFS?
Ans. HDFS is filing system use to store large data files. It handles streaming data and running clusters on the commodity hardware. 
Q15. What are the main features of HDFS>
Ans. Great fault tolerance, high throughput, suitability for handling large data sets, and streaming access to file system data are the main features of HDFS. It can be built with commodity hardware. 
Q16. Why replication is pursued in HDFS though it may cause data redundancy?
Ans. Systems with average configuration are vulnerable to crash at any time. HDFS replicates and stores data at three different locations that makes the system highly fault tolerant. If data at one location becomes corrupt and is inaccessible it can be retrieved from another location. 
Q17. Would the calculations made on one node be replicated to others in HDFS?
Ans. No! The calculation would be made on the original node only. In case the node fails then only the master node would replicate the calculation on to a second node. 
Q18. What is meant by streaming access?
Ans. HDFS works on the principle of “write once, read many” and the focus is on fast and accurate data retrieval. Steaming access refers to reading the complete data instead of retrieving single record from the database. 
Q19. What is meant by ‘commodity hardware’? Can  Hadoop work on them?
Ans. Average and non-expensive systems are known as commodity hardware and Hadoop can be installed on any of them. Hadoop does not require high end hardware to function. 
Q20. Which one is the master node in HDFS? Can it be commodity?
Ans. Name node is the master node in HDFS and job tracker runs on it. The node contains metadata and works as high availability machine and single pint of failure in HDFS. It cannot be commodity as the entire HDFS works on it. 
Q21. What is meant by Data node?
Ans. Data node is the slave deployed in each of the systems and provides the actual storage locations and serves read and writer requests for clients. 
Q22. What is daemon?
Ans. Daemon is the process that runs in background in the UNIX environment. In Windows it is ‘services’ and in DOS it is ‘TSR’. 
Q23. What is the function of ‘job tracker’?
Ans. Job tracker is one of the daemons that runs on name node and submits and tracks the MapReduce tasks in Hadoop. There is only one job tracker who distributes the task to various task trackers. When it goes down all running jobs comes to a halt. 
Q24. What is the role played by task trackers?
Ans. Daemons that run on What data nodes, the task tracers take care of individual tasks on slave node as entrusted to them by job tracker. 
Q25.What is meant by heartbeat in HDFS?
Ans. Data nodes and task trackers send heartbeat signals to Name node and Job tracker respectively to inform that they are alive. If the signal is not received it would indicate problems with the node or task tracker. 
Q26. Is it necessary that Name node and job tracker should be on the same host?
Ans. No! They can be on different hosts. 
Q.27. What is meant by ‘block’ in HDFS?
Ans.  Block in HDFS refers to minimum quantum of data for reading or writing. Default block size is 64 MB in HDFS. If a file is 52 MB then HDFS would store it and leave 12 MB empty and ready to use. 
Q.28. Can blocks be broken down by HDFS if a machine does not have the capacity to copy as many blocks as the user wants?
Ans.  Blocks in HDFS cannot be broken. Master node calculates the required space and how data would be transferred to a machine having lower space. 
Q.29. What is the process of indexing in HDFS?
Ans. Once data is stored HDFS will depend on the last part to find out where the next part of data would be stored. 
Q.30. How a data node is identified as saturated?
Ans. When a data node is full and has no space left the name node will identify it. 
Q31. What type of data is processed by Hadoop?
Ans. Hadoop processes the digital data only. 
Q32. How Name node determines which data node to write on?
Ans. Name node contains metadata or information in respect of all the data nodes and it will decide which data node to be used for storing data. 
Q33. Who is the ‘user’ in HDFS?
Ans. Anyone who tries to retrieve data from database using HDFS is the user. Client is not end user but an application that uses job tracker and task tracker to retrieve data. 
Q.34. How the client communicates with Name node and Data node in HDFS?
Ans. The communication mode for clients with name node and data node in HDFS is SSH. 
Q.35. What is a rack in HDFS?
Ans. Rack is the storage location where all the data nodes are put together. Thus it is a physical collection of data nodes stored in a single location.