Before we move ahead lets learn a bit on Setup Apache Spark,
So, What is Apache Spark?
Apache Spark is a fast, real time and extremely expressive computing system which executes job in distributed (clustered) environment.
It is quite compatible with Apache Hadoop and more almost 10x faster than Hadoop MapReduce on Disk Computing and 100x faster using in memory computations. It provides rich APIs in Java, Scala and Python along with Functional Programming capabilities. Continue reading “6 Steps to Setup Apache Spark 1.0.1 (Multi Node Cluster) on CentOS”
Many of us are very curious about the fact that why functional references introduced in Java 8 are know as Lambda Expressions. So let’s reveal the fact of word “Lambda Expressions” and make it very simple to understand.
This fact is very much related to the Evaluation process of any expression. Many of us may have solved the mathematical problems and while solving the problems we may have observed how we are reducing the mathematical expression to some values. Similarly we also may have observed in many programming languages how any expression is evaluated. Continue reading “Why Functional References are called “Lambda Expressions” in Java 8?”
Java 8 is completing almost 2 years now and this topic is widely discussed on Internet now a days. Java was originally designed to work OOP as it was the primary paradigm for software development at that time. But today’s programming directed towards concurrent and reactive programming which focuses on functional modularity which is provided by functional languages like Scala.
Concept of functional programming is completely a new era for the hard core Java Programmers and lambda expressions are bridging OOP and FP by adding functional language capabilities to Java. You will find this concept roaming around anonymous classes and functional programming. Continue reading “Java 8: Lambda Expressions”
Scala is a widely used Functional Programming language in today’s era of programming. It is a hybrid language which combines the feature of functional and object oriented programming together. Same as Java it also convert the source code to byte code hence it require JVM in order to execute the program. You can learn more about it from this link. Continue reading “4 Steps to Setup Scala On CentOS”
Today is the era of parallel computation and whenever we talk about processing very large chunk of datasets the first word that comes in everyone’s mind is HADOOP. Apache Hadoop sits at the peak of Apache Project lists. In this post I’ll explain you all steps of setting up a Bazic Multi Node Hadoop Cluster (we’ll setup two node cluster).
Here I have used two machines for cluster setup you can repeat the steps of setting up slave nodes on more machines in order to create bigger Hadoop cluster. Continue reading “Setup Multi Node Hadoop 2.6.0 Cluster with YARN”
For many of us term Syntactic Sugar could be something out of this world and the term is too much heavy that anyone would wonder about this alien word. But many times the concept behind such terms is very simple. Now let’s make this word easily understandable. In such cases, I always split the term into separate words and try to understand the meanings of individual words in programming world prior to understand actual meaning the term.
So below is the diagram how I understand this term, Continue reading “What do you mean by term “Syntactic Sugar”?”
Before we understand practical capacities of Scala language or attempt to learn core ideas of lambda expressions in Java, it is more crucial to understand the concept of Functional Programming first.
Most of the popular programming languages (like C, C++ and Java) are primarily created to follow the methodology of Imperative Programming, which implies that programmer generally write a code block (sequence of statements) which will change the state of the program(value of the variables) as each line is executed step by step. Continue reading “Introduction to Functional Programming”
Most of the when we start to setup any kind of cluster (for Hadoop, Spark etc…) from scratch, we as a beginner often face certain problems while performing steps (like setup Static IP address or setup FQDN) after installation of Linux flavour OS on our VM or Machine. There are very less forums where you can get all of these bazics steps together along with full understanding. Here I have tried to explain you the process of setting up static IP address and FQDN (Fully Qualified Domain Name) in 5 steps. All of these steps are tested on VM/Machine available straight after minimal installation of CentOS 6.4. Continue reading “5 Steps to Setup Static IP And FQDN to CentOS Machine”
Setting up keyless SSH is quite easy on CentOS but still sometimes there are chances that after following all the steps mentioned in How to setup Keyless SSH with non root users in CentOS post it is still not setup properly. There could be many possibilities as a root cause like improper permissions, invalid configuration etc. I have mentioned mainly four debug points which you should follow if your keyless ssh setup seems to be misbehaving. Continue reading “4 Debug Points to check why your Keyless SSH is not working”
While setting up Hadoop or Spark Cluster, primary objective is always to setup keyless SSH between master and slave machines after you set Static IP and FQDN properly. First of all lets understand why it is required to setup passwordless SSH is required for any distributed cluster. Continue reading “How to setup Keyless SSH with non root users in CentOS”