While setting up Hadoop or Spark Cluster, primary objective is always to setup keyless SSH between master and slave machines after you set Static IP and FQDN properly. First of all lets understand why it is required to setup passwordless SSH is required for any distributed cluster.
Why do we need Passwordless SSH for Hadoop and Spark?
Normally distributed architectures like Hadoop and Spark Master/Slave model. Master node starts or stops demon processes on slave and/or secondary nodes using SSH. And it is always preferable to start all demon processes in slaves from master node itself. Now If SSH is not passwordless than you have to login to each individual slave nodes and manage the process manually rather than master node doing it automatically.
Normally there is one more question which bothers to many of us is,
Is Passwordless SSH should ONLY be configured from Master to each slave node or it is required to be between slaves also?
The straight answer to this question is NO. There is not any requirement for passwordless SSH among the slave nodes internally.
Let’s look at the steps for creating new non root users on CentOS first. Here I have used two machines to SSH setup.
IP 192.168.1.10, HOSTNAME master, FQDN master.backtobazics.com
IP 192.168.1.11, HOSTNAME slave1, FQDN slave1.backtobazics.com
Step 1: Create the non-root user using below commands
$ useradd [USER] $ passwd [USER]
Step 2: Add the user to wheel group
$ usermod -a -G wheel [USER]
Step 3: Open
/etc/sudoers file using vi and uncomment the following lines
$ vi /etc/sudoers %wheel ALL=(ALL) ALL %wheel ALL=(ALL) NOPASSWD: ALL
Step 4: Assign permission to base or any other directory
$ chown -R [GROUP]:[USER] ~ $ chown -R [GROUP]:[USER] /data $ chmod 775 -R ~ $ chmod 775 -R /data
Step 5: Switch user with below command
$ su [USER]
Passwordless SSH Setup
Installing SSH and firewall settings
Prior to this setup first we need to check whether SSH is installed on the machine or not using
ssh command. If you have installed minimal version of CentOS than SSH might need to be installed separately using below command after logging in with root user.
$ yum -y install openssh-server openssh-clients
Before move forward check your firewall settings in
/etc/sysconfig/iptables file. Port 22 needs to be filtered in your firewall. If you don’t find any entry with port 22 than add below line to the file and restart service.
$ vi /etc/sysconfig/iptables
Append below line if not found in file.
iptables -A INPUT -p tcp -s 126.96.36.199 --dport 22 -j ACCEPT
iptables service with below command
$ service iptable restart
Check entries in
/etc/hosts file on all machines
Make sure that
/etc/hosts file should have entries of all machine needs to be used in cluster setup. Here we are using two machines so my
/etc/hosts file on both the machines looks like below
$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.10 master.backtobazics.com master 192.168.1.11 slave1.backtobazics.com slave1
Generate SSH public private keys with
Next step is to generate public private key pairs using
ssh-keygen command. Here I am generating key pairs using RSA algorithm you can also generate it using DSA algorithm with same steps. Now we’ll login with non root user which we just have created using above steps and generate SSH key pairs.
$ su varun #Login with non root user $ ssh-keygen -t rsa #Use –t dsa for generating DSA key pairs Generating public/private rsa key pair. Enter file in which to save the key (/home/varun/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/varun/.ssh/id_rsa. Your public key has been saved in /home/varun/.ssh/id_rsa.pub. The key fingerprint is: c7:b6:b0:ec:b4:e7:67:ea:18:82:a8:b2:00:fd:e8:fa email@example.com The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | . . | |. . S + | |. + . . = . | |. o o . = . | |oo + +. o | |=+E ++++ | +-----------------+
To check for generated keys by listing files under
$ ls -Z ~/.ssh/ -rw-------. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub
Repeat above steps on both of the machines.
Update public entries in “authorized_keys” file with proper file permissions
Now update create new file authorized_keys under
~/.ssh/ directory and add public keys of both machines in both files.
$ vi ~/.ssh/authorized_keys
Add content of
id_rsa.pub file from both the machines
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3qObeu256FJXIV+/zEPe6ObVHhEZlrkcvm2OoZiMArGEtcUDOaybjJLbtwiKEL54qFNrzYggDoO6bmCf6wZRjcqHTgR22ViQI0z6A8wHQo8EzIEq8jf3e43IxC6Y96NeI8tqrnaULJoT8tHUzQnGapA0BP0FZfqj3e6Knr1CqpBpsr64IbcVR/B8fdDcwpIJWGD3cYT95yFG3mfJ/B32biohok9GZUGG2OPzPsI1/INNZ3yw5mRxYWDkPD6domMOl885FaICdpQXyZo17wWX5tUbs8C/2ZFZ6ISrz6UpP1Hq/ypjwdMmGkFGsEy82GPwSar1kaL4fXfuf96GLnUmmw== firstname.lastname@example.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtlewtGPgA7Os3nXrZdhbZSRcA7/Hv/HE8hn+/lt1yeXtLFzNGjIrlJv3Gsq+ICVNmUV/nIFkXNj5c0CTB/7al8CkkA19lOtAjO0W3fBZNON00q4Q3YYz1nwHnuqgMDSDJEt5Aoku74158hplhWrEuBlb5lVW66BMsp8Y13+1hORAVi1yDPLoWZeZmmwCMUE8qIYOaGbpH5hFWnki4l70e6UNzqq4ojUKxUV3KKv1sTKmgQXQcvYE9U7UgOVac1Vqe++nMRfnvc8Cq1TiyKLBVpicUYwDofj5cda0MGqOZdg6bpiQIx9E0thQD7R9BLRn6a3Mz4q7hJYdMozO+1wEVQ== email@example.com
Make sure your files should have proper permissions as mentioned below,
id_rsa => 0600
authorized_keys => 0644
pub => 0644
Run following commands if permissions are not proper
$ chmod 0600 ~/.ssh/id_rsa $ chmod 0644 ~/.ssh/id_rsa.pub $ chmod 0644 ~/.ssh/authorized_keys
Now it’s time to connect slave machine using below step from your master node.
$ chmod 0600 ~/.ssh/id_rsa $ ssh varun@slave1
While connecting for the first time you would get below message, just type yes and you will be connected to slave node without any password.
$ chmod 0600 ~/.ssh/id_rsa The authenticity of host 'master (192.168.1.10)' can't be established. RSA key fingerprint is bf:2a:bc:de:4b:68:54:36:ad:d3:1f:fb:65:c1:4d:f8. Are you sure you want to continue connecting (yes/no)? yes
Congratulations…..!!!!! You have successfully completed the task…..!!!!!
What if still my Keyless SSH is not working properly?
There are certain debugging steps which I have mentioned in my post 4 Debug Steps to check why your Keyless SSH is not working post which will help you to correct your settings.
Even while setting up large cluster in production environment, you should restrict password login to your secondary and slave nodes so that everything should be changes to your slave nodes via your master node and/or backup node.