Setting up keyless SSH is quite easy on CentOS but still sometimes there are chances that after following all the steps mentioned in How to setup Keyless SSH with non root users in CentOS post it is still not setup properly. There could be many possibilities as a root cause like improper permissions, invalid configuration etc. I have mentioned mainly four debug points which you should follow if your keyless ssh setup seems to be misbehaving.
Debug Point 1: Check content of file
As a first step we should check the content of
~/.ssh/authorized_keys file on master and slave.
- On master it should contain public key (will be available in
~/.ssh/id_rsa.pub. file) of that machine as well as all slave nodes
- At the same time on each slave it should only contain public key of that node and public key of master node
- Make sure that all keys are generate using the same user using with you are trying to connect to other server. User name is available at the end of each public key in [USE]>@[HOSTNAME/FQDN/IP] format.
Debug Point 2: Check & set the proper permissions of all files in
This step is already mentioned in my previous post regarding keyless SSH setup, still we should make sure by checking the permissions all files under this directory those should be as following,
$ ls -Z ~/.ssh/ -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 authorized_keys -rw-------. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub If permission are not proper than execute below steps,
$ cd ~/.ssh $ chmod 0600 id_rsa $ chmod 0644 id_rsa.pub $ chmod 0644 authorized_keys $ chmod 0644 known_hosts
Debug Point 3: Connect to other node using ssh command in debug mode
This step is very important as a debug point if you are still not able to login on other node over SSH without password. You can execute command using –v option which will display debug log for all steps and file for which SSH is looking for. Every time you add one more v (upto 3) in debug option it will show statements of next depth.
$ ssh –v [USER]@[HOSTNAME] # It will show debug log $ ssh –vv [USER]@[HOSTNAME] # More debug details $ ssh –vvv [USER]@[HOSTNAME] # Even more details than above
Debug Point 4: CentOS 6 Bug when SELinux is set to Enforcing
One more possibility is a bug in CentOS 6 related to SELinux permission. According to this bug
~/.ssh/authorized_key should have proper permission if SELinux is enable otherwise selinux will not allow SSH to access this file.
To check this permission execute,
$ ls -lZ ~/.ssh/ -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 authorized_keys -rw-------. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 id_rsa.pub -rw-r--r--. varun varun unconfined_u:object_r:ssh_home_t:s0 known_hosts
Above command will show SElinux permissions for each file. If any of above file has permission like
unconfined_u:object_r:user_home_t:s0 we should reset it using below command.
$ restorecon -Rv ~/.ssh
$ chcon -R unconfined_u:object_r:ssh_home_t:s0 ~/.ssh
Here is the link if you wish to learn more about SELinux.
There also one best practice for setting of Hadoop/Spark cluster.
Every slave node in a cluster should not have direct SSH access using password authentication it should be accessible thru master and/or stand by node only.
Do you want to learn how to disable password authentication and keeping only Keyless SSH login enabled? You can learn it from my subsequent post.