1

My question is, I'm wondering if it's possible Hadoop can do ssh access with a different port number(not 22) for slaves when it makes a Hadoop cluster.

Detail)
I tried to make Hadoop cluster with two slaves. The master node is on my local computer, but the other nodes are on my AWS ec2 as Docker containers.

local : master node
AWS EC2(33.162.168.105) : first Hadoop container as slave #1, second Hadoop container as slave #2
(btw, this ip addr is fake.)

The first slave container opened 1111 port number for 22 with -p 1111:22 option,
The second slave container opened 1112 port number for 22 with -p 1112:22 option
so that remote computer can do ssh access into slave#1 with 1111 port number.

Actually, when I tried to do ssh access into slave#1 with 1111 port number from my local computer terminal, it worked well!

The problem is Hadoop ssh connection as slaves to make cluster.
I'm using Hadoop 3.1.1 version so I had to put slaves ip into /etc/hadoop/workers like,

33.162.168.105:1111

But after run start-all.sh, I showed errors.

server@9130cf720e0f:/usr/local/hadoop/etc/hadoop# start-all.sh
Starting namenodes on [master]    
Starting datanodes
33.162.168.105:1111: ssh: Could not resolve hostname 33.162.168.105:1111: Name or service not known
Starting secondary namenodes [9130cf720e0f]
Starting resourcemanager
Starting nodemanagers
33.162.168.105:1111: ssh: Could not resolve hostname 33.162.168.105:1111: Name or service not known

I know I can't add ":1111" at the end of the ip addr.
So I really wanna know if it's possible Hadoop can do ssh access with a different port number(1111 in this case) for slaves when it makes a Hadoop cluster.
If you know how to, help me, please.

Thank you for your reading and help in advance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.