java - hbase connection refused

Question

I am new to HBase and Hadoop. I have completely setup HBase and started perfectly. Now when I tried to connect from p1 to HBase (where HBase setup is on p2) using a Java client, then it is throwing a strange exception.

12/04/17 14:36:37 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.15.20:2181 sessionTimeout=180000 watcher=hconnection
12/04/17 14:36:38 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.20:2181
12/04/17 14:36:38 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:39 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:39 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:41 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:41 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:43 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:43 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:44 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:44 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:45 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:45 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:47 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:47 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:48 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:48 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
12/04/17 14:36:49 INFO zookeeper.ClientCnxn: Opening socket connection to server hbase.local/192.168.15.20:2181
12/04/17 14:36:49 INFO zookeeper.ZooKeeper: Session: 0x0 closed
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
    at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1258)
    at HBaseDemo.main(HBaseDemo.java:55)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 7 more
12/04/17 14:36:49 INFO zookeeper.ClientCnxn: EventThread shut down
Execution script exiting 

---

/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.name=xyz
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/xyz
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/xyz/hbase-sample
12/04/17 17:41:35 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.15.169:2181 sessionTimeout=180000 watcher=hconnection
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.15.169:2181
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Socket connection established to hbase.xyz.com/192.168.15.169:2181, initiating session
12/04/17 17:41:35 INFO zookeeper.ClientCnxn: Session establishment complete on server hbase.xyz.com/192.168.15.169:2181, sessionid = 0x136c00d59ac0009, negotiated timeout = 180000
12/04/17 17:41:36 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 1 failed; no more retrying.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:604)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy4.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:360)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:1258)
at HBaseDemo.main(HBaseDemo.java:55)
HBase is not running!
Execution script exiting 

Answer 1

The accepted answer here might work, but only as a side effect.

The problem is that HBase is opening a port listening on localhost (i.e., on loopback) rather than on the external facing interface. Removing the localhost entry in /etc/hosts forces HBase to bind to the external interface -- but only because it can't find localhost. This is likely to become a problem when some daemon wants to legitimately listen on or connect to localhost (via lo) alone.

You can test that you have this problem by using telnet to connect to port 60000 via localhost or the IP assigned to the machine. It's likely you'll be able to connect via localhost (because that's where HBase is bound) and not to the IP.

Better to use your machine name in your HBase conf.