Hbase异常宕机，NoServerForRegionException: Unable to find region（1）

  • javenzhen

    2015-09-12 10:59:35
  • Hadoop
  • 原创
1、在创建了表mytab之后，物理机出故障，导致虚拟机异常宕机重启，Hadoop->Zookeeper->Hbase启动之后查看表
  1. hbase(main):001:0> scan 'mytab'
  2. ROW COLUMN+CELL

  4. ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for mytab,,99999999999999 after 7 tries.

  6. Here is some help for this command:
  7. Scan a table; pass table name and optionally a dictionary of scanner
  8. specifications. Scanner specifications may include one or more of:
  9. TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, TIMESTAMP, MAXLENGTH,
  10. or COLUMNS, CACHE

  12. If no columns are specified, all columns will be scanned.
  13. To scan all members of a column family, leave the qualifier empty as in
  14. 'col_family:'.
2、查看Hbase UI，只有ROOT表，META元数据表和自己创建的mytab、test表都没了，这里没有截图
3、使用hbck修复META元数据
  1. grid@master1:~$ hbase hbck -fixMeta
  2. ........................................
  3. 15/09/11 16:05:32 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
  4. 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@deb8755; serverName=slave3,60020,1441810138459
  5. 15/09/11 16:06:04 DEBUG client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=slave3, port=60020}, attempt=10 of 14 failed; retrying after sleep of 64200 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
  6.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3609)
  7.         at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:2202)
  8.         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  9.         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  10.         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  11.         at java.lang.reflect.Method.invoke(Method.java:606)
  12.         at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:323)
  13.         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
  14. ........................................
修复不了，报错org.apache.hadoop.hbase.NotServingRegionException: Region is not online
4、查看Hadoop文件是否存在
  1. grid@master1:~$ hadoop fs -lsr /hbase
  2. drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/-ROOT-
  3. -rw-r--r-- 2 grid supergroup 728 2015-09-08 13:06 /hbase/-ROOT-/.tableinfo.0000000001
  4. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/.tmp
  5. drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/-ROOT-/70236052
  6. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs
  7. -rw-r--r-- 2 grid supergroup 421 2015-09-08 13:06 /hbase/-ROOT-/70236052/.oldlogs/hlog.1441717607322
  8. -rw-r--r-- 2 grid supergroup 109 2015-09-08 13:06 /hbase/-ROOT-/70236052/.regioninfo
  9. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/.tmp
  10. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/-ROOT-/70236052/info
  11. -rw-r--r-- 3 grid supergroup 1596 2015-09-11 19:21 /hbase/-ROOT-/70236052/info/924e206e22754690b08c8ec8bbf84692
  12. drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/-ROOT-/70236052/recovered.edits
  13. drwxr-xr-x - grid supergroup 0 2015-09-09 15:50 /hbase/.META.
  14. drwxr-xr-x - grid supergroup 0 2015-09-11 18:05 /hbase/.META./1028785192
  15. drwxr-xr-x - grid supergroup 0 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs
  16. -rw-r--r-- 2 grid supergroup 134 2015-09-08 13:06 /hbase/.META./1028785192/.oldlogs/hlog.1441717607577
  17. -rw-r--r-- 2 grid supergroup 111 2015-09-08 13:06 /hbase/.META./1028785192/.regioninfo
  18. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/.tmp
  19. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.META./1028785192/info
  20. -rw-r--r-- 3 grid supergroup 3443 2015-09-11 19:21 /hbase/.META./1028785192/info/d735b80efd454d9caca81e4392e8d0d2
  21. drwxr-xr-x - grid supergroup 0 2015-09-09 14:09 /hbase/.META./1028785192/recovered.edits
  22. drwxr-xr-x - grid supergroup 0 2015-09-11 19:21 /hbase/.archive
  23. drwxr-xr-x - grid supergroup 0 2015-09-09 10:58 /hbase/.corrupt
  24. drwxr-xr-x - grid supergroup 0 2015-09-11 17:35 /hbase/.logs
  25. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388
  26. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave1,60020,1441989279388/slave1%2C60020%2C1441989279388.1442007288767
  27. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626
  28. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave2,60020,1441989279626/slave2%2C60020%2C1441989279626.1442007288761
  29. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275
  30. -rw-r--r-- 3 grid supergroup 0 2015-09-11 21:34 /hbase/.logs/slave3,60020,1441989280275/slave3%2C60020%2C1441989280275.1442007288199
  31. drwxr-xr-x - grid supergroup 0 2015-09-11 21:34 /hbase/.oldlogs
  32. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave1%2C60020%2C1441989279388.1442003688318
  33. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave2%2C60020%2C1441989279626.1442003688444
  34. -rw-r--r-- 3 grid supergroup 134 2015-09-11 21:34 /hbase/.oldlogs/slave3%2C60020%2C1441989280275.1442003687774
  35. drwxr-xr-x - grid supergroup 0 2015-09-11 16:35 /hbase/.tmp
  36. -rw-r--r-- 2 grid supergroup 19 2015-09-11 16:35 /hbase/.tmp/hbase-hbck.lock
  37. -rw-r--r-- 2 grid supergroup 38 2015-09-08 13:06 /hbase/hbase.id
  38. -rw-r--r-- 2 grid supergroup 3 2015-09-08 13:06 /hbase/hbase.version
  39. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest
  40. -rw-r--r-- 2 grid supergroup 671 2015-09-09 13:34 /hbase/hivetest/.tableinfo.0000000001
  41. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/.tmp
  42. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e
  43. -rw-r--r-- 2 grid supergroup 231 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/.regioninfo
  44. drwxr-xr-x - grid supergroup 0 2015-09-09 13:34 /hbase/hivetest/42b9d2fff35c18393058668f77e7b86e/cf1
  45. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab
  46. -rw-r--r-- 2 grid supergroup 665 2015-09-09 17:26 /hbase/mytab/.tableinfo.0000000001
  47. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/.tmp
  48. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d
  49. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/.regioninfo
  50. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/4c6f25b2a4469ba63be2a28e9f79681d/cf1
  51. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224
  52. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/.regioninfo
  53. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/58be7af75c6d124aad1a8385f7410224/cf1
  54. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582
  55. -rw-r--r-- 2 grid supergroup 230 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/.regioninfo
  56. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/76ce4f3cde98abd2e7679b099489d582/cf1
  57. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d
  58. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/.regioninfo
  59. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/97e21ce593d16c77d35c253c447c176d/cf1
  60. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910
  61. -rw-r--r-- 2 grid supergroup 236 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/.regioninfo
  62. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/9c6d2a8e69c5aea308b79debf82cc910/cf1
  63. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185
  64. -rw-r--r-- 2 grid supergroup 238 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/.regioninfo
  65. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/accd15db4e78d03c8a53cd6b1e17f185/cf1
  66. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7
  67. -rw-r--r-- 2 grid supergroup 226 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/.regioninfo
  68. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/b640abdb82a91acb98844d109ad03ba7/cf1
  69. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d
  70. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/.regioninfo
  71. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/c17b2121db16f1f8938077016bfc524d/cf1
  72. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a
  73. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/.regioninfo
  74. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/cf803d46bd2fd225cce52d2aad18314a/cf1
  75. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b
  76. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/.regioninfo
  77. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/f81699b3cefb992fa5c8e5a0fd1f133b/cf1
  78. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30
  79. -rw-r--r-- 2 grid supergroup 234 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/.regioninfo
  80. drwxr-xr-x - grid supergroup 0 2015-09-09 17:26 /hbase/mytab/fc2ed8a43e2eaf02238d7e1bc398cc30/cf1
  81. drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test
  82. -rw-r--r-- 2 grid supergroup 663 2015-09-08 13:15 /hbase/test/.tableinfo.0000000001
  83. drwxr-xr-x - grid supergroup 0 2015-09-08 13:15 /hbase/test/.tmp
  84. drwxr-xr-x - grid supergroup 0 2015-09-09 13:30 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad
  85. -rw-r--r-- 2 grid supergroup 219 2015-09-08 13:15 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/.regioninfo
  86. drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1
  87. -rw-r--r-- 3 grid supergroup 1018 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/cf1/b8cd45592ea64e278a12b3f0b95eb220
  88. drwxr-xr-x - grid supergroup 0 2015-09-09 12:41 /hbase/test/edca99f7a3a627594f2ab1af7e7cb1ad/recovered.edits
文件一切正常，有之前创建mytab表的文件
5、检查Hadoop文件是否损坏
  1. grid@master1:~$ hadoop fsck /
  2. FSCK started by grid from /10.10.10.70 for path / at Fri Sep 11 16:14:13 UTC 2015
  3. .............................Status: HEALTHY
  4.  Total size: 12829 B
  5.  Total dirs: 70
  6.  Total files: 29 (Files currently being written: 6)
  7.  Total blocks (validated): 29 (avg. block size 442 B) (Total open file blocks (not validated): 6)
  8.  Minimally replicated blocks: 29 (100.0 %)
  9.  Over-replicated blocks: 0 (0.0 %)
  10.  Under-replicated blocks: 0 (0.0 %)
  11.  Mis-replicated blocks: 0 (0.0 %)
  12.  Default replication factor: 2
  13.  Average block replication: 2.0344827
  14.  Corrupt blocks: 0
  15.  Missing replicas: 0 (0.0 %)
  16.  Number of data-nodes: 3
  17.  Number of racks: 1
  18. FSCK ended at Fri Sep 11 16:14:13 UTC 2015 in 30 milliseconds


  21. The filesystem under path '/' is HEALTHY
Hadoop文件也正常，那么，region没有在线，是不是zookeeper集群存在问题
6、停止集群，删除zookeeper文件，相当于重建zookeeper集群
  1. grid@master1:~$ stop-hbase.sh
  2. stopping hbase...
  3. grid@master1:~$ zkServer.sh stop
  4. JMX enabled by default
  5. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  6. Stopping zookeeper ... STOPPED
  7. grid@master1:~$ cd /data/zk/
  8. grid@master1:/data/zk$ ls
  9. myid version-2
  10. grid@master1:/data/zk$ ls -l
  11. total 8
  12. -rw-r--r-- 1 grid hadoop 3 Sep 8 20:33 myid
  13. drwxr-xr-x 2 grid hadoop 4096 Sep 11 23:10 version-2
  14. grid@master1:/data/zk$ rm -rf version-2/
  15. grid@master1:/data/zk$ cd ~
  16. grid@master1:~$

  18. grid@master1:~$ zkServer.sh start
  19. JMX enabled by default
  20. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  21. Starting zookeeper ... STARTED
  22. grid@master1:~$ zkServer.sh status
  23. JMX enabled by default
  24. Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
  25. Mode: follower
  26. grid@master1:~$ start-hbase.sh
  27. starting master, logging to /usr/local/hbase/logs/hbase-grid-master-master1.out
  28. slave1: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave1.out
  29. slave2: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave2.out
  30. slave3: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-grid-regionserver-slave3.out
  31. grid@master1:~$ jps
  32. 7544 Jps
  33. 1473 SecondaryNameNode
  34. 6257 QuorumPeerMain
  35. 1556 JobTracker
  36. 1283 NameNode
  37. 6463 HMaster
7、再次使用hbck修复META元数据表
  1. grid@master1:~$ hbase hbck -fixMeta
  2. .....................................
  3. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
  4. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
  5. 15/09/11 16:36:00 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
  6. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
  7. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Removed .META.,,1.1028785192 for tableName=.META. from cache because of
  8. 15/09/11 16:36:01 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is slave3:60020
  9. 15/09/11 16:36:03 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@756a162a; serverName=slave3,60020,1441989280275
  10. ......................................
这次修复貌似正常了，但是日志中提示remove了一些在cache中的文件
请使用浏览器的分享功能分享到微信等