GBase 8a 节点替换_财经热点

GBase 8a 节点替换

admin

2024-02-29 22:52:21

节点替换包括替换gcware、替换gcluster、替换gnode ，新节点可以是freenode节点（替换后节点IP改变），也可以是全新未安装过gbase的节点（替换后节点IP不变），本次内容是使用全新节点替换gnode，其他情况以后有时间再补充

原始集群 192.168.61.1 【8a-1】 (gcware + gcluster + gnode)

192.168.61.2 【8a-2】 (gnode) 待替换节点

新节点 192.168.61.3 【8a-3】

原始状态

设置被替换节点状态为 unavailable

删除被替换节点的 feventlog

生成新的distribution

将被替换的集群节点（原 192.168.61.2）机器网线拔出（或者直接关机），并将待替换的新机器（192.168.61.3）IP改为192.168.61.2

执行节点替换

再次数据重分布，将数据分布到新节点

删除旧的分布方案

问题记录

原始状态

[gbase@8a-1 gcinstall]$ gcadmin
CLUSTER STATE:         ACTIVE====================================
| GBASE GCWARE CLUSTER INFORMATION |
====================================
| NodeName |  IpAddress   | gcware |
------------------------------------
| gcware1  | 192.168.61.1 |  OPEN  |
------------------------------------
======================================================
|       GBASE COORDINATOR CLUSTER INFORMATION        |
======================================================
|   NodeName   |  IpAddress   | gcluster | DataState |
------------------------------------------------------
| coordinator1 | 192.168.61.1 |   OPEN   |     0     |
------------------------------------------------------
=============================================================
|         GBASE CLUSTER FREE DATA NODE INFORMATION          |
=============================================================
| NodeName  |  IpAddress   | gnode | syncserver | DataState |
-------------------------------------------------------------
| FreeNode1 | 192.168.61.2 | OPEN  |    OPEN    |     0     |
-------------------------------------------------------------
| FreeNode2 | 192.168.61.1 | OPEN  |    OPEN    |     0     |
-------------------------------------------------------------0 virtual cluster
1 coordinator node
2 free data node

设置被替换节点状态为 unavailable

当节点状态置为unavaliable时该节点不再接收gcluster下发的SQL

[gbase@8a-1 gcinstall]$ gcadmin setnodestate 192.168.61.2 unavailable
[gbase@8a-1 gcinstall]$ gcadmin
CLUSTER STATE:         ACTIVE
VIRTUAL CLUSTER MODE:  NORMAL====================================
| GBASE GCWARE CLUSTER INFORMATION |
====================================
| NodeName |  IpAddress   | gcware |
------------------------------------
| gcware1  | 192.168.61.1 |  OPEN  |
------------------------------------
======================================================
|       GBASE COORDINATOR CLUSTER INFORMATION        |
======================================================
|   NodeName   |  IpAddress   | gcluster | DataState |
------------------------------------------------------
| coordinator1 | 192.168.61.1 |   OPEN   |     0     |
------------------------------------------------------
===============================================================================================================
|                                       GBASE DATA CLUSTER INFORMATION                                        |
===============================================================================================================
| NodeName |                IpAddress                 | DistributionId |    gnode    | syncserver | DataState |
---------------------------------------------------------------------------------------------------------------
|  node1   |               192.168.61.2               |       1        | UNAVAILABLE |            |           |
---------------------------------------------------------------------------------------------------------------
|  node2   |               192.168.61.1               |       1        |    OPEN     |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------------

删除被替换节点的 feventlog

当集群各节点状态不一致时会生成feventlog，这个feventlog也是判断集群各节点是否一致的标准，当我们将节点状态置为unavailable时这个节点已经与其他节点不一致了，需要手工删除feventlog，虽然再后面替换节点repalce.py这个里面也会删除feventlog, 但是我们选择先手工删除，这样在节点替换过程中再次删除feventlog的时候会很快，可以保证在节点替换过程中不会因为feventlog太大而耗费时间过长，只要feventlog一致，那么集群就可以正常运转，同时也保证了替换节点过程中，瞬间删除feventlog ，集群可以正常工作，同时在节点替换过程中，如果没有先将feventlog删除的话，那么节点替换过程中也会自动删除，只不过这个时间可能会很长，但是在这个时间段，下发过来的sql不会拒收，而是将SQL置为等待状态，等节点替换完继续执行SQL，这个特性也说明了节点替换过程中，业务不受影响

[gbase@8a-1 gcinstall]$ gcadmin rmfeventlog 192.168.61.2

生成新的distribution

由于192.168.61.2将被替换，所以要将192.168.61.2上面的数据先转移，转移的方法就是先生成新的分布方案，再按照新的分布方案进行数据重分布。比如原集群的分布方案是192.168.61.1和192.168.61.2两个节点，192.168.61.2将被替换，那么新的分布方案就不能包含192.168.61.2了

原始集群的分布方案（Distribution ID = 1）

[gbase@8a-1 gcinstall]$ gcadmin showdistribution nodeDistribution ID: 1 | State: new | Total segment num: 2============================================================================================
|  nodes   |             192.168.61.2              |             192.168.61.1              |
--------------------------------------------------------------------------------------------
| primary  |                  1                    |                  2                    |
| segments |                                       |                                       |
--------------------------------------------------------------------------------------------
|duplicate |                  2                    |                  1                    |
|segments 1|                                       |                                       |
============================================================================================

复制原始的分布方案

[gbase@8a-1 gcinstall]$ gcadmin getdistribution 1 distribution_info_1.xml
[gbase@8a-1 gcinstall]$ cat distribution_info_1.xml

修改新的分布方案

[gbase@8a-1 gcinstall]$ vi distribution_info_1.xml

如果被替换节点是主分片，那么将ip由被替换的节点改为改为该分片上的备份分片的IP，当前分片的备份分片为192.168.61.1，所以主分片这里改为 192.168.61.1

~~因为上面主分片已经改为192.168.61.1，这里的备份分片可以直接删掉~~

~~如果被替换节点是备份分片，那么这几行直接删除~~

[gbase@8a-1 gcinstall]$ vi gcChangeInfo_dis1.xml

生成新的分布方案

[gbase@8a-1 gcinstall]$ gcadmin distribution gcChangeInfo_dis1.xml

查看新的分布方案

此时多了distribution ID = 2 ,这个分布方案里面不包含被替换节点

[gbase@8a-1 gcinstall]$ gcadmin showdistribution nodeDistribution ID: 2 | State: new | Total segment num: 2====================================================
|  nodes   |             192.168.61.1              |
----------------------------------------------------
| primary  |                  1                    |
| segments |                  2                    |
====================================================Distribution ID: 1 | State: old | Total segment num: 2============================================================================================
|  nodes   |             192.168.61.2              |             192.168.61.1              |
--------------------------------------------------------------------------------------------
| primary  |                  1                    |                  2                    |
| segments |                                       |                                       |
--------------------------------------------------------------------------------------------
|duplicate |                  2                    |                  1                    |
|segments 1|                                       |                                       |
============================================================================================

使用Distribution = 2 ,将数据重新分布但不给被替换节点分配数据

没有重分布时数据的分布情况如下（数据全部使用data_distribution_id = 1）

gbase> select index_name,tbname,data_distribution_id,vc_id from gbase.table_distribution;
+-------------------------------+--------------------+----------------------+---------+
| index_name                    | tbname             | data_distribution_id | vc_id   |
+-------------------------------+--------------------+----------------------+---------+
| gclusterdb.rebalancing_status | rebalancing_status |                    1 | vc00001 |
| gclusterdb.dual               | dual               |                    1 | vc00001 |
| testdb.t1                     | t1                 |                    1 | vc00001 |
+-------------------------------+--------------------+----------------------+---------+
3 rows in set (Elapsed: 00:00:00.00)

重分布后数据的分布情况如下（数据改为使用 data_distribution_id = 2）

gbase> initnodedatamap;
Query OK, 0 rows affected, 3 warnings (Elapsed: 00:00:00.11)gbase> rebalance instance to 2;    这里 to 2 表示使用Distribution = 2 的分布方案
Query OK, 1 row affected (Elapsed: 00:00:00.03)gbase> select index_name,status,percentage,priority,host,distribution_id from gclusterdb.rebalancing_status;
+------------+-----------+------------+----------+--------------+-----------------+
| index_name | status    | percentage | priority | host         | distribution_id |
+------------+-----------+------------+----------+--------------+-----------------+
| testdb.t1  | COMPLETED |        100 |        5 | 192.168.61.1 |               2 |
+------------+-----------+------------+----------+--------------+-----------------+
1 row in set (Elapsed: 00:00:00.01)

将被替换的集群节点（原 192.168.61.2）机器网线拔出（或者直接关机），并将待替换的新机器（192.168.61.3）IP改为192.168.61.2

[root@8a-3 ~]# ip a
2: ens33:  mtu 1500 qdisc fq_codel state UP group default qlen 1000link/ether 00:0c:29:c6:d3:8e brd ff:ff:ff:ff:ff:ffinet 192.168.61.2/24 brd 192.168.61.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::61d5:26ac:7834:3674/64 scope link noprefixroute 
[root@8a-3 ~]# cd /opt/gbase/
[root@8a-3 gbase]# ll  #新节点原始状态 当前目录为空
total 0
[root@8a-3 gbase]# ll  #replace.py 完成之后 再次查看这个目录 发现已经安装了gbase
total 0
drwxrwxr-x 2 gbase gbase 6 Nov 28 23:58 192.168.61.2

自己在虚拟机测试时，这一步非常重要，如果忘记这步，那么替换前后集群是没有变化的，只有先将旧节点关机，新节点开机才可以达到正常替换的效果，不然替换完了，结果一看新节点192.168.61.3上什么都没有

执行节点替换

[gbase@8a-1 gcinstall]$ ./replace.py --host=192.168.61.2 --type=data --dbaUser=gbase --dbaUserPwd=gbase --generalDBUser=gbase --generalDBPwd=gbase20110531 --overwrite  这里的host表示被替换节点的IP

再次检查分布情况（替换后的分布方案 Distribution = 3）

[gbase@8a-1 gcinstall]$ gcadmin showdistributionDistribution ID: 3 | State: new | Total segment num: 2Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                  192.168.61.2                  |         1          |                  192.168.61.1                  |
------------------------------------------------------------------------------------------------------------------------
|                  192.168.61.1                  |         2          |                  192.168.61.2                  |
========================================================================================================================Distribution ID: 2 | State: old | Total segment num: 2Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                  192.168.61.1                  |         1          |                                                |
------------------------------------------------------------------------------------------------------------------------
|                  192.168.61.1                  |         2          |                                                |
========================================================================================================================

再次数据重分布，将数据分布到新节点

[gbase@8a-1 gcinstall]$ gccli -uroot -p
Enter password: GBase client 9.5.3.27.14_patch.1b41b5c1. Copyright (c) 2004-2022, GBase.  All Rights Reserved.gbase> rebalance instance to 3;
Query OK, 1 row affected (Elapsed: 00:00:00.02)gbase> select * from gclusterdb.rebalancing_status;
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
| index_name | db_name | table_name | tmptable | start_time                 | end_time | status  | percentage | priority | host         | distribution_id |
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
| test.t     | test    | t          |          | 2022-11-29 00:08:30.294000 | NULL     | RUNNING |          0 |        5 | 192.168.61.1 |               3 |
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
1 row in set (Elapsed: 00:00:00.00)

删除旧的分布方案

[gbase@8a-1 gcinstall]$ gcadmin rmdistribution 2

问题记录

删除旧的分布方案时无法删除，提示还在使用

[gbase@8a-1 gcinstall]$ gcadmin rmdistribution 2
cluster distribution ID [2]
it will be removed now
please ensure this is ok, input [Y,y] or [N,n]: y
select count(*) from gbase.nodedatamap where data_distribution_id=2 result is not 0
refreshnodedatamap drop 2 failed, gcluster command error: Can not drop nodedatamap 2. Some table are using it.
gcadmin remove distribution: check whether distribution [2] is using failed
gcadmin remove distribution [2] failedgbase> select index_name,tbname,data_distribution_id,vc_id from gbase.table_distribution;
+-------------------------------+--------------------+----------------------+---------+
| index_name                    | tbname             | data_distribution_id | vc_id   |
+-------------------------------+--------------------+----------------------+---------+
| gclusterdb.rebalancing_status | rebalancing_status |                    3 | vc00001 |
| test.t                        | t                  |                    2 | vc00001 |
| gclusterdb.dual               | dual               |                    3 | vc00001 |
+-------------------------------+--------------------+----------------------+---------+
3 rows in set (Elapsed: 00:00:00.00)gbase> rebalance table test.t to 3;
ERROR 1707 (HY000): gcluster command error: target table has been rebalancing.

查看重分布状态发现重分布一直处于running状态

gbase> select index_name,status,percentage,priority,host,distribution_id from gclusterdb.rebalancing_status;
+------------+---------+------------+----------+--------------+-----------------+
| index_name | status  | percentage | priority | host         | distribution_id |
+------------+---------+------------+----------+--------------+-----------------+
| test.t     | RUNNING |          0 |        1 | 192.168.61.1 |               3 |
+------------+---------+------------+----------+--------------+-----------------+

判断是不是锁住了，查看锁信息，果然被锁了

[gbase@8a-1 gcinstall]$ gcadmin showlock+=====================================================================================+|                                    GCLUSTER LOCK                                    |+=====================================================================================++-----------------------------+------------+---------------+--------------+------+----+|          Lock name          |   owner    |    content    | create time  |locked|type|+-----------------------------+------------+---------------+--------------+------+----+|        gc-event-lock        |192.168.61.1| global master |20221129002719| TRUE | E  |+-----------------------------+------------+---------------+--------------+------+----+|    vc00001.hashmap_lock     |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  |+-----------------------------+------------+---------------+--------------+------+----+|    vc00001.test.db_lock     |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  |+-----------------------------+------------+---------------+--------------+------+----+|  vc00001.test.t.meta_lock   |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  |+-----------------------------+------------+---------------+--------------+------+----+|vc00001.test.t.rebalance_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | E  |+-----------------------------+------------+---------------+--------------+------+----+|vc00001.test.table_space_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  |+-----------------------------+------------+---------------+--------------+------+----+Total : 6

找出当前被锁住的进程ID为3388

gbase> show processlist;
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
| Id   | User            | Host               | db   | Command | Time | State                       | Info                                                              |
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
|    1 | event_scheduler | localhost          | NULL | Daemon  | 1780 | Waiting for next activation | NULL                                                              |
| 3388 | gbase           | 192.168.61.1:36000 | NULL | Query   |  667 | Move table's slice          | REBALANCE /*+ synchronous,vcid */ TABLE "vc00001"."test"."t" TO 3 |
| 3391 | root            | localhost          | NULL | Query   |    0 | NULL                        | show processlist                                                  |
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
3 rows in set (Elapsed: 00:00:00.00)

查看被锁进程的锁状态

[gbase@8a-1 gcinstall]$ gcadmin showlock | grep 3388|    vc00001.hashmap_lock     |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  ||    vc00001.test.db_lock     |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  ||  vc00001.test.t.meta_lock   |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  ||vc00001.test.t.rebalance_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | E  ||vc00001.test.table_space_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S  |

后面就是排查哪个进程持有了锁没有释放，然后杀掉就好

有时需要先删除datamap

refreshnodedatamap drop 2

GBase 8a扩容完成后refreshnodedatamap drop报错：Can not drop nodedatamap ,Some table are using it. – 老紫竹的家

词库加载错误:未能找到文件“E:\highferrum_mysql\Configuration\Dict_Stopwords.txt”。

上一篇：杭州今年首批土拍揽金111.6亿七宗宅地均被本地房企竞得杭州2023年宅地揽金1780亿杭州土拍2019超1000亿

下一篇：重大通报369河南麻将究竟有没有挂(原来真的有挂)-知乎

GBase 8a 节点替换

原始状态

设置被替换节点状态为 unavailable

删除被替换节点的 feventlog

生成新的distribution

将被替换的集群节点（原 192.168.61.2）机器网线拔出（或者直接关机），并将待替换的新机器（192.168.61.3）IP改为192.168.61.2

执行节点替换

再次数据重分布，将数据分布到新节点

删除旧的分布方案

问题记录

相关内容

热门资讯