pxc-55 error failed to join cluster

I had problem my pxc-55, one of 5 node has error because package percona-xtradb-cluster-55 removed by MySQL-client when i try install on the node, after repaired i want to join again to the cluster, but i got problem when join process.

The error messages has

WSREP_SST: [INFO] Waiting for SST streaming to complete! (20170825 18:32:41.575)
´┐Żxbstream: Can't create/write to file './xtrabackup_logfile' (Errcode: 17 - File exists)
xbstream: failed to create file.
2017/08/25 18:32:51 socat[46336] E write(1, 0x10b2410, 8192): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20170825 18:32:51.682)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20170825 18:32:51.685)
170825 18:32:51 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.19.17' --auth 'sstuser:qwerty123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '46034': 32 (Broken pipe)
170825 18:32:51 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
170825 18:32:51 [ERROR] WSREP: SST failed: 32 (Broken pipe)
170825 18:32:51 [ERROR] Aborting

It looks like i have stalled .sst dir in the data directory. This usually happens if previous SST try fails.

Solution for this problem is :
remove data on /var/lib/mysql include .sst dir and try to join again

Advertisements

Perform Failover testing Postgresql replication

On this article i will show the failover testing postgresql replication with repmgr, repmgr already configure automatic failover, reference from this article

Begin failover

Stop service postgresql on master node

$ pg_ctl -D /var/lib/postgresql/data -m fast stop

See log on slave node, like bellow. The log will show error cause master down until standby promote as master

[2017-08-09 04:05:14] [DEBUG] get_upstream_connection(): conninfo is "host=172.17.0.2 user=repmgr dbname=repmgr"
[2017-08-09 04:05:14] [DEBUG] connecting to: 'host=172.17.0.2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:05:14] [ERROR] connection to database failed: FATAL:  the database system is shutting down

[2017-08-09 04:05:14] [ERROR] unable to connect to upstream node: FATAL:  the database system is shutting down

[2017-08-09 04:05:14] [DEBUG] connecting to: 'host=172.17.0.2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:05:14] [ERROR] connection to database failed: FATAL:  the database system is shutting down

[2017-08-09 04:05:14] [WARNING] connection to master has been lost, trying to recover... 60 seconds before failover decision
[2017-08-09 04:05:24] [WARNING] connection to master has been lost, trying to recover... 50 seconds before failover decision
[2017-08-09 04:05:34] [WARNING] connection to master has been lost, trying to recover... 40 seconds before failover decision
[2017-08-09 04:05:44] [WARNING] connection to master has been lost, trying to recover... 30 seconds before failover decision
[2017-08-09 04:05:54] [WARNING] connection to master has been lost, trying to recover... 20 seconds before failover decision
[2017-08-09 04:06:04] [WARNING] connection to master has been lost, trying to recover... 10 seconds before failover decision
[2017-08-09 04:06:14] [ERROR] unable to reconnect to master (timeout 60 seconds)...
[2017-08-09 04:06:14] [DEBUG] get_node_record():
SELECT id, type, upstream_node_id, name, conninfo,        slot_name, priority, active  FROM "repmgr_db_cluster".repl_nodes  WHERE cluster = 'db_cluster'    AND id = 1
[2017-08-09 04:06:14] [DEBUG] failure detected on master node (1); attempting to promote a standby
[2017-08-09 04:06:14] [DEBUG] 2 active nodes registered
[2017-08-09 04:06:14] [DEBUG] node=1 conninfo="host=172.17.0.2 user=repmgr dbname=repmgr" type=master
[2017-08-09 04:06:14] [DEBUG] node=2 conninfo="host=node2 user=repmgr dbname=repmgr" type=standby
[2017-08-09 04:06:14] [DEBUG] connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:06:14] [DEBUG] set_config():
SET synchronous_commit TO 'local'
[2017-08-09 04:06:14] [DEBUG] total nodes counted: registered=2, visible=1
[2017-08-09 04:06:14] [DEBUG] checking node 1...
[2017-08-09 04:06:14] [DEBUG] checking node 2...
[2017-08-09 04:06:14] [DEBUG] connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:06:14] [DEBUG] set_config():
SET synchronous_commit TO 'local'
[2017-08-09 04:06:14] [DEBUG] LSN of node 2 is: 0/6000098
[2017-08-09 04:06:14] [DEBUG] connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:06:14] [DEBUG] set_config():
SET synchronous_commit TO 'local'
[2017-08-09 04:06:14] [DEBUG] LSN of node 2 is: 0/6000098
[2017-08-09 04:06:14] [DEBUG] best candidate node id is 2
[2017-08-09 04:06:19] [NOTICE] this node is the best candidate to be the new master, promoting...
[2017-08-09 04:06:19] [DEBUG] promote command is: "/opt/pgsql-9.6/bin/repmgr standby promote -f /etc/repmgr/repmgr.conf"
INFO: connecting to standby database
DEBUG: connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
INFO: retrieving node list for cluster 'db_cluster'
DEBUG: connecting to: 'host=172.17.0.2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
ERROR: connection to database failed: could not connect to server: Connection refused
	Is the server running on host "172.17.0.2" and accepting
	TCP/IP connections on port 5432?

DEBUG: connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
NOTICE: promoting standby
NOTICE: promoting server using '/opt/pgsql-9.6/bin/pg_ctl -D /var/lib/postgres/data promote'
INFO: reconnecting to promoted server
DEBUG: connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
DEBUG: setting node 2 as master and marking existing master as failed
NOTICE: STANDBY PROMOTE successful
[2017-08-09 04:06:21] [DEBUG] connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
[2017-08-09 04:06:21] [DEBUG] set_config():
SET synchronous_commit TO 'local'
[2017-08-09 04:06:21] [DEBUG] get_node_record():
SELECT id, type, upstream_node_id, name, conninfo,        slot_name, priority, active  FROM "repmgr_db_cluster".repl_nodes  WHERE cluster = 'db_cluster'    AND id = 2
[2017-08-09 04:06:21] [DEBUG] create_event_record():
 INSERT INTO "repmgr_db_cluster".repl_events (              node_id,              event,              successful,              details             )       VALUES ($1, $2, $3, $4)    RETURNING event_timestamp 
[2017-08-09 04:06:21] [DEBUG] create_event_record(): Event timestamp is "2017-08-09 04:06:21.880298+00"
[2017-08-09 04:06:23] [DEBUG] standby check loop will terminate
[2017-08-09 04:06:23] [INFO] checking cluster configuration with schema 'repmgr_db_cluster'
[2017-08-09 04:06:23] [INFO] checking node 2 in cluster 'db_cluster'
[2017-08-09 04:06:23] [INFO] reloading configuration file
[2017-08-09 04:06:23] [INFO] configuration has not changed
[2017-08-09 04:06:23] [INFO] starting continuous master connection check

Bring old master as new standby

Before run standby clone, please run this query on new master

repmgr=# select * from repmgr_db_cluster.repl_nodes ;
id | type | upstream_node_id | cluster | name | conninfo
| slot_name | priority | active
----+--------+------------------+------------+-------+--------------------------------------
-----+---------------+----------+--------
1 | master | | db_cluster | node1 | host=node1 user=repmgr dbname=re
pmgr | repmgr_slot_1 | 100 | f
2 | master | | db_cluster | node2 | host=node2 user=repmgr dbname=repmgr
| repmgr_slot_2 | 100 | t

Delete old master from repmgr_db_cluster.repl_nodes

repmgr=# delete from repmgr_db_cluster.repl_nodes where id =1;
DELETE 1

Next run repmgr to clone data from master

$ /opt/pgsql-9.5/bin/repmgr -h node2 -U repmgr -d repmgr -D /var/lib/postgresql/data -f /etc/repmgr/repmgr.conf standby clone
NOTICE: destination directory '/var/lib/postgres/data' provided
INFO: connecting to upstream node
INFO: Successfully connected to upstream node. Current installation size is 41 MB
INFO: checking and correcting permissions on existing directory /var/lib/postgres/data/ ...
DEBUG: standby clone:   WITH dd AS (     SELECT setting AS data_directory      FROM pg_catalog.pg_settings      WHERE name = 'data_directory'   )     SELECT DISTINCT(sourcefile),            regexp_replace(sourcefile, '^.*\/', '') AS filename,            sourcefile ~ ('^' || dd.data_directory) AS in_data_dir       FROM dd, pg_catalog.pg_settings ps      WHERE sourcefile IS NOT NULL   ORDER BY 1 
DEBUG: standby clone:   WITH dd AS (     SELECT setting AS data_directory      FROM pg_catalog.pg_settings      WHERE name = 'data_directory'   )     SELECT ps.setting,            regexp_replace(setting, '^.*\/', '') AS filename,            ps.setting ~ ('^' || dd.data_directory) AS in_data_dir       FROM dd, pg_catalog.pg_settings ps      WHERE ps.name IN ('hba_file', 'ident_file')   ORDER BY 1 
DEBUG: create_replication_slot(): Creating slot 'repmgr_slot_1' on master
NOTICE: starting backup (using pg_basebackup)...
HINT: this may take some time; consider using the -c/--fast-checkpoint option
INFO: executing: '/opt/pgsql-9.6/bin/pg_basebackup -l "repmgr base backup"  -D /var/lib/postgres/data/ -h node2 -p 5432 -U repmgr -X stream -S repmgr_slot_1 '
DEBUG: create_recovery_file(): creating '/var/lib/postgres/data/recovery.conf'...
DEBUG: recovery.conf: standby_mode = 'on'
DEBUG: recovery.conf: primary_conninfo = 'user=repmgr port=5432 sslmode=disable sslcompression=1 host=node2 application_name=node1'
DEBUG: recovery.conf: recovery_target_timeline = 'latest'
DEBUG: recovery.conf: primary_slot_name = repmgr_slot_1
NOTICE: standby clone (using pg_basebackup) complete
NOTICE: you can now start your PostgreSQL server
HINT: for example : pg_ctl -D /var/lib/postgres/data/ start
HINT: After starting the server, you need to register this standby with "repmgr standby register"

Start service postgresql and register server as new standby

$ pg_ctl -D /var/lib/postgres/data/ start       
server starting
$ /opt/pgsql-9.5/bin/repmgr -h node2 -U repmgr -d repmgr -D /var/lib/postgres/data/ -f /etc/repmgr/repmgr.conf standby register

WARNING: master connection parameters not required when executing STANDBY REGISTER
WARNING: destination directory not required when executing STANDBY REGISTER
INFO: connecting to standby database
DEBUG: connecting to: 'host=node1 user=repmgr dbname=repmgr fallback_application_name='repmgr''
INFO: connecting to master database
INFO: retrieving node list for cluster 'db_cluster'
DEBUG: connecting to: 'host=node2 user=repmgr dbname=repmgr fallback_application_name='repmgr''
DEBUG: get_master_connection(): current master node is 2
INFO: registering the standby
INFO: standby registration complete
NOTICE: standby node correctly registered for cluster db_cluster with id 1 (conninfo: host=node1 user=repmgr dbname=repmgr)

Failover testing done

CMIIW

Upgrade Percona-xtradb-cluster-5.5 to 5.6

Spec :

node1 = 172.17.0.2
node2 = 172.17.0.3
node3 = 172.17.0.4

pxc version = 5.5.41-37.0-55-log Percona XtraDB Cluster
OS = Debian wheezy (7.x)

This is steps to doing upgrade percona-xtradb-cluster-5.5 to 5.6. I was try upgrade on the fly but failed the cluster can’t sync because different version of galera lib, So must stop service mysql on all nodes of pxc-5.5 and bootstrap-pxc again.

First step, list all package related with percona-xtradb-cluster-5.5

# dpkg-query -l 'percona*';
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                               Version                Architecture           Description
+++-==================================-======================-======================-=========================================================================
un  percona-server-client-5.1                                                  (no description available)
un  percona-server-client-5.5                                                  (no description available)
ii  percona-server-common-5.5          5.5.50-rel38.0-1.wheez amd64                  Percona Server database common files (e.g. /etc/mysql/my.cnf)
un  percona-server-server-5.1                                                  (no description available)
un  percona-server-server-5.5                                                  (no description available)
ii  percona-xtrabackup                 2.3.5-1.wheezy         amd64                  Open source backup tool for InnoDB and XtraDB
un  percona-xtrabackup-21                                                      (no description available)
un  percona-xtrabackup-22                                                      (no description available)
un  percona-xtradb-client-5.0                                                  (no description available)
ii  percona-xtradb-cluster-55          5.5.41-25.11-853.wheez amd64                  Percona XtraDB Cluster with Galera
ii  percona-xtradb-cluster-client-5.5  5.5.41-25.11-853.wheez amd64                  Percona XtraDB Cluster database client binaries
ii  percona-xtradb-cluster-common-5.5  5.5.41-25.11-853.wheez all                    Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
un  percona-xtradb-cluster-galera                                              (no description available)
ii  percona-xtradb-cluster-galera-2    1:2.12.2682.wheezy     amd64                  Metapackage for latest version of galera2.
ii  percona-xtradb-cluster-galera-2.x  1:2.12.2682.wheezy     amd64                  Galera components of Percona XtraDB Cluster
un  percona-xtradb-cluster-galera-25                                           (no description available)
ii  percona-xtradb-cluster-server-5.5  5.5.41-25.11-853.wheez amd64                  Percona XtraDB Cluster database server binaries
un  percona-xtradb-server-5.0                                                  (no description available)

Stop service mysql on each node pxc-5.5 one by one

# /etc/init.d/mysql stop
[ ok ] Stopping MySQL (Percona XtraDB Cluster): mysqld.

Remove all packages related with pxc-5.5

# apt-get remove percona-xtradb-cluster-server-5.5 percona-xtradb-cluster-galera-2.x percona-xtradb-cluster-galera-2 percona-xtradb-cluster-common-5.5 percona-xtradb-cluster-client-5.5 percona-xtradb-cluster-55 percona-xtrabackup percona-server-common-5.5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  binutils cpp cpp-4.7 gcc gcc-4.7 libaio1 libc-dev-bin libc6-dev libdbd-mysql-perl
  libdbi-perl libev4 libgmp10 libgomp1 libitm1 libmpc2 libmpfr4 libquadmath0 linux-libc-dev
  lsof manpages manpages-dev socat zlib1g-dev
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  libperconaserverclient18 libperconaserverclient18-dev percona-server-common-5.5
  percona-xtrabackup percona-xtradb-cluster-55 percona-xtradb-cluster-client-5.5
  percona-xtradb-cluster-common-5.5 percona-xtradb-cluster-galera-2
  percona-xtradb-cluster-galera-2.x percona-xtradb-cluster-server-5.5
0 upgraded, 0 newly installed, 10 to remove and 0 not upgraded.
After this operation, 118 MB disk space will be freed.
Do you want to continue [Y/n]? Y

# apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  binutils cpp cpp-4.7 gcc gcc-4.7 libaio1 libc-dev-bin libc6-dev libdbd-mysql-perl
  libdbi-perl libev4 libgmp10 libgomp1 libitm1 libmpc2 libmpfr4 libquadmath0 linux-libc-dev
  lsof manpages manpages-dev socat zlib1g-dev
0 upgraded, 0 newly installed, 23 to remove and 39 not upgraded.
After this operation, 69.3 MB disk space will be freed.
Do you want to continue [Y/n]? Y

Doing Upgrade

Install package percona-xtradb-cluster-5.6 one by one per node

# apt-get install percona-xtradb-cluster-56 -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libaio1 libdbd-mysql-perl libdbi-perl libev4 libnuma1 lsof percona-xtrabackup
  percona-xtradb-cluster-client-5.6 percona-xtradb-cluster-common-5.6
  percona-xtradb-cluster-galera-3 percona-xtradb-cluster-galera-3.x
  percona-xtradb-cluster-server-5.6 socat
Suggested packages:
  tinyca netcat-openbsd pv qpress
The following NEW packages will be installed:
  libaio1 libdbd-mysql-perl libdbi-perl libev4 libnuma1 lsof percona-xtrabackup
  percona-xtradb-cluster-56 percona-xtradb-cluster-client-5.6
  percona-xtradb-cluster-common-5.6 percona-xtradb-cluster-galera-3
  percona-xtradb-cluster-galera-3.x percona-xtradb-cluster-server-5.6 socat
0 upgraded, 14 newly installed, 0 to remove and 39 not upgraded.
Need to get 39.3 MB of archives.
After this operation, 136 MB of additional disk space will be used.

And then open file my.cnf mark line bellow, because was deprecated on release pxc-5.6

innodb_locks_unsafe_for_binlog = 1

Then bootstrap-pxc start from node (the last stop of service mysql)

~# /etc/init.d/mysql bootstrap-pxc
[ ok ] Bootstrapping Percona XtraDB Cluster database server: mysqld ..

Run command mysql_upgrade to examines all tables on all databases for incompatibilities with the current version of percona-xtradb-cluster, mysql_upgrade also upgrades the system tables so that you can take advantage of new privileges or capabilities that might have been added

# mysql_upgrade -u root -p
Enter password: 
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Running 'mysql_fix_privilege_tables'...
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
Running 'mysqlcheck with default connection arguments
Warning: Using a password on the command line interface can be insecure.
solusidb.access_tokens                             OK
solusidb.cross_temporary_tokens                    OK
solusidb.delayed_jobs                              OK
solusidb.ip_whitelists                             OK
solusidb.merchants                                 OK
solusidb.old_passwords                             OK
solusidb.roles                                     OK
solusidb.schema_migrations                         OK
solusidb.sessions                                  OK
solusidb.users                                     OK
OK

For next step start service mysql on node2 and node1 alternate and then do mysql_upgrade

# /etc/init.d/mysql start

# mysql_upgrade -u root -p

Finaly run command show status on mysql to check wsrep of cluster with new version of pxc

# mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.36-82.0-56-log Percona XtraDB Cluster (GPL), Release rel82.0, Revision de7a681, WSREP version 26.20, wsrep_26.20

Copyright (c) 2009-2017 Percona LLC and/or its affiliates
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show status like '%wsrep%';
+------------------------------+-------------------------------------------------+
| Variable_name                | Value                                           |
+------------------------------+-------------------------------------------------+
| wsrep_local_state_uuid       | a8f38ef0-78eb-11e7-85e1-536ad920c921            |
| wsrep_protocol_version       | 7                                               |
| wsrep_last_committed         | 841                                             |
| wsrep_replicated             | 0                                               |
| wsrep_replicated_bytes       | 0                                               |
| wsrep_repl_keys              | 0                                               |
| wsrep_repl_keys_bytes        | 0                                               |
| wsrep_repl_data_bytes        | 0                                               |
| wsrep_repl_other_bytes       | 0                                               |
| wsrep_received               | 2                                               |
| wsrep_received_bytes         | 273                                             |
| wsrep_local_commits          | 0                                               |
| wsrep_local_cert_failures    | 0                                               |
| wsrep_local_replays          | 0                                               |
| wsrep_local_send_queue       | 0                                               |
| wsrep_local_send_queue_max   | 2                                               |
| wsrep_local_send_queue_min   | 0                                               |
| wsrep_local_send_queue_avg   | 0.500000                                        |
| wsrep_local_recv_queue       | 0                                               |
| wsrep_local_recv_queue_max   | 1                                               |
| wsrep_local_recv_queue_min   | 0                                               |
| wsrep_local_recv_queue_avg   | 0.000000                                        |
| wsrep_local_cached_downto    | 0                                               |
| wsrep_flow_control_paused_ns | 0                                               |
| wsrep_flow_control_paused    | 0.000000                                        |
| wsrep_flow_control_sent      | 0                                               |
| wsrep_flow_control_recv      | 0                                               |
| wsrep_cert_deps_distance     | 0.000000                                        |
| wsrep_apply_oooe             | 0.000000                                        |
| wsrep_apply_oool             | 0.000000                                        |
| wsrep_apply_window           | 0.000000                                        |
| wsrep_commit_oooe            | 0.000000                                        |
| wsrep_commit_oool            | 0.000000                                        |
| wsrep_commit_window          | 0.000000                                        |
| wsrep_local_state            | 4                                               |
| wsrep_local_state_comment    | Synced                                          |
| wsrep_cert_index_size        | 0                                               |
| wsrep_cert_bucket_count      | 22                                              |
| wsrep_gcache_pool_size       | 1320                                            |
| wsrep_causal_reads           | 0                                               |
| wsrep_cert_interval          | 0.000000                                        |
| wsrep_incoming_addresses     | 172.17.0.4:3306,172.17.0.3:3306,172.17.0.2:3306 |
| wsrep_desync_count           | 0                                               |
| wsrep_evs_delayed            |                                                 |
| wsrep_evs_evict_list         |                                                 |
| wsrep_evs_repl_latency       | 0.00054281/0.00103473/0.00128215/0.000347841/3  |
| wsrep_evs_state              | OPERATIONAL                                     |
| wsrep_gcomm_uuid             | a9813327-78f8-11e7-9682-97718b75bc11            |
| wsrep_cluster_conf_id        | 9                                               |
| wsrep_cluster_size           | 3                                               |
| wsrep_cluster_state_uuid     | a8f38ef0-78eb-11e7-85e1-536ad920c921            |
| wsrep_cluster_status         | Primary                                         |
| wsrep_connected              | ON                                              |
| wsrep_local_bf_aborts        | 0                                               |
| wsrep_local_index            | 2                                               |
| wsrep_provider_name          | Galera                                          |
| wsrep_provider_vendor        | Codership Oy                |
| wsrep_provider_version       | 3.20(r7e383f7)                                  |
| wsrep_ready                  | ON                                              |
+------------------------------+-------------------------------------------------+
59 rows in set (0.01 sec)

And upgrade pxc done

CMIIW

Source :
howto percona