ClickHouse Cluster Deployment

This article explains how to use clickhouse_ansible to perform a ClickHouse cluster or standalone mode deployment.

The commands in this article are based on the portable Ansible distribution by default and have performed the following initialization:

cd /usr/local/dbbot/portable-ansible-v0.5.0-py3
bash setup_portable_ansible.sh
source ~/.bashrc

Once completed, you can use ansible-playbook and ansible directly.

1. Scope of application

Supports the following entry playbooks:

playbooks/deploy_cluster.yml: cluster deployment.
playbooks/deploy_single.yml: rapid deployment for a single node or an experimental environment.

The default documentation example uses 3 shard 2 replicas, but the inventory design supports expansion to other x shard y replica topologies.

2. Preconditions

It is recommended to complete the following preparations before deployment:

Install dbbot on the control node and complete the portable distribution Ansible registration.
Plan the ClickHouse node and ZooKeeper/Keeper node.
Confirm that the target node network, time zone, disk directory, and operating system baseline meet the requirements.
Confirm that the control node can SSH to the target host without interaction.
Prepare the ClickHouse installation package, or confirm that the control node can connect to the Internet to download the required package.

3. Recommended inventory division of labor

clickhouse_ansible splits inventory by scenario by default:

inventory/hosts.deploy.ini: Deployment-specific.
inventory/hosts.dr_deploy.ini: dedicated for disaster recovery cluster deployment.
inventory/hosts.backup.ini: dedicated for backup.
inventory/hosts.dr_backup.ini: dedicated for disaster recovery cluster backup disk preparation.
inventory/hosts.restore.ini: dedicated to recovery.
inventory/hosts.nfs_server.ini: dedicated to NFS server.
inventory/hosts.single.ini: dedicated for single-node validation.

In deployment scenarios, it is recommended to set it in [all:vars]:

dbbot_inventory_purpose=deploy
ansible_python_interpreter=auto_silent
ansible_user=root
ansible_ssh_pass="'<your_ssh_password>'"

4. Deploy inventory minimal example

[clickhouse_deploy]
ck-131-1 ansible_host=192.0.2.131 shard=1 replica=1 instance_id=1 replica_name=ck-s1-r1 system_hostname=ck-node1
ck-131-2 ansible_host=192.0.2.131 shard=3 replica=2 instance_id=2 replica_name=ck-s3-r2 system_hostname=ck-node1
ck-132-1 ansible_host=192.0.2.132 shard=1 replica=2 instance_id=1 replica_name=ck-s1-r2 system_hostname=ck-node2
ck-132-2 ansible_host=192.0.2.132 shard=2 replica=1 instance_id=2 replica_name=ck-s2-r1 system_hostname=ck-node2
ck-133-1 ansible_host=192.0.2.133 shard=2 replica=2 instance_id=1 replica_name=ck-s2-r2 system_hostname=ck-node3
ck-133-2 ansible_host=192.0.2.133 shard=3 replica=1 instance_id=2 replica_name=ck-s3-r1 system_hostname=ck-node3

[clickhouse_cluster:children]
clickhouse_deploy

[zookeeper_deploy]
ck-131-1 zk_id=1
ck-132-1 zk_id=2
ck-133-1 zk_id=3

[zookeeper_cluster:children]
zookeeper_deploy

[all:vars]
dbbot_inventory_purpose=deploy
ansible_python_interpreter=auto_silent
ansible_user=root
ansible_ssh_pass="'<your_ssh_password>'"

Description:

instance_id is recommended to be filled in explicitly to facilitate port, directory and service name derivation.
If fcs_set_hostname=true is enabled, system_hostname can be additionally configured for the host line.
In Keeper mode, it is still necessary to ensure that the coordination node is completely defined.

5. Key variable file

Before deployment, focus on the following files:

playbooks/vars/common_config.yml: Common parameters such as version, user, directory, password, Keeper switch, etc.
playbooks/vars/cluster_config.yml: cluster name, port baseline and topology related parameters.
playbooks/vars/standalone_config.yml: Standalone mode behavior.

Common parameters that need to be confirmed:

clickhouse_version
clickhouse_cluster_name
use_clickhouse_keeper
clickhouse_enable_ssl
clickhouse_install_debug_package
clickhouse_default_password
fcs_allow_dbbot_default_passwd
fcs_set_hostname
deploy_require_manual_confirm

For a more complete parameter description, see: ClickHouse variable and parameter description

Password guardrails:

The public default password for clickhouse_default_password is Dbbot_default@8888.
Default fcs_allow_dbbot_default_passwd: false, so deployment-, backup-, and restore-related playbooks will intercept this public default password in the pre_tasks stage.
If you want to verify quickly in the experimental environment, you can explicitly set fcs_allow_dbbot_default_passwd: true; this is not recommended in the production environment.

6. Prepare installation package

If you turn off automatic downloading, you need to put the installation package into downloads/ in advance:

cd /usr/local/dbbot/clickhouse_ansible/downloads
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-common-static-23.6.1.1524-amd64.tgz"
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-server-23.6.1.1524-amd64.tgz"
wget "https://packages.clickhouse.com/tgz/stable/clickhouse-client-23.6.1.1524-amd64.tgz"
wget "https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz"

Description:

clickhouse-common-static-dbg has been changed to optional, requiring preparation and installation only when clickhouse_install_debug_package=true.
It is recommended to keep clickhouse_install_debug_package=false by default for official releases to reduce package size and distribution time.

7. Execute deployment

7.1 Grammar check

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml --syntax-check

7.2 Cluster deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml

7.3 Keeper mode deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.deploy.ini \
  deploy_cluster.yml \
  -e "use_clickhouse_keeper=true"

7.4 Stand-alone mode deployment

cd /usr/local/dbbot/clickhouse_ansible/playbooks
ansible-playbook \
  -i ../inventory/hosts.single.ini \
  deploy_single.yml

8. System changes

Deploying a playbook not only installs ClickHouse, but also performs system-side initialization on the target host. Defaults may involve:

Set the time zone to Asia/Shanghai
Modify the mapping between hostname and /etc/hosts
Disable swap and comment the swap entry in /etc/fstab
Adjust limits.conf / limits.d
Adjust ClickHouse-specific parameters under sysctl.d
Tweak THP, SELinux and Firewall

If you don’t want something to take effect, at least evaluate in advance:

fcs_set_hostname
clickhouse_uid / clickhouse_gid
fcs_auto_download_packages
clickhouse_install_debug_package

9. Post-deployment validation

8.1 Port monitoring

ansible \
  all -i ../inventory/hosts.deploy.ini -m shell \
  -a "ss -lntp | egrep ':8123 |:8124 |:8443 |:8444 |:9000 |:9001 |:9440 |:9441 '"

8.2 Cluster topology

SELECT cluster, shard_num, replica_num, host_name, port
FROM system.clusters
WHERE cluster = 'example_3shards_2replicas'
ORDER BY shard_num, replica_num;

8.3 Copy status

SELECT table, is_readonly, queue_size, absolute_delay
FROM system.replicas
ORDER BY table;

10. Common precautions

Deploy the default requirement dbbot_inventory_purpose=deploy to avoid misuse of backup or recovery inventory.
ansible_ssh_pass is recommended to be written in "'password'" to avoid Ansible inventory parsing exceptions.
Manual access control is enabled by default and is not recommended to be closed for a long time during production.
If you do not want to modify the system hostname, you can set fcs_set_hostname to false.
If the target machine already has an account with the same UID/GID, you need to adjust clickhouse_uid / clickhouse_gid first.
If you want to back up or restore later, please continue reading ClickHouse Backup and ClickHouse Recovery to complete NFS and backup disk preparation first.