======  Ceph deploy via cephadm  [Ubuntu 20.04 ]  ======

Post about how deploy ceph deploy via cephadm on ubuntu 20.04 on 3 nodes

==== REQUIREMENTS ====

  * 3 nodes, with additional drive(in this example /dev/sdb)
  * ubuntu/debian
  * Systemd
  * Podman or Docker for running containers
  * Time synchronization (such as chrony or NTP)

==== PREPARE ====

Edit /etc/hosts

<code bash>
192.168.56.101 node1 node1.linuxdemo.local
192.168.56.102 node2 node2.linuxdemo.local
192.168.56.103 node3 node3.linuxdemo.local


</code>

Set hostnames

<code bash>
sudo hostnamectl set-hostname node1
sudo hostnamectl set-hostname node2
sudo hostnamectl set-hostname node3


</code>

**Hostnames must be resolved by ipv4, not localhost loop!!!**

timesync install

<code bash>
sudo apt-get install ntp ntpdate -y
timedatectl set-ntp on


</code>

tune /etc/sysctl.conf

<code bash>
kernel.pid_max = 4194303
fs.aio-max-nr=1048576


</code>

<code bash>
sysctl -p


</code>

User ceph-admin add

<code bash>
sudo useradd -d /home/ceph-admin -m ceph-admin -s /bin/bash
sudo passwd ceph-admin
echo "ceph-admin ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph-admin
sudo chmod 0440 /etc/sudoers.d/ceph-admin


</code>

Ssh configure

Create ssh config for user

<code bash>
su ceph-admin


</code>

<code bash>
tee -a ~/.ssh/config <<EOF
Host *
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   IdentitiesOnly yes
   ConnectTimeout 0
   ServerAliveInterval 300
Host node1
   Hostname node1
   User ceph-admin
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   IdentitiesOnly yes
   ConnectTimeout 0
   ServerAliveInterval 300
Host node2
   Hostname node2
   User ceph-admin
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   IdentitiesOnly yes
   ConnectTimeout 0
   ServerAliveInterval 300
Host node3
   Hostname node3
   User ceph-admin
   UserKnownHostsFile /dev/null
   StrictHostKeyChecking no
   IdentitiesOnly yes
   ConnectTimeout 0
   ServerAliveInterval 300
EOF


</code>

Generate and copy keys on all nodes

<code bash>
su ceph-admin
ssh-keygen
ssh-copy-id ceph-admin@node1 && ssh-copy-id ceph-admin@node2 && ssh-copy-id ceph-admin@node3


</code>

==== Ceph initial deploy ====

From node1(192.168.56.101)

<code bash>
su ceph-admin
sudo curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm -o /usr/local/bin/cephadm
sudo chmod +x /usr/local/bin/cephadm
sudo rm /etc/apt/sources.list.d/ceph.list
sudo rm /etc/apt/trusted.gpg.d/ceph.release.gpg
sudo /usr/local/bin/cephadm add-repo --release octopus
sudo /usr/local/bin/cephadm install
if [[ ! -d "/etc/ceph" |]]; then sudo mkdir -p /etc/ceph;fi
sudo /usr/local/bin/cephadm bootstrap --mon-ip  192.168.56.101 --initial-dashboard-user itclife  --initial-dashboard-password itclife --dashboard-password-noupdate


</code>

Output like

<code bash>
INFO:cephadm:Ceph Dashboard is now available at:
   URL: https://vm-1:8443/
        User: admin
    Password: fv8b98gzeb
INFO:cephadm:You can access the Ceph CLI with:
    sudo /usr/local/bin/cephadm shell --fsid a3f8eb22-edbc-11ea-a98b-09a89b8358a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
INFO:cephadm:Please consider enabling telemetry to help improve Ceph:
    ceph telemetry on
For more information see:
    https://docs.ceph.com/docs/master/mgr/telemetry/
INFO:cephadm:Bootstrap complete.


</code>

Login in shell(we log in in docker)

<code bash>
sudo /usr/local/bin/cephadm shell --fsid a3f8eb22-edbc-11ea-a98b-09a89b8358a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring


</code>

exit from docker

<code bash>
exit


</code>

Install packages other machines(from cluster) to mount ceph(node2,node3)

<code bash>
sudo cephadm install ceph-common


</code>

Copy keys to root user to other nodes

<code bash>
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node3


</code>

or

<code bash>
cat /etc/ceph/ceph.pub


</code>

and copy to /root/authorized_keys on node2,node3

=== Ceph nodes add via orch(orchestrator) ===

<code bash>
sudo ceph orch host add node2
sudo ceph orch host add node3


</code>

=== Set public network for our ceph ===

<code bash>
sudo ceph config set mon public_network 192.168.1.0/24


</code>

=== Deploy monitors on a specific set of hosts: ===

<code bash>
sudo ceph orch apply mon "node1,node2,node3"


</code>

or If you want to adjust the default of 5 monitors:

<code bash>
sudo ceph orch apply mon 5


</code>

=== Ceph add label mon ===

<code bash>
sudo ceph orch host label add node1 mon
sudo ceph orch host label add node2 mon
sudo ceph orch host label add node3 mon


</code>

=== ceph list hosts ===

<code bash>
ceph orch host ls


</code>

=== Ceph create osd ===

Ceph list osds

<code bash>
sudo ceph osd tree | column -t -s $'\t'


</code>

**Exec command on nodes to clean drives(not necessary)**

<code bash>
sudo wipefs --all /dev/sdb


</code>

Add osd on host's

<code bash>
sudo ceph orch daemon add osd node1:/dev/sdb
sudo ceph orch daemon add osd node2:/dev/sdb
sudo ceph orch daemon add osd node3:/dev/sdb


</code>

=== Ceph create fs and mds, and set placement groups (version1) ===

Create fs with name volume1

<code bash>
sudo ceph fs volume create volume1


</code>

Create 3 mds for fs with name volume1

<code bash>
sudo ceph orch apply mds volume1 3


</code>

When we create fs with name volume1, pools will be created automatically

List pools

<code bash>
sudo ceph osd lspools


</code>

output

<code bash>
1 device_health_metrics
2 cephfs.volume1.meta
3 cephfs.volume1.data


</code>

SET THE NUMBER OF PLACEMENT GROUPS

get pg

<code bash>
ceph osd pool get cephfs.volume1.data pg_num
ceph osd pool get cephfs.volume1.meta pgp_num


</code>

and set pg

<code bash>
sudo ceph osd pool set cephfs.volume1.data pg_num 128
sudo ceph osd pool set cephfs.volume1.data pgp_num 128


</code>

it is mandatory to choose the value of pg_num because it cannot be calculated automatically. Here are a few values commonly used:

Less than 5 OSDs set pg_num to 128 \\ Between 5 and 10 OSDs set pg_num to 512 \\ Between 10 and 50 OSDs set pg_num to 4096 \\ If you have more than 50 OSDs, you need to understand the tradeoffs and how to calculate the pg_num value by yourself \\ For calculating pg_num value by yourself please take help of [[https://ceph.com/pgcalc/|pgcalc]] tool

List fs

<code bash>
sudo ceph mds stat
sudo ceph fs ls


</code>

=== Ceph create fs and mds(version1) ===

Create fs with name cephfs

<code bash>
sudo ceph osd pool create cephfs_data 64 64
sudo ceph osd pool create cephfs_metadata 64 64
sudo ceph fs new cephfs cephfs_metadata cephfs_data
sudo ceph orch apply mds cephfs 3


</code>

Where:

  * {pool-name} is the name of the Ceph pool you are creating.
  * pg_num is the total number of placement groups for the pool. It is recommended that pg_num is set to 128 when using less than 5 OSDs in your Ceph cluster.
  * pgp_num specifies total number of placement groups for placement purposes. Should be equal to the total number of placement groups.

[[https://docs.ceph.com/docs/mimic/rados/configuration/pool-pg-config-ref/|Info about pg]]

<code bash>
# By default, Ceph makes 3 replicas of objects. If you want to make four
# copies of an object the default value--a primary copy and three replica
# copies--reset the default values as shown in 'osd pool default size'.
# If you want to allow Ceph to write a lesser number of copies in a degraded
# state, set 'osd pool default min size' to a number less than the
# 'osd pool default size' value.
osd pool default size = 4  # Write an object 4 times.
osd pool default min size = 1 # Allow writing one copy in a degraded state.
# Ensure you have a realistic number of placement groups. We recommend
# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
# divided by the number of replicas (i.e., osd pool default size). So for
# 10 OSDs and osd pool default size = 4, we'd recommend approximately
# (100 * 10) / 4 = 250.


</code>

=== Ceph mount cephfs ===

Create creds

Volume name volume1

<code bash>
sudo ceph fs authorize volume1 client.user / rw | sudo tee /etc/ceph/ceph.client.user.keyring
sudo chmod 600 /etc/ceph/ceph.client.user.keyring
sudo cat /etc/ceph/ceph.client.user.keyring


</code>

Output

<code bash>
[client.user]
    key = AQDSxDNfzpmkIhAAGyKz2EIqZsaVrXWP0ZNRAQ==


</code>

mount via sh

<code bash>
sudo mkdir /mnt/cephfs
sudo mount -t ceph node3:/ /mnt/cephfs -o name=user,secret=AQDSxDNfzpmkIhAAGyKz2EIqZsaVrXWP0ZNRAQ==
cd /mnt/cephfs


</code>

mount via fstab

<code bash>
192.168.56.101,192.168.56.102,192.168.56.103:/     /mnt/cephfs    ceph    name=user,secret=AQDSxDNfzpmkIhAAGyKz2EIqZsaVrXWP0ZNRAQ==,noatime,_netdev    0   2


</code>

===== FAQ =====

=== Deploy ceph client ===

<code bash>
ceph-deploy install ceph-client


</code>

=== Ceph test ===

<code bash>
cd /mnt/cephfs
sudo dd if=/dev/zero of=./test3.img bs=20M count=30


</code>

=== Ceph repair ===

<code bash>
sudo ceph osd crush rule dump # Shows you the current crush maps
sudo ceph osd getcrushmap -o comp_crush_map.cm  # Get crush map
crushtool -d comp_crush_map.cm -o crush_map.cm  # Decompile map
vim crush_map.cm   # Make and changes you need (host -> osd)
crushtool -c crush_map.cm -o new_crush_map.cm  # Compile map
sudo ceph osd setcrushmap -i new_crush_map.cm  # Load the new map


</code>

=== Ceph clean all packages and wipe data, umount fstab ===

<code bash>
cd /etc/systemd/system
rm -rf $(ls | grep ceph)
systemctl daemon-reload
docker rm -f $(docker ps | grep ceph | awk '{print $NF}')
apt-get purge ceph-*
 -y -f; apt-get autoremove -y
rm -rf /var/lib/ceph/*
sudo rm /etc/apt/sources.list.d/ceph.list
sudo rm /etc/apt/trusted.gpg.d/ceph.release.gpg
sudo rm /etc/apt/sources.list.d/docker-ce.list
sudo apt-get update
lvremove $(lvdisplay  | grep Path | awk '{print $NF}' | grep ceph ) --force  --verbose
vgremove $(vgdisplay  | grep Name | awk '{print $NF}' | grep ceph ) --force  --verbose
wipefs --all /dev/sdb


</code>

Delete record from fstab

<code bash>
sed -i '/^.*ceph.*$/d' /etc/fstab ; cat /etc/fstab


</code>

=== Ceph commands ===

<code bash>
sudo ceph status
sudo ceph orch <start|stop|restart|redeploy|reconfig> <service_name>
sudo ceph orch restart osd
sudo ceph orch resume # CEPHADM_PAUSED: cephadm background work is paused


</code>

get pools

<code bash>
sudo ceph osd lspools


</code>

output

<code bash>
1 device_health_metrics
2 cephfs_data
3 cephfs_meta


</code>

delete pool

<code bash>
systemctl stop ceph-mds.target
killall ceph-mds
ceph mds cluster_down
ceph mds fail 0
ceph fs rm cephfs name --yes-i-really-mean-it
ceph osd pool delete cephfs data pool cephfs data pool --yes-i-really-really-mean-it
ceph osd pool delete cephfs metadata pool cephfs metadata pool --yes-i-really-really-mean-it
rm -rf "/var/lib/ceph/mds/cluster-metadata server"
ceph auth del mds."$hostname"


</code>

**Mon configure**

list mon

<code bash>
ceph mon dump


</code>

rm mon from node3

<code bash>
ceph mon rm node3


</code>

**Osd configure**

Set min_size data

<code bash>
ceph osd pool set cephfs_data  min_size 1


</code>

Osd tree

<code bash>
ceph osd tree | column -t -s $'\t'


</code>

Remove osd

<code bash>
ceph osd crush remove osd.0
ceph auth del osd.0
ceph osd rm 0


</code>

**Ceph stop cluster**

Umount share on all nodes and exec on all nodes

<code bash>
sudo systemctl stop ceph\*.service ceph\*.target


</code>

==== Fix ceph after crush ====

On all nodes

<code bash>
systemctl stop ceph\*.service ceph\*.target


</code>

An then restart

<code bash>
systemctl restart ceph-11d4341a-dde8-11ea-9cb6-063fcf8afd33.target


</code>

===== Mans =====

[[https://docs.ceph.com/docs/master/cephadm/install/|https://docs.ceph.com/docs/master/cephadm/install/]]

[[https://medium.com/@balderscape/setting-up-a-virtual-single-node-ceph-storage-cluster-d86d6a6c658e|https://medium.com/@balderscape/setting-up-a-virtual-single-node-ceph-storage-cluster-d86d6a6c658e]]

[[https://github.com/ceph/ceph/pull/35587/files|https://github.com/ceph/ceph/pull/35587/files]]

[[https://docs.ceph.com/docs/giant/rados/operations/control/|https://docs.ceph.com/docs/giant/rados/operations/control/]]

[[http://onreader.mdl.ru/LearningCeph/content/Ch02.html#From_zero|http://onreader.mdl.ru/LearningCeph/content/Ch02.html#From_zero]]

[[https://programming.vip/docs/ceph-delete-add-osd.html|https://programming.vip/docs/ceph-delete-add-osd.html]]

[[https://docs.ceph.com/docs/mimic/cephfs/fstab/|https://docs.ceph.com/docs/mimic/cephfs/fstab/]]

[[https://docs.ceph.com/docs/master/cephfs/fs-volumes/|https://docs.ceph.com/docs/master/cephfs/fs-volumes/]]

[[https://kifarunix.com/install-and-configure-ceph-block-device-on-ubuntu-18-04/|Ceph block storage]]

[[https://docs.ceph.com/docs/master/cephadm/adoption/|CONVERTING AN EXISTING CLUSTER TO CEPHADM]]

[[https://ceph.io/planet/debug-ceph-docker-containers/|debug-ceph-docker-containers]]

[[https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/container_guide/administering-ceph-clusters-that-run-in-containers|administering-ceph-clusters-that-run-in-containers]] \\
[[http://onreader.mdl.ru/LearningCeph/content/Ch04.html|PG]]