Detecting swap usage by Ceph daemons
Detecting swap usage by Ceph daemons TL;DR: In good Ceph architectures we prefer daemons to be killed, instead of being swapped. This behaviour is much more predictable, detectable and usually informs you about where to fix the actual problem. We use this method to detect whether Ceph daemons are being swapped by the kernel. To […]
How to handle large omap objects
Every once in a while a customer will ask me what to do with these messages: health: HEALTH_WARN1 large omap objects First let’s see what this means: Most of the time when seeing the large omap warning is in relation to the RGW workload. The RGW or Rados Gateway provides a S3 and/or SWIFT compatible […]
Creating a Management Routing Instance (VRF) on Juniper QFX5100
For a Ceph cluster I have two Juniper QFX5100 switches running as a Virtual Chassis. This Virtual Chassis is currently only performing L2 forwarding, but I want to move this to a L3 setup where the QFX switches use Dynamic Routing (BGP) and thus become the gateway(s) for the Ceph servers. This should work, but […]
Setting noout flag per Ceph OSD
Prior to Ceph Luminous you could only set the noout flag cluster-wide which means that none of your OSDs will be marked as out. On large(r) cluster this isn’t always what you want as you might be performing maintenance on a part of the cluster, but you sill want other OSDs which go down to […]
Comparing two Ceph CRUSH maps
Sometimes you want to test if changes you are about to make to a CRUSH map will cause data to move or not. In this case I wanted to change a rule in CRUSH where it would use device classes, but I didn’t want any of the ~1PB of data in that cluster to move. […]
HAProxy in front of Ceph Manager dashboard
The Ceph Mgr dashboard plugin allows for an easy dashboard which can show you how your Ceph cluster is performing. In certain situations you can’t contact the Mgr daemons directly and you have to place a Proxy server between your computer and the Mgr daemons. This can be done easily with HAProxy and the following […]
VXLAN with VyOS and Ubuntu 18.04
VXLAN Virtual Extensible LAN uses encapsulation technique to encapsulate OSI layer 2 Ethernet frames within layer 4 UDP datagrams. More on this can be found on the link provided. For a Ceph and CloudStack environment I needed to set up a Proof-of-Concept using VXLAN and some refurbished hardware. The main purpose of this PoC is […]
Placement Groups with Ceph Luminous stay in activating state
Placement Groups stuck in activating When migrating from FileStore with BlueStore with Ceph Luminuous you might run into the problem that certain Placement Groups stay stuck in the activating state. 44 activating+undersized+degraded+remapped PG Overdose This is a side-effect of the new PG overdose protection in Ceph Luminous. Too many PGs on your OSDs can cause […]
Quick overview of Ceph version running on OSDs
When checking a Ceph cluster it’s useful to know which versions you OSDs in the cluster are running. There is a very simple on-line command to do this: ceph osd metadata|jq ‘.[].ceph_version’|sort|uniq -c Running this on a cluster which is currently being upgraded to Jewel to Luminous it shows: 10 “ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)” 1670 […]
Do not use SMR disks with Ceph
Many new disks like the Seagate He8 disks are using a technique called Shingled Magnetic Recording to increase capacity. As these disks offer a very low price per Gigabyte they seem interesting to use in a Ceph cluster. Performance Due to the nature of SMR these disks are very, very, very bad when it comes […]