Ceph RBD latency with QD=1 bs=4k
You might be asking yourself, why talk about QD=1 bs=4k? Well, from the experience of Wido den Hollander, our colleague at 42on, a single thread IO latency is very important for many applications. What we see with benchmarks is that people focus on high amounts of bandwidth and large amounts of IOps. They simply go […]
Cephalocon 2021, Let’s talk Ceph!
At this moment we are busy with the organization of another Cephalocon. Cephalocon is an annual event organized by the Ceph Foundation. By organizing Cephalocon, Ceph aims to bring together technologists and adopters from across the globe to showcase Ceph’s history and its future, demonstrate real-world applications, and highlight vendor solutions. By doing this, Ceph […]
HAProxy in front of Ceph Manager dashboard
The Ceph Mgr dashboard plugin allows for an easy dashboard which can show you how your Ceph cluster is performing. In certain situations you can’t contact the Mgr daemons directly and you have to place a Proxy server between your computer and the Mgr daemons. This can be done easily with HAProxy and the following […]
VXLAN with VyOS and Ubuntu 18.04
VXLAN Virtual Extensible LAN uses encapsulation technique to encapsulate OSI layer 2 Ethernet frames within layer 4 UDP datagrams. More on this can be found on the link provided. For a Ceph and CloudStack environment I needed to set up a Proof-of-Concept using VXLAN and some refurbished hardware. The main purpose of this PoC is […]
Placement Groups with Ceph Luminous stay in activating state
Placement Groups stuck in activating When migrating from FileStore with BlueStore with Ceph Luminuous you might run into the problem that certain Placement Groups stay stuck in the activating state. 44 activating+undersized+degraded+remapped PG Overdose This is a side-effect of the new PG overdose protection in Ceph Luminous. Too many PGs on your OSDs can cause […]
Quick overview of Ceph version running on OSDs
When checking a Ceph cluster it’s useful to know which versions you OSDs in the cluster are running. There is a very simple on-line command to do this: ceph osd metadata|jq ‘.[].ceph_version’|sort|uniq -c Running this on a cluster which is currently being upgraded to Jewel to Luminous it shows: 10 “ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)” 1670 […]
Do not use SMR disks with Ceph
Many new disks like the Seagate He8 disks are using a technique called Shingled Magnetic Recording to increase capacity. As these disks offer a very low price per Gigabyte they seem interesting to use in a Ceph cluster. Performance Due to the nature of SMR these disks are very, very, very bad when it comes […]
Testing Ceph BlueStore with the Kraken release
Ceph version Kraken (11.2.0) has been released and the Release Notes tell us that the new BlueStore backend for the OSDs is now available. BlueStore The current backend for the OSDs is the FileStore which mainly uses the XFS filesystem to store it’s data. To overcome several limitations of XFS and POSIX in general the […]
Running headless VirtualBox inside Nested KVM
For the Ceph training at 42on I use VirtualBox to build Virtual Machines. This is because they work under MacOS, Windows, and Linux. For the internal Git at 42on we use Gitlab and I wanted to use Gitlab’s CI to build my Virtual Machines automatically. As we don’t have any physical hardware at 42on (everything […]
Slow requests with Ceph: ‘waiting for rw locks’
Slow requests in Ceph When a I/O operating inside Ceph is taking more than X seconds, which is 30 by default, it will be logged as a slow request. This is to show you as a admin that something is wrong inside the cluster and you have to take action. Origin of slow requests Slow […]