Using TRIM/DISCARD with Ceph RBD and libvirt
TRIM/DISCARD Using TRIM/DISCARD you can give back free space to a Ceph cluster. Normally, any thin provisioned block device will keep on growing until its maximum size while being used. Using the DISCARD command a underlying block device can be instructed to discard blocks which do not contain data. In the case of Ceph’s RBD […]
The Ceph Trafficlight
At PCextreme we have a 700TB Ceph cluster which is used behind our public cloud Aurora Compute which runs Apache CloudStack. Ceph health One of the things we monitor of the Ceph cluster is it’s health. This can be OK, WARN or ERR. It speaks for itself that you always want to see OK, but […]
Ceph Monitors are laggy or clock might be skewed
This weekend I got to investigate a Ceph cluster which had issues where the Monitors were constantly performing new elections. After some investigation on of the three monitors was eating 100% CPU on a single core and kept printing this in the logs: mon.charlie@2(peon).paxos(paxos updating c 106399655..106400232) lease_expire from mon.0 [2a00:XXX:121:XXX::6789:1]:6789/0 is 2.380296 seconds in […]
Protecting your Ceph pools against removal or property changes
One of the dangers of Ceph was that by accident you could remove a multi TerraByte pool and loose all the data. Although the CLI tools asked you for conformation, librados and all it’s bindings did not. Imagine explaining that you just removed a 200TB pool from your storage system due to a typo in […]
Rebuilding libvirt under CentOS 7.1 with RBD storage pool support
If you want to use CentOS 7.1 for your hypervisors with Apache CloudStack and Ceph’s RBD as Primary Storage you need to rebuild libvirt. CloudStack requires libvirt to be built with RBD storage pool support. It uses libvirt to manage RBD volumes. By default libvirt under CentOS is not built with this support. (On Ubuntu […]
NFS-Ganesha with libcephfs on Ubuntu 14.04
This week I’m testing a lot with CephFS and one of the things I never tried was re-exporting CephFS using NFS-Ganesha and libcephfs. NFS-Ganesha is a NFS server which runs in userspace. It has multiple backends (FSALs) it can use and libcephfs is one of them. libcephfs is a userspace library which you can use […]
Ceph with a cluster and public network on IPv6
I’m a big fan of Ceph and IPv6, so I always try to deploy Ceph over IPv6 when possible. Ceph is the future, just like IPv6 is. Why implement legacy? Recently I did a deployment of Ceph with a public and cluster network running over IPv6. It has a small catch, so I let me […]
PowerDNS backend for a global RADOS Gateway namespace
At my hosting company PCextreme we are building a cloud offering based on Ceph and CloudStack. We call our cloud services Aurora. Our cloud services are composed out of two components: Compute and Objects. For our Aurora Objects service we use the RADOS Gateway from Ceph and we are using the Federated Config to create […]
Deploying Ceph over IPv6
I like to deploy Ceph clusters over IPv6. I actually think that’s the way forward. IPv4 is legacy just like iSCSI and NFS are. Last week I was at a customer deploying a new Ceph cluster and they wanted to deploy with IPv6! Most deployment I did with IPv6 were done manually and not with […]
Calculating RADOS objects for RBD images
Ceph’s RBD (RADOS Block Device) is just a thin wrapper on top of RADOS, the object store of Ceph. It stripes (by default) over 4MB objects in RADOS. It’s very simple to calculate which RADOS object corresponds with which sector on your RBD image/block device. First you have to find out the block device’s object […]