Selfhosting your calendars, todolists and contacts
As you may or may not know, I have decided to part from Google. And by that I don’t mean to find another service provider: I want to host my services myself.
As you may or may not have guessed, I’m going to talk in this post about setting up a server (and the corresponding clients) to sync my calendars, my todolists and my contacts.
Let’s get to it.
Minimal necessary information
This paragraph reminds the absolute basics. If the CalDAV and CardDAV protocol already are something you heard of, you can jump to the next title
The choice of describing how to setup servers for calendars, todolists and contacts all at once may seem rather odd, and maybe you are wondering why I’m doing one post instead of three. The reason is simple: all of these services are actually (almost) one and the same. Let me explain.
First, a todolist is just a special calendar. You have some different fields, such as priority, and the dates of events are not mandatory, but basically it’s the same thing. So todolists and calendars are actually the same thing, stored in ics files, the iCalendar format.
For the contacts things are a little different (hence the precautionnary almost above). Contacts are stored using the vCard format, in vcf files. The vCard format, as the iCalendar format, is a textual format and both look very much alike. But what truly put them together is the protocol used to sync them with a remote server. Contacts are synced using CardDAV, calendars and todolists are synced using CalDAV. Both protocols are extensions of the WebDAV protocol, itself an extension of the HTTP protocol.
Consequently, servers generally choose to implement both protocols and therefore act as CalDAV and CardDAV servers. Which means you can install and configure just one server and have calendars, contacts and todolists synchronized. Cool, huh?
Installation and configuration of the Radicale server
For ease of installation, deinstallation, update and concistency, I insist that all my self-hosted services run inside Docker containers, and this one will be no exception. If you don’t know about Docker yet, I encourage you to check out their website and read through the tutorial pages. For the rest of this post, I will assume you know at least what Docker is and how it runs.
There are a lot of different CalDAV/CardDAV projects ouit there. After browsing them for a while 1, I settled on Radicale. The features are basically the same than the other projects. What decided me was :
it has an official Dockerfile as part of the project ;
it’s written in python.
Anyway, the website is here, if you want to take a look.
I said that there was an official Dockerfile, but there are no official Docker repository on te Docker hub2. So the first thing we need to do is build it ourselves:
$> mkdir Kozea $> cd Kozea/ $> git clone https://github.com/Kozea/Radicale [ ... cloning occurs ..] $> cd Radicale $> docker build --tag lertsenem/radicale .
You should of course replace lertsenem with whatever your own Docker login is.
After the docker image is built, you can run it using the following command:
$> mkdir /tmp/radicaledata $> docker run -it \ -p 5232:5232 \ -v /tmp/radicaledata/:/data/config \ lertsenem/radicale
Radicale uses the port 5232 by default, we will need to change that later. The interesting directory of the docker image is /data/config, which holds the server configuration. We mapped it here to /tmp/radicaledata on our host system, now we are going to fill it.
The git repo contains a default example configuration for Radicale, which we can use as a starting point. Here is my final config file.
# -*- mode: conf -*- # vim:ft=cfg # Config file for Radicale - A simple calendar server # # Place it into /etc/radicale/config (global) # or ~/.config/radicale/config (user) [server] # SSL flag, enable HTTPS protocol #ssl = False # Message displayed in the client when a password is needed realm = Password Required [auth] # Authentication method # Value: None | htpasswd | IMAP | LDAP | PAM | courier | http | remote_user | custom type = htpasswd # Htpasswd filename htpasswd_filename = /data/config/users # Htpasswd encryption method # Value: plain | sha1 | ssha | crypt | bcrypt | md5 htpasswd_encryption = bcrypt [rights] # Rights backend # Value: None | authenticated | owner_only | owner_write | from_file | custom type = from_file # File for rights management from_file #file = ~/.config/radicale/rights
First, one important note: I leave the ssl setting to False, but that’s because I’m using an http proxy that manage all the SSL encryption.
|Be aware that you should not authenticate on a server without a proper, secure, encrypted communication channel. Especially a server you are going to log in to automatically on a regular basis.|
Also note that the default protocole suit is PROTOCOL_SSLv23, which is good for interoperability, but not so much for security. Unless you are using an oudated web browser3 you will be better using PROTOCOL_TLSv1.2.
In the [auth] section, you will then need to specify the authentication method you intend to use for you users. If you are very tech savvy and love playing sysadmin on your free time, maybe you already have an LDAP server that you can use for authentication. If you want to use the unix accounts of your machine, use the PAM setting4. But if you are someone simple and direct, you can use the htpasswd setting. You can then add a htpasswd_filename and a htpasswd_encryption5, and all you are needing is a htpasswd file with some users in it. You can create one thusly:
$> htpasswd -c -B /tmp/radicaledata/users email@example.com $> htpasswd -B /tmp/radicaledata/users anotheruser
Finally, you have to give rights upon different collections to your users. Some easy settings for that are type = owner_only or owner_write. They both give users full permissions on directories matching their logins, and the second one also gives read permissions to every authenticated users on every directory. The authenticated setting is easier even, giving full permissions on every collection to every user authenticated. But that does not satisfy us, we want security and privacy, with shared collections, and we are going to achieve that using type = from_file.
Of course we now need to create a rights file.
Here is mine.
[owner-write] user: .+ collection: ^%(login)s(/.*)*$ permission: rw [shared] user: .+ collection: ^shared(-.*)*-%(login)s(-.*)*(/.*)*$ permission: rw [public-read] user: .* collection: ^public-.*(/.*)*$ permission: r [public-write] user: .+ collection: ^public(-.*)*-%(login)s(-.*)*(/.)*$ permission: rw
The rights file is simple to understand. Each section of the file is a rule, and you can give it any name you want. Then for each rule, you have to specify:
who it concerns (user)
what it concerns (collection)
what it allows (permission)
Every permission that is not exmplicitly autorized is forbidden.
The syntax uses regular exressions and some specific symbols. Here for example, the first rule [owner-write] specifies that any user with a login matching . (which means more than one character: no anonymous) has full rw permissions on collections matching their login. You could guess it with the name, that’s a reimplementation of the +owner-write setting we saw earlier.
The [shared] rule is more complex and specifies that any authenticated user6 has full read and write permissions on collections starting with shared and containing their login (separated by dashes). This allows both my users firstname.lastname@example.org and anotheruser to access and modify the collections under email@example.com/*.
And with that our server is configured. Let’s create some empty collections.
$> mkdir -p /firstname.lastname@example.org $> touch /email@example.com/Main\ calendar $> touch /firstname.lastname@example.org/TODO
And now our server is working.
The docker command line I used earlier was ugly. If you want to do things properly, I recommand you use the nginx-proxy docker image to handle the HTTP connections, create a dedicated volume-only docker container for your configuration and your data (with someting along $> docker create -v /data --name radicale-data lertsenem/radicale /bin/true), and finally you start your docker container properly:
$> docker run -d \ --name radicale \ --volumes-from radicale-data \ --expose=5232 \ -e "VIRTUAL_HOST=radicale.mydomain.example.com" \ lertsenem/radicale
Configuration of an android client: DavDroid
We have the server, now we will need clients. I will start with the client I’m going to use the most: the one on my smartphone.
The Radicale website propose several alternatives for android clients. I picked DavDroid, because why not. The icon seems clean enough that someone put some effort in the app, the updates seem regulat, and it’s available on F-Droid.
|If like me you want to use the Tasks application to manage your todo lists, you have to install it before installing CalDAV, for some reason I was not inclined to investigate. Permissions problems, something like that.|
The configuration of DavDroid is very straightforward, but the apps is not exempt of problems. For example, you have to delete an account to register new calendars/todolists. Yep, that sucks. Also, you cannot create new calendars/todolists from the app. Sorry.
|The DavDroid app reached version 1.0 recently, and those two problems have apparently been corrected. Cool huh? :)|
Keep in mind that if the application does not seem to detect your collections, it’s almost certainly a permissions problem on your collections, on the Radicale side. Go check the Radicale logs.
Once DavDroid is correctly configured, you should have access to your calendars and contacts through the regular Android applications. Remember to spend some time in the settings of your account in the DavDroid application, especially to check the synchronization delays. By default DavDroid syncs contacts, calendars and tasks once every day and on each local modification. I found that not to be enough for the tasks and calendars I share with my girlfriend, so I set it to sync more often (every hour for the calendars and every 15 minutes for the tasks lists).
Configuration of vdirsyncer
Now, a smartphone is the main device I’m using to manage my agenda and contacts, but I still want to have a valid alternative for my desktop computer. Oh, and I want it with a command line interface too, because, you know, command line is fun.
Searching through the web for an application that would suit my needs, I stumble upon khal and vdirsyncer, which are so similar in principle to my alot/offlineimap setup that I immediately adopted them.
vdirsyncer will play the part of offlineimap, syncing my remote collections (contacts and calendars/todolists) to a local folder. Here is the configuration I use (description follows):
[general] status_path = ~/.local/share/vdirsyncer/status/ [pair my_calendars] a = my_calendars_local b = my_calendars_remote conflict_resolution = b wins collections = ["from b"] [storage my_calendars_local] type = filesystem path = ~/Calendarsemail@example.com/ fileext = .ics [storage my_calendars_remote] verify_fingerprint = "68:32:27:AC:FB:93:FC:DA:8B:33:75:7C:15:77:D3:52:49:45:02:EF:03:3D:87:75:55:05:1B:7D:FC:D0:DB:DE" verify = false type = caldav url = "https://radicale.lertsenem.com/" username.fetch = ["command", "~/.config/vdirsyncer/gnome-keyring_helper.py", "get", "firstname.lastname@example.org", "username" ] password.fetch = ["command", "~/.config/vdirsyncer/gnome-keyring_helper.py", "get", "email@example.com", "password" ] [pair my_contacts] a = my_contacts_local b = my_contacts_remote conflict_resolution = b wins collections = ["from b"] [storage my_contacts_local] type = filesystem path = ~/Contactsfirstname.lastname@example.org/ fileext = .vcf [storage my_contacts_remote] verify_fingerprint = "68:32:27:AC:FB:93:FC:DA:8B:33:75:7C:15:77:D3:52:49:45:02:EF:03:3D:87:75:55:05:1B:7D:FC:D0:DB:DE" verify = false type = carddav url = "https://radicale.lertsenem.com/" username.fetch = ["command", "~/.config/vdirsyncer/gnome-keyring_helper.py", "get", "email@example.com", "username" ] password.fetch = ["command", "~/.config/vdirsyncer/gnome-keyring_helper.py", "get", "firstname.lastname@example.org", "password" ]
As you can see, the configuration file is very similar to the offlineimap one.
First, in the [general] section you will find global settings. Here the status_path indicates where vdirsyncer should put its database.
For each connection, you will then have a [pair xxx] section describing the syncing of your remote and local calendars or contacts (and how to solve conflicts). You need to define two matching [storage xxx] sections to describe how to access your local and remote collections. As you can see, I chose to put my local collections under /Contacts/ and /Calendars/.
Also, note for the remote parts the verify=false and verify_fingerprint settings. They are here because I was too lazy to generate a real certificate and instead reused an old one, meant for another domain name. Thus vdirsyncer cannot really verify it (verify=false) and instead verifies its fingerprint. Of course this is not ideal and only meant as a temporary fix7.
On the remote side, you also need to specify a username and a password. You can pass them directly in the config file, using the username and password settings, but that would be leaving your credentials in clear text on your disk, which is unsafe (especially if you are not the only user of your computer). Instead, you can use the username.fetch and password.fetch to specify a command line that will get the credentials, and put them in a dedicated keyring app. Here I’m using a python script I got (and improved) from the Archlinux website to get my credentials from the gnome keyring, which is included with Ubuntu8.
Once your configuration is done, use vdirsyncer discover to initialise your base. Until your collections change, you will then just need to run vdirsyncer sync to synchronize your pairs.
Configuration of khal
While vdirsyncer was an offlineimap-like, khal is an alot-like. It’s a command-line frontend to navigate the local Calendars folder. It’s written in Python and packaged for pypi, so you can install it easily. Just remember to use pip3 to get the up-to-date python3 version.
$> pip3 install khal
The configuration is pretty simple, you just need to indicate the location of your synced Calendars on your disk. Optionally you may specify colors to prettify your interface, and that’s about it.
[default] days = 10 highlight_event_days = True [calendars] [[main]] path = "~/Calendarsemail@example.com/Main calendar/" color = dark green [[shared]] path = "~/Calendarsfirstname.lastname@example.org/Shared calendar/" color = dark red [[public]] path = "~/Calendarsemail@example.com/Public calendar/" color = light blue [view] theme = light [highlight_days] method = foreground default_color = black
You can use khal as a simple command line tool: typing khal in console will print a view of the current month, with colors marking days with events and a summary of the events of the present day. You can also start khal interactive and navigate throiugh each day, viewing, editing, adding and removing events with vi-like controls. Of course, you will need to vdirsyncer sync once you are done to synchronize your local modifications with your remote server.
Configuration of todoman
Finally I was looking for a frontend like khal to read and write my todo lists. I found it with todoman.
Like khal, todoman is a Python app, and you can install it using pip3. Like khal, the configuration is very easy (you have your content saved locally after all): just point the path toward the directory containing your collections and you are done.
[main] path = ~/Calendarsfirstname.lastname@example.org/*
todoman is easy to use. Here are the basics: use todo to get a list of all your tasks. You can refine this using todo list MyList. Once a todo command has printed out some numbered tasks, you can act on them base on their number. Thus todo done 3 will check the third task you last printed with a todo or todo list command. There’s more features available, but for a basic usage you are all set. Use todo --help for more pointers. :)
I’m pretty satisfied with my setup. Both the configuration and daily usage are very similar to my mail setup, which makes it easier to use. My calendars and contacts (and todolists, but that’s just a type of calendar) are now saved on my personnal server. I’ll wait to have a backup solution before I completely migrate my data though. :]
I didn’t speak about it in the article, but there’s a setting in the client mail alot to use contact completion. I wrote a short9 shell script to parse the VCF files in my local ~/Contacts/ directory:
#!/bin/sh ARG_PATTERN="$1" ARG_LOC="$2" [ -z "$ARG_LOC" ] && ARG_LOC="$HOME/Contacts" if [ -d "$ARG_LOC" ]; then all_vcfs="$(find "$ARG_LOC" -type f -iname "*.vcf")" else all_vcfs="$ARG_LOC" fi for vcf in $all_vcfs; do fn="$( grep '^FN:' "$vcf" | sed 's/^FN://' )" #nickname="$( grep 'NICKNAME:' "$vcf" | sed 's/NICKNAME://' )" emails="$( grep '^EMAIL\(;TYPE=[^;:]*\)\?:' "$vcf" | sed 's/^EMAIL.*://' )" for email in $emails; do echo "\"$fn\" <$email>" | grep -i "$ARG_PATTERN" done done exit 0
Now I can add the following section to my alot configuration:
[...] [[[abook]]] type = shellcommand command = /home/lertsenem/.config/alot/vcfparse regexp = '\"(?P<name>.+?)\"\s*<(?P<email>.+?)>'
And boom: when writing a mail I can autocomplete the recipients fields using <Tab>.
That’s it for today, next time I will probably talk about a nice solution of backup, or about how to reinstall you server from scratch quickly after a fatal crash. :]
As always for comments, suggestions or insults, please use my twitter account.
- http://alternativesto.net is a pretty good website for that purpose ↩
- or elsewhere, that I’m aware of, really ↩
- which is horrible and you absolutely should not do that, update now! ↩
- though beware that I’m not sure how this setting will react with docker… ↩
- which should be bcrypt, it’s the safest available ↩
- with a login at least one character long ↩
- Of course, as every temporary fix, this one has been going on for quite a while now… ↩
- See my previous article about a mail setup for more information on this script ↩
- and ugly, it probably does not scale to a lot of contacts ↩