A command line mail setup with OfflineImap, NotMuch and Alot
Some times ago, I decided to part as much as possible with Google and its like, and go down the difficult road of self-hosted services. This was not an easy decision, but it is not one I’m regretting, at least for now.
A bit of background
When I decided to stop using online services and self-host them instead, I started by renting a machine and buying a domain name3 and hosting some services on my machine. Now I I’m a big believer in "transmitting experience", so when the time came to switch my mail service, I googled4 "self-hosting IMAP SMTP" and read about it. That is how I came to the realisation that I should not self-host my IMAP and SMTP servers.
Hosting a mail service is a real, full-time job. Mail is, at least for me, critically important, and I cannot bear downtime, mails considered as spam and other whatnots. Therefore I chose to go with my hosting provider, which gave me access to IMAP POP3 and SMTP servers as part of my hosting contract.
The provider also gives me access to a RoundUp webmail. I tried it, it’s clean and nice, but I don’t want to use it for everyday mails: it’s not fun enough5. I already tried a lot of heavy mail clients: Outlook, Thunderbird and its clones, Evolution,… None is really fun6. I could see but one path left to try: a command-line client.
Command line is fun!
Yes it is. Don’t deny it. Unfortunately the vast majority of command-line mail clients are a little dated, hard to use and hard to configure. So I had to do a little bit of digging to search past the Mutt and (Al)Pine. If you are interested in them you will find plenty of information on the web. For myself, I tried them and didn’t enjoy it.
So, as the blog title says, the solution I came up with is based around notmuch, offlineimap and alot. I want to add that I like the fact that my mail setup is separated in smaller, more dedicated parts, staying true to the KISS philosophy.
List of parts
I will now describe the parts and their configuration. Again, I’m no expert with these programs so it may contain mistakes. Nonetheless, I always find it useful when someone shares its own setup: it’s more practical than just documentation examples. Here it is.
OfflineImap
OfflineImap is a python program for syncing a remote IMAP account with a local Maildir folder7. It accepts multiple accounts and works both ways: a mail added or deleted locally will also be added or deleted on the IMAP server.
# Sample minimal config file. Copy this to ~/.offlineimaprc and edit to
# get started fast.
[general]
accounts = lertsenem
pythonfile = ~/.config/offlineimap/gnome-keyring_helper.py
[Account lertsenem]
localrepository = lertsenem-local
remoterepository = lertsenem-remote
postsynchook = ~/.config/offlineimap/hooks/postsync_lertsenem
[Repository lertsenem-local]
type = Maildir
localfolders = ~/Mail/lertsenem
[Repository lertsenem-remote]
type = IMAP
ssl = yes
sslcacertfile = ~/.config/offlineimap/certs/mail.gandi.net.chain.crt
remotehost = mail.gandi.net
remoteport = 993
remoteusereval = get_username("lertsenem")
remotepasseval = get_password("lertsenem")
The most important part of the offlineimap configuration is the .offlineimaprc dotfile. The important parts are:
-
you can use multiple account (but I left only one for privacy reasons) ;
-
you can use hooks ;
-
you can store your credentials any way you like.
So, here I connect to my IMAP server provider, which is Gandi, using the credentials returned by the functions get_username() and get_password(). These functions are not defined by offlineimap, but by the python file I specified in the pythonfile configuration directive. You can read it here:
#!/usr/bin/env python
import gnomekeyring as gkey
def set_credentials(repo, user, pw):
KEYRING_NAME = "mails"
attrs = { "repo": repo, "user": user }
keyring = gkey.get_default_keyring_sync()
gkey.item_create_sync(keyring, gkey.ITEM_NETWORK_PASSWORD,
KEYRING_NAME, attrs, pw, True)
def get_credentials(repo):
keyring = gkey.get_default_keyring_sync()
attrs = {"repo": repo}
items = gkey.find_items_sync(gkey.ITEM_NETWORK_PASSWORD, attrs)
return (items[0].attributes["user"], items[0].secret)
def get_username(repo):
return get_credentials(repo)[0]
def get_password(repo):
return get_credentials(repo)[1]
if __name__ == "__main__":
import sys
import os
import getpass
if len(sys.argv) < 2:
print "Usage: %s (get|set)" \
% (os.path.basename(sys.argv[0]))
sys.exit(0)
method = sys.argv[1]
if method == "set":
if len(sys.argv) != 4:
print "Usage: %s set <repository> <username>" \
% (os.path.basename(sys.argv[0]))
sys.exit(0)
repo, username = sys.argv[2:]
password = getpass.getpass("Enter password for user '%s': " % username)
password_confirmation = getpass.getpass("Confirm password: ")
if password != password_confirmation:
print "Error: password confirmation does not match"
sys.exit(1)
set_credentials(repo, username, password)
elif method == "get":
if len(sys.argv) != 4:
print "Usage: %s get <repository> (username|password)" \
% (os.path.basename(sys.argv[0]))
sys.exit(0)
repo, toget = sys.argv[2:]
if toget == "username":
print(get_username(repo))
elif toget == "password":
print(get_password(repo))
As you can see, this script8 uses the gnome-keyring to fetch a username and a password, in the default keyring (which name is Login most of the time) with the name mails. If you don’t know how to use credentials in your gnome-keyring do not frown: the script also allows you to store the credentials by running it.
The hooks are defined on a per-account basis with the postsynchook directive9. They can be any script you like. Mine is written in shell and goes like this:
#!/bin/sh # Initial import notmuch new # Tagging notmuch tag -inbox +archive folder:Archive notmuch tag -inbox +draft folder:Brouillons notmuch tag -inbox +trash folder:Corbeille
As you can see, I make use of the notmuch command to tag the new mails according to specific rules. What is notmuch you ask? That’s exactly what I’m going to talk to you about right now.
Notmuch
Notmuch is a C program dedicated to index and search through a mail database. It works with tags rather than folders: you can add and remove tags (arbitrary strings) to each mail and then filter them quickly based on those tags. For example, the default configuration adds an inbox tag to all messages present in your INBOX/ folder, an unread tag to all your unread mails, an attachment tag to all your mail containing an attachment, etc.
Here is my notmuch configuration:
[database] path=/home/lertsenem/Mail/<my_email_address> [user] name=Lertsenem primary_email=<my_email_address> [new] tags=unread;inbox; ignore= [search] exclude_tags=deleted;spam; [maildir] synchronize_flags=true
The default configuration file is under ~/.notmuch-config. There are a lot more options you can use, but for now I chose to take a minimalist approach.
-
[database] indicates where my mails are stored (Maildir standard). Basically, it’s the output of the offlineimap command.
-
[new] and [search] define respectively what tags are automatically set for new mails, and what tags are ignored from searches.
-
synchronize_flags indicates to synchronize some specific tags, such as the unread tag, with the corresponding IMAP flags. By doing this, I ensure that my mail is marked as read on the IMAP server whenever I remove the unread tag of a mail10.
Now my mail is synchronized and indexed. I have everything I need and I’m happy.
Okay, I may sometimes need a way to read it easily and send some… Here comes alot!
Alot
Notmuch itself is only a search engine. There are some default front-end like an Emacs plug-in11, a vim one12 or an extension for Mutt, and then there’s Alot.
Alot is a python front-end for Notmuch. It’s beautiful13, fast and modern. The only drawbacks is that the default configuration is non-existent, and it’s hard to find real life example online14.
Here is mine.
theme = "solarized"
[accounts]
[[lertsenem@lertsenem.com]]
realname = Lertsenem
address = <my_email_address>
sent_box = maildir:///home/lertsenem/Mail/<my_email_address>/INBOX
draft_box = maildir:///home/lertsenem/Mail/<my_email_address>/Brouillons
sent_tags = inbox,
sendmail_command = msmtp --account=lertsenem -t
The alot default configuration is in ~/.config/alot/config, and it’s great to see an application actually giving a damn about the .config directory.
First of all, let’s use a civilized theme with solarized. The theme is included in the alot repo on github, get it there if you need it (it’s not included in the deb package distributed by Ubuntu).
The definition of the theme is enough to run alot and read you mail15, because alot looks at the .notmuch-config file to get the informations it needs. If you want to send mail with it though, you have to define at least one account. The configuration is self-explicit, except maybe the sent_tags parameter, which lists the tags to add to a sent mail. The sendmail_command is the command to use to… well… send your mail. As you can see, I use msmtp, which is the last program of my setup I’m going to share with you.
Msmtp
msmtp is just a sendmail command, a way to send e-mail via the command line. The configuration file is straightforward:
defaults
auth on
tls on
syslog on
account lertsenem
host mail.gandi.net
port 465
tls_trust_file ~/.config/msmtp/certs/mail.gandi.net.chain.crt
tls_starttls off
tls on
from <my_email_address>
user <my_email_address>
passwordeval "~/.config/msmtp/gnome-keyring_helper.py get lertsenem password"
Here again, like offlineimap, you can use a script and the passwordeval parameter to fetch securely a password. Here I use the same script I presented earlier and run it normally to get my password from the gnome-keyring.
Time to play!
Now I have all the parts configured, let’s play!
In a shell, I use the command offlineimap to synchronize my emails. The notmuch command is automatically invoked by a hook to index the synchronized emails. Finally I use alot and I can read my mails like a boss16
When I’m done reading and writing mails, I quit alot and I use offlineimap again to synchronize the files I just created (when writing mails) and to update the read status of the mails I just read.
And done.
Now, can we do better? Sure.
Hooks in alot
Having to use offlineimap separately from alot is tedious. To solve that, there are two ways to go: you can make offlineimap a service or a crontask, but according to a lot of people on the Internet it’s kind of buggy and do not always work well. Or you can use alot hooks to automatically run offlineimap every time it’s needed —and nothing more.
Like offlineimap, alot allows you to use hooks, written in python. There are a lot of hooks you can set up (take a look at the documentation, it is really well written), I use two of them to automatically run the offlineimap command.
#!/usr/bin/env python
import logging
import subprocess
from alot.settings import settings
def pre_global_exit(**kwargs):
accounts = settings.get_accounts()
logging.info('Syncing mail with offlineimap')
subprocess.call(["offlineimap"])
if accounts:
logging.info('goodbye, %s!' % accounts[0].realname)
else:
logging.info('goodbye!')
def pre_global_refresh(**kwargs):
logging.info('Syncing mail with offlineimap')
subprocess.call(["offlineimap"])
The hooks file is by default ~/config/alot/hooks.py. You can write the hooks by defining functions with specific names. Here I define pre_global_exit() to run offlineimap before exiting, and pre_global_refresh() to run again offlineimap every time I use the :refresh command in alot (default shortcut is @).
That way, no need to run offlineimap by itself anymore: I just fire alot, press @ to refresh, and I am ready to go!
Multiple accounts
There are two ways to deal with multi-accounts with this setup, depending on how you use notmuch.
Every other programs I showed you has a notion of accounts, which mean you can set up different accounts, but not notmuch.
The first solution is to index all your mails together. For example, I sync my mail1@example.com and mail2@example.com in /Mail/mail1/ and /Mail/mail2 respectively using different offlineimap accounts. Then I setup notmuch to run in ~/Mail/ and it will index all of my mails, for both mail1 and mail2. When I then run alot, I see all of my mail at once (and I can reply/write mail with any account I set up in my alot configuration).
The second solution, which is the one I use, is to completely separate the accounts and the notmuch databases by writing two different .notmuch-config files. To do that, we make use of the --config option in notmuch which allows us to specify a configuration file name (rather than the ~/.notmuch-config default). Use this option in the offlineimap postsync hooks I showed you earlier (remember how these +postsynchook+s directive are defined on a per-account basis?) and you can run notmuch for each account independently.
Now for alot, which uses the notmuch config file, you can use the -n option to specify the file to use. To separate the accounts furthermore, you can also write different alot config files for each account (rather than ~/.config/alot/config default) and specify them with the -c option. That way you won’t risk to send a mail from one account with another address.
To simplify running alot, you can finally define some aliases, such as the one following.
$> alias alot-lertsenem="alot \
-c ~/config/alot/config-lertsenem \
-n ~/.notmuch-config-lertsenem"
Add this to your .{bash,zsh}rc file and you are good to go. :]
Conclusion
That’s it for now, I’m pretty satisfied with my setup, it’s at least as fun to play with than GMail. And I can write my mail with vim, which is plainly awesome!
I’m sure there are tons of ways to improve it (especially with all the hooks I can define in alot, yummy), but for this will be done over time. If by chance you have some ideas, do not hesitate to share them with me!
As always for comments, suggestions or insults, please use my twitter account.
- Probably the best, apart from their search engine of course. ↩
- Be warned that this post is by no mean a definitive solution, I was just giving a little bit of context so that you may understand what I like and what I’m looking for in mail UI. ↩
- It’s lertsenem.com. Come on, it’s written right in your address bar. Please make an effort to follow. ↩
- Yes. It’s a long and difficult road, have mercy. ↩
- Yes, fun. I can’t translate any better the satisfaction feeling I got when managing my emails with GMail ↩
- And a lot of them are actually pretty sad to use… ↩
- Which is just a standardized way to store mails as files ↩
- which I did not write from scratch, I merely added the get command to some script found on the Archlinux website ↩
- There are of course more, depending on when you want the hooks to be run; go take a look at the documentation if you need them ↩
- I will need to re-synchronize my local mail with the remote server using offlineimap of course. ↩
- Urk. ↩
- Which I never managed to get working. ↩
- Okay, I may oversell it. But no, really, I like it. :) ↩
- The program is still quite young. Fortunately, the documentation is clear and well-written. ↩
- Okay, you don’t even have to change the theme. But you should. ↩
- The important command at this point is ? to prompt for help. ↩