I’ve been asked to help backup a friends email from their existing provider. Since we’re not sure what’ll happen to the existing email after the request to change service providers has been made, it seems reasonable to backup her existing email automatically.
Since I run my own mail server, I thought it’d be quite easy to run something like offlineimap to backup the existing service. One caveat to that has been the existing server prefixes “INBOX” to the mail directory, which my setup doesn’t support. It looks like this is a difference between Courier and Dovecot. I also want to rename “Sent Items” to “.Sent” to follow my Dovecot setup.
Fortunately, offlineimap offers the ability to translate folders on the fly.
.offlineimaprc
[general]
accounts = alice
[Account alice]
localrepository = Local
remoterepository = Remote
[Repository Local]
type = Maildir
localfolders = ~/Mail/Alice
[Repository Remote]
type = IMAP
readonly = true
remotehost = mail.example.com
remoteuser = alice@example.com
remotepass = secret
# Necessary as of OfflineIMAP 6.5.4
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)
ssl_version = tls1_2
nametrans = lambda foldername: re.sub('^INBOX\.*', '.',
re.sub('^Sent Items\.*', '.Sent', foldername))
Once I’ve got this setup, I can use crontabs to automatically sync it every x minutes.
Note that after pulling the email, Dovecot may need the quota recalculated manually:
doveadm quota get -u alice@example.com doveadm quota recalc -u alice@example.com
https://wiki2.dovecot.org/Tools/Doveadm/Quota
Here are some good resources:
https://github.com/OfflineIMAP/offlineimap/issues/417
[general]
metadata = $XDG_DATA_HOME/offlineimap
accounts = darnirmlist
maxsyncaccounts = 1
ui = basic
pythonfile = $XDG_DATA_HOME/offlineimap/passkey.py
socktimeout = 20
fsync = false
[DEFAULT]
localrepositoryhome = ~/.local/var/lib/mail
[Account darnirmlist]
localrepository = darnirmlist-local
remoterepository = darnirmlist-remote
status_backend = sqlite
autorefresh = 1
quick = 20
synclabels = yes
ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important
[Repository darnirmlist-local]
type = GmailMaildir
localfolders = %(localrepositoryhome)s/darnirmlist
[Repository darnirmlist-remote]
type = Gmail
remoteuser = darnirmlist@gmail.com
auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id('darnirmlist@gmail.com')
oauth2_client_secret_eval = get_client_secret('darnirmlist@gmail.com')
oauth2_refresh_token_eval = get_refresh_token('darnirmlist@gmail.com')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
nametrans = lambda folder: re.sub('.*Spam$', 'Spam',
re.sub('.*INBOX$', 'Inbox',
re.sub('.*Drafts$', 'Drafts',
re.sub('.*Sent Mail$', 'Sent',
re.sub('.*All Mail', 'Archive',
re.sub('.*Trash$', 'Trash', folder))))))
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail',
'[Gmail]/Drafts',
'[Gmail]/Sent Mail',
'[Gmail]/Spam']
createfolders = False
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
realdelete = no
holdconnectionopen = yes
keepalive = 50
http://www.offlineimap.org/doc/use_cases.html
# -*- mode: conf; -*-
#
# NOTE: Settings generally support python interpolation. This means
# values can contain python format strings which refer to other values
# in the same section, or values in a special DEFAULT section. This
# allows you for example to use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.
# NOTE2: This implies that any '%' needs to be encoded as '%%'
##################################################
# General definitions
##################################################
[general]
# This specifies where offlineimap is to store its metadata.
# This directory will be created if it does not already exist.
#metadata = ~/.offlineimap
# This variable specifies which accounts are defined. Separate them
# with commas. Account names should be alphanumeric only.
# You will need to specify one section per account below. You may
# not use "general" for an account name.
accounts = BoostPro
# Offlineimap can synchronize more than one account at a time. If you
# want to enable this feature, set the below value to something
# greater than 1. To force it to synchronize only one account at a
# time, set it to 1.
#
# Note: if you are using autorefresh and have more than one account,
# you must set this number to be >= to the number of accounts you have;
# since any given sync run never "finishes" due to a timer, you will never
# sync your additional accounts if this is 1.
#maxsyncaccounts = 1
# You can specify one or more user interface modules for OfflineIMAP
# to use. OfflineIMAP will try the first in the list, and if it
# fails, the second, and so forth.
#
# The pre-defined options are:
# Blinkenlights -- A fancy (terminal) interface
# TTYUI -- a text-based (terminal) interface
# Basic -- Noninteractive interface suitable for cron'ing
# Quiet -- Noninteractive interface, generates no output
# except for errors.
# MachineUI -- Interactive interface suitable for machine
# parsing.
#
# You can override this with a command-line option -u.
#ui = basic
# If you try to synchronize messages to a folder which the IMAP server
# considers read-only, OfflineIMAP will generate a warning. If you want
# to suppress these warnings, set ignore-readonly to yes. Read-only
# IMAP folders allow reading but not modification, so if you try to
# change messages in the local copy of such a folder, the IMAP server
# will prevent OfflineIMAP from propagating those changes to the IMAP
# server. Note that ignore-readonly is unrelated to the "readonly"
# setting which prevents a repository from being modified at all.
#ignore-readonly = no
########## Advanced settings
# You can give a Python source filename here and all config file
# python snippets will be evaluated in the context of that file.
# This allows you to e.g. define helper functions in the Python
# source file and call them from this config file. You can find
# an example of this in the manual.
#
# pythonfile = ~/.offlineimap.py
pythonfile = ~/.offlineimap/offlineimap_config.py
# By default, OfflineIMAP will not exit due to a network error until
# the operating system returns an error code. Operating systems can sometimes
# take forever to notice this. Here you can activate a timeout on the
# socket. This timeout applies to individual socket reads and writes,
# not to an overall sync operation. You could perfectly well have a 30s
# timeout here and your sync still take minutes.
#
# Values in the 30-120 second range are reasonable.
#
# The default is to have no timeout beyond the OS. Times are given in seconds.
#
# socktimeout = 60
# By default, OfflineIMAP will use fsync() to force data out to disk at
# opportune times to ensure consistency. This can, however, reduce
# performance. Users where /home is on SSD (Flash) may also wish to reduce
# write cycles. Therefore, you can disable OfflineIMAP's use of fsync().
# Doing so will come at the expense of greater risk of message duplication
# in the event of a system crash or power loss. Default is fsync = true.
# Set fsync = false ot disable fsync.
#
# fsync = true
##################################################
# Mailbox name recorder
##################################################
[mbnames]
# offlineimap can record your mailbox names in a format you specify.
# You can define the header, each mailbox item, the separator,
# and the footer. Here is an example for Mutt.
# If enabled is yes, all six setting must be specified, even if they
# are just the empty string "".
#
# The header, peritem, sep, and footer are all Python expressions passed
# through eval, so you can (and must) use Python quoting.
enabled = no
filename = ~/Mutt/muttrc.mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"
# You can also specify a folderfilter. It will apply to the
# *translated* folder name here, and it takes TWO arguments:
# accountname and foldername. In all other ways, it will
# behave identically to the folderfilter for accounts. Please see
# that section for more information and examples.
#
# Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter.
##################################################
# Accounts
##################################################
# This is an account definition clause. You'll have one of these
# for each account listed in general/accounts above.
[Account BoostPro]
########## Basic settings
# These settings specify the two folders that you will be syncing.
# You'll need to have a "Repository ..." section for each one.
localrepository = BoostProLocal
remoterepository = BoostProRemote
########## Advanced settings
# You can have offlineimap continue running indefinitely, automatically
# syncing your mail periodically. If you want that, specify how
# frequently to do that (in minutes) here. You can also specify
# fractional minutes (ie, 3.25).
autorefresh = 5
# OfflineImap can replace a number of full updates by quick
# synchronizations. It only synchronizes a folder if 1) a Maildir
# folder has changed, or 2) if an IMAP folder has received new messages
# or had messages deleted, ie it does not update if only IMAP flags have
# changed. Full updates need to fetch ALL flags for all messages, so
# this makes quite a performance difference (especially if syncing
# between two IMAP servers).
# Specify 0 for never, -1 for always (works even in non-autorefresh
# mode), or a positive integer <n> to do <n> quick updates before doing
# another full synchronization (requires autorefresh). Updates are
# always performed after <autorefresh> minutes, be they quick or full.
quick = 24
# You can specify a pre and post sync hook to execute a external command.
# In this case a call to imapfilter to filter mail before the sync process
# starts and a custom shell script after the sync completes.
# The pre sync script has to complete before a sync to the account will
# start.
# presynchook = imapfilter
# postsynchook = notifysync.sh
postsynchook = /Users/dave/bin/imap-postsync localhost dave --soft-errors
# You can also specify parameters to the commands
# presynchook = imapfilter -c someotherconfig.lua
# OfflineImap caches the state of the synchronisation to e.g. be able to
# determine if a mail has been deleted on one side or added on the
# other.
#
# The default and historical backend is 'plain' which writes out the
# state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the
# complete file for each change. Another new backend 'sqlite' is
# available which stores the status in sqlite databases.
#
# If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually
# once you are sure that things work.
#
status_backend = sqlite
# If you have a limited amount of bandwidth available you can exclude larger
# messages (e.g. those with large attachments etc). If you do this it
# will appear to offlineimap that these messages do not exist at all. They
# will not be copied, have flags changed etc. For this to work on an IMAP
# server the server must have server side search enabled. This works with gmail
# and most imap servers (e.g. cyrus etc)
# The maximum size should be specified in bytes - e.g. 2000000 for approx 2MB
# maxsize = 2000000
# When you are starting to sync an already existing account you can tell
# offlineimap to sync messages from only the last x days. When you do
# this messages older than x days will be completely ignored. This can
# be useful for importing existing accounts when you do not want to
# download large amounts of archive email.
#
# Messages older than maxage days will not be synced, their flags will
# not be changed, they will not be deleted etc. For offlineimap it will
# be like these messages do not exist. This will perform an IMAP search
# in the case of IMAP or Gmail and therefor requires that the server
# support server side searching. This will calculate the earliest day
# that would be included in the search and include all messages from
# that day until today. e.g. maxage = 3 to sync only the last 3 days
# mail
#
# maxage =
maxage = 365
# Maildir file format uses colon (:) separator between uniq name and info.
# Unfortunatelly colon is not allowed character in windows file name. If you
# enable maildir-windows-compatible option, offlineimap will be able to store
# messages on windows drive, but you will probably loose compatibility with
# other programs working with the maildir
#
#maildir-windows-compatible = no
[Repository BoostProLocal]
# Each repository requires a "type" declaration. The types supported for
# local repositories are Maildir and IMAP.
type = IMAP
# Specify local repository. Your IMAP folders will be synchronized
# to maildirs created under this path. OfflineIMAP will create the
# maildirs for you as needed.
# localfolders = ~/Test
# You can specify the "folder separator character" used for your Maildir
# folders. It is inserted in-between the components of the tree. If you
# want your folders to be nested directories, set it to "/". 'sep' is
# ignored for IMAP repositories, as it is queried automatically.
#
#sep = .
# Some users may not want the atime (last access time) of folders to be
# modified by OfflineIMAP. If 'restoreatime' is set to yes, OfflineIMAP
# will restore the atime of the "new" and "cur" folders in each maildir
# folder to their original value after each sync.
#
# In nearly all cases, the default should be fine.
#
#restoreatime = no
# This setting will be forced to "yes" by the fact because we have
# enabled IDLE support
#
# holdconnectionopen = yes
preauthtunnel = /usr/local/libexec/dovecot/imap
[Repository BoostProRemote]
# And this is the remote repository. We only support IMAP or Gmail here.
type = Gmail
# The following can fetch the account credentials via a python expression that
# is parsed from the pythonfile parameter. For example, a function called
# "getcredentials" that parses a file "filename" and returns the account
# details for "hostname".
# remotehosteval = getcredentials("filename", "hostname", "hostname")
# remoteusereval = getcredentials("filename", "hostname", "user")
# remotepasseval = getcredentials("filename", "hostname", "passwd")
remotepasseval = osxkeychain('imap.gmail.com', account='dave@boostpro.com')
# Specify the remote hostname.
# remotehost = examplehost
# Whether or not to use SSL.
ssl = yes
# SSL Client certificate (optional)
# sslclientcert = /path/to/file.crt
# SSL Client key (optional)
# sslclientkey = /path/to/file.key
# SSL CA Cert(s) to verify the server cert against (optional).
# No SSL verification is done without this option. If it is
# specified, the CA Cert(s) need to verify the Server cert AND
# match the hostname (* wildcard allowed on the left hand side)
# The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt
# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways.
#cert_fingerprint = <SHA1_of_server_certificate_here>
cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1
# Specify the port. If not specified, use a default port.
# remoteport = 993
# Specify the remote user name.
remoteuser = dave@boostpro.com
# There are six ways to specify the password for the IMAP server:
#
# 1. No password at all specified in the config file.
# If a matching entry is found in ~/.netrc (see netrc (5) for
# information) this password will be used. Do note that netrc only
# allows one entry per hostname. If there is no ~/.netrc file but
# there is an /etc/netrc file, the password will instead be taken
# from there. Otherwise you will be prompted for the password when
# OfflineIMAP starts when using a UI that supports this.
#
# 2. The remote password stored in this file with the remotepass
# option. Any '%' needs to be encoded as '%%'. Example:
# remotepass = mypassword
#
# 3. The remote password stored as a single line in an external
# file, which is referenced by the remotefile option. Example:
# remotepassfile = ~/Password.IMAP.Account1
#
# 4. With a preauth tunnel. With this method, you invoke an external
# program that is guaranteed *NOT* to ask for a password, but rather
# to read from stdin and write to stdout an IMAP procotol stream that
# begins life in the PREAUTH state. When you use a tunnel, you do
# NOT specify a user or password (if you do, they'll be ignored.)
# Instead, you specify a preauthtunnel, as this example illustrates
# for Courier IMAP on Debian:
# preauthtunnel = ssh -q imaphost '/usr/bin/imapd ./Maildir'
#
# 5. If you are using Kerberos and have the Python Kerberos package
# installed, you should not specify a remotepass. If the user has a
# valid Kerberos TGT, OfflineIMAP will figure out the rest all by
# itself, and fall back to password authentication if needed.
#
# 6. Using arbitrary python code. With this method, you invoke a
# function from your pythonfile. To use this method assign the name
# of the function to the variable 'remotepasseval'. Example:
# remotepasseval = get_password("imap.example.net")
# You can also query for the username:
# remoteusereval = get_username("imap.example.net")
# This method can be used to design more elaborate setups, e.g. by
# querying the gnome-keyring via its python bindings.
########## Advanced settings
# Some IMAP servers need a "reference" which often refers to the "folder
# root". This is most commonly needed with UW IMAP, where you might
# need to specify the directory in which your mail is stored. The
# 'reference' value will be prefixed to all folder paths refering to
# that repository. E.g. accessing folder 'INBOX' with reference = Mail
# will try to access Mail/INBOX. Note that the nametrans and
# folderfilter functions will still apply the full path including the
# reference prefix. Most users will not need this.
#
# reference = Mail
# In between synchronisations, OfflineIMAP can monitor mailboxes for new
# messages using the IDLE command. If you want to enable this, specify here
# the folders you wish to monitor. Note that the IMAP protocol requires a
# separate connection for each folder monitored in this way, so setting
# this option will force settings for:
# maxconnections - to be at least the number of folders you give
# holdconnectionopen - to be true
# keepalive - to be 29 minutes unless you specify otherwise
#
# This feature isn't complete and may well have problems. See the manual
# for more details.
#
# This option should return a Python list. For example
#
# idlefolders = ['INBOX', 'INBOX.Alerts']
#
idlefolders = %(folderincludes)s
# OfflineIMAP can use multiple connections to the server in order
# to perform multiple synchronization actions simultaneously.
# This may place a higher burden on the server. In most cases,
# setting this value to 2 or 3 will speed up the sync, but in some
# cases, it may slow things down. The safe answer is 1. You should
# probably never set it to a value more than 5.
#maxconnections = 2
# OfflineIMAP normally closes IMAP server connections between refreshes if
# the global option autorefresh is specified. If you wish it to keep the
# connection open, set this to true. If not specified, the default is
# false. Keeping the connection open means a faster sync start the
# next time and may use fewer server resources on connection, but uses
# more server memory. This setting has no effect if autorefresh is not set.
#
#holdconnectionopen = no
# If you want to have "keepalives" sent while waiting between syncs,
# specify the amount of time IN SECONDS between keepalives here. Note that
# sometimes more than this amount of time might pass, so don't make it
# tight. This setting has no effect if autorefresh and holdconnectionopen
# are not both set.
#
# keepalive = 60
# Normally, OfflineIMAP will expunge deleted messages from the server.
# You can disable that if you wish. This means that OfflineIMAP will
# mark them deleted on the server, but not actually delete them.
# You must use some other IMAP client to delete them if you use this
# setting; otherwise, the messgaes will just pile up there forever.
# Therefore, this setting is definitely NOT recommended.
#
#expunge = no
# Specify whether to process all mail folders on the server, or only
# those listed as "subscribed".
#
#subscribedonly = no
# You can specify a folder translator. This must be a eval-able
# Python expression that takes a foldername arg and returns the new
# value. I suggest a lambda. This example below will remove "INBOX." from
# the leading edge of folders (great for Courier IMAP users)
#
# See the user documentation for details and use cases. They are also
# online at:
# http://docs.offlineimap.org/en/latest/nametrans.html
#
# WARNING: you MUST construct this such that it NEVER returns
# the same value for two folders, UNLESS the second values are
# filtered out by folderfilter below. Failure to follow this rule
# will result in undefined behavior
#
# nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)
# Using Courier remotely and want to duplicate its mailbox naming
# locally? Try this:
#
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
# You can specify which folders to sync using the folderfilter
# setting. You can provide any python function (e.g. a lambda function)
# which will be invoked for each foldername. If the filter function
# returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
#
# Example 1: synchronizing only INBOX and Sent.
#
# folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
#
# Example 2: synchronizing everything except Trash.
#
# folderfilter = lambda foldername: foldername not in ['Trash']
#
# Example 3: Using a regular expression to exclude Trash and all folders
# containing the characters "Del".
#
# folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
#
# If folderfilter is not specified, ALL remote folders will be
# synchronized.
#
# You can span multiple lines by indenting the others. (Use backslashes
# at the end when required by Python syntax) For instance:
#
# folderfilter = lambda foldername: foldername in
# ['INBOX', 'Sent Mail', 'Deleted Items',
# 'Received']
folderfilter = lambda foldername: False
# You can specify folderincludes to include additional folders. It
# should return a Python list. This might be used to include a folder
# that was excluded by your folderfilter rule, to include a folder that
# your server does not specify with its LIST option, or to include a
# folder that is outside your basic reference. The 'reference' value
# will not be prefixed to this folder name, even if you have specified
# one. For example:
# folderincludes = ['debian.user', 'debian.personal']
folderincludes = ['INBOX', '[Gmail]/All Mail']
# If you do not want to have any folders created on this repository,
# set the createfolders variable to False, the default is True. Using
# this feature you can e.g. disable the propagation of new folders to
# the new repository.
#createfolders = True
# You can specify 'foldersort' to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression
# should return -1, 0, or 1, as the default Python cmp() does. The two
# arguments, x and y, are strings representing the names of the folders
# to be sorted. The sorting is applied *AFTER* nametrans, if any. The
# default is to sort IMAP folders alphabetically
# (case-insensitive). Usually, you should never have to modify this. To
# eg. reverse the sort:
#
# foldersort = lambda x, y: -cmp(x, y)
# Enable 1-way synchronization. When setting 'readonly' to True, this
# repository will not be modified during synchronization. Use to
# e.g. backup an IMAP server. The readonly setting can be applied to any
# type of Repository (Maildir, Imap, etc).
#
#readonly = False
# [Repository GmailExample]
# A repository using Gmail's IMAP interface. Any configuration
# parameter of `IMAP` type repositories can be used here. Only
# `remoteuser` (or `remoteusereval` ) is mandatory. Default values
# for other parameters are OK, and you should not need fiddle with
# those.
#
# The Gmail repository will use hard-coded values for `remotehost`,
# `remoteport`, `tunnel` and `ssl`. (See
# http://mail.google.com/support/bin/answer.py?answer=78799&topic=12814)
# Any attempt to set those parameters will be silently ignored.
# type = Gmail
# Specify the Gmail user name. This is the only mandatory parameter.
# remoteuser = username@gmail.com
# The trash folder name may be different from [Gmail]/Trash
# for example on german googlemail, this setting should be
#
# trashfolder = [Google Mail]/Papierkorb
#
# The same is valid for the spam folder
#
# spamfolder = [Google Mail]/Spam
# Enable 1-way synchronization. See above for explanation.
#
#readonly = False