Misc Archives - apt get life

Handling Failure: What I’ve learned with a failed systems architecture change

2020/04/25 by sudo Leave a Comment

Docker. It’s everywhere. I use it at work and at home. It’s amazing for doing development in environments that more closely match production (such as running a full Laravel stack with queues, Redis, database, local mailmog for catching test email).

In order to learn more about docker, and improve my ability to roll out sites I host, I decided that I wanted to move a whole bunch of WordPress sites to docker. The existing setup is based on Ubuntu 16.04 LTS running on Nginx and PHP-FPM with per user resource pools for better security and site resource allocation. PHP 7.0 has been end of life for some time and it’s definitely time to update. Ubuntu LTS likely won’t track current PHP versions due to the way PHP’s release cycle has changed to be effectively 2 years. That means before Ubuntu 20.04 (released just days ago) is end of Life, PHP 7.4 will have been in end of life for over a year and won’t have had active support for over 2 years! Docker, I think, will allow me to better update the environments and keep them in line with the PHP release cycle. Hopefully it also makes them easier to migrate to new operating systems later.

I’ve already had experience with Traefik as a reverse proxy and it’s fantastic for handling Lets Encrypt SSL certificates with multiple sites out of the box. I can easily add docker containers with labels and they’ll appear automatically in Traefik; magic!

So here’s what I had in my head when I started:

VM running Ubuntu Server 20.04 LTS
A docker user that manages and has permissions over everything
A docker-compose file in that user’s home folder. This file runs the core config for Traefik and any other main containers I need (maybe fail2ban too).
Each site exists in a subdirectory named after it’s domain name. Within that there’s a docker-compose file related to that site and any files are stored there too.
Database instances either per site or provided by the host

So that gives you something that looks like:

/home/docker/docker-compose.yml # traefik

/home/docker/aptgetlife/docker-compose.yml # wordpress and MySQL

/home/docker/aptgetlife/public_html/ # site files

/home/docker/aptgetlife/mysql/ # database files

Problem: The docker PPA for ubuntu doesn’t exist for 20.04!

Ubuntu’s apt packages are often out of date, by default I jump straight into the docker doc website in order to get the latest possible version. Or not. There was no PPA available for 20.04 yet!

Fallback was to use apt. Since it’s a new release it was an up to date docker package. It may be worth changing to the PPA once it’s available.

Problem: Packet loss on ubuntu 20.4

While editing the master docker compose file, my ssh connection kept hanging and dropping. Following some pings I discovered that there was an intermittent network connection. It isn’t clear if this was caused by the docker networking packages, KVM drivers or Ubuntu 20.04 itself. It has only been out a day so there’s possible issues with the OS itself.

Fallback was to go back to Ubuntu 18.04 which didn’t have any issues! I’ll jump back to 20.04 after it’s bedded in a little and hopefully the issue will go away.

Problem: The traefik network can’t be seen by individual sites

This is a new one to me and I’ve never done this before so I didn’t have any experience of the setup. I have named my internal network traefik in my main docker-compose file. This works great. Traefik will create the network and the traefik container will connect to it fine. What didn’t work was the per-site docker-compose file connecting to the network. It wanted to create it’s own version named after the folder it was in. I discovered that as of docker-compose format version 3.5 (https://github.com/docker/compose/issues/3736) it allows you to use named containers.

networks:
    aptgetlife: # network for this container and associated resources like MySQL
    traefik: # link to Traefik for inbound traffic
        external:
            name: traefik

Problem: wordpress site URLS

Okay. This is one of my pet hates with WordPress. WordPress requires a site URL. This apparently was “www.” and I’d set it up on the new system without. It turned out easier to change the configuration to use “www.” instead of convincing WordPress to change the URL. This worked. Except no style sheets or Javascript would load. This, as it turns out, is due to WordPress loading insecure URLs for these assets. I attempted to use tools to update the SQL file and edited the wp-config.php file but neither would solve this problem.

sed -i 's|http://www.aptgetlife.co.uk|https://www.aptgetlife.co.uk|g' wp_aptgetlife.sql

This actually defeated me. I really don’t understand WordPress and how insistent it is to load resourced on particular URLs.

What have I learned?

Well, I’ve learned a lot about docker-compose, networking, override files. I know my architecture will work. I have also learned that I dislike WordPress. Alot. I’m sure the site asset problem is fixable, but I don’t have the patience to deal with it. I’m not interested in fixing WordPress related problems. Even though this project failed, and it is something I wanted to use for moving my hosted sites to, I have gained a lot of knowledge in the process. So instead of taking the failure “I have not deployed my sites using docker”, I am trying to look at the benefits of the knowledge I’ve gained and reflecting on the project as a learning experience. Hopefully a little bit of a retrospective will embed some key technical details in my brain for future DevOps! It’s also important to research what’s possible with systems architecture and I’d have looked into this before starting if it were a work project, but because it’s a personal project I didn’t feel the need. This was almost a playground, a trial run to see if it was even feasible. I think I learnt more and more quickly through this make, fail, make, fail, make, fail iterative approach. I had a definable success for each of the failures and a learning experience from each while overcoming them.

My big takeaway is that docker-compose has some great features in 3.5+ that I didn’t know existed. There’s some great information about networking (and in particular the section at the end about “external” or pre-existing docker networks) on their website https://docs.docker.com/compose/networking/

Backing up email using offlineimap

2019/03/06 by sudo

I’ve been asked to help backup a friends email from their existing provider. Since we’re not sure what’ll happen to the existing email after the request to change service providers has been made, it seems reasonable to backup her existing email automatically.

Since I run my own mail server, I thought it’d be quite easy to run something like offlineimap to backup the existing service. One caveat to that has been the existing server prefixes “INBOX” to the mail directory, which my setup doesn’t support. It looks like this is a difference between Courier and Dovecot. I also want to rename “Sent Items” to “.Sent” to follow my Dovecot setup.

Fortunately, offlineimap offers the ability to translate folders on the fly.

.offlineimaprc

[general]
accounts = alice

[Account alice]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/Mail/Alice

[Repository Remote]
type = IMAP
readonly = true
remotehost = mail.example.com
remoteuser = alice@example.com
remotepass = secret
# Necessary as of OfflineIMAP 6.5.4
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)
ssl_version = tls1_2
nametrans = lambda foldername: re.sub('^INBOX\.*', '.',
                               re.sub('^Sent Items\.*', '.Sent', foldername))

Once I’ve got this setup, I can use crontabs to automatically sync it every x minutes.

Note that after pulling the email, Dovecot may need the quota recalculated manually:

doveadm quota get -u alice@example.com
doveadm quota recalc -u alice@example.com

https://wiki2.dovecot.org/Tools/Doveadm/Quota

Here are some good resources:

https://github.com/OfflineIMAP/offlineimap/issues/417

[general]

metadata = $XDG_DATA_HOME/offlineimap
accounts = darnirmlist
maxsyncaccounts = 1
ui = basic
pythonfile = $XDG_DATA_HOME/offlineimap/passkey.py

socktimeout = 20
fsync = false

[DEFAULT]
localrepositoryhome = ~/.local/var/lib/mail

[Account darnirmlist]
localrepository = darnirmlist-local
remoterepository = darnirmlist-remote
status_backend = sqlite
autorefresh = 1
quick = 20
synclabels = yes
ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important

[Repository darnirmlist-local]
type = GmailMaildir
localfolders = %(localrepositoryhome)s/darnirmlist

[Repository darnirmlist-remote]
type = Gmail
remoteuser = darnirmlist@gmail.com

auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id('darnirmlist@gmail.com')
oauth2_client_secret_eval = get_client_secret('darnirmlist@gmail.com')
oauth2_refresh_token_eval = get_refresh_token('darnirmlist@gmail.com')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
nametrans = lambda folder: re.sub('.*Spam$', 'Spam',
                           re.sub('.*INBOX$', 'Inbox',
                           re.sub('.*Drafts$', 'Drafts',
                           re.sub('.*Sent Mail$', 'Sent',
                           re.sub('.*All Mail', 'Archive',
                           re.sub('.*Trash$', 'Trash', folder))))))
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail',
                                                     '[Gmail]/Drafts',
                                                     '[Gmail]/Sent Mail',
                                                     '[Gmail]/Spam']
createfolders = False
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
realdelete = no
holdconnectionopen = yes
keepalive = 50

http://www.offlineimap.org/doc/use_cases.html

https://gist.githubusercontent.com/dabrahams/3030332/raw/6e0cfab9982a97d0921d774d9f0d070dccb3b061/gistfile1.cfg

# -*- mode: conf; -*-
# 
# NOTE: Settings generally support python interpolation. This means
# values can contain python format strings which refer to other values
# in the same section, or values in a special DEFAULT section. This
# allows you for example to use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.

# NOTE2: This implies that any '%' needs to be encoded as '%%'

##################################################
# General definitions
##################################################

[general]

# This specifies where offlineimap is to store its metadata.
# This directory will be created if it does not already exist.

#metadata = ~/.offlineimap

# This variable specifies which accounts are defined.  Separate them
# with commas.  Account names should be alphanumeric only.
# You will need to specify one section per account below.  You may
# not use "general" for an account name.

accounts = BoostPro

# Offlineimap can synchronize more than one account at a time.  If you
# want to enable this feature, set the below value to something
# greater than 1.  To force it to synchronize only one account at a
# time, set it to 1.
#
# Note: if you are using autorefresh and have more than one account,
# you must set this number to be >= to the number of accounts you have;
# since any given sync run never "finishes" due to a timer, you will never
# sync your additional accounts if this is 1.

#maxsyncaccounts = 1

# You can specify one or more user interface modules for OfflineIMAP
# to use.  OfflineIMAP will try the first in the list, and if it
# fails, the second, and so forth.
#
# The pre-defined options are:
# Blinkenlights -- A fancy (terminal) interface
# TTYUI         -- a text-based (terminal) interface
# Basic         -- Noninteractive interface suitable for cron'ing
# Quiet         -- Noninteractive interface, generates no output
#                  except for errors.
# MachineUI     -- Interactive interface suitable for machine
#                  parsing.
#
# You can override this with a command-line option -u.

#ui = basic

# If you try to synchronize messages to a folder which the IMAP server
# considers read-only, OfflineIMAP will generate a warning.  If you want
# to suppress these warnings, set ignore-readonly to yes.  Read-only
# IMAP folders allow reading but not modification, so if you try to
# change messages in the local copy of such a folder, the IMAP server
# will prevent OfflineIMAP from propagating those changes to the IMAP
# server.  Note that ignore-readonly is unrelated to the "readonly"
# setting which prevents a repository from being modified at all.

#ignore-readonly = no

########## Advanced settings

# You can give a Python source filename here and all config file
# python snippets will be evaluated in the context of that file.
# This allows you to e.g. define helper functions in the Python
# source file and call them from this config file.  You can find
# an example of this in the manual.
#
# pythonfile = ~/.offlineimap.py
pythonfile = ~/.offlineimap/offlineimap_config.py

# By default, OfflineIMAP will not exit due to a network error until
# the operating system returns an error code.  Operating systems can sometimes
# take forever to notice this.  Here you can activate a timeout on the
# socket.  This timeout applies to individual socket reads and writes,
# not to an overall sync operation.  You could perfectly well have a 30s
# timeout here and your sync still take minutes.
#
# Values in the 30-120 second range are reasonable.
#
# The default is to have no timeout beyond the OS.  Times are given in seconds.
#
# socktimeout = 60

# By default, OfflineIMAP will use fsync() to force data out to disk at
# opportune times to ensure consistency.  This can, however, reduce
# performance.  Users where /home is on SSD (Flash) may also wish to reduce
# write cycles.  Therefore, you can disable OfflineIMAP's use of fsync().
# Doing so will come at the expense of greater risk of message duplication
# in the event of a system crash or power loss.  Default is fsync = true.
# Set fsync = false ot disable fsync.
#
# fsync = true

##################################################
# Mailbox name recorder
##################################################

[mbnames]

# offlineimap can record your mailbox names in a format you specify.
# You can define the header, each mailbox item, the separator,
# and the footer.  Here is an example for Mutt.
# If enabled is yes, all six setting must be specified, even if they
# are just the empty string "".
#
# The header, peritem, sep, and footer are all Python expressions passed
# through eval, so you can (and must) use Python quoting.

enabled = no
filename = ~/Mutt/muttrc.mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

# You can also specify a folderfilter.  It will apply to the
# *translated* folder name here, and it takes TWO arguments:
# accountname and foldername.  In all other ways, it will
# behave identically to the folderfilter for accounts.  Please see
# that section for more information and examples.
#
# Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter.


##################################################
# Accounts
##################################################

# This is an account definition clause.  You'll have one of these
# for each account listed in general/accounts above.

[Account BoostPro]
########## Basic settings

# These settings specify the two folders that you will be syncing.
# You'll need to have a "Repository ..." section for each one.

localrepository = BoostProLocal
remoterepository = BoostProRemote

########## Advanced settings

# You can have offlineimap continue running indefinitely, automatically
# syncing your mail periodically.  If you want that, specify how
# frequently to do that (in minutes) here.  You can also specify
# fractional minutes (ie, 3.25).

autorefresh = 5

# OfflineImap can replace a number of full updates by quick
# synchronizations.  It only synchronizes a folder if 1) a Maildir
# folder has changed, or 2) if an IMAP folder has received new messages
# or had messages deleted, ie it does not update if only IMAP flags have
# changed.  Full updates need to fetch ALL flags for all messages, so
# this makes quite a performance difference (especially if syncing
# between two IMAP servers).
# Specify 0 for never, -1 for always (works even in non-autorefresh
# mode), or a positive integer <n> to do <n> quick updates before doing
# another full synchronization (requires autorefresh).  Updates are
# always performed after <autorefresh> minutes, be they quick or full.

quick = 24

# You can specify a pre and post sync hook to execute a external command.
# In this case a call to imapfilter to filter mail before the sync process
# starts and a custom shell script after the sync completes.
# The pre sync script has to complete before a sync to the account will
# start. 

# presynchook = imapfilter
# postsynchook = notifysync.sh
postsynchook = /Users/dave/bin/imap-postsync localhost dave --soft-errors

# You can also specify parameters to the commands
# presynchook = imapfilter -c someotherconfig.lua

# OfflineImap caches the state of the synchronisation to e.g. be able to
# determine if a mail has been deleted on one side or added on the
# other.
#
# The default and historical backend is 'plain' which writes out the
# state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the
# complete file for each change.  Another new backend 'sqlite' is
# available which stores the status in sqlite databases.
#
# If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually
# once you are sure that things work.
#
status_backend = sqlite

# If you have a limited amount of bandwidth available you can exclude larger
# messages (e.g. those with large attachments etc).  If you do this it
# will appear to offlineimap that these messages do not exist at all.  They
# will not be copied, have flags changed etc.  For this to work on an IMAP
# server the server must have server side search enabled.  This works with gmail
# and most imap servers (e.g. cyrus etc)
# The maximum size should be specified in bytes - e.g. 2000000 for approx 2MB

# maxsize = 2000000


# When you are starting to sync an already existing account you can tell
# offlineimap to sync messages from only the last x days.  When you do
# this messages older than x days will be completely ignored.  This can
# be useful for importing existing accounts when you do not want to
# download large amounts of archive email.
#
# Messages older than maxage days will not be synced, their flags will
# not be changed, they will not be deleted etc.  For offlineimap it will
# be like these messages do not exist.  This will perform an IMAP search
# in the case of IMAP or Gmail and therefor requires that the server
# support server side searching.  This will calculate the earliest day
# that would be included in the search and include all messages from
# that day until today.  e.g. maxage = 3 to sync only the last 3 days
# mail
#
# maxage =
maxage = 365

# Maildir file format uses colon (:) separator between uniq name and info.
# Unfortunatelly colon is not allowed character in windows file name. If you
# enable maildir-windows-compatible option, offlineimap will be able to store
# messages on windows drive, but you will probably loose compatibility with
# other programs working with the maildir
#
#maildir-windows-compatible = no


[Repository BoostProLocal]

# Each repository requires a "type" declaration. The types supported for
# local repositories are Maildir and IMAP.

type = IMAP

# Specify local repository.  Your IMAP folders will be synchronized
# to maildirs created under this path.  OfflineIMAP will create the
# maildirs for you as needed.

# localfolders = ~/Test

# You can specify the "folder separator character" used for your Maildir
# folders.  It is inserted in-between the components of the tree. If you
# want your folders to be nested directories, set it to "/". 'sep' is
# ignored for IMAP repositories, as it is queried automatically.
#
#sep = .

# Some users may not want the atime (last access time) of folders to be
# modified by OfflineIMAP.  If 'restoreatime' is set to yes, OfflineIMAP
# will restore the atime of the "new" and "cur" folders in each maildir
# folder to their original value after each sync.
#
# In nearly all cases, the default should be fine.
#
#restoreatime = no

# This setting will be forced to "yes" by the fact because we have
# enabled IDLE support
#
# holdconnectionopen = yes
preauthtunnel = /usr/local/libexec/dovecot/imap

[Repository BoostProRemote]
# And this is the remote repository.  We only support IMAP or Gmail here.

type = Gmail

# The following can fetch the account credentials via a python expression that
# is parsed from the pythonfile parameter. For example, a function called
# "getcredentials" that parses a file "filename" and returns the account
# details for "hostname".
# remotehosteval = getcredentials("filename", "hostname", "hostname")
# remoteusereval = getcredentials("filename", "hostname", "user")
# remotepasseval = getcredentials("filename", "hostname", "passwd")
remotepasseval = osxkeychain('imap.gmail.com', account='dave@boostpro.com')

# Specify the remote hostname.
# remotehost = examplehost

# Whether or not to use SSL.
ssl = yes

# SSL Client certificate (optional)
# sslclientcert = /path/to/file.crt

# SSL Client key (optional)
# sslclientkey = /path/to/file.key

# SSL CA Cert(s) to verify the server cert against (optional).
# No SSL verification is done without this option. If it is
# specified, the CA Cert(s) need to verify the Server cert AND
# match the hostname (* wildcard allowed on the left hand side)
# The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt

# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways.

#cert_fingerprint = <SHA1_of_server_certificate_here>
cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1

# Specify the port.  If not specified, use a default port.
# remoteport = 993

# Specify the remote user name.
remoteuser = dave@boostpro.com

# There are six ways to specify the password for the IMAP server:
#
# 1. No password at all specified in the config file.
#    If a matching entry is found in ~/.netrc (see netrc (5) for
#    information) this password will be used. Do note that netrc only
#    allows one entry per hostname. If there is no ~/.netrc file but
#    there is an /etc/netrc file, the password will instead be taken
#    from there. Otherwise you will be prompted for the password when
#    OfflineIMAP starts when using a UI that supports this.
#
# 2. The remote password stored in this file with the remotepass
#    option. Any '%' needs to be encoded as '%%'. Example:
#    remotepass = mypassword
#
# 3. The remote password stored as a single line in an external
#    file, which is referenced by the remotefile option.  Example:
#    remotepassfile = ~/Password.IMAP.Account1
#
# 4. With a preauth tunnel.  With this method, you invoke an external
#    program that is guaranteed *NOT* to ask for a password, but rather
#    to read from stdin and write to stdout an IMAP procotol stream that
#    begins life in the PREAUTH state.  When you use a tunnel, you do
#    NOT specify a user or password (if you do, they'll be ignored.)
#    Instead, you specify a preauthtunnel, as this example illustrates
#    for Courier IMAP on Debian:
#    preauthtunnel = ssh -q imaphost '/usr/bin/imapd ./Maildir'
#
# 5. If you are using Kerberos and have the Python Kerberos package
#    installed, you should not specify a remotepass.  If the user has a
#    valid Kerberos TGT, OfflineIMAP will figure out the rest all by
#    itself, and fall back to password authentication if needed.
#
# 6. Using arbitrary python code.  With this method, you invoke a
#    function from your pythonfile.  To use this method assign the name
#    of the function to the variable 'remotepasseval'.  Example:
#    remotepasseval = get_password("imap.example.net")
#    You can also query for the username:
#    remoteusereval = get_username("imap.example.net")
#    This method can be used to design more elaborate setups, e.g. by
#    querying the gnome-keyring via its python bindings.

########## Advanced settings

# Some IMAP servers need a "reference" which often refers to the "folder
# root".  This is most commonly needed with UW IMAP, where you might
# need to specify the directory in which your mail is stored. The
# 'reference' value will be prefixed to all folder paths refering to
# that repository. E.g. accessing folder 'INBOX' with reference = Mail
# will try to access Mail/INBOX. Note that the nametrans and
# folderfilter functions will still apply the full path including the
# reference prefix.  Most users will not need this.
#
# reference = Mail

# In between synchronisations, OfflineIMAP can monitor mailboxes for new
# messages using the IDLE command. If you want to enable this, specify here
# the folders you wish to monitor. Note that the IMAP protocol requires a
# separate connection for each folder monitored in this way, so setting
# this option will force settings for:
#     maxconnections - to be at least the number of folders you give
#     holdconnectionopen - to be true
#     keepalive - to be 29 minutes unless you specify otherwise
#
# This feature isn't complete and may well have problems. See the manual
# for more details.
#
# This option should return a Python list. For example
#
# idlefolders = ['INBOX', 'INBOX.Alerts']
#
idlefolders = %(folderincludes)s

# OfflineIMAP can use multiple connections to the server in order
# to perform multiple synchronization actions simultaneously.
# This may place a higher burden on the server.  In most cases,
# setting this value to 2 or 3 will speed up the sync, but in some
# cases, it may slow things down.  The safe answer is 1.  You should
# probably never set it to a value more than 5.

#maxconnections = 2

# OfflineIMAP normally closes IMAP server connections between refreshes if
# the global option autorefresh is specified.  If you wish it to keep the
# connection open, set this to true.  If not specified, the default is
# false.  Keeping the connection open means a faster sync start the
# next time and may use fewer server resources on connection, but uses
# more server memory.  This setting has no effect if autorefresh is not set.
#
#holdconnectionopen = no

# If you want to have "keepalives" sent while waiting between syncs,
# specify the amount of time IN SECONDS between keepalives here.  Note that
# sometimes more than this amount of time might pass, so don't make it
# tight.  This setting has no effect if autorefresh and holdconnectionopen
# are not both set.
#
# keepalive = 60

# Normally, OfflineIMAP will expunge deleted messages from the server.
# You can disable that if you wish.  This means that OfflineIMAP will
# mark them deleted on the server, but not actually delete them.
# You must use some other IMAP client to delete them if you use this
# setting; otherwise, the messgaes will just pile up there forever.
# Therefore, this setting is definitely NOT recommended.
#
#expunge = no

# Specify whether to process all mail folders on the server, or only
# those listed as "subscribed".
#
#subscribedonly = no

# You can specify a folder translator.  This must be a eval-able
# Python expression that takes a foldername arg and returns the new
# value.  I suggest a lambda.  This example below will remove "INBOX." from
# the leading edge of folders (great for Courier IMAP users)
#
# See the user documentation for details and use cases. They are also
# online at:
# http://docs.offlineimap.org/en/latest/nametrans.html
#
# WARNING: you MUST construct this such that it NEVER returns
# the same value for two folders, UNLESS the second values are
# filtered out by folderfilter below.  Failure to follow this rule
# will result in undefined behavior
#
# nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

# Using Courier remotely and want to duplicate its mailbox naming
# locally?  Try this:
#
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)

# You can specify which folders to sync using the folderfilter
# setting. You can provide any python function (e.g. a lambda function)
# which will be invoked for each foldername. If the filter function
# returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
#
# Example 1: synchronizing only INBOX and Sent.
#
# folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
#
# Example 2: synchronizing everything except Trash.
#
# folderfilter = lambda foldername: foldername not in ['Trash']
#
# Example 3: Using a regular expression to exclude Trash and all folders
# containing the characters "Del".
#
# folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
#
# If folderfilter is not specified, ALL remote folders will be
# synchronized.
#
# You can span multiple lines by indenting the others.  (Use backslashes
# at the end when required by Python syntax)  For instance:
#
# folderfilter = lambda foldername: foldername in
#        ['INBOX', 'Sent Mail', 'Deleted Items',
#         'Received']
folderfilter = lambda foldername: False

# You can specify folderincludes to include additional folders.  It
# should return a Python list.  This might be used to include a folder
# that was excluded by your folderfilter rule, to include a folder that
# your server does not specify with its LIST option, or to include a
# folder that is outside your basic reference. The 'reference' value
# will not be prefixed to this folder name, even if you have specified
# one.  For example:
# folderincludes = ['debian.user', 'debian.personal']
folderincludes = ['INBOX', '[Gmail]/All Mail']

# If you do not want to have any folders created on this repository,
# set the createfolders variable to False, the default is True. Using
# this feature you can e.g. disable the propagation of new folders to
# the new repository.
#createfolders = True


# You can specify 'foldersort' to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression
# should return -1, 0, or 1, as the default Python cmp() does.  The two
# arguments, x and y, are strings representing the names of the folders
# to be sorted.  The sorting is applied *AFTER* nametrans, if any.  The
# default is to sort IMAP folders alphabetically
# (case-insensitive). Usually, you should never have to modify this.  To
# eg. reverse the sort:
#
# foldersort = lambda x, y: -cmp(x, y)

# Enable 1-way synchronization. When setting 'readonly' to True, this
# repository will not be modified during synchronization. Use to
# e.g. backup an IMAP server. The readonly setting can be applied to any
# type of Repository (Maildir, Imap, etc).
#
#readonly = False

# [Repository GmailExample]

# A repository using Gmail's IMAP interface.  Any configuration
# parameter of `IMAP` type repositories can be used here.  Only
# `remoteuser` (or `remoteusereval` ) is mandatory.  Default values
# for other parameters are OK, and you should not need fiddle with
# those.
#
# The Gmail repository will use hard-coded values for `remotehost`,
# `remoteport`, `tunnel` and `ssl`.  (See
# http://mail.google.com/support/bin/answer.py?answer=78799&topic=12814)
# Any attempt to set those parameters will be silently ignored.

# type = Gmail

# Specify the Gmail user name. This is the only mandatory parameter.
# remoteuser = username@gmail.com

# The trash folder name may be different from [Gmail]/Trash
# for example on german googlemail, this setting should be
#
# trashfolder = [Google Mail]/Papierkorb
#
# The same is valid for the spam folder
#
# spamfolder = [Google Mail]/Spam

# Enable 1-way synchronization. See above for explanation.
#
#readonly = False

Fixing “Cannot remount block device is write protected” on Ubuntu

2017/02/26 by sudo

While working on an old hard drive, I came across a problem that it would only mount as read only, even when specifying that it should mount as read write. Here’s an example:

mount /dev/sdb1 /media/aptgetlife/datadisk -o remount,rw
mount: cannot remount block device /dev/sdb1 read-write, is write-protected

By this point I’d already got the data disk mounted to the local user media folder, so I could read data from /media/aptgetlife/datadisk but not write data to it.

After some investigation I was able to found this post on stack overflow regarding a similar problem and a solution using blockdev:

blockdev --setrw /dev/sdb1

After which it was possible to run the mount command again, which didn’t present an error and allowed me to write data to the disk.

Linux servers – using ClamAV to find malware

2016/04/11 by sudo

ClamAV is an open source anti-virus program that can be run from the command line, making it incredibly useful for locating any viruses and malware on Linux based servers. Recently someone I’ve previously worked with reported that they’d had reports of abuse originating form one of their servers. Given the quantity of sites, it was difficult to locate any potential vulnerabilities.

grep -RPl --include=*.{php,txt} "(passthru|shell_exec|system|phpinfo|base64_decode|chmod|mkdir|fopen|fclose|readfile) *\(" /var/www/

Blindly grepping for potentially malicious strings such as “base64_decode” and “exec” was getting tired fast, as these can be legitimately used for some applications. I stumbled across reports that ClamAV works well for locating potential threats

nice -n 19 clamscan ./ -r -i | grep " FOUND" >> possible_exploits.txt

You can then review these files as you see fit, editing the file to remove ones that are false positives. I then run a command to delete the infected files:

while read f; do rm $f ; done<$possible_exploits.txt

HP Microserver Mods

2016/04/03 by sudo

HP Microservers are a low cost, small form factor cube-sh PC that are great for using as media centres or NAS boxes. I’ve already got a few microservers running FreeNas or Ubuntu server providing shared network drives or running as mass-backup boxes. Since this microserver was broken and relatively low cost, my first action was to investigate what’d gone wrong with it.

Amber power light: Failed PSU

When connected to the mains, the server’s power button was constant amber. It didn’t respond to button presses and wouldn’t power up. I did some research and it suggested that the most likely reason for the amber light was a stuck power button. I pulled the power switch connector from the mother board and used a multimetre on continuity test mode to check each pin pair until I found one that connected when the button was pressed. This indicated that the power switch was working fine, as it would complete a circuit when held down, and disconnect when released. I shorted the matching pins on the mother board, effectively becoming the power switch and the server still didn’t boot, which confirmed that this was not the problem. Delving deeper, I started dismantling the microserver to see if I could work out what had gone wrong – sure enough a quick visual inspection found the CD drive SATA power connector was a melted mess, and scorch marks were on the metal frame. The CD drive was dead, pins on the molex to SATA adapter having shorted inside the plastic housing. I pulled the PSU and found it to be dead, it wouldn’t turn on when shorting the pins on the ATX header so a new power supply was in order.

Since HP microserver power supplies are quite expensive (~70 GBP), I wanted to make sure the motherboard still worked before paying for a replacement PSU. I used an old ATX power supply which I had floating around and connected it to the microserver. It’d never fit inside the case, so I just left it hanging around next to the microserver. It booted off a memory stick fine, and that was proof enough for me that I needed a new power supply.

I ended up finding this blog: http://chrisstark.co/2013/08/hp-microserver-n36-replacement-power-supply-psu/ which was someone in a similar position to me. They had a microserver which needed a new power supply, but found that LinITX had a similar replacement unit which was significantly cheaper than a genuine spare. LinITX is a fantastic site dedicated to embedded hardware and I’ve often browsed wishing I had enough cash to buy goodies from them. In this case, since the PSU was only £30 (GBP) I bought one straight away. This is a link to the PSU https://linitx.com/product/fsp-150w-1u-compact-power-supply-fsp15050gub/13062 and the quickfind code is FSP150-50GUB.

All back-plane cables are grouped by colour into a chocbloc, then a female molex cable attached to the other side.

With the PSU connected, the only problem I now had was the SATA backplane requires 4 molex power connectors and the PSU only has one. Since this is likely going to be a “toy” microserver due to the cost and not-properly-tested nature of it, instead of investing in molex splitters, I decided to hack off the existnig backplane molex cables, group the connectors into colours, strip the ends and solder them together. Once this was done I used standard electrical choc blocks to connect one end into a standard molex female that I’d cut off previously. It looks messy but is reasonably electrically sound.

Case Mods

Now that’s done, I decided to have some fun. I’ve had an air brush for years and never properly gotten into it – mastering a skill takes time which I’ve never seemed to have. Plus it’s messy so you’ve got to have dedicated time to flush through cleaner when you’re done, and set up a dirty work area. While tempted to companion cube my micro server I decided in the end to just do black and yellow hazard lines. First off, I opened the door and lifted it slightly to release it from the hinge. I was then able to use a pair of pliers to remove the locking mechanism’s nut on the inside of the door and put this to one side for safety. Once the lock is off (this can take some slight wiggling of the lock cylendar) I was able to unclip the plastic from the back metal frame. I only wanted to paint the plastic, and was worried there would be ghosting effects if I left the metal frame inside the door. I masked off the area using masking tape, following the lines of the air holes on the door. I mixed by yellow paint 50/50 with water and tried to evenly coat the door. That didn’t go so well. After 2 coats I gave up and instead removed the masking tape. I then got out some aluminium paint and dry-brushed the edges of the door to give it a weathered look. My method for dry-brushing, since I’d never done it before, was to get a little bit of paint onto a dry paint brush, dab it on my cardboard floor covering to thin it out, and then lightly brush the edges of the door. It actually came out really well – yes I am surprised by that! Finally I reassembled the server with the new door, power supply and hacked molex connector. I still have plans for what to do with the CD drive bay – possibly an LCD screen indicating IP address and drive status, I might replace the lock with a handle of some kind and I may paint the rest of the server. I also need to decide what to install on it!

Almost there – Microserver with new PSU, hacked back-plane power connector and painted front door.