akeil.de (Posts about linux)

InfluxDB for TinyCore Linux

Alexander Keil — Sat, 12 Aug 2017 22:00:00 GMT

InfluxDB for TinyCore Linux (ARM)

author: akeil
date: 2017-08-13
version: 1

This post shows how to set up InfluxDB 1 on a RaspberryPi running TinyCore Linux 2.

This is intended as part of a setup for home automation with where InfluxDB is used as a storage backend for statistics.

Tiny Core Extension

Before we can install InfluxDB, we need to create a TinyCore Extension 3 for it.

Download and extract the prebuilt binary from the InfluxDB Downloads 4 site:

$ wget https://dl.influxdata.com/influxdb/releases/influxdb-1.3.2_linux_armhf.tar.gz
$ tar -xzf influxdb-1.3.2_linux_armhf.tar.gz

Note

The download URL is different for each version.

The extracted archive creates a directory tree like this:

etc/
    influxdb/           # default config
    logrotate.d/
usr/
    bin/                # binaries
    lib/
    share/
var/
    lib/
        influxdb/
    log/
        influxdb/

Only a minimal selection of files goes into the TinyCore extension. Create a temporary directory (/tmp/influxdb/usr/local) and place selected files there. The complete destination directory looks like this:

/tmp/influxdb/
    usr/
        local/
            bin/            # binaries
            etc/            # sample config
            tce.installed/  # post-install script

Binaries

From the source usr/ directory, only the binaries are used. This omits the man pages (under share/) and start scripts (under lib/).

Configuration File

The InfluxDB distribution contains a default configuration file which is the starting point for a custom configuration.

Reduce cache sizes to play nice with the available memory (see InfluxDB docs ):

[data]
cache-max-memory-size = 20971520
cache-snapshot-memory-size = 10485760

Disable usage statistics:

reporting-disabled = true

Place the configuration file in the temporary etc directory, naming it influxdb.sample.conf.

Post-Install Script

TinyCore supports a tce.installed script which is run after the extension was loaded. Our post-install script will

if no config file is present, copy the sample config to the default location for the config file.
create the influxdb user.
create the default data directories with correct permissions.

Create a tce.installed script like this:

#!/bin/sh

CFG=/usr/local/etc/influxdb/influxdb.conf
SAMPLE_CFG=/usr/local/etc/influxdb/influxdb.sample.conf
DATADIR=/var/lib/influxdb
USER=influxdb

# if no config exists, use the sample config
if [[ ! -e $CFG ]]; then
    cp $SAMPLE_CFG $CFG
fi

# create the influxdb user
grep -q $USER /etc/passwd
if [[ $? -ne 0 ]]; then
    adduser -S -H $USER
fi

# create the default data directory
if [[ ! -e $DATADIR ]]; then
    mkdir $DATADIR
    chown $USER:nogroup $DATADIR
fi

The script is added to the temporary installation dir under tce.installed/influxdb.

Package Script

To make the process easily repeatable for new versions of InfluxDB, create a package.sh script. The script includes all steps except downloading the distribution files for InfluxDB.

#!/bin/sh
SRC=influxdb-1.3.2-1
ROOT=/tmp/influxdb/usr/local

# download and install required tools
tce-load -w squashfs-tools
tce-load -i squashfs-tools

# cleanup
sudo rm -rf $ROOT

mkdir -p $ROOT/bin
cp -a $SRC/usr/bin/* $ROOT/bin

mkdir -p $ROOT/etc/influxdb
cp influxdb.conf $ROOT/etc/influxdb/influxdb.sample.conf

# startup script
mkdir -p $ROOT/tce.installed
cp tce.installed $ROOT/tce.installed/influxdb


# default permissions
sudo chown -R root:root $ROOT

# special perms for startup script
sudo chown root:staff $ROOT/tce.installed
sudo chown 775 $ROOT/tce.installed

sudo chown root:staff $ROOT/tce.installed/influxdb
sudo chmod 755 $ROOT/tce.installed/influxdb

# pack and cleanup
mksquashfs /tmp/influxdb /tmp/influxdb.tcz
sudo rm -rf /tmp/influxdb

mv /tmp/influxdb.tcz /etc/sysconfig/tcedir/optional/

Configuration

Persistence

We need to persist two locations, using opt/.filetool.lst:

usr/local/etc/influxdb/influxdb.conf
var/lib/influxdb

This will include the configuration file and the data directories in the filetool.sh backup (you still need to remember to execute this before shutdown). That backup is restored during boot.

Storage

The default storage location is /var/lib/influxdb. on TinyCore, this resides in memory and can be persisted using filetool.sh. Since the data files can become quite large, this is not the best option. Both backup and restore will take a long time and the data will consume available memory.

Whether or not to place any part of the database on volatile storage of course depends on whether or not (partial) data loss is a problem. In this case, it is assumed that this is non-critical data and that it is acceptable if some of the data is lost if the system shuts down unexpectedly.

Note

Deciding whether to store the InfluxDB on "disk" (SD-Card) or in memory (with backup/restore on boot) depends on the InfluxDB Storage Engine 5. there are two important types of storage:

WAL - Write Ahead Log: This is stored under /var/lib/influxdb/wal/<database-name> (the wal-dir= directive from config). New writes go into the WAL, i.e. these files receive many writes and change often.
TSM - Time Structured Merge Tree: These are read-only files which contain the bulk of the data. They are stored under /var/lib/influxdb/data/<database-name> (the dir= directive from config). During a process named Compaction, data from the WAL is periodically written to TSM files. Also during compaction, existing TSM files are optimized, i.e. rewritten.

The TSM files will probably become too large to keep in memory and are therefore placed on a separate storage (in this case, additional storage that is mounted under (/mnt/storage).

The WAL is kept at the default location, in memory. This has the benefit of faster access times and also reduces writes to the RaspberryPi's SD-card. If the WAL is lost, we lose data from the most recent writes.

[data]
...
dir= "/mnt/storage/influxdb/data"
wal-dir = "/var/lib/influxdb/wal"
...

Start and Stop Scripts

Create two shell scripts to start and stop InfluxDB.

The start script is included in /opt/bootlocal.sh, the stop script in /opt/shutdown.sh.

The start script does the following:

set permissions on pid file
set permissions on log file
start influxd with parameters for pid file and configuration file and redirect stderr to the logfile

The stop script simply reads the process id from the pid file and kills that process.

Start

CONFIG=/usr/local/etc/influxdb/influxdb.conf
PIDFILE=/var/run/influxd.pid
LOGFILE=/var/log/influxdb.log

touch $PIDFILE
chown influxdb:nogroup $PIDFILE

touch $LOGFILE
chown influxdb:staff $LOGFILE

su influxdb -s /bin/sh -c "influxd run -pidfile $PIDFILE -config $CONFIG 2>$LOGFILE"

Stop

pid=$(cat /var/run/influxd.pid)
if [ -n "$pid" ]; then
    kill "$pid"
fi

Backup

There a several options for backing up InfluxDB. On can backup the full database(s) or only selected retention policies. It is also possible to only back up data since a specific timestamp.

The backup can be run locally or over the network and it can be performed while InfluxDB is running.

For the RaspberryPi, we create full backups, using a local cron job. Backups are created daily and a number of the most recent backups is retained.

The basic steps are as follows: - backup the Metastore and selected Databases to a temporary location - if all went well, compress the result (.tar.gz) - delete the oldest backup

To backup the Metastore:

influxd backup /path/to/backup

Each Database is backed up individually by calling

influxd backup -database <NAME> /path/to/backup

See InfluxDB backup and restore 6 for details.

Monitoring with Monit

Use the following checks with monit 7:

Check whether the process exists using the pidfile
Make sure it is run by the influxdb user
Call the HTTP interface on the /ping endpoint
Check if there is a recent backup

These are the monit checks:

check process influxd pidfile /var/run/influxd.pid
    group influxdb
    if failed uid influxdb then alert
    if failed
        port 8086
        protocol http
        request /ping
        status = 204
    then alert

check file influxdbbackup path /mnt/storage/influxdb/backups/snapshot.0.tar.gz
    group backup
    if does not exist then alert
    if timestamp > 25 hours then alert

1: https://www.influxdata.com/time-series-platform/influxdb/
2: http://tinycorelinux.net/
3: http://wiki.tinycorelinux.net/wiki:creating_extensions
4: https://portal.influxdata.com/downloads#influxdb
5: https://docs.influxdata.com/influxdb/v1.3/concepts/storage_engine/
6: https://docs.influxdata.com/influxdb/v1.3/administration/backup_and_restore/
7: https://mmonit.com/monit/documentation/monit.html

MQTT Bridge with Mosquitto and nginx

Alexander Keil — Sun, 05 Mar 2017 10:50:00 GMT

MQTT Bridge with Mosquitto and nginx

author: akeil
date: 2017-03-05
version: 1

Set up a Mosquitto 1 MQTT broker which is available in the internet. nginx 2 is used as a reverse proxy and to handle SSL encryption. An additional Mosquitto instance on the local network is used as a bridge to forward MQTT messages from the local network to the internet and vice versa.

This allows to easily connect devices which do not support authentication or encryption on the local network with with other devices or services connected via the internet.

Overview for an MQTT bridge scenario. Clients on the local network use an insecure connection to talk to the local MQTT broker while clients on the internet use a secure connection.

Mosquitto (public)

The public Mosquitto instance needs some configuration changes. The configuration file is located at /etc/mosquitto/mosquitto.conf.

Since we will be running behind nginx, make the MQTT service listen on localhost only:

bind_address localhost

Change MQTT Service Port (optional)

Change the default port number from 1883 to something else:

listener 9883

This is strictly not necessary as the default is normally used for unencrypted connections and we will configure nginx to listen on port 8883 (the default port for MQTT over SSL).

Finally, set up mosquitto to require authentication with username and password:

allow_anonymous false
password_file = /etc/mosquitto/passwords

Create the password file and the first user with:

# mosquitto_passwd -c /etc/mosquitto/passwords username

(You will be prompted for a password.)

To test authentication, attempt to subscribe to any topic. A valid username and password should work, a subscription attempt with invalid credentials should be rejected.

$ mosquitto_sub -p 9883 -t '#' -u username -P secret

nginx

Configure nginx as a reverse proxy for TCP streams 3 and to terminate the SSL encryption 4.

First, define Mosquitto as an upstream service. Use either 1883 or whatever alternative port is set for the default (unencrypted) MQTT service.

Next, set up a server to listen on port 8883 (MQTT over SSL) and tell it to pass requests to the upstream service.

To enable SSL, configure the location of the SSL certificate and private key. Set additional SSL options as needed; in this case taken from cipherli.st 5.

Note

The example assumes a Let's Encrypt 6 certificate.

The nginx configuration looks like this (with stream being a toplevel-directive):

stream {

    upstream mosquitto {
        server localhost:9883;
    }

    server {
        listen 8883 ssl;
        proxy_pass mosquitto;

        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

        # from https://cipherli.st/
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
        ssl_ecdh_curve secp384r1;
        ssl_session_cache shared:SSL:10m;
        ssl_session_tickets off;

        ssl_dhparam /etc/ssl/certs/dhparam.pem;
    }

}

MQTT-Bridge

On the local (LAN) MQTT broker, edit the configuration to add a bridge:

connection bridge
address <ip-or-hostname>:8883

remote_username foo
remote_password xxx

remote_clientid LAN-broker
local_client public-broker

# must specify cafile to enable SSL
bridge_cafile /etc/ssl/certs/DST_Root_CA_X3.pem
bridge_insecure false

# topic mappings
# topic PATTERN [out|in|both] QOS LOCAL-PREFIX REMOTE-PREFIX
topic foo both 0

Remember to specify a port with the bridge connection, otherwise 1883 is used. The bridge_cafile setting is required to enable SSL. See Mosquitto conf 7 for more details.

Restart mosquitto to apply the new settings.

The Mosquitto log file for the local broker should contain a line like this:

Connecting bridge bridge (<ip-or-hostname>:8883)

On the remote side, the bridge looks like a normal client connection:

New client connected from ::1 as <remote_clientid> (c0, k60, u'<remote_username>').

To test, subscribe to foo on the local broker...

$ mosquitto_sub -h <localbroker> -t foo

... and publish to foo on the remote broker:

$ mosquitto_pub -h <remotebroker> -p 8883 -u username -P secret -t foo -m test

You should see the "test" message appear on the session subscribed to the local broker.

Topic Mappings

A topic mapping consists of a pattern to subscribe to, the direction of the messages (in, out or both and a QoS setting. Additionally, a local prefix and remote prefix can be defined.

Only the pattern is required. Direction defaults to out, QoS defaults to 0.

Examples:

# forward all 'foo' messages to the bridge
topic foo
topic foo out 0  # same as above

# two-way forwarding for everything under "foo"
topic foo/# both

# (only) receive all foo messages
topic foo/# in

This can be used to forward only selected topics and to insert topics from the bridged broker into the local topic tree.

If no topic directive is present, no messages are forwarded across the bridge.

1: https://mosquitto.org/
2: https://nginx.com/
3: https://www.nginx.com/resources/admin-guide/tcp-load-balancing/
4: https://www.nginx.com/resources/admin-guide/nginx-tcp-ssl-termination/
5: https://cipherli.st/
6: https://letsencrypt.org/
7: https://mosquitto.org/man/mosquitto-conf-5.html

Mosquitto MQTT on TinyCore Linux

Alexander Keil — Sat, 14 Jan 2017 07:37:47 GMT

Mosquitto MQTT on TinyCore Linux

author: akeil
date: 2017-01-14
version: 1

This post describes how to install the Mosquitto 1 MQTT 2 broker on a Raspberry Pi with TinyCore 3 Linux.

Build an Extension

There is no prebuilt extension for Mosquitto which means that have to build the program from source and create a custom extension for TinyCore Linux.

First, download the source and unpack:

$ wget http://mosquitto.org/files/source/mosquitto-1.4.10.tar.gz
$ tar -xzf mosquitto-1.4.10.tar.gz

Next, install build dependencies.

$ tce-load -w make
$ tce-load -w gcc
$ tce-load -w compiletc
$ tce-load -w openssl-dev
$ tce-load -w squashfs-tools

$ tce-load -i make
$ tce-load -i gcc
$ tce-load -i compiletc
$ tce-load -i openssl-dev
$ tce-load -i squashfs-tools

Edit Makefiles

There are several Makefiles in the Mosquitto project directory. Some of these contain the command install -s --strip-program foo but the --strip-program option is not supported by busybox install. Remove that option (not the call) in all affected Makefiles:

./Makefile
./lib/Makefile
./lib/cpp/Makefile
./client/Makefile
./src/Makefile

Build and Install

According to the Mosquitto README, optional dependencies are c-ares and libuuid, both of which are not available. So we will call make WITH_SRV=no WITH_UUID=no to build.

Note

The README says to use WITH_DNS_SRV=no to build without ares. We need to do this if we do not want to build ares ourselves. This is not quite correct, it is WITH_SRV=no.

$ cd mosquitto-1.4.10/
$ make
$ make WITH_SRV=no WITH_UUID=no

Install into /tmp:

$ sudo make install WITH_DOCS=no DESTDIR=/tmp/mosquitto

Installation is done as root so we have the correct permissions on all installed files.

TinyCore Linux recommends to not include the docs in the extension (optionally package them in a separate -docs extension). The WITH_DOCS=no option does exactly this.

The TinyCore Linux guide recommends to strip the created binaries. This has already been done in the Makefiles with install -s.

After installation, all relevant files are located in /tmp/mosquitto. Execute /tmp/mosquitto/usr/local/sbin/mosquitto to make a test-run.

Create the Extension

To pack the extension:

$ mksquashfs /tmp/mosquitto/ /tmp/mosquitto.tcz
$ rm -rf /tmp/mosquitto
$ mv /tmp/mosquitto.tcz /etc/sysconfig/tcedir/optional/

Then to install

$ tce-load -i mosquitto

To load on boot:

$ echo mosquitto.tcz >> /etc/sysconfig/tcedir/onboot.lst

Create User

To run Mosquitto with a dedicated user, create that user:

$ sudo adduser -S mosquitto

And edit configuration accordingly.

Configuration

The config file goes into /usr/local/etc/mosquitto.conf. This is not the default location, so we need to use the --config-file option when starting Mosquitto to tell it where to look for configuration.

Of course, the config file should be included in the list of files to be backed up and restored on boot. Include it in /opt/.filetool.lst:

usr/local/etc/mosquitto.conf

Logging

TinyCore Linux does not come with a syslog daemon. If logging is desired, write to a log file:

log_dest file /var/log/mosquitto.log

On boot, make sure that the file is writable for the mosquitto user.

Persistence

Mosquitto can write connection, subscription and message data to disk and reload it on every restart. To enable this in combination with TinyCore Linux, configure Mosquitto with persistence and include the mosquitto.db in backup/restore.

Configuration:

persistence true
persistence_location /var/lib/mosquitto/
persistence_file mosquitto.db
autosave_interval 1800

The autosave_interval means that data is saved every 1800 seconds (30 minutes). Additionally, state is stored on exit.

The persistence directory should be created and chown`ed to mosquitto on boot.

To include it in backup and restore, include it in /opt/.filetool.lst:

var/lib/mosquitto

PID File

To help with shutdown and monitoring, we let Mosquitto write a pidfile:

pid_file /var/run/mosquitto.pid

Start on boot

To start the service on boot, include this in /opt/bootlocal.sh:

# start Mosquitto in background
touch /var/log/mosquitto.log
chown mosquitto:staff /var/log/mosquitto.log
mkdir -p /var/lib/mosquitto
chown mosquitto:staff /var/lib/mosquitto
mosquitto --daemon --config-file /usr/local/etc/mosquitto.conf

This will start Mosquitto as root unless the configuration file defines the user to run as:

user mosquitto

Which is recommended.

Shutdown

To shutdown gracefully, include this in /opt/shutdown.sh:

# stop mosquitto
mosquitto_pid=$(cat /var/run/mosquitto.pid)
if [ -n "$mosquitto_pid" ]; then
    kill "$pid"
fi

Monitoring with monit

To use monit 6 to monitor Mosquitto, add these checks to monitrc (or in a separate file in monit.d/):

check process mosquitto pidfile /var/run/mosquitto.pid
    group mqtt
    if failed uid mosquitto then alert

check program mqtt-connect
with path "/usr/local/bin/mosquitto_sub --quiet -C 1 --id monit --topic $SYS/broker/version"
    group mqtt
    if status != 0 then alert

In order to check if the broker is running, connect to it and retrieve one message. We request a message from the $SYS hierarchy and pick one that is static so that we can be sure that it exists. The $SYS hierarchy is described in the Mosquitto docs 7.

mosquitto_sub comes with the Mosquitto installation an we can use it like this:

$ mosquitto_sub --quiet -C 1 --id monit --topic \$SYS/broker/version

This should be sufficient, i.e. exit with "0" if the connection was possible. The -C 1 options is necessary to exit the program after the first message. Note that you need to escape the "$" for the command line but not in monitrc.

1: http://mosquitto.org/
2: http://mqtt.org/
3: http://tinycorelinux.net/
4: http://distro.ibiblio.org/tinycorelinux/corebook.pdf
5: http://wiki.tinycorelinux.net/wiki:creating_extensions
6: https://mmonit.com/monit/
7: https://mosquitto.org/man/mosquitto-8.html#idm46187459232016

Node-RED on TinyCore Linux

Alexander Keil — Thu, 12 Jan 2017 20:51:19 GMT

Node-RED on TinyCore Linux

author: akeil
date: 2017-01-14
version: 2

Node-RED 1 is a flow based tool which can be used to connect services and devices with each other and to small automation flows.

Node-RED runs as a NodeJS 2 application and is managed through the browser.

This post shows how to install it on a Raspberry Pi 3 running TinyCore Linux 4

Installation

First, install Node.js:

$ tce-load -w- i node

Then install Node-RED through the Node Package Manager 5 (npm):

$ sudo npm install -g --unsafe-perm node-red

Note

The -g option installs it globally. The recommended procedure is to install packages which contain executable command (like Node-RED) globally and other packages locally (simply without the -g).

The Default location for global NPM packages on TinyCore Linux is /usr/local/lib/node_modules/<package>. Local packages are installed in a node_modules directory inside the current working directory.

The npm install command will also be used to install additional packages (flows or nodes) for Node-RED.

Unfortunately, having a custom package manager does not go well with the philosophy of TinyCore extensions. Packages installed through npm will not be persisted and are lost on reboot.

Two things can be done to prevent this:

install packages into a location which is backed up and restored, e.g. one of the /home/* directories. This will be the solution for additional node packages.
Create tce's for every package. We will do this for the Node-RED package only.

To create a tce for the node-red package:

$ mkdir -p /tmp/node-red/usr/local/lib/node_modules
$ sudo mv /usr/local/lib/node_modules/node-red /tmp/node-red/usr/local/lib/node_modules
$ mkdir -p /tmp/node-red/usr/local/bin
$ cd /tmp/node-red/usr/local/bin
$ ln -s ../lib/node_modules/node-red/red.js node-red
$ ln -s ../lib/node_modules/node-red/bin/node-red-pi node-red-pi
$ cd ~
$ sudo chown root:root -R /tmp/node-red/
$ tce-load -wo squashfs-tools
$ tce-load -il squashfs-tools
$ mksquashfs /tmp/node-red/ /tmp/node-red.tcz
$ sudo rm -r /tmp/node-red/
$ mv /tmp/node-red.tcz /etc/sysconfig/tcedir/optional/

Now, after a reboot, the node-red installation should be gone. But it can now be installed with:

$ tce-load -i node-red

To make sure that Node-RED is reinstalled on boot, add it to onboot.lst:

$ echo node-red.tcz >> /etc/sysconfig/tcedir/onboot.lst

Finally, we need to secure the Node-RED configuration and data files. These are stored in $HOME/.node-red and we add it to /opt/.filetool.lst. Or rather: make sure that home directories are present (they are, by default).

Additional Node Packages

Additional node packages will be installed in the /home/nodered/.node-red, assuming the the nodered user will run the application. This is done like this:

$ cd /home/nodered/.node.red
$ npm install node-red-contrib-foo

User

Next, we need to create the nodered user which we will use to run the service.

$ sudo adduser -S nodered

Start Script

To start the service at boot, edit /opt/bootlocal.sh. Since Node-RED only implements logging to the console, we redirect its output into a log file:

# start Node-RED with nodered-user in a subshell
NR_LOG=/var/log/node-red.log
touch "$NR_LOG"
chown nodered:root "$NR_LOG"
(su nodered -c /usr/local/bin/node-red -s /bin/sh >> "$NR_LOG") &

Monitoring (with monit)

This is how monit 6 can be used to monitor that Node-RED is up and running. This assumes a working installation of monit on TinyCore Linux.

Adding checks for Node-RED is done by editing a configuration file. The file is located in /usr/local/etc/monit.d. Actually, monit's configuration file is /usr/local/etc/monitrc but it is possible to split the configuration into several files by adding this line to monitrc:

include /usr/local/etc/monit.d/*

So, inside monit.d/, create a new file named node-red and add a check which looks for the Node-red process and make a HTTP request against the Node-RED Admin API 7, specifically its auth endpoint. The auth endpoint lists the currently active authentication scheme and is available even if authentication is on.

The endpoint should return either a JSON object with details on the authentication scheme or an empty JSON object. In any case, a pair of curly braces.

We will also make sure that the application is running with the intended user.

The check looks like this:

# list authentication scheme from Admin-API - see
# http://nodered.org/docs/api/admin/oauth
check process node-red matching node-red
    group node-red
    if failed
        port 1880
        with protocol http
        request /auth/login
        content = "\{.*?\}"
    then alert

    if failed uid nodered then alert

1: https://nodered.org/
2: https://nodejs.org/
3: https://www.raspberrypi.org/
4: http://tinycorelinux.net/
5: https://www.npmjs.com/
6: https://mmonit.com/monit/
7: http://nodered.org/docs/api/admin/

Firefox Sync Server - Backup

Alexander Keil — Thu, 15 Sep 2016 21:45:00 GMT

Firefox Sync Server - Backup

author: akeil
date: 2016-09-15
version: 1

This is a follow up on a previous post. where I installed a Firefox Sync Server on a Raspberry Pi running piCore. After having the Sync Service basically running, it is time to set up regular backups.

Filesystem Backup

This is a variant on a backup-scheme described in an older post. Most backups work by creating a backup locally and then pushing it to a remote server. In this case there is no remote server and the backup is to be stored on a desktop computer. The problem is that the desktop computer is not always running.

Therefore the backup will be initiated from the destination machine. It assumes that the source machine(s) will always be reachable.

TinyCore Linux already has a backup facility 1, namely filetool.sh. This creates a backup of selected files under mydata.tgz. Data is restored on every boot.

Use

$ filetool.sh -bs

The -s option tells filetool to create a safe backup. "Safe" means that the previous backup is secured before it is overwritten with a new one.

Still, mydata.tgz is stored in the tce directory on a local drive. And for a Raspberry Pi this means the SD card or an attached USB removable drive. Not exactly reliable storage media.

We will use rsync 2 to periodically download data from the tce directory (and possibly other locations) to another computer. The basic command should look like this:

$ rsync --archive rsync://box/<src> <dst>

Install and Configure rsync

Install rsync on the TinyCore Linux machine with:

$ tce-load -wi rsync

Install with -wi to have it added to the onboot.lst.

We need rsync to run in daemon mode so that the client can connect to it at any time.

First, create a minimal rsyncd configuration 3 at /etc/rsyncd.conf:

uid = nobody
gid = nogroup
pid file = /var/run/rsyncd.pid
log file = /var/log/rsyncd.log
use chroot = no

[tce]
  comment = tce directory backup
  path = /mnt/mmcblk0p2/tce
  read only = yes
  uid = tc

The config defines a module named tce. This means clients will see files under /mnt/mmcblk0p2/tce when they request rsync://box/tce. Since it is only intended for backup it is marked as read only.

Add as many modules for other locations as required.

Note

Remember to add etc/rsyncd.conf to /opt/.filetool.lst.

Start it on boot in bootlocal.sh:

# start rsyncd
rsync --daemon

Stop it on shutdown in shutdown.sh:

# stop rsyncd - location of pidfile from /etc/rsyncd.conf
pid=$(cat /var/run/rsyncd.pid)
if [ -n "$pid" ]; then
    kill "$pid"
fi

The Backup Destination

The actual backup task runs on the computer which is the backup destination. It will be set up so that it connects to a configured list of hosts (sources) via rsync.

The script retrieves a list of rsync modules from each host. If a specific KEYWORD (in this case: "backup") is part of the module name or comment, all files from that module will be included in the backup.

The last KEEP backups are kept, the oldest backup is removed.

Any host that is registered with the backup script can define any number of rsync modules and if they contain the keyword, they will be backed up. Hosts can also use the exclude option in a module definition to control which files are backed up.

On the destination this will result in a directory structure like this:

BASE/
    foo/           # backups from the "foo" host
        files.0/   # the most recent backup of "files"
            [...]
        files.1/   # an older version of "files"
            [...]
        files.2/
            [...]
        data.0/
            [...]
        data.1/
            [...]
        data.2/
            [...]
    bar/
        data.0/
        data.1/

Where we have two hosts "foo" and "bar". The "foo" hosts has two rsync modules included in the backup and we have collected three backups of each (0..2). The "bar" host has only one module of which two backups were collected.

Systemd Timer and Service

To control the periodic backup through systemd a .timer and .service file are needed.

The timer goes into /etc/systemd/system/pi-backup.timer:

[Unit]
Description=Timer for Rasperry Pi backup

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=default.target

And the service in /etc/systemd/system/pi-backup.service:

[Unit]
Description=Backup for Raspberry Pis

[Service]
ExecStart=/usr/local/bin/pi-backup.sh

Assuming that the backup script is located at /usr/local/bin/pi-backup.sh.

The timer is activated with:

$ sudo systemctl enable pi-backup.timer

Backups can be started manually with:

$ sudo systemctl start pi-backup.service

Backup Script

The complete script looks like this. The UPPERCASE variables at the top of the script are meant for configuration:

#!/bin/sh

# basedir for all backups
ROOT=/path/to/backups

# hosts to consider for backup
HOSTS=( foo bar )

# modules with KEYWORD in name or description are backed up
KEYWORD=backup

# the number of old backups to keep (not including the current backup)
KEEP=6


function main {
    for host in "${HOSTS[@]}"; do
        backup_host $host
    done
}

# fetch a list of modules
# filter all modules with "backup" in their description
# args: host
function backup_host {
    local host=$1
    # get a list of all modules for that host,
    # filter the ones containing "KEYWORD"
    # and keep only the module name
    rsync rsync://$host/ | grep "$KEYWORD" | cut -f1 | while read -r module; do
        backup_module $host $module
    done
}


# backup a single module
# args: host module
function backup_module {
    local host=$1
    local module=$2
    local basedst=$ROOT/$host/$module
    local dst=$basedst.0
    local lndst=../$module.1
    rotate $basedst
    echo Backup $host/$module to $dst
    mkdir -p $dst
    rsync --archive --delete --link-dest=$lndst rsync://$host/$module $dst
}


# drop the oldest backup,
# move other backups up one place (e.g. `backup.0` to `backup.1`)
# args: basedst
function rotate {
    local counter=$KEEP
    local dst=$1
    # going backwards: n, n-1, ... 0
    while [ $counter -ge 0 ]; do
        ahead=$(($counter+1))
        local current=$dst.$counter
        local older=$dst.$ahead

        if [ -d $current ]; then
            if [ $counter -eq $KEEP ]; then
                echo delete $current
                rm -r $current
            else
                echo move $current to $older
                mv $current $older
            fi
        fi

        counter=$(($counter-1))
    done
}

# run it
main

SQL Dumps

The Firefox sync server was installed using MariaDB as a database backend. We will perform logical backups of the database. That is, the backup takes the form of SQL statements which, when executed, restore the original data.

mysqldump 4 is used to dump the complete database into SQL statements.

The backup script for the ffsync database looks like this:

#!/bin/sh
dst=/mnt/storage/mysql/dumps/ffsync.dump
cfg=/opt/ffsync.db.cnf

mysqldump --defaults-extra-file=$cfg --lock-tables ffsync > $dst.temp

status=$?
if [ $status -eq 0 ]; then
    mv $dst.temp $dst
fi

exit $status

The --lock-tables option ensures that we retrieve a consistent state of the database. For InnoDB table types, --single-transaction is recommended to achieve that but table locking will work with any storage engine.

To avoid having the database password in the command line, a configuration file is used to keep credentials. It is passed with --defaults-extra-file.

The config file looks like this:

[mysqldump]
user=ffsync
password=<secret>

Set the permissions for the config file so that only the owner can read it and make it owned by the mysql user.

$ sudo chown mysql:nogroup /opt/ffsync.db.cnf
$ sudo chmod 600 /opt/ffsync.db.cnf

Both, script and config file are kept in /opt which means they should already be included in filetool.lst for backup and restore.

Finally, add the script to mysql's crontab:

$ sudo crontab -eu mysql

12 1 * * * /opt/backup-ffsync-db.sh

The backup will be executed with the user we created to run the mysqld service (mysql) so make sure that the backup destination is writable for that user.

To include the SQL dump in the filesystem backup, create an rsync module for it in rsyncd.conf:

...
[sqldumps]
  comment = SQL dumps backup
  path = /mnt/storage/mysql/dumps
  read only = yes
  uid = mysql

Cron

We have made a crontab entry but cron might not be enabled. To enable it, we must add the Tiny Core bootcode 5 "cron".

Bootcodes can be set in a file named cmdline.txt. the file is located in the boot partition, which is normally unmounted after boot. Mount it and edit the file:

$ sudo mount /dev/mmcblk0p1 /mnt/mmcblk0p1
$ sudo vi /mnt/mmcblk0p1/cmdline.txt
(append bootcode "cron")
$ sudo umount /mnt/mmcblk0p1

cmdline.txt contains a space separated list. Simply append " cron" to the end of that list.

Warning

With piCore 8.x and Raspberry Pi 3, the file is cmdline3.txt, not cmdline.txt. See http://forum.tinycorelinux.net/index.php?topic=20107.0

After a reboot, check if it worked. This should return the PID of the running crond daemon:

$ pgrep crond

Now make sure that crontabs are persisted across boots. Add the crontabs directory to /opt/.filetool.lst:

var/spool/cron/crontabs

Timezone

If you did not set it, piCore's time zone is UTC. And your crontab entry would refer to UTC as well.

This is a good opportunity to add another bootcode (tz) for setting the timezone.

The TZ bootcode 6 is a bit more involved it (not as complicated as it looks, though). For Central European Time the timezone is encoded like this:

tz=CET-1CEST,M3.5.0,M10.5.0/3
   |  | |    || | | ||  | | `- 03:00:00
   |  | |    || | | ||  | `--- Sunday (day 0)
   |  | |    || | | ||  `----- Last week (week 5)
   |  | |    || | | |`-------- October (month 10)
   |  | |    || | | `--------- End of DST in format m.w.d[/h]
   |  | |    || | `----------- Sunday (day 0)
   |  | |    || `------------- Last week (week 5)
   |  | |    |`--------------- March (month 3)
   |  | |    `---------------- Begin of DST in format m.w.d[/h]
   |  | `--------------------- TZ name during DST
   |  `----------------------- UTC offset w/o DST
   `-------------------------- TZ name w/o DST

Start and end times for DST are specified in the format:

Mm.w.d[/h]
 ^ ^ ^  ^
m: month, 1=January
w: week of the month (1=first, 5=last)
d: day of the week, 0=Sunday, 6=Monday
h: [optional] hour of the day, 3 for 03:00:00

If no time parameter is given for DST start or end time, 02:00:00 is assumed.

Add the parameter to cmdline[3].txt (for CET):

tz=CET-1CEST,M3.5.0,M10.5.0/3

Whether setting the timezone this way (or setting it at all) is a good idea is debatable. For me - living in a country with daylight savings time - it felt more comfortable having the same time on all computers.

1: http://wiki.tinycorelinux.net/wiki:backup
2: https://rsync.samba.org/
3: https://download.samba.org/pub/rsync/rsyncd.conf.html
4: https://mariadb.com/kb/en/mariadb/mysqldump/
5: http://wiki.tinycorelinux.net/wiki:boot_codes_explained
6: http://wiki.tinycorelinux.net/wiki:time_zone

Firefox Sync Server on a Raspberry Pi

Alexander Keil — Wed, 14 Sep 2016 15:50:27 GMT

Firefox Sync Server on a Raspberry Pi

author: akeil
date: 2016-09-14
version: 1.2
updated: 2016-09-15

The Firefox browser can use a Sync Service to synchronize settings, bookmarks and other stuff across multiple Firefox installations. By default, Mozilla's public sync server is used but it is possible to run your own sync server 1.

We will install piCore Linux and the Firefox Sync Server on a new Raspberry Pi.

Install piCore OS

We will use piCore 2, the Raspberry Pi version of Tiny Core Linux 3. The benefit of using piCore is that it runs entirely from memory which should reduce strain on the SD card.

piCore is installed by downloading the image and putting it on an SD card.

$ wget http://tinycorelinux.net/8.x/armv6/releases/RPi/piCore-8.0.zip
$ unzip piCore-8.0.zip
$ sudo dd if=piCore-8.0.img of=/dev/sdX bs=1M
$ sudo sync

Where /dev/sdX is the empty SD card. Don't forget to unmount all partitions from the SD card before writing to it.

After that, put the SD card into the Raspberry Pi, plug in power and network and log in with user tc and password piCore.

Extend Partition

TinyCore Linux knows two modes - "Cloud Mode" and "Mounted Mode". Mounted Mode allows to have some persistence, e.g. for installed applications. For this, a second partition is used. The second partition is already available after installation but it should be enlarged.

Use fdisk to delete the existing partition and create a new, larger one:

The second partition will normally be mounted. Unmount it first.

$ sudo umount /dev/mmcblk0p2
$ sudo fdisk -u /dev/mmcblk0

List partitions with p and note down the start and end sectors of the second partition. Delete the second partition with d, then create a new one with n. Use the same start sector as previously but different end sector. The default end sector uses all available space. Save with w.

After a reboot, log in again an expand the filesystem to fill up the second partition:

$ sudo reboot
$ sudo resize2fs /dev/mmcblk0p2

This should leave us with two partitions on the SD card. A small partition 1 (mmcblk0p1) which holds the OS files and will be unmounted after boot and a larger partition 2 (mmcblk0p2) which takes most of the SD cards capacity and holds installed applications and data.

Add Additional Storage

We will use the No. 2 partition to persist applications and settings on the SD card. We will also set up a removable USB drive to hold bulk data. We create a filesystem on it and set it up to be mounted at boot.

Plug the USB storage into the Raspberry Pi. It should show up in /proc/partitions, blkid can be used to get more info.

$ cat /proc/partitions
$ blkid /dev/sdX

We will put an ext2 filesystem on it. ext2 is non-journaling which should generate less writes and extend the life of the flash drive

First, we create a single partition for the whole disk:

$ sudo fdisk /dev/sdX
> d
> n
> p
> 1
> w

Then, create an ext2 filesystem 4:

$ mke2fs /dev/sdX1

Since the device is a plugged in drive, we mount it by its UUID. Find the UUID with blkid /dev/sdX1. Also we choose a different directory name for the mountpoint.

To mount it automatically on boot, add the following lines to /opt/bootlocal.sh:

mkdir /mnt/storage
chmod 777 /mnt/storage
mount UUID=b0ee2fe8-c54a-4c67-9a40-a9c2c4cb734c /mnt/storage

And the respective unmount command to /opt/shutdown.sh:

umount /mnt/storage

Tiny Core Persistence

By default, TinyCore Linux has no persistence. That means all changes made in configuration files or newly installed applications are lost when the system reboots. There are two options to keep data between two boots:

backup
persistent partitions

Software packages for TinyCore Linux are called "extensions" and they are stored in the /tce directory.

The preconfigured image already comes with the /tce directory on the second partition so there is not much to do here.

Verify that /mnt/mmcblk0p2/tce exists; it should contain some preinstalled applications (like openssh). Look at /etc/fstab to see how /dev/mmcblk0p2 is mounted.

The backup utility compresses selected files or directories and stores them as mydata.tgz inside the /tce directory.

The file /opt/.filetool.lst defines which files or directories will be included. /opt/.xfiletool.lst can be used to exclude previously included files (e.g. include a complete directory tree but exclude single large files).

The default list includes the /opt and /home directories, ssh host keys and files related to user management (/etc/passwd for example).

Warning

The backup script does not run automatically. One must include a call to filetool.sh -b in /opt/shutdown.sh. Also, shutdown.sh will only run when the system is shut down with exitcheck.sh.

Install Firefox Sync

The server is installed into a python virtualenv 5 which is located in the syncserver installation directory. The server will later run as a Pyramid 6 app inside a Gunicorn 7 application server.

There is no prebuilt package for Firefox Sync on piCore so we need to download the source and build the software locally.

Install Prerequisites

Start by installing prerequisites:

$ tce-load -wi python
$ tce-load -wo python-dev
$ tce-load -wo make
$ tce-load -wo git
$ tce-load -wo gcc
$ tce-load -wo compiletc
$ tce-load -i python-dev make git gcc compiletc

The -wi switch adds the extension to the OnBoot list which means it is loaded on every boot. The -wo switch creates an OnDemand item which we can load manually with tce-load -i. We install only the base python with the OnBoot option because it is the only package we need for running the app. All other packages are only required for the build process.

The installation additionally requires Python virtualenv for which we do not have a piCore package. But we can install virtualenv from source 8:

$ wget https://pypi.python.org/packages/8b/2c/c0d3e47709d0458816167002e1aa3d64d03bdeb2a9d57c5bd18448fd24cd/virtualenv-15.0.3.tar.gz
$ tar -xzf virtualenv-15.0.3.tar.gz
$ cd virtualenv-15.0.3
$ sudo python setup.py install
$ cd ..
$ rm virtualenv-15.0.3.tar.gz
$ rm -r virtualenv-15.0.3

The download URL can be obtained from PyPi.

Note

Installing virtualenv in this way places installation files in their default locations. They will be lost on shutdown. This does not matter (much) as we only need virtualenv for the installation, not for running the application.

Create an ffsync User

We want to run the sync service under a dedicated ffsync user. Create it like so:

$ sudo adduser ffsync -SH

Installation Location

Decide upon an installation location.

The default installation procedure is to clone the syncserver git repository 9 and build the application inside the repo directory. Building the application entails creation of a virtual Python environment. The virtualenv is basically a copy of the system-wide Python install plus additional packages installed in a local directory, in this case the syncserver installation directory.

This results in a large number of files and directories. If we keep the installation inside the /home directory, we have the benefit of running it from memory but the disadvantage that the backup and restore for the home directory takes quite long.

If we install to a persistent partition, we lose the "running from memory" advantage.

The proper(?) solution is to create a TinyCore Linux extension for it. This is described in detail in chapter 14 and 15 of the Tiny Core book 10 [PDF]. The basic idea is to create the desired directory structure and files in some temporary directory and then "pack" it with squashfs. TinyCore Linux will later mount/symlink it into the desired location.

That is, if we want to have a file /usr/share/myfile, create /tmp/myextension/usr/share/myfile and then use squashfs to create myextension.tcz from /tmp/myextension.

The syncserver build process (more precisely: virtualenv) will generate some files with hardcoded paths so it is tricky to move the installation to some place different than where it was created. There is an option to make a relocatable virtualenv but it does not seem fully supported and we are not going to use it.

The good thing is that the virtualenv installation keeps all required files under a single base directory. This makes it possible to install the application at the desired location then pack it up and remove the original install.

We will install under /usr/local/syncserver.

First create the required directory and clone the syncserver repo into it:

$ cd /usr/local
$ sudo mkdir syncserver
$ sudo chown syncserver tc:staff
$ git clone --depth=1 https://github.com/mozilla-services/syncserver syncserver
$ cd syncserver
$ make build

Note

Root permissions are only required to create a new directory in /usr/local. The installation itself works with normal permissions.

Use git clone with --depth=1 to retrieve as little history as possible, but still keep it as a git repo.

Delete the syncserver/.git directory to save additional space.

Next, we need to install the Python database driver. This is done using the Python package manager (pip) that is part of the virtualenv. So, from the syncserver directory:

$ ./local/bin/pip install pysqlite

Install pysqlite for Usage with SQLite, PyMySQL for MySQL/MariaDB.

Note

The make build and the installation of pysqlite/PyMySQL with pip will leave several files in ~/.cache/pip. You may want to delete those so they will not be included in the backup.

We should now have a complete installation at /usr/share/syncserver. Running sudo make serve from that directory should start the server with default settings on port 5000.

Look at ./local/bin/gunicorn. On the top it should say:

#!/usr/local/syncserver/local/bin/python2

Config

The default setup assumes that the syncserver is started from inside the installation directory with make serve and that the configuration file is kept there (it is referenced with ./syncserver.ini). If we want to change config separately from the installation, we need to copy it to another location:

$ cp syncserver.ini /etc

And also include it in /opt/.filetool.lst:

etc/syncserver.ini

We will later use a custom command like this to run the service:

$ ./local/bin/gunicorn --paste /etc/syncserver.ini

Create the Extension

First off, install squashfs-tools:

$ tce-load -wo squashfs-tools

To produce a TinyCore Linux extension, we need to transfer everything to a temporary directory structure, pack it and move the resulting .tcz file to the tce directory:

$ cd /tmp
$ mkdir -p syncserver/usr/local
$ sudo mv /usr/local/syncserver syncserver/usr/local
$ sudo chown root:root -R syncserver/
$ mksquashfs syncserver syncserver.tcz
$ mv syncserver.tcz /etc/sysconfig/tcedir/optional
$ sudo rm -r syncserver/

Note

It is a good idea to place a backup of the syncserver.tcz file somewhere other then the tce/optional directory.

We can now start the application with:

$ tce-load -i syncserver

You should see the syncserver installation (as a collection of symlinks) under /usr/local/syncserver.

Note that tce-run will not actually start the service. It will only make the installed application available. Running sudo make serve from the installation directory should now work as before.

To have the application available on boot, add it to the onboot.lst (inside the tce directory):

$ echo syncserver.tcz >> /etc/sysconfig/tcedir/onboot.lst

Start and Stop Script

We need a custom script to start the service. The start script is included in bootlocal.sh to start the service on boot and performs three tasks:

change into the correct working directory (/usr/local/syncserver)
switch to the ffsync user
start the syncserver with our configuration file at /etc/syncserver.ini

BASEDIR=/usr/local/syncserver
GUNICORN=$BASEDIR/local/bin/gunicorn
CONFIG=/etc/syncserver.ini
COMMAND="$GUNICORN --paste $CONFIG --log-file /tmp/syncserver.log > /dev/null &"
(cd $BASEDIR && su ffsync -c "$COMMAND" -s /bin/sh)

We still need to change the working directory before we execute the sync server. If we do this inside the bootlocal.sh script we might affect subsequent actions in that script. One way to avoid this, invoke everything in a subshell.

Also, since the gunicorn command will not exit, "disown" it afterwards with "&".

To stop the service, use:

pkill syncserver

(not pretty)

Configuration

Edit the config file to reflect your settings:

[syncserver]
public_url = http://box:5000/
sqluri = sqlite:////mnt/storage/syncserver/syncserver.db
secret = xxx
allow_new_users = false  # set to true to create initial user

public_url is set to what the client sees as the URL of the service.

If you do not configure a sqluri, the default applies - which is an in-memory database (which is lost when the service is shut down).

The secret is used to generate authentication tokens. It can be any random string. Generate one like this:

$ head -c20 /dev/urandom | sha1sum

allow_new_users should be set to true initially. When the service is fully installed and all required Firefox accounts are registered, it can be set to false.

Install a Database

The syncserver works with SQLite, MySQL/MariaDB or PostgreSQL. There are piCore packages for SQLite and MariaDB.

SQLite

For SQLite, make sure the sqlite3 package is installed. Then decide where the database file should live and update syncserver.ini accordingly:

[syncserver]
...
sqluri = sqlite:////mnt/storage/syncserver/syncserver.db

Remember to make the database directory accessible for the ffsync user.

$ sudo mkdir /mnt/storage/syncserver
$ sudo chown ffsync:staff /mnt/storage/syncserver

MariaDB

There is a (somewhat outdated?) MariaDB installation guide 11 in the TinyCore Linux wiki.

Start by installing MariaDB

$ tce-load -wi mariadb
$ tce-load -wi mariadb-client

Next, we make sure that the DB service starts on boot and stops on shutdown. The package comes only with a partially complete configuration and some effort is required to make it work.

Start by creating a system account to run mariadb:

$ sudo adduser -SH mysql

Next, add commands to start and stop MariaDB 12 to bootlocal.sh and shutdown.sh:

# bootlocal.sh
/usr/local/share/mysql/mysql.server start

# shutdown.sh
/usr/local/share/mysql/mysql.server stop

Note

Make sure that the database service is started before the syncserver, and shut down after the syncserver.

Then create a minimal configuration file for MariaDB 13 file at /etc/my.cnf:

[client]
socket = /tmp/mysql.sock

[mysqld]
socket = /tmp/mysql.sock
basedir = /usr/local
user = mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
log_error = /tmp/mysql.error.log
pid_file = /tmp/mysql.pid

It is important to set the socket option in the [server] and [client] sections. Otherwise tools like mysqladmin will not see the socket option and will fail to connect.

Make sure that the datadir is accessible for the mysql user:

$ sudo chown -R mysql:nogroup /usr/local/mysql/data

Manually starting and stopping mysqld should now work:

$ sudo /usr/local/share/mysql/mysql.server start
Starting MySQL. SUCCESS!
$ sudo /usr/local/share/mysql/mysql.server stop
Shutting down MySQL.. SUCCESS!

Persistence

For now, the config file and data directory only exist in memory. To persist the config file between reboots, add it to the backup list in /opt/.filetool.lst:

etc/my.cnf

The data directory is where the database files are stored. By default, this is /usr/local/mysql/data.

If this would be kept in memory and backed up/restored when the system restarts, we have a risk of losing data when the system shuts down unexpectedly and the backup script does not run (e.g. when power is unplugged).

To avoid this, keep the database on a separate partition and create a symlink in the original location which points to the new location. Stop the mysqld service before moving the data directory.

$ sudo /usr/local/share/mysql/mysql.server stop
$ sudo mkdir /mnt/storage/mysql
$ sudo chown mysql:nogroup /mnt/storage/mysql
$ sudo mv /usr/local/mysql/data /mnt/storage/mysql
$ sudo ln -s /mnt/storage/mysql/data /usr/local/mysql
$ sudo /usr/local/share/mysql/mysql.server start

We need to repeat parts of this on every reboot. So add the following to bootlocal.sh (before starting mysqld):

# replace the standard datadir with persisted data
MYSQL_DATADIR=/usr/local/mysql/data
rm -rf $MYSQL_DATADIR
ln -s /mnt/storage/mysql/data $MYSQL_DATADIR

Setup a Database User

First, set a password for mysql admin

$ mysqladmin -u root password <secret>

$ mysql -u root -p
(enter password when prompted)
mysql> CREATE USER 'ffsync'@'localhost' IDENTIFIED BY '<secret>';
mysql> CREATE DATABASE ffsync;
mysql> GRANT ALL ON ffsync.* TO 'ffsync'@'localhost';

Finally, configure Firefox Sync Server to use the MariaDB database:

[syncserver]
...
sqluri = pymysql://ffsync:<secret>@localhost/ffsync

Configure Firefox

Go to about:config and set:

identity.sync.tokenserver.uri = http://box:5000/token/1.0/sync/1.5

(the default is https://token.services.mozilla.com/1.0/sync/1.5)

You will still need a firefox account to use the sync feature.

Troubleshooting

syncserver complains that public_url does not match application url. Public URL is set to "http://hostname:5000/", error says that app url is "http://hostname/". Change public_url in config file to hostname w/o port, get the same error but the other way round. Set force_wsgi_environ = true fixed this.

With SQLite - errors when clients try to sync:

(OperationalError) cannot start a transaction within a transaction

Tried to limit gunicorn to a single worker thread - no help. Installed MariaDB to work around this.

Note

Type about:sync-log into Firefox address bar to see ...sync logs.

Some additional links: - http://blog.sysbite.org/run-a-firefox-sync-server-on-the-raspberry-pi/ - http://www.raspberry-pi-geek.com/Archive/2015/11/The-new-Firefox-synchronizer - https://wiki.archlinux.org/index.php/Mozilla_Firefox_Sync_Server

Coordinate Scripts With systemd

Alexander Keil — Sun, 14 Dec 2014 13:43:38 GMT

Coordinate Scripts With systemd

author: akeil
date: 2013-12-14
version: 1

Use systemd to execute scripts, controlling the order of execution.

Assuming we have:

a single preparation unit - prep.service
multiple "task" units - task-N.service
a single cleanup script - done.service

We want to run the prep unit once, then run all our task units and finally the done unit once.

Defining dependencies between tasks is relatively simple when each task is represented by a systemd unit 1. Dependencies are defined using Requires= or Wants= and order of execution is defined with After= and Before=.

The only thing left is to make sure that the "preparation" and "finish" steps are executed only once, even if multiple tasks are run (and not once for each task).

To achieve this a .target is created and all tasks are associated to that target.

Note

Steps:

Create .service files for the commands that must be executed before
Create .service for commands that should be executed after
Create as single .target for all tasks
Create a .service for each task, make it WantedBy= the target.

Target Unit

A systemd target 2 is used to group several units together.

The target file goes into /etc/systemd/system and looks like this:

# tasks.target ---------------------------------------
[Unit]
Description = Tasks Target

The target can be started manually with:

# systemctl start tasks.target

The task units are associated with the target using WantedBy=.

This means, if we enable one or more task units with

# systemctl enable task-1.service task-2.service

... starting the tasks.target will start all of the associated units.

This allows us to start all tasks with a single invocation which in turn means that we execute any number of tasks and the prep and done units only once.

Task Units

Each task unit specifies the prep.service as a precondition using Requires= and After=. It also specifies the done.service using Wants= and Before= to have it run after the tasks.

# task.service ---------------------------------------
[Unit]
Description = A task unit
Requires = prep.service
After = prep.service
Wants = done.service
Before = done.service

[Service]
User = someuser
Group = somegroup
Type = oneshot
ExecStart = /usr/local/bin/script.sh

[Install]
WantedBy = tasks.target

The Requires=prep.service directive means that the task is not started if the prep.service failed. Using Wants=prep.service would try to start the prep unit first but continue with the tasks regardless of whether preparation was successful or not.

The User= and Group= properties can be used to run the command under a different user than root.

Not all tasks have to specify the same dependencies. It is also possible that tasks depend on each other.

Prep and Done Units

The prep.service and the done.service are plain units and do not define any dependencies.

the prep unit should be configured with Type=oneshot to have it complete before the first task can be started.

Service Type

By default, Before= and After= only control the order in which services are started. To make sure that the prep unit completes before the tasks are started, use Type=oneshot in the [Service] definition.

From the systemd service 3 documentation:

Behavior of oneshot is similar to simple; however, it is expected that the process has to exit before systemd starts follow-up units.

The same goes for the task units if the done unit should only be executed after the tasks are complete.

Locations

Custom unit files go into:

/etc/systemd/system for system-wide scope
~/.config/systemd/user for user-specific scope

1: http://www.freedesktop.org/software/systemd/man/systemd.unit.html
2: http://www.freedesktop.org/software/systemd/man/systemd.target.html
3: http://www.freedesktop.org/software/systemd/man/systemd.service.html

Syncthing on Arch Linux

Alexander Keil — Wed, 10 Dec 2014 20:40:00 GMT

Syncthing on Arch Linux

author: akeil
date: 2014-12-10
version: 1

Syncthing 1 is an open source tool which keeps files on different machines in sync.

Installation

For Arch Linux Syncthing is available in the community repository 3 and can be installed easily:

# pacman -S syncthing

The Syncthing package installs a .service file at /usr/lib/systemd/system/syncthing@.service.

[Unit]
Description=Syncthing service for %i
After=network.target

[Service]
User=%i
Environment=STNORESTART=yes
ExecStart=/usr/bin/syncthing
Restart=always

[Install]
WantedBy=multi-user.target

When enabling or starting the service, supply the user under which syncthing will run:

# systemctl start syncthing@yourname
# systemctl enable syncthing@yourname

Configuration

Configuration is located at ~/.config/syncthing/. The directory and config files inside it will be created when Syncthing is first started. It is described in detail in the Syncthing documentation 2.

The config.xml can either be edited via the web GUI or manually. By default, the Web GUI is only reachable from the same machine.

If Syncthing is installed on a machine without a desktop, one can either edit the configuration manually for all tasks or at least to make the GUI available on the network. Manual configuration will also be necessary if you want to change the port number under which the GUI is available, for example when the default port 8080 is already in use.

<configuration version="6">
    ...
    <gui enabled="true" tls="true">
        <!-- default: 127.0.0.1:8080 -->
        <address>0.0.0.0:8888</address>
    </gui>
    ...
</configuration>

The first thing you might want to do is to remove the ~/Sync/ directory that was created on startup and replace it with something else.

# systemctl stop syncthing@yourname
$ rm -r ~/Sync

Edit the config file to include the directories to sync:

<configuration version="6">
    <folder id="default" path="/home/yourname/sync-folder" ro="false" rescanIntervalS="60" ignorePerms="false">
        <device id="[ID-1]"></device>
        <versioning></versioning>
        <lenientMtimes>false</lenientMtimes>
        <copiers>0</copiers>
        <pullers>0</pullers>
        <finishers>0</finishers>
    </folder>
    ...
</configuration>

Create more <folder> elements if you want to sync other directories.

Changes will take effect when Syncthing is restarted:

# systemctl restart syncthing@yourname

Start Syncing

Repeat installation and configuration on other machines you which to keep in sync. On each machine, Syncthing must be started at least once to determine the device ID for that machine.

Edit the config file to include the new devices and add the device to each folder you want to sync:

<configuration version="6">
    <folder id="notes" path="/home/akeil/notes" ro="false" rescanIntervalS="60" ignorePerms="false">
        <!-- the ID of the local device -->
        <device id="[ID-1]"></device>
        <!-- one or more IDs of remote devices -->
        <device id="[ID-2]"></device>
        <versioning></versioning>
        <lenientMtimes>false</lenientMtimes>
        <copiers>1</copiers>
        <pullers>16</pullers>
        <finishers>1</finishers>
    </folder>
    <!-- the local device -->
    <device id="[ID-1]" name="other-machine" compression="true" introducer="false">
        <address>dynamic</address>
    </device>
    <!-- remote devices to sync with -->
    <device id="[ID-1] name="home" compression="true" introducer="false">
        <address>dynamic</address>
    </device>
    ...
</configuration>

Restart syncthing:

# systemctl restart syncthing@yourname

Note

You must create <device> elements on both machines. (inside the <folder>'s you wish to sync and in the general section of known devices.).

If not, entries like this appear in the log and nothing is synced:

[...] Connection from 192.168.1.100:12345 with unknown device ID [ID-1];

Logfiles (Journal)

View logs with:

$ journalctl --unit syncthing@yourname

$ systemctl status syncthing@yourname

Misc

There is a Syncthing Android App 4.
And another blog post 5 on Syncthing.
There is Syncthing for Arch Linux ARM 6 which means it can run on a RaspberryPi 7

1: http://syncthing.net/
2: https://discourse.syncthing.net/c/documentation
3: https://www.archlinux.org/packages/community/x86_64/syncthing/
4: https://f-droid.org/repository/browse/?fdid=com.nutomic.syncthingandroid
5: http://jasonwryan.com/blog/2014/05/10/syncthing/
6: http://archlinuxarm.org/packages?arch=&search=syncthing
7: http://www.raspberrypi.org/

Spellcheck with aspell

Alexander Keil — Sat, 10 May 2014 10:00:00 GMT

Spellcheck with aspell

author: akeil
date: 2014-05-10
version: 1

aspell 1 is an open-source spell checker which can be used from the command line. Aspell can be used when working with text-based files such as ReStructured Text or markdown (or plain text, of course).

aspell has filters to spellcheck E-Mails or HTML documents.

There are also Python bindings for aspell 2.

Invoke aspell like this to open an interactive terminal:

$ aspell check document.txt

By default, aspell uses its built-in dictionary and an additional user dictionary located at ~/.aspell.en.pws (for an English localization). The user dictionary holds any words that were selected as "add to dictionary" during an aspell session and it can be edited manually.

If the file to be checked contains specific terms which should not be treated as misspellings but should also not be included in the global or user based dictionary, one can define an additional dictionary on the command line. This is useful if the document contains for example technical or domain-specific terms, abbreviations, etc.

To check using an additional dictionary:

$ aspell --add-extra-dicts=project-dict.pws check document.txt

This can be useful if the file to be checked contains specific terms that should not be treated as misspellings but should also not be included in the global dictionary (like technical or domain-specific terms, abbreviations, ...).

If the file is modified during the aspell session, a backup with the original content is created. To run without backing up the uncorrected original:

$ aspell -x check document.txt

To list misspelled words from a file, use list.

$ aspell list < document.txt

Aspell does not accept shell-wildcards in arguments, so it is not possible to aspell check directory/*.txt. To spellcheck multiple files, use find (from paulbradley 3):

$ find *.txt -exec aspell check {} \;

Interesting Command Line Options

-c FILE, check FILE: specify a file to be checked
list: Generate a list of misspelled words. will not correct anything.
clean: Clean up a wordlist and output a new wordlist where every line is a valid word.

--add-extra-dicts FILE: Use FILE as an additional wordlist. The absolute path must be specified and the path may not contain shell variables.
-b, --backup: Create a backup-file (.bak) with the uncorrected file.
-x, --dont-backup: Do not create a backup (.bak) file.
-l LANGUAGE, --lang LANGUAGE: Specify the language to use with the two-letter language code (e.g. "en") optionally followed by the country code (e.g. "en-US").

Install

Install with:

# apt-get install aspell

on a Debian based system. Or

# pacman -S aspell

on ArchLinux.

Configure

Relevant configuration files for aspell are:

/etc/aspell.conf: The System-wide configuration.
~/.aspell.conf: The per-user configuration file.
~/.aspell.en.pws: The user-specific wordlist (for the en locale).

Use additional wordlists by specifying a path in the config file:

# ~/.aspell.conf
# -------------------------------------------------------
add-extra-dicts /home/USERNAME/.aspell-tech-terms.en.pws
add-extra-dicts /home/USERNAME/.aspell-names.en.pws

Use multiple lines of add-extra-dicts to add several wordlists. The wordlists will be used in every invocation of aspell.

Note

The paths must be absolute. Shell variables/expansions like ~, /path/* or $HOME will not work.

Wordlists

aspell wordlists are simple text files with one word per line.

The first line must be:

personal_ws-1.1 en 10

Where en is your locale and 10 would be the number of words in the wordlist. The number of words does not have to be exact but should hint the actual number of words in the file.

The conventional file extension is .pws but any filename can be used.

Scripts

A Spellcheck Command

A small shell script to invoke aspell and use an additional per-file wordlist and/or a project specific wordlist.

The script will look for a wordlist with the same name as the file to be checked and it will look for a wordlist in the current directory. This allows to use a list of allowed words without adding them to any of the globally used wordlists.

Usage:

$ spellcheck document.txt

Code

spellcheck.sh (Source)

#!/bin/bash

set -o nounset
set -o errexit


# Commands -----------------------------------------------
ASPELL=/usr/bin/aspell
BASENAME=/usr/bin/basename
DIRNAME=/usr/bin/dirname
PWD=/bin/pwd


# Script -------------------------------------------------
extra=""
switch="--add-extra-dicts"

# file specific dictionary
# if there is a file with the same name
# but ending in `.pws`
# use it as an extra dict
directory=`$DIRNAME "$1"`
filename=`$BASENAME "$1"`
name="${filename%.*}"
dictpath=$directory/$name.pws
if [ -f "$dictpath" ]
then
    extra="--add-extra-dicts=`$PWD`/${dictpath}"
fi

# extra dicts from the current directory
# all files in the working directory
# ending in `.pws` are used as extra dicts

# return nothing if *.pws does not match anything
shopt -s nullglob

for filename in *.pws;
do
    extra="${extra} --add-extra-dicts=`$PWD`/${filename}"
done

$ASPELL $extra check "$1"

Create a Dictionary from Misspelled Words

Say you have a file which contains multiple domain-specific terms. You want to spellcheck it regularly and you want to exclude these terms from the spellcheck for this file only.

Keeping it separate from your general dictionary is especially useful if these "special terms" are similar to typical typos.

To easily generate this dictionary, produce a list of words that aspell considers misspelled, and edit that list so that it contains the "special terms" only.

mk-aspell-dict.sh (Source)

#!/bin/bash
# generates an *aspell* "dictionary"
# from a list of misspelled words found by aspell.

set -o errexit


# Configuration ----------------------------------------------------
HEADER=personal_ws-1.1
LANG=en


# Commands ---------------------------------------------------------
ASPELL=/usr/bin/aspell
CAT=/bin/cat
ECHO=/bin/echo
READLINK=/bin/readlink
RM=/bin/rm
SORT=/usr/bin/sort
UNIQ=/usr/bin/uniq
WC=/usr/bin/wc


# Script -----------------------------------------------------------
if [ -z "$1" ];
then
    $ECHO Error: no source specified.
    exit 1
fi

# normalize paths
# aspell will not resolve shell variables or perform expansion
src=`readlink -fn "$1"`

if [ ! -r "$src" ];
then
    $ECHO Error: "$src"  does not exist or is not readable
    exit 1
fi

if [ -z "$2" ];
then
    dest="$1.pws"
    $ECHO Destination not specified, using \'$dest\'
else
    # use `readlink` to get absolute path.
    dest=`$READLINK -fn "$2"`
fi

templist=/tmp/wordlist

# generate a wordlist from words considered misspelled by aspell
# remove duplicates with `sort | uniq`
$ASPELL --rem-extra-dicts "$dest" list < "$src"\
 | $ASPELL clean\
 | $SORT\
 | $UNIQ > "$templist"

# generate the header for the pws file with the number of words
# wc -l will output:
# [numlines] [filenames]
# (separated by space)
wordcount=`$WC -l "$templist"`
$ECHO "$HEADER $LANG ${wordcount% *}" > "$dest"
$CAT "$templist" >> "$dest"
$RM "$templist"
$ECHO Generated dictionary at \'$dest\'.

The command (aspell list < document.txt) generates a list of words that would be considered misspelled by aspell. The command aspell clean takes that list from and cleans it up so that every line is a valid entry for an aspell wordlist (this might not be necessary as aspell list does probably not return invalid words).

sort and uniq are used to remove duplicate entries and sort alphabetically. The result is written to a temporary file.

The last part of the script generates the header-line for the wordlist, using wc -l to count the number of words (lines, actually). The header is written to the dictionary file and then the list of words is cat into the dictionary file.

To just update the header of a wordlist file:

$ echo personal_ws-1.1 en `tail -n +2 wordlist.pws | wc -l`\n > /tmp/wordlist.pws
$ tail -n +2 wordlist.pws >> /tmp/wordlist.pws
$ mv /tmp/wordlist.pws wordlist.pws

One could additionally add a sort | uniq pipe to the tail-calls. This would remove duplicates.

Create a Wordlist from Radicale Address Book

Add the names of your contacts to the list of correctly spelled words.

This assumes that the address book is available over the network as CardDAV - for example via Radicale.

This script fetches an address book from a CardDAV server and filters out the names and nicknames of all contacts.

It writes them to a wordlist named ~/.aspell-names.en.pws which should be included via --add-extra-dict in in ~/.aspell.conf.

mk-aspell-names.sh (Source)

#!/bin/bash
set -o errexit
set -o nounset

user="username"
pass="password"
host="hostname"
port=5232
url="https://$host:$port/$user/contacts.vcf"


# check if the addressbook is available
curl --silent --insecure\
 --basic\
 --user "$user:$pass"\
 --request HEAD\
 --header "Accept: text/vcard"\
 --write-out "%{http_code}"\
 "$url"\
 | grep --silent -E "2[0-9]{2}"

if [ $? -gt 0 ];
then
    echo No access to adressbook at $url.
    exit 1
fi


# Write the complete addressbook to a temp-file
curl --silent --insecure\
 --basic --user "$user:$pass"\
 --request GET\
 --header "Accept: text/vcard"\
 "$url"\
> /tmp/vcards.vcf


# Reduce to the fields we are interested in:
#   N:Name;Part;More;parts;;
#   FN:Full Name
#   NICKNAME: Nicky
sed -r\
 -e "/^(FN|N|NICKNAME):.+$/!d"\
 -e "s/(N|FN|NICKNAME)://"\
 -e "s/;+/\n/g"\
 /tmp/vcards.vcf\
 > /tmp/names

# will filter the list and include only
# names that would be regarded as spelling errors
~/scripts/mk-aspell-dict.sh /tmp/names /home/USERNAME/.aspell-names.en.pws

rm /tmp/vcards.vcf
rm /tmp/names

1: http://aspell.net/
2: https://github.com/WojciechMula/aspell-python
3: https://paulbradley.org/aspell/

Music with MPD and beets

Alexander Keil — Mon, 24 Mar 2014 23:30:00 GMT

MPD and beets

author: akeil
date: 2014-03-24
version: 1

MPD 5 (Music Player Daemon) is a music player that runs in the background (as a daemon process). MPD does not have a user interface but requires a separate client application to control playback. One interesting thing about MPD is that it can be controlled over the network. MPD can also stream music over HTTP.

beets 1 is a command line application to manage a music library. It downloads album/track-information and cover art from various web services and manages your music-file's metadata ("tags").

The resulting setup will be a system where you can playback and manage music without the need for a graphical user interface (or any graphics). It can run on your desktop PC and be controlled by a locally installed (GUI-) client or it can run on a different machine and be controlled over the network. When controlled via network, no display is required on the computer running thee beets/MPD combo.

Install beets

The installation procedure is also described in the beets documentation 2.

Install From Repository

Ubuntu/Linux Mint:

# apt-get install beets

Arch Linux:

# pacman -S beets

Install Manually

beets can also be installed manually inside a virtualenv 4. This allows to install it without root privileges (for a single user only) and also keeps the system-wide python installation clean. However, when installed like this, one must update it manually..

Install python-virtualenv if it is not already installed. beets will run with Python 2.7 only, so install that version of Python as well.

On Ubuntu:

# apt-get install python2.7 python2.7-dev python-virtualenv

On Arch Linux

# pacman -S python2 python2-virtualenv

Next, create a virtualenv and install beets:

$ mkdir ~/.virtualenvs
$ cd ~/.virtualenvs
$ virtualenv --python python2.7 beets
$ source beets/bin/activate
(beets)$ pip install beets

Now beets can be executed with the beet command - as long as you have the beets virtualenv activated. To execute outside the virtualenv, link the beet executable into ~/bin (assuming that ~/bin is in your $PATH):

$ ln -s ~/.virtualenvs/beets/bin/beet ~/bin/beet

Updating the virtualenv-install is done like this:

$ source ~/.virtualenvs/beets/bin/activate
(beets)$ pip install --upgrade beets

Plugins

Beets offers several plugins. For some of them, additional packages must be installed.

Some interesting of the many beets plugins 3:

discogs: Use discogs 7 as an additional source for metadata (default is musicbrainz 6).
lyrics: Fetches song lyrics.
echonest: Fetches tempo (bpm) from echonest 11
fetchart: Fetches cover art. Requires ImageMagick or PIL to re-size downloaded art.
embedart: Embeds cover art into the music file. Also requires ImageMagick 8 or PIL 9 to re-size images.
lastgenre: Fetches genre from last.fm 10 and optionally reduces genres to a canonized list.
replaygain: Calculates Replay Gain, using either mp3gain or replaygain.
scrub: Removes unwanted tags from music files.
mbsync: Updates your library with new metadata from MusicBrainz.
missing: Finds missing tracks in your collection.
duplicates: Finds and optionally deletes duplicate albums/tracks.

Plugin requirements:

Plugin	Requirement	Install with
discogs	discogs-client	pip
echonest_tempo	pyechonest	pip
lastgenre	pylast	pip
mpdstats	python-mpd	pip
fetchart	imagemagick	(package manager)
replaygain	mp3gain replaygain	(package manager)

To install the python requirements (assuming that beets was installed in a virtualenv):

(beets)$ pip install discogs-client pyechonest pylast python-mpd

To install the additional packages required by beets or plugins:

# apt-get install mp3gain replaygain imagemagick

A simple update script for the complete beets virtualenv:

#!/bin/sh
VENV="$HOME/.virtualenvs/beets"
$VENV/bin/pip install --upgrade beets discogs-client pyechonest pylast

Configure

The beets config file is ~/.config/beets/config.yaml. The beets config is always user-based (this cannot be changed - however, since version 1.3.3, an additional configuration file can be supplied via the --config option).

Change the base directory where beets stores music files:

directory: /mnt/storage/music/

A complete list of beets configuration options 12 can be found in the beets documentation:

The default config from the installation directory can be used a a starting point for a personal config.

$ cp ~/.virtualenvs/beets/lib/python2.7/site-packages/beets/config_default.yaml \
  ~/.config/beets/config.yaml

Import Settings

Files to be imported will be placed in a special "import" folder, which is intended to contain to-be-imported files only. beets should clean up this directory after import. Settings are:

import:
    move: yes

Use the strong_rec_thresh setting to allow more or less imperfectly matched albums to be imported without prompt:

strong_rec_thresh: 0.1  # match 90% or better for auto-import

Genres

The lastgenre plugin will fetch genres from last.fm Additionally, the last.fm genre is compared against a whitelist, to exclude some of the genres. It is also possible to canonicalize the genres to a more coarser set of genres using a YAML-configuration file.

For both options, default files are included at:

~/.virtualenvs/beets/lib/python2.7/site-packages/beetsplug/lastgenre/genres.txt
~/.virtualenvs/beets/lib/python2.7/site-packages/beetsplug/lastgenre/genres-tree.yaml

Copy these to ~/.config/beets/ and configure beets to use these files. Edit the files as required.

lastgenre:
    whitelist: ~/.config/beets/genres.txt
    canonical: ~/.config/beets/genres-tree.yaml

MPD

Install

Arch Linux:

# pacman -S mpd

Ubuntu/Linux Mint:

# apt-get install mpd

Configure

The system wide config file for MPD is /etc/mpd.conf

relevant settings:

port                "6600"
music_directory     "/var/lib/mpd/music"
playlist_directory  "/var/lib/mpd/playlists"
db_file             "/var/lib/mpd/mpd.db"
state_file          "/var/lib/mpd/state"
sticker_file        "/var/lib/mpd/sticker.sql"
log_file            "/var/log/mpd.log"
pid_file            "/run/mpd/mpd.pid"
user                "mpd"
replaygain          "album"
metadata_to_use     "artist,album,title,track,name,genre,date,disc,albumartist"

To ensure that MPD runs smoothly, create the required files and chown them to the mpd-user.

# mkdir -p /var/lib/mpd/playlists
# touch /var/lib/mpd/{mpd.db,state,sticker.sql}
# chown -R mpd:mpd /var/lib/mpd
# touch /var/log/mpd.log
# chown mpd:mpd /var/log/mpd.log
# mkdir /run/mpd
# chown mpd:mpd /run/mpd
# ln -s /mnt/storage/music /var/lib/mpd

The music_directory is the only directory defined in the config that is not created during setup. Instead we want MPD to use the same directory as beets and symlink the music directory for MPD. Make sure that the mpd user has read permissions on all files as well as read+execute permissions on all directories within the music directory.

Port 6600 is the default for MPD. If possible, leave this unchanged as most clients will expect MPD to listen at this port.

user (and group) control the permissions under which MPD is run. On some systems, MPD must be a member of the audio group in order to use the sound card.

The var/lib/mpd directory should look as follows:

$ ls -l /var/lib/mpd/
total 4
-rw-r--r-- 1 mpd  mpd     0 Nov 23 18:32 mpd.db
lrwxrwxrwx 1 root root   20 Nov 23 18:35 music -> /mnt/storage/music
drwxr-xr-x 2 mpd  mpd  4096 Nov 17 19:26 playlists
-rw-r--r-- 1 mpd  mpd     0 Nov 23 18:32 state
-rw-r--r-- 1 mpd  mpd     0 Nov 23 18:32 sticker.sql

There is an extensive article in the Arch wiki on setting up MPD 13.

Metadata

This config-setting:

metadata_to_use     "artist,album,title,track,name,genre,date,disc,albumartist"

tells MPD which tags to include in its database. You can select tracks by any of these tags. The albumartist is not enabled by default.

If the metadata_to_use setting is changed later, MPD must rebuild its database for the change to take effect. MPD will rebuild the database if you delete (or rename) the existing one. The database is a plain text file; have a look at it.

Stream over HTTP

MPD version 0.15 or up has built in HTTP-streaming.

To enable it, add the following to /etc/mpd.conf:

audio_output {
    type    "httpd"
    name    "HTTP Stream"
    encoder "vorbis"
    port    "8000"
    quality "7.0"
    #bitrate "128"
    format  "44100:16:2"
}

quality settings for for variable bitrate (VBR) range from -1 (poor) to 10 (high). Alternatively, specify a constant bitrate, e.g. "128".

format 44100:16:2 means 44100khz 16 Bit Stereo

The URL for MPD's stream is then http://192.168.1.2:8000 (if 192.168.1.2 is the IP-address of the machine MPD is running on). For example:

$ mplayer http://192.168.1.2:8000

will have mplayer start playing whatever MPD currently plays.

MPD Clients

mpc 17

a minimalist command line client. Good for testing or quick commands.

Ario 14

A desktop client with a GUI.

GMPC 16

The Gnome Music Player Client is another GUI client.

mpDris2 15

Adapts the dbus MPRIS interface for MPD. This makes MPD to occur in (and be controlled from) the sound menu of your desktop environment like "normal" music players.

MPDroid 19

MPD client for Android.

MPDroid has a setting to list tracks by Albumartist instead of (Track-)Artist. The albumartist must be included in MPD's config as metadata_to_use or this setting will not work.

The MPD wiki has a large list of MPD clients 18.

Misc

Automatically Update Metadata

beets metadata is based primarily on MusicBrainz 6, which is permanently updated. To periodically and automatically sync your music collection, run a script like this (e.g. weekly via cron):

#!/bin/bash
$VENV=/home/username/.virtualenvs/beets

# sync metadata
$VENV/bin/beet mbsync > /dev/null

# fetch coverart
$VENV/bin/beet fetchart > /dev/null

Albumart for MPDroid

MPDroid can display the cover of the currently playing song. The cover is not provided by MPD but instead

fetched from the internet (last.fm 10)
provided by a web server, running on the same machine as MPD (see MPDroid wiki 20)

Set up a web server like this

# pacman -S nginx
# systemctl enable nginx

# apt-get install nginx

The relevant nginx-configuration is this:

http {
    server{
        listen 80;
        root /mnt/storage/music;
        location /cover-art/ {
            rewrite /cover-art/(.*) /$1 break;
            try_files $uri $uri
            allow 192.168.1.0/24    # local network
            deny  all               # everybody else
        }
    }
}

Sample Beets Configuration

Complete sample configuration for beets:

beets.config.yaml (Source)

# Beets configuration --------------------------------------------------------

directory: /mnt/storage-1/music

import:
    # write metadata to music files
    write: yes

    # move imported files from source to the music directory
    move: yes
    copy: no
    delete: no

    # use auto-tagging where possible
    # do not require confirmation on strong matches
    autotag: yes
    timid: no

    resume: ask
    incremental: no
    none_rec_action: ask
    log:

    quiet: no  # enable with command line option
    quiet_fallback: skip
    default_action: apply

    singletons: no
    languages: []
    detail: no
    flat: no

# use the release-date of the original (first) release of an album?
original_date: no

# on multi-disk releases, assign track numbers for the whole album.
# If "per disk", make sure tracknames do not collide ("paths" setting).
per_disc_numbering: no

# files matching these patterns are deleted from source after import
clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", ".pls", "*.jpg"]

# files/directories matching one of these patterns are ignored during import
ignore: [".*", "*~", "System Volume Information"]


# Paths ----------------------------------------------------------------------

# Paths and filenames for music files
# relative to music directory
paths:
    default: %asciify{$albumartist}/%asciify{$album}%aunique{}/$track %asciify{$title}
    singleton: Non-Album/%asciify{$artist}/%asciify{$title}
    comp: Compilations/%asciify{$album}%aunique{}/$track %asciify{$title}

# replace special characters in generated filenames
replace:
    '[\\/]': _
    '^\.': _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    '\.$': _
    '\s+$': ''

path_sep_replace: _

# filename for the album art
art_filename: cover  # results in "cover.jpg"

max_filename_length: 0  # unlimited


# General --------------------------------------------------------------------

# use mutliple threads during import
threaded: yes
timeout: 5.0
verbose: no


# User Interface -------------------------------------------------------------

color: yes
list_format_item: %upper{$artist} - $album - $track. $title
list_format_album: %upper{$albumartist} - $album
time_format: '%Y-%m-%d %H:%M:%S'
terminal_encoding: utf8

ui:
    terminal_width: 80
    length_diff_thresh: 10.0


# Auto Tagger ----------------------------------------------------------------

match:
    strong_rec_thresh: 0.1      # match 90% or better for auto import
    medium_rec_thresh: 0.25
    rec_gap_thresh: 0.25
    max_rec:
        missing_tracks: medium
        unmatched_tracks: medium
    distance_weights:
        source: 2.0
        artist: 3.0
        album: 3.0
        media: 1.0
        mediums: 1.0
        year: 1.0
        country: 0.5
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 5.0
        tracks: 2.0
        missing_tracks: 0.9
        unmatched_tracks: 0.6
        track_title: 3.0
        track_artist: 2.0
        track_index: 1.0
        track_length: 2.0
        track_id: 5.0
    preferred:
        countries: []
        media: []
        original_year: no
    ignored: []
    track_length_grace: 10
    track_length_max: 30


# Plugins --------------------------------------------------------------------

plugins: [
    discogs,
    lyrics,
    echonest_tempo,
    fetchart,
    embedart,
    lastgenre,
    scrub,
    mbsync,
    mpdupdate,
    #mpdstats,
    missing,
    duplicates
]
pluginpath: []


# Plugins Config -------------------------------------------------------------

lyrics:
    auto: yes
    fallback: ''

echonest_tempo:
    auto: yes

lastgenre:
    whitelist: ~/.config/beets/genres.txt
    canoncical: ~/.config/beets/genres-tree.yaml
    fallback: ''
    source: artist

fetchart:
    auto: yes
    maxwidth: 300
    cautious: true
    cover_names: cover folder

embedart:
    auto: yes
    maxwidth: 300

replaygain:
    auto: yes
    overwrite: yes
    albumgain: yes

scrub:
    auto: yes

# required for mpdstats
mpd:
    host: localhost
    port: 6600
    password:  # none

mpdstats:
    rating: False
    # two ratings are calculated:
    # "rolling" based on recent development
    # "stable" based on all-time development
    # rating mix 0.0 is all "rolling", 1.0 is all "stable"
    rating_mix: 0.75

missing:
    format: $albumartist - $album - $track - $title
    count: no
    total: no

duplicates:
    checksum: no  # expensive