This is intended as part of a setup for home automation with where InfluxDB is used as a storage backend for statistics.
Tiny Core Extension
$ wget https://dl.influxdata.com/influxdb/releases/influxdb-1.3.2_linux_armhf.tar.gz $ tar -xzf influxdb-1.3.2_linux_armhf.tar.gz
The download URL is different for each version.
The extracted archive creates a directory tree like this:
etc/ influxdb/ # default config logrotate.d/ usr/ bin/ # binaries lib/ share/ var/ lib/ influxdb/ log/ influxdb/
Only a minimal selection of files goes into the TinyCore extension. Create a temporary directory (/tmp/influxdb/usr/local) and place selected files there. The complete destination directory looks like this:
/tmp/influxdb/ usr/ local/ bin/ # binaries etc/ # sample config tce.installed/ # post-install script
From the source usr/ directory, only the binaries are used. This omits the man pages (under share/) and start scripts (under lib/).
The InfluxDB distribution contains a default configuration file which is the starting point for a custom configuration.
Reduce cache sizes to play nice with the available memory (see InfluxDB docs ):
Disable usage statistics:
Place the configuration file in the temporary etc directory, naming it influxdb.sample.conf.
TinyCore supports a tce.installed script which is run after the extension was loaded. Our post-install script will
- if no config file is present, copy the sample config to the default location for the config file.
- create the influxdb user.
- create the default data directories with correct permissions.
Create a tce.installed script like this:
#!/bin/sh CFG=/usr/local/etc/influxdb/influxdb.conf SAMPLE_CFG=/usr/local/etc/influxdb/influxdb.sample.conf DATADIR=/var/lib/influxdb USER=influxdb # if no config exists, use the sample config if [[ ! -e $CFG ]]; then cp $SAMPLE_CFG $CFG fi # create the influxdb user grep -q $USER /etc/passwd if [[ $? -ne 0 ]]; then adduser -S -H $USER fi # create the default data directory if [[ ! -e $DATADIR ]]; then mkdir $DATADIR chown $USER:nogroup $DATADIR fi
The script is added to the temporary installation dir under tce.installed/influxdb.
To make the process easily repeatable for new versions of InfluxDB, create a package.sh script. The script includes all steps except downloading the distribution files for InfluxDB.
#!/bin/sh SRC=influxdb-1.3.2-1 ROOT=/tmp/influxdb/usr/local # download and install required tools tce-load -w squashfs-tools tce-load -i squashfs-tools # cleanup sudo rm -rf $ROOT mkdir -p $ROOT/bin cp -a $SRC/usr/bin/* $ROOT/bin mkdir -p $ROOT/etc/influxdb cp influxdb.conf $ROOT/etc/influxdb/influxdb.sample.conf # startup script mkdir -p $ROOT/tce.installed cp tce.installed $ROOT/tce.installed/influxdb # default permissions sudo chown -R root:root $ROOT # special perms for startup script sudo chown root:staff $ROOT/tce.installed sudo chown 775 $ROOT/tce.installed sudo chown root:staff $ROOT/tce.installed/influxdb sudo chmod 755 $ROOT/tce.installed/influxdb # pack and cleanup mksquashfs /tmp/influxdb /tmp/influxdb.tcz sudo rm -rf /tmp/influxdb mv /tmp/influxdb.tcz /etc/sysconfig/tcedir/optional/
We need to persist two locations, using opt/.filetool.lst:
This will include the configuration file and the data directories in the filetool.sh backup (you still need to remember to execute this before shutdown). That backup is restored during boot.
The default storage location is /var/lib/influxdb. on TinyCore, this resides in memory and can be persisted using filetool.sh. Since the data files can become quite large, this is not the best option. Both backup and restore will take a long time and the data will consume available memory.
Whether or not to place any part of the database on volatile storage of course depends on whether or not (partial) data loss is a problem. In this case, it is assumed that this is non-critical data and that it is acceptable if some of the data is lost if the system shuts down unexpectedly.
|WAL - Write Ahead Log:|
|This is stored under /var/lib/influxdb/wal/<database-name> (the wal-dir= directive from config). New writes go into the WAL, i.e. these files receive many writes and change often.|
|TSM - Time Structured Merge Tree:|
|These are read-only files which contain the bulk of the data. They are stored under /var/lib/influxdb/data/<database-name> (the dir= directive from config). During a process named Compaction, data from the WAL is periodically written to TSM files. Also during compaction, existing TSM files are optimized, i.e. rewritten.|
The TSM files will probably become too large to keep in memory and are therefore placed on a separate storage (in this case, additional storage that is mounted under (/mnt/storage).
The WAL is kept at the default location, in memory. This has the benefit of faster access times and also reduces writes to the RaspberryPi's SD-card. If the WAL is lost, we lose data from the most recent writes.
Start and Stop Scripts
Create two shell scripts to start and stop InfluxDB.
The start script is included in /opt/bootlocal.sh, the stop script in /opt/shutdown.sh.
The start script does the following:
- set permissions on pid file
- set permissions on log file
- start influxd with parameters for pid file and configuration file and redirect stderr to the logfile
The stop script simply reads the process id from the pid file and kills that process.
CONFIG=/usr/local/etc/influxdb/influxdb.conf PIDFILE=/var/run/influxd.pid LOGFILE=/var/log/influxdb.log touch $PIDFILE chown influxdb:nogroup $PIDFILE touch $LOGFILE chown influxdb:staff $LOGFILE su influxdb -s /bin/sh -c "influxd run -pidfile $PIDFILE -config $CONFIG 2>$LOGFILE"
There a several options for backing up InfluxDB. On can backup the full database(s) or only selected retention policies. It is also possible to only back up data since a specific timestamp.
The backup can be run locally or over the network and it can be performed while InfluxDB is running.
For the RaspberryPi, we create full backups, using a local cron job. Backups are created daily and a number of the most recent backups is retained.
The basic steps are as follows: - backup the Metastore and selected Databases to a temporary location - if all went well, compress the result (.tar.gz) - delete the oldest backup
To backup the Metastore:
Each Database is backed up individually by calling
Monitoring with Monit
- Check whether the process exists using the pidfile
- Make sure it is run by the influxdb user
- Call the HTTP interface on the /ping endpoint
- Check if there is a recent backup
These are the monit checks:
check process influxd pidfile /var/run/influxd.pid group influxdb if failed uid influxdb then alert if failed port 8086 protocol http request /ping status = 204 then alert check file influxdbbackup path /mnt/storage/influxdb/backups/snapshot.0.tar.gz group backup if does not exist then alert if timestamp > 25 hours then alert