InfluxDB for TinyCore Linux (ARM)
This is intended as part of a setup for home automation with where InfluxDB is used as a storage backend for statistics.
Tiny Core Extension
$ wget https://dl.influxdata.com/influxdb/releases/influxdb-1.3.2_linux_armhf.tar.gz $ tar -xzf influxdb-1.3.2_linux_armhf.tar.gz
The download URL is different for each version.
The extracted archive creates a directory tree like this:
etc/ influxdb/ # default config logrotate.d/ usr/ bin/ # binaries lib/ share/ var/ lib/ influxdb/ log/ influxdb/
Only a minimal selection of files goes into the TinyCore extension.
Create a temporary directory (
and place selected files there.
The complete destination directory looks like this:
/tmp/influxdb/ usr/ local/ bin/ # binaries etc/ # sample config tce.installed/ # post-install script
From the source
usr/ directory, only the binaries are used. This omits
the man pages (under
share/) and start scripts (under
The InfluxDB distribution contains a default configuration file which is the starting point for a custom configuration.
Reduce cache sizes to play nice with the available memory (see InfluxDB docs ):
Disable usage statistics:
Place the configuration file in the temporary
TinyCore supports a tce.installed script which is run after the extension was loaded. Our post-install script will
if no config file is present, copy the sample config to the default location for the config file.
create the default data directories with correct permissions.
Create a tce.installed script like this:
#!/bin/sh CFG=/usr/local/etc/influxdb/influxdb.conf SAMPLE_CFG=/usr/local/etc/influxdb/influxdb.sample.conf DATADIR=/var/lib/influxdb USER=influxdb # if no config exists, use the sample config if [[ ! -e $CFG ]]; then cp $SAMPLE_CFG $CFG fi # create the influxdb user grep -q $USER /etc/passwd if [[ $? -ne 0 ]]; then adduser -S -H $USER fi # create the default data directory if [[ ! -e $DATADIR ]]; then mkdir $DATADIR chown $USER:nogroup $DATADIR fi
The script is added to the temporary installation dir under
To make the process easily repeatable for new versions of InfluxDB,
The script includes all steps except downloading the distribution files
#!/bin/sh SRC=influxdb-1.3.2-1 ROOT=/tmp/influxdb/usr/local # download and install required tools tce-load -w squashfs-tools tce-load -i squashfs-tools # cleanup sudo rm -rf $ROOT mkdir -p $ROOT/bin cp -a $SRC/usr/bin/* $ROOT/bin mkdir -p $ROOT/etc/influxdb cp influxdb.conf $ROOT/etc/influxdb/influxdb.sample.conf # startup script mkdir -p $ROOT/tce.installed cp tce.installed $ROOT/tce.installed/influxdb # default permissions sudo chown -R root:root $ROOT # special perms for startup script sudo chown root:staff $ROOT/tce.installed sudo chown 775 $ROOT/tce.installed sudo chown root:staff $ROOT/tce.installed/influxdb sudo chmod 755 $ROOT/tce.installed/influxdb # pack and cleanup mksquashfs /tmp/influxdb /tmp/influxdb.tcz sudo rm -rf /tmp/influxdb mv /tmp/influxdb.tcz /etc/sysconfig/tcedir/optional/
We need to persist two locations, using
This will include the configuration file and the data directories
filetool.sh backup (you still need to remember
to execute this before shutdown). That backup is restored during boot.
The default storage location is
on TinyCore, this resides in memory and can be persisted using
Since the data files can become quite large, this is not the best option.
Both backup and restore will take a long time and the data will consume
Whether or not to place any part of the database on volatile storage of course depends on whether or not (partial) data loss is a problem. In this case, it is assumed that this is non-critical data and that it is acceptable if some of the data is lost if the system shuts down unexpectedly.
- WAL - Write Ahead Log
This is stored under
wal-dir=directive from config). New writes go into the WAL, i.e. these files receive many writes and change often.
- TSM - Time Structured Merge Tree
These are read-only files which contain the bulk of the data. They are stored under
dir=directive from config). During a process named Compaction, data from the WAL is periodically written to TSM files. Also during compaction, existing TSM files are optimized, i.e. rewritten.
The TSM files will probably become too large to keep in memory and are therefore
placed on a separate storage
(in this case, additional storage that is mounted under (
The WAL is kept at the default location, in memory. This has the benefit of faster access times and also reduces writes to the RaspberryPi's SD-card. If the WAL is lost, we lose data from the most recent writes.
Start and Stop Scripts
Create two shell scripts to start and stop InfluxDB.
The start script is included in
the stop script in
The start script does the following:
set permissions on pid file
set permissions on log file
influxdwith parameters for pid file and configuration file and redirect stderr to the logfile
The stop script simply reads the process id from the pid file and kills that process.
CONFIG=/usr/local/etc/influxdb/influxdb.conf PIDFILE=/var/run/influxd.pid LOGFILE=/var/log/influxdb.log touch $PIDFILE chown influxdb:nogroup $PIDFILE touch $LOGFILE chown influxdb:staff $LOGFILE su influxdb -s /bin/sh -c "influxd run -pidfile $PIDFILE -config $CONFIG 2>$LOGFILE"
There a several options for backing up InfluxDB. On can backup the full database(s) or only selected retention policies. It is also possible to only back up data since a specific timestamp.
The backup can be run locally or over the network and it can be performed while InfluxDB is running.
For the RaspberryPi, we create full backups, using a local cron job. Backups are created daily and a number of the most recent backups is retained.
The basic steps are as follows: - backup the Metastore and selected Databases to a temporary location - if all went well, compress the result (.tar.gz) - delete the oldest backup
To backup the Metastore:
Each Database is backed up individually by calling
Monitoring with Monit
Check whether the process exists using the
Make sure it is run by the
Call the HTTP interface on the
Check if there is a recent backup
These are the
check process influxd pidfile /var/run/influxd.pid group influxdb if failed uid influxdb then alert if failed port 8086 protocol http request /ping status = 204 then alert check file influxdbbackup path /mnt/storage/influxdb/backups/snapshot.0.tar.gz group backup if does not exist then alert if timestamp > 25 hours then alert