InfluxDB for TinyCore Linux (ARM)
Author: | akeil |
---|---|
Date: | 2017-08-12 |
Version: | 1 |
This post shows how to set up InfluxDB [1] on a RasperryPi running TinyCore Linux [2].
Extension
First, cCreate a TinyCore Extension [3] for InfluxDB.
Download and extract the prebuilt binary from the InfluxDB Downloads [4] site:
$ wget https://dl.influxdata.com/influxdb/releases/influxdb-1.3.2_linux_armhf.tar.gz $ tar -xzf influxdb-1.3.2_linux_armhf.tar.gz
The download URL is different for each version.
The extracted archive creates a directory tree like this:
etc/ influxdb/ # default config logrotate.d/ usr/ bin/ # binaries lib/ share/ var/ lib/ influxdb/ log/ influxdb/
We create a temporary install directory (/tmp/influxdb/usr/local) and place selected files there.
Custom Config
The InfluxDB distribution contains a default configuration file which is the starting point for a custom configuration.
Reduce cache sizes to play nice with the available memory (see InfluxDB docs ):
[data] cache-max-memory-size = 20971520 cache-snapshot-memory-size = 10485760
Disable usage statistics:
reporting-disabled = true
Place the configuration file in the temporary install, naming it influxdb.sample.conf.
Binaries
From the usr/ directory, only the binaries are used. This omits the man pages (under share/) and start scripts (under lib/).
Post-Install Script
TincYore supports a tce.installed script which is run after the extension was loaded. Our post-install script will
- if no config file is present, copy the sample config to the default location for the config file.
- create the influxdb user.
- create the default data directories with correct permissions.
Create a tce.installed script like this:
#!/bin/sh CFG=/usr/local/etc/influxdb/influxdb.conf SAMPLE_CFG=/usr/local/etc/influxdb/influxdb.sample.conf DATADIR=/var/lib/influxdb USER=influxdb # if no config exists, use the sample config if [[ ! -e $CFG ]]; then cp $SAMPLE_CFG $CFG fi # create the influxdb user grep -q $USER /etc/passwd if [[ $? -ne 0 ]]; then adduser -S -H $USER fi # create the default data directory if [[ ! -e $DATADIR ]]; then mkdir $DATADIR chown $USER:nogroup $DATADIR fi
The script is added to the temporary installation dir under tce.installed/influxdb.
Package Script
To make the process easily repeatable for new versions of InfluxDB, create a package.sh script. The script includes all steps except downloading the distribution files for InfluxDB.
#!/bin/sh SRC=influxdb-1.3.2-1 ROOT=/tmp/influxdb/usr/local # download and install required tools tce-load -w squashfs-tools tce-load -w compiletc tce-load -i squashfs-tools tce-load -i compiletc # cleanup sudo rm -rf $ROOT mkdir -p $ROOT/bin cp -a $SRC/usr/bin/* $ROOT/bin mkdir -p $ROOT/etc/influxdb cp influxdb.conf $ROOT/etc/influxdb/influxdb.sample.conf # startup script mkdir -p $ROOT/tce.installed cp tce.installed $ROOT/tce.installed/influxdb # default permissions sudo chown -R root:root $ROOT # special perms for startup script sudo chown root:staff $ROOT/tce.installed sudo chown 775 $ROOT/tce.installed sudo chown root:staff $ROOT/tce.installed/influxdb sudo chmod 755 $ROOT/tce.installed/influxdb # pack and cleanup mksquashfs /tmp/influxdb /tmp/influxdb.tcz sudo rm -rf /tmp/influxdb mv /tmp/influxdb.tcz /etc/sysconfig/tcedir/optional/
Start and Stop Scripts
Create start and stop scripts (see below) and place them in usr/local/lib/influxdb/scripts
Note
This does NOT seem to work. Scripts are installed, but calling them from bootlocal.sh gives an error
TODO: checkout start-stop-daemon with init scripts?
mkdir -p $ROOT/lib/influxdb/scripts cp scripts/* $ROOT/lib/influxdb/scripts
The start script does the following:
- set permissions on pid file
- set permissions on log file
- start influxd with parameters for pid file and configuratio file and redirect stderr to the logfile
Since this is not the default location for influxdb, pass the -config option when starting:
influxd -config /usr/local/etc/influxdb/influxdb.conf
The stop script simply reads the process id from the pid file and kills that process.
Configuration
Persistance
We need to persist two locations, using opt/.filetool.lst:
usr/local/etc/influxdb/influxdb.conf var/lib/influxdb
Storage
The default storage location is /var/lib/influxdb. By default, this directory is not persisted and its contents will be lost on reboot. If we add it to the .filetool.lst it will be backed up and restored. However, since some of the data files become quite large, this is not the best option.
Note
Deciding whether to store the InfluxDB on "disk" (SD-Card) or in memory (with backup/restore on boot) depends on the InfluxDB Storage Engine [5].
Influx stores data in read-only files which are organized into Shards. In the default setup, each shard contains data from 7 days. This means a shard will not change once its completly written.
We can keep these shards in memory and maintain a backup on disk. Since the live data will never change, we can replace it with a backup version any time without further considerations.
The write ahead log (WAL) is constantly written to. Several WAL files exist, named 0001.wal, 0002.wal, ... Only the highest number file is in write-use
The shards are stored in the data directory (default: /var/lib/influxdb/data) and may become quite large. We will place these directly on persistent storage and not in memory. Open the config file. In the [data] section, change the dir= directive to a different location (in this case, additional storage that is mounted under (/mnt/storage).
[data] ... dir=/mnt/storage/influxdb/data ...
Backup
Backups are always full backups, so we need to:
- create a snapshot of the current state to a temporary location.
- when successful, rotate snapshots
We will keep the n most recent snapshots.
Backup is split in two parts:
- Metastore
- Databases
For the Metastore:
influxd backup /path/to/backup
Each Database is backed up individually by calling
influxd backup -database <NAME> /path/to/backup
Both can be made while the database is running.
See: https://docs.influxdata.com/influxdb/v1.2/administration/backup_and_restore/
The backup script is scheduled as a cron job running once every day.
Monitoring with Monit
Use the following checks with monit [6]:
- Check whether the process exists using the pidfile
- Make sure it is run by the influxdb user
- Call the HTTP interface on the /ping endpoint
- Check if there is a recent backup
These are the monit checks:
check process influxd pidfile /var/run/influxd.pid group influxdb if failed uid influxdb then alert if failed port 8086 protocol http request /ping status = 204 then alert check directory influxdbbackup path /mnt/storage/influxdb/backups/snapshot.0 group backup if does not exist then alert if timestamp > 25 hours then alert
[1] | https://www.influxdata.com/time-series-platform/influxdb/ |
[2] | http://tinycorelinux.net/ |
[3] | http://wiki.tinycorelinux.net/wiki:creating_extensions |
[4] | https://portal.influxdata.com/downloads#influxdb |
[5] | https://docs.influxdata.com/influxdb/v1.3/concepts/storage_engine/ |
[6] | https://mmonit.com/monit/documentation/monit.html |