Automating sett

Batch packaging and transferring of files

While sett does not offer “batch” processing natively, you can use pkg-catapult, a companion tool that automates data packaging and transfer.

Specifically, pkg-catapult has a “batch” mode that allows to specify different sets of files to package and transfer via a simple text file input. Please refer to the pkg-catapult documentation for more details.

Note that the term “batch” processing here refers to the packaging and transfer or multiple file sets as independent packages - sett can already natively package multiple files in a single package.

Dealing with passwords when automating sett tasks

A difficulty in automating sett tasks is the handling of passwords that are required during data encryption (password for the private PGP key used to sign packages) and transfer (password for the private SSH key used to connect to the SFTP server where data is being transferred). The following section offers some guidance on how to deal with passwords when automating sett tasks.

Warning

Security warning: automating a workflow that involves secrets (i.e. the need to enter a password) almost always means weakening the security chain. Please consider the following points carefully before starting to automate tasks with sett:

  • Against whom should the data be protected?
  • Is the protection sufficient for that purpose on both ends of the communication channel?

Example 1: Fully automated - unencrypted password file

Given

  • The following connections section in the sett configuration:

    {
      "connections": {
        "area51": {
          "parameters": {
            "host": "area51.org",
            "username": "chucknorris",
            "destination_dir": "upload",
            "pkey": "/home/chucknorris/.ssh/pkey",
            "pkey_password": ""
          },
          "protocol": "sftp"
        }
      }
    }
    
  • An unprotected ⚠ private ssh key /home/chucknorris/.ssh/pkey

  • A gpg key protected with the password contained in a file .pw

Then, the following will automate packaging and upload (and could potentially be put into a cron job):

#!/bin/bash

sett encrypt -r bob@example.com --dtr-id 42 --purpose PRODUCTION -o output ./files --passphrase-cmd "cat .pw"
sett transfer --connection area51 output/*

Example 2: Semi automated

  • Set "pkey_password": null in the above example

  • Use a password protected ssh private key instead (still /home/chucknorris/.ssh/pkey)

  • As above, use a gpg key protected with the password contained in a file .pw

  • Before activating the cron job / the shell script, load the ssh key into the ssh agent (have to repeat this on every login to the machine):

    ssh-add /home/chucknorris/.ssh/pkey
    

Then activate a cron job / invoke the above commands.