sett packaging file format specifications

sett compresses, encrypts and packages files in a single .zip file whose specifications are described below. Only files adhering to these specifications can be transferred or decrypted by sett, and failure to comply with the specification will generate an error.

File structure

sett .zip files have the following structure:

YYYYMMDDThhmmss.zip
├── metadata.json
├── metadata.json.sig
└── data.tar.gz.gpg
    └── data.tar.gz
        ├── content/
        |   ├── [file1]
        |   ├── [file2]
        |   └── ...
        └── checksum.sha256
metadata.json

Metadata file containing the following information:

  • transfer_id: unsigned integer indicating the transfer ID, or null.

  • sender: fingerprint of the sender’s public PGP key.

  • recipients: list of fingerprints of the recipients’ public PGP keys.

  • timestamp: datetime of metadata generation. Uses the datetime format YYYY-MM-DDThh:mm:ss±ZZZZ (where ZZZZ is the UTC offset).

  • checksum: sha256 checksum of the encrypted data.tar.gz.gpg file. This allows verifying the integrity of the transferred file without decrypting it, and is used to make sure nothing was corrupted during the transfer process.

  • checksum_algorithm: algorithm used to compute the checksum. Currently sha256.

  • compression_algorithm: algorithm used to compress the inner tarball. The following values are supported:

    • zstandard, for data compressed with zstandard. This is the data encryption algorithm used by default in sett.

    • gzip, for data compressed with gzip.

    • stored, for data that are not compressed, but simply packaged with tar.

  • purpose: one of PRODUCTION, TEST or null.

  • version: the version of the metadata specifications.

metadata.json.sig

Detached PGP signature for the metadata.json file.

data.tar.gz.gpg

A tarball encrypted using the receiver’s public PGP key and signed with the sender’s private key. It is compressed with compression_algorithm.

[file1], [file2]

One or several data files can be transferred. Data to be sent can be in any format, e.g. .txt, .csv, .dat.

checksum.sha256

sha256 checksum file of all files present in data.tar.gz. This is used to make sure nothing was corrupted during the encryption/transfer/decryption process.

Important: the .zip file itself must use the stored compression method, i.e. it should not do any compression of its content. This is because the content of the zip archive (the data.tar.gz.gpg file) is already compressed, so compressing it a second time is unnecessary.

File example

Examples of the content and structure of the metadata and checksum files.

metadata.json
{
    "transfer_id": 42,
    "sender": "AAABFBC698539AB6CE60BDBE8220117C2F906548",
    "recipients": ["D99AD936FC83C9BABDE7C33E1CF8C1A2076818C3"],
    "timestamp": "2020-01-29T15:31:42+0100",
    "checksum": "a8eb0ee5a6a53326b1c6f9bf94136da5d98a1dc6dceee21d62f694d71c4cf184",
    "checksum_algorithm": "SHA256",
    "compression_algorithm": "gzip",
    "purpose": "PRODUCTION",
    "version": "0.7"
}
checksum.sha256
41421f0c4a5353a5a0cdd37de3fd80a840a190ca997ad8044a67c4c1683f7b63 file1.csv
35ba157ed1c3269d731a438c466790a4f481bb49805e2d1f380df0c636792ff6 folder1/file.txt
fd9ebdbcc1a5fc35ded6e78a6b16ef658502c9d0b05dd4a2185d0f94ccf165cf folder1/folder2/file.txt