Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

archive_sender

archive_sender ships the batches of a timeranger2 archive to a backup, and then removes them. An archive is a directory where retention keeps the files that it purges.

To use it:

  1. Give the retention of the store an archive directory on the filesystem of the store (see Configuration below).

  2. Run one archive_sender for each archive directory. Set archive_directory and transport in its configuration.

  3. To see what waits, run list-pending, or read the stats bytes_pending and oldest_pending_t.

Retention can keep what it purges. tranger2_archive_key_files() does the same purge as tranger2_delete_key_files(), and it leaves the purged files in a batch of an archive directory (design: kernel/c/timeranger2/DESIGN-retention.md §12). What happens to the batches is not the job of the store. It is the job of this yuno.

store ──retention──> archive_directory ──archive_sender──> backup
        (tranger2_archive_key_files)        native ──> archive_keeper (another machine)
                                            rsync  ──> rsync destination

The node opens the connection to the backup. Nothing connects to the node, so a sealed node can also ship.

The archive it reads

<archive_directory>/
    .ready/<mark>                                   one empty file per batch
    <database>/<topic>/<key>/<upto>@<t>/            one batch per purge of a key
        <file_id>.json  <file_id>.md2  ...          the files the purge took
        topic_desc.json  topic_cols.json            the schema of the topic
        manifest.json                               what the batch holds

The two transports

transport selects one of them. They are equal in one point: the yuno removes a batch here only when the backup has all of it.

native: to an archive_keeper

The batches go to an archive_keeper (yunos/c/archive_keeper/) over the C_IOGATE that output_service names: C_CHANNEL → C_PROT_TCP4H → C_TCP, with TLS when the url is tcps://. There is one batch in flight. Its data goes in frames of chunk_size bytes, with at most window frames not yet acknowledged. Thus the yuno never holds a whole file in memory.

The link is mutual TLS. The node shows its client certificate, and the keeper takes a batch only when its node is the name of that certificate.

A frame is a compact json. A data frame adds a NUL byte and then the bytes of the file.

FrameFromFieldsAnswer
beginsendernode, batch (<database>/<topic>/<key>/<batch>), files ([{name, size}], manifest.json last)begin-ack, or refused
datasenderbatch, file (index in files), offset, then NUL and the bytes. The files go in their order, each one whole before the nextdata-ack (offset = the end of the bytes), or refused
endsenderbatch, sha256 (one per file of files, in its order: 64 lowercase hex characters)stored, when every file is whole, has its sha256, and is fsync’ed, or when the keeper already has this batch with this content. Else refused
refusedkeeperbatch, msgthe batch fails here. The sender tries it again after retry_period
{"cmd": "begin", "node": "node-b", "batch": "events/raw_events/DEV42/2026-10-01@1793164943",
 "files": [{"name": "2026-10-01.json", "size": 3512018}, {"name": "2026-10-01.md2", "size": 288000},
           {"name": "topic_cols.json", "size": 612}, {"name": "topic_desc.json", "size": 455},
           {"name": "manifest.json", "size": 301}]}
{"cmd": "end", "batch": "events/raw_events/DEV42/2026-10-01@1793164943",
 "sha256": ["9f2c...", "41d0...", "e7a5...", "0b3e...", "c8f1..."]}

(The sha256 are 64 hex characters each, cut here.) The sender hashes each file while it reads the file for its data frames (sha256_update() of gobj-c). Thus the sha256 cost no additional read, but they are known only at the end. That is why they go in the end and not in the begin.

A link that closes while a batch is in flight is not a failure of the batch. When the link opens, the sender ships the batch again, whole. The keeper removes what it got of it.

rsync

rsync_path runs two times for each batch:

rsync -a --mkpath --protect-args --timeout=<rsync_timeout> <rsync_options...> --exclude=/manifest.json <batch>/ <dest>/<node>/<database>/<topic>/<key>/<batch>/
rsync -a --mkpath --protect-args --timeout=<rsync_timeout> <rsync_options...> <batch>/manifest.json   <dest>/<node>/<database>/<topic>/<key>/<batch>/

When the backup is not there

A batch takes the disk space that its files took in the store, because they are hard links and the store link is gone. Thus an archive that is never shipped is a store that is never purged.

max_archive_bytes (0: no limit) is the limit. Above it, the yuno removes the oldest batches WITHOUT shipping them, but never the one in flight. Each one gets a WARNING (“Batch removed WITHOUT being shipped: the archive is over max_archive_bytes”). The node stays alive, and the backup loses its oldest part, not the newest. The stats bytes_pending and oldest_pending_t show it before it happens.

Configuration

AttributeDefaultMeaning
archive_directory—Absolute. The archive_directory given to tranger2_archive_key_files()
transportnativenative or rsync
node_namethe host nameThe first directory of the batches of this node in the backup. native: the cn (or a DNS name) of the client certificate of the node
output_service__output_side__native: the C_IOGATE to the keeper
chunk_size262144native: bytes of a file in one frame
window4native: data frames not yet acknowledged
rsync_path/usr/bin/rsyncrsync: the program
rsync_destination—rsync: where the batches go
rsync_options[]rsync: more arguments, before the paths
rsync_timeout300rsync: --timeout, seconds with no I/O
max_archive_bytes0Pending bytes above which the yuno drops the oldest batches. 0: no limit. Writable
retry_period60Seconds with no new try after a batch failed. Writable

Native, in the batch configuration of the node:

{
    "global": {
        "archive_sender.archive_directory": "/yuneta/archive",
        "archive_sender.node_name": "node-b",
        "archive_sender.max_archive_bytes": 200000000000,
        "__output_side__.__json_config_variables__": {
            "__output_url__": "tcps://backup.example.com:1999"
        }
    },
    "services": [
        {
            "name": "__output_side__",
            "gclass": "C_IOGATE",
            "autostart": false,
            "autoplay": false,
            "children": [
                {
                    "name": "output",
                    "gclass": "C_CHANNEL",
                    "children": [
                        {
                            "name": "output",
                            "gclass": "C_PROT_TCP4H",
                            "children": [
                                {
                                    "name": "output",
                                    "gclass": "C_TCP",
                                    "kw": {
                                        "url": "(^^__output_url__^^)",
                                        "crypto": {
                                            "library": "(^^__tls_library__^^)",
                                            "ssl_verify_mode": "required",
                                            "ssl_use_system_ca": true,
                                            "ssl_certificate": "/yuneta/store/certs/node-b.crt",
                                            "ssl_certificate_key": "/yuneta/store/certs/private/node-b.key"
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
}

The ssl_certificate of the C_TCP is the client certificate of the node, from the CA of the nodes that the keeper trusts. Its cn must be the node_name, because the keeper refuses the batches of another name. To make one, see the README of archive_keeper.

rsync over ssh:

{
    "global": {
        "archive_sender.archive_directory": "/yuneta/archive",
        "archive_sender.transport": "rsync",
        "archive_sender.rsync_destination": "backup@backup.example.com:/backups",
        "archive_sender.rsync_options": ["-e", "ssh -i /yuneta/store/certs/private/backup_key"]
    }
}

The archive directory must be on the filesystem of the store (the batches are hard links). The retention of the owner of the store names the same directory, for example raw_events_archive_directory in the configuration of an application.

Crash safety

CrashWhat happens
Before the batch is whole (.new-*)Not a batch: the next archive of the key removes it. Its purge did not happen
After the batch, before its markThe scan finds it when the yuno plays
While shipping (native)The keeper removes what it got when the channel closes. The sender ships the batch again
While shipping (rsync)The next rsync completes the copy. A batch with no manifest at the destination is not whole
After the keeper said stored, before the removal hereThe sender ships the batch again, whole. The keeper compares it with the one that it has, and answers stored at its end
Between the batch and the base of the store (SDK)The purge did not happen, and retention archives the files again later. Thus the backup gets them two times. Until that purge, the first batch holds hard links to live files. A late append, a user flag, or a deleted instance changes them after the manifest of the batch. Thus the two copies can differ. Take a file id (manifest.json, files) from the batch with the latest @<t>

Commands

CommandDoes
list-pending limit=100The batches waiting, the oldest first, with t, bytes and failures
rescanScan the archive tree again, and ship what it finds

Stats

batches_pending, bytes_pending, oldest_pending_t, batches_shipped, bytes_shipped, batches_failed, batches_dropped, bytes_dropped.

Traces

The trace levels are batches (found, shipped, failed, dropped, each rsync run) and frames (each frame to and from the keeper):

ycommand -c 'command-yuno id=<id> service=__yuno__ command=set-gclass-trace gclass=C_ARCHIVE_SENDER set=1 level=batches'

Tests

tests/c/archive_shipping/ runs both transports in one process: a native sender and a keeper over tcp://127.0.0.1:7755, and an rsync sender to a local directory. It tests these cases: