Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

timeranger2

Append-only time-series storage for Yuneta. Records are grouped into topics, keyed by a primary id, and indexed for fast access by time range or row id.

Higher-level stores built on top

Core API (selected)

tranger2_startup(...) / tranger2_shutdown(...)
tranger2_open_topic(...) / tranger2_close_topic(...)
tranger2_append_record(...)
tranger2_open_iterator(...) / tranger2_iterator_get_page(...) / tranger2_close_iterator(...)
tranger2_open_list(...) / tranger2_close_list(...)

Persistence semantics

Every call to tranger2_append_record() writes one record to disk, increments the per-key g_rowid (monotonic, never resets) and i_rowid (row position in the .md2 index file).

Limits

timeranger2.h defines two limits for values from outside: a record’s time, and a version.

TRANGER2_MAX_TIME (0x00000FFFFFFFFFFF, 17592186044415) is the largest time that an md2 row keeps, as __t__ and as __tm__. The row keeps 44 bits of the time. The 16 bits above them hold the user flag and the system flag, so tranger2_append_record() writes the flags over the high bits of a larger time. Then the record keeps another time than the name of its file. In seconds the last time is 559444-03-08T09:40:15 UTC. In milliseconds (sf_t_ms, sf_tm_ms), it is 2527-06-23T06:20:44.415 UTC. The time of the tkey is a time too, and a string tkey is a DATE with no year limit (approxidate_careful() reads "@99999999999999 +0000" as that many seconds, times 1000 on an sf_tm_ms topic). tranger2_append_record() keeps a time that does not fit CUT, as it always did, and logs an ERROR since SDK 7.27.0. A caller that takes times from outside asks first, with tranger2_record_tkey_time() for the tkey: C_TRANGER’s add-record (see C_TRANGER), and the treedb’s treedb_create_node() and treedb_update_node() refuse such a record:

BOOL fits = TRUE;
tranger2_record_tkey_time(topic, record, &fits);
if(__t__ < 0 || (uint64_t)__t__ > TRANGER2_MAX_TIME || !fits) {
    gobj_log_warning(gobj, 0,
        "function",     "%s", __FUNCTION__,
        "msgset",       "%s", MSGSET_PARAMETER,
        "msg",          "%s", "a time that a record cannot keep, refused",
        NULL
    );
    return -1;
}

TRANGER2_VERSION_MAX (0x7FFFFFFF, 2147483647) is the largest topic_version or schema_version that a command takes from outside. A version goes up one at a time, and save-schema writes the version plus 1, so a version near the end of a json_int_t overflowed there. The code keeps and compares versions as json_int_t, whatever their size. It is asked by tranger2_topic_var_fault() and by C_TREEDB’s open-treedb and create-topic for a peer’s schema, and a peer’s save-schema is refused when it tries to raise the schema_version or a topic_version past it (since SDK 7.27.0). So a treedb a peer opens AT the maximum cannot save an edit again: give a peer’s schema versions far below it.

# from a peer: refused
command-yuno id=<id> service=treedbs command=create-topic treedb_name=treedb_x topic_name=t2 topic_version=2147483648 cols='{"id":{"header":"Id","type":"string","flag":["persistent"]}}'
# -> -1 <role^name>: cannot create topic 't2' in treedb 'treedb_x': a topic_version that is not an integer from 0 to 2147483647
# a peer's treedb opened at "schema_version": 2147483647, and a column edited in __system__
command-yuno id=<id> service=treedbs command=save-schema treedb_name=treedb_x
# -> -1 <role^name>: cannot save the schema of 'treedb_x': the version 'schema_version' would pass 2147483647, the largest version a command can publish

Filesystem watcher

src/fs_watcher.c implements an inotify-based watcher (fs_event_t) used by root-linux/C_FS and by the stores themselves to react to on-disk changes. See the fs_watcher page in the sidebar.

CLI companions

ToolPurpose
utils/c/tr2listList records in a topic
utils/c/tr2keysList keys in a topic
utils/c/tr2searchSearch by filter
utils/c/tr2migrateMigrate from legacy timeranger v1
utils/c/list_queue_msgs2List a tr_queue queue
utils/c/msg2db_listList a tr_msg2db topic
utils/c/treedb_listList TreeDB nodes

Tests

tests/c/timeranger2, tests/c/tr_dt_unknown, tests/c/tr_msg, tests/c/tr_msg2db, tests/c/tr_queue, tests/c/tr_treedb and the other tests/c/tr_treedb_* directories (tr_treedb_load_failed, tr_treedb_failed_save, tr_treedb_link_events, tr_treedb_files, tr_treedb_hook_hygiene, ...), and, through C_TREEDB, tests/c/c_treedb_literal_wins. The test suite lists them, and each directory has a README.md.

Source code

Function reference

tranger2_append_record()

Appends a new record to a topic in the TimeRanger database. Master-only. If __t__ is zero a timestamp is assigned (milliseconds when the topic is sf_t_ms, else seconds). The new record’s metadata is returned through the required md_record_ex out-parameter — the return value is a status code, not the metadata. jn_record is owned (consumed, even on error).

int tranger2_append_record(
    json_t *tranger,
    const char *topic_name,
    uint64_t __t__,
    uint16_t user_flag,
    md2_record_ex_t *md_record_ex,
    json_t *jn_record
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic where the record will be appended.
__t__uint64_tTimestamp of the record. If set to 0, the current time is used.
user_flaguint16_tUser-defined flag associated with the record.
md_record_exmd2_record_ex_t *Pointer to the metadata structure of the record. This field is required.
jn_recordjson_t *JSON object containing the record data. Ownership is transferred to the function.

Returns

Returns 0 on success, or a negative value on failure.

Notes

The function makes sure that the record is appended to the specified topic in tranger2_startup(). If the topic does not exist, it must be created using tranger2_create_topic() before calling this function.

A time past the 44 bits of an md2 row is kept CUT, and said. The row keeps 44 bits of __t__ and of __tm__ (TRANGER2_MAX_TIME). The flags use the other bits. A __t__, or a time of the tkey (tranger2_record_tkey_time()), that does not fit is appended cut, as it always was, and logged as an ERROR (without a stack, since SDK 7.27.0, where before it was cut without a word): “Record time past what an md2 row keeps (44 bits): appended CUT”, with topic, key, __t__, __tm__ and tm_wrapped (the multiplication by 1000 of a date of an sf_tm_ms topic passed 64 bits). The append returns 0. A caller that takes the time from a peer refuses the record before the append:

/*  topic `raw_t`, tkey `when`, sf_string_key  */
tranger2_append_record(tranger, "raw_t", 0, 0, &md,
    json_pack("{s:s, s:s}", "id", "r1", "when", "@99999999999999 +0000"));
/*  0: appended, its __tm__ cut; ERROR "Record time past what an md2 row keeps (44 bits): appended CUT"  */

A tkey that is a text is read by tranger2_time_of_json_text(), the rule of a text time bound: "" is no tkey value (0), the text of a JSON integer is that integer in the unit of tm, a text written as another number (" 5", "5 ", "10.5", "007", "1,5", "5 5", "1,5 days ago") is refused, a string with a NUL inside ("5\u0000junk") is refused, and any other text is a date. A text that is refused is refused with a WARNING (the text is from a peer or a user) and nothing written: “Cannot append record, its tkey is a text that is no date”, with why, topic, tkey and the first 64 bytes of the text in value. The append returns -1, and the trace shows the first 256 bytes of the record, not all of it. The same in tranger2_append_record_text(). Up to SDK 7.29.3 approxidate() read a text that is no date as NOW, in silence, and "5" as the 5th of this month. On an sf_tm_ms topic, "1761000000000" was stored as NOW times 1000 (up to 7.29.2 as the year 57773, cut). Now it is the integer 1761000000000, as a json integer tkey is.

/*  topic `raw_tm_ms`, tkey `tm`, sf_int_key|sf_tm_ms  */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
    json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "garbage"));
/*  -1, nothing written; WARNING "Cannot append record, its tkey is a text that is no date",
    why "a text that is no date"  */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
    json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "10.5"));
/*  -1, nothing written; the same WARNING, why "a number that is not the text of a JSON integer"  */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
    json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "1761000000000"));
/*  0: md.__tm__ 1761000000000  */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
    json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "2000-01-03T12:00:00+0000"));
/*  0: md.__tm__ 946900800000  */

Reading the metadata back. The metadata of the new record is returned in md_record_ex, and that is where a caller reads it: md.g_rowid is the key’s global rowid (the one that never resets), md.rowid the row’s position inside its .md2 file (i_rowid), then __t__, __tm__, __offset__, __size__, system_flag, user_flag. Hand the function your only reference:

md2_record_ex_t md;
if(tranger2_append_record(tranger, "topic", 0, 0, &md, record) == 0) {
    json_int_t g_rowid = (json_int_t)md.g_rowid;
}

A string of a record can hold NUL characters. json_dumps() writes each one as the escape \u0000, and a read hands it back (the content is loaded with JSON_ALLOW_NUL). A consumer that reads the string with kw_get_str() gets the C string up to its first NUL, but json_string_length() gives the whole length:

json_t *rec = json_pack("{s:s, s:s#}", "id", "A", "blob", "a\0b", 3);
tranger2_append_record(tranger, "topic", 0, 0, &md, rec);
/*  a load hands {"id": "A", "blob": "a\u0000b"}: json_string_length() is 3  */

A content that is not json when it is read logs “Bad data, the content of the record is not json” (a CRITICAL, with topic, key, __t__, __offset__, __size__, the jansson error and its position), and the record is not handed. A content that this process has not the memory to parse logs “Cannot read the record, this process has not the memory to parse its content (MEM_MAX_BLOCK)” (the same fields, and the jansson error is “not enough memory”), and the record is not handed either. The largest memory block (gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and a content smaller than that block can need more to parse: jansson doubles the buffer of a string as it reads it, and the table of an array or an object. A string of 40 000 bytes does not parse with a block of 64 KiB, nor does an array of 10 000 numbers. A yuno with a larger block reads the record. Up to 7.25.4 such a string crashed the process (jansson wrote past its buffer, see the lexer fix of Yuneta’s jansson in kernel/c/linux-ext-libs/HACKS.md).

The function does not add a __md_tranger__ to the record for the caller. It adds one (g_rowid, i_rowid, t, tm, offset, size, system_flag, user_flag, the same dict a load attaches) only to the record it hands to a realtime list that takes it: one that wants the key and is not an only_md feed (an only_md feed is handed NULL). A list open on another key does not make the append build it. Nor is a __md_tranger__ ever stored: one the record carries in, from an earlier append or from a load, is the metadata of THAT record and is dropped before the content is written.

The record is owned by the function, and these changes are made to the object itself, not to a copy: a list hands it on without copying it, on the hot path of every append. So a caller that keeps a reference of its own sees them, and reads the metadata from md_record_ex anyway:

json_incref(record);                    // the caller keeps one
tranger2_append_record(tranger, "topic", 0, 0, &md, record);
// No list takes the key:   record has no __md_tranger__ (a carried one is gone)
// A list takes the record: record HAS the list's fresh __md_tranger__
// On an sf_rowid_key topic: record also has "__rowid__"
json_decref(record);

A __t__ that belongs to an earlier file. The record goes to the file that its __t__ selects (through the topic’s filename_mask), even if that file is not the newest one. A key’s global rowid counts the records in the order of its files, so the new record takes its place in that order, and every record of the later files moves one place up:

/*  filename_mask "%Y-%m-%d"; key 1 has A (2000-01-01), B (2000-01-02)  */
tranger2_append_record(tranger, "topic", t_2000_01_01 + 10, 0, &md, jn_c);
/*  C gets g_rowid 2, and B is now 3: an iterator serves A, C, B  */

The g_rowid in md_record_ex is its place, the same number a reload gives (and stores in the loaded record’s __md_tranger__). If you keep global rowids, for example as a page position, a record written into an earlier file makes the rowids of the later records stale. Records written with __t__ = 0 (now) always go to the last file, and move nothing. Until 2026-09-15 the cache in memory disagreed with the disk until the next reload: it served the wrong record, and a follower re-published the whole file.

An append that fails is not half written. The content (.json) is written first and the md2 row after. When the md2 cannot be opened or created, sought or written, the function returns -1 and cuts the content file back to where the record began (a row written in part is cut from the .md2 too): the file keeps no record that no row names. If the cut fails as well, it logs “Cannot cut back the content of an append whose md2 row was not written: the content file keeps a record no row names”. A kill or a power cut between the two writes still leaves that shape. The next open ignores it with a warning (see After a restart too, under tranger2_open_iterator()). Up to 7.25.4 the content was left behind.

The cut comes BEFORE the critical that reports the failure. That matters with the exit bit of the tranger’s on_critical_error (LOG_OPT_EXIT_ZERO, 2, the default of C_TRANGER, and what C_TREEDB passes with its exit_on_error): gobj_log_critical() then ends the process inside the log call, and nothing after it runs. When the md2 cannot be opened or created, the cause is logged without leaving, the content is cut back, and then one more critical exits:

CRITICAL create_file: Cannot create json file  path=<store>/items/keys/A/2000-01-06.md2
CRITICAL get_topic_wr_fd: Cannot open file to write  path=<store>/items/keys/A/2000-01-06.md2
CRITICAL tranger2_append_record: Cannot append record, its md2 file cannot be opened: its content
         was cut back  topic=items key=A file_id=2000-01-06          <- exit(0) here, with 2

The same order holds for a content write that fails part way: its part is cut back before the critical.

A write that stops part way (the file size limit, a full disk) is not said as a write FAILED with an errno: write() returned a count, and errno says nothing then. It is said as a SHORT write, with the bytes written and the bytes expected. A read that returns fewer bytes than a row is said as a SHORT read the same way:

CRITICAL tranger2_append_record: Cannot append record, short write of its content: the file size
         limit or the disk is full  topic=items written=13 expected=45
CRITICAL tranger2_append_record: Cannot save record metadata, short write: the file size limit or
         the disk is full  topic=items written=13 expected=32
CRITICAL read_md2_row: Cannot read a record of md2 file, short read
         path=<store>/items/keys/A/2000-01-02.md2 row=last offset=64 read=16 expected=32

The same holds for the files that are replaced whole (topic_var.json, topic_cols.json) and for the zero wipe of tranger2_delete_instance():

ERROR    replace_json_file: Cannot replace topic_var.json, short write: the file size limit or
         the disk is full  path=<store>/items/topic_var.json
CRITICAL tranger2_delete_instance: Cannot zero the payload, short write: the file size limit or
         the disk is full  topic=items key=A remaining=53 written=4 expected=53

A row always goes on a row boundary. When the md2 row of a failed append was written in part and the cut of that part failed too (“Cannot cut back the md2 of an append whose row was not written whole: its size is not a whole number of rows”), the md2 ends in a torn row. The next append cuts the md2 back to its whole rows first, with the same warning the open gives (see A .md2 whose last row is torn under tranger2_open_iterator()), and writes its row there:

WARNING tranger2_append_record: md2 file of the key ends in a part of a row: an append that was
        never acknowledged was cut back  topic=items key=A file_id=2000-01-02
        old_size=141 new_size=128

If that cut fails, the append is refused with -1 and its content is cut back (“Cannot append record, its md2 file ends in a part of a row that cannot be cut back: the append is refused”). The whole rows stay readable, and the next append tries the cut again. The append cuts only a tail that is a torn row after good rows, the same check as the open (see tranger2_open_iterator()). For any other tail it does not cut: it is refused with -1, its content is cut back, the md2 does not change, and the file is flagged as the open flags it. Every load of the key in the same process says load_failed, and the next append into the file counts it again first, so it is refused too (see below) until the file is repaired. The logs, after the CRITICAL that names the shape:

ERROR    flag_file_unreadable_at_append: md2 file of the key flagged unreadable at
         an append: every load of the key says load_failed
         topic=items key=A file_id=2000-01-02
CRITICAL tranger2_append_record: Cannot append record, its md2 file ends in a part
         of a row that must not be cut back: the append is refused
         topic=items key=A file_id=2000-01-02 md2_size=161

The file is flagged BEFORE that CRITICAL. With the exit bit of on_critical_error (the default, see above) the yuno ends inside the log call, and the next start reads the file again and flags it at its open. So the flag in memory matters only when on_critical_error does not exit.

When the check cannot RUN (the .json cannot be opened, for example with EMFILE or ENFILE, or cannot be read, or there is no memory), it found nothing wrong with the file. The append is refused with -1 and its content is cut back, but the file is NOT flagged: the whole rows stay readable, and the next append checks again. With the default on_critical_error the refusal’s CRITICAL exits the yuno, and the next start checks the file at its open (and flags it when the check still cannot run). The CRITICAL of the check comes first (one of the three listed in tranger2_open_iterator()), then the refusal:

CRITICAL check_torn_md2_tail: Cannot check the torn tail of a md2 file, its content file
         cannot be read: not cut  topic=items key=A file_id=2000-01-02
         path=<store>/items/keys/A/2000-01-02.json errno=24 serrno="Too many open files"
CRITICAL tranger2_append_record: Cannot append record, the torn tail of its md2 file
         cannot be checked now: the append is refused, the file is not flagged
         topic=items key=A file_id=2000-01-02 md2_size=161

Up to 7.25.4 nothing cut a torn row back: an append wrote its row after the torn bytes, and the cache build left the whole file out of the key, so no read found that row.

An append into a file flagged unreadable (a .md2 that the cache build was not able to count, see the same section) counts the file again first. If it can be read now, it gets its cell, the flag of that file goes (“md2 file of the key readable again: it is counted, and the key is not flagged for it”), and the append goes on. If not, the append is refused with -1 and writes nothing (“Cannot append record, its file is flagged unreadable: its row would follow rows no cell counts”). Without this refusal, its row is counted as the file’s first, and a load reads an OLD row of the file in its place.

/*  key A: the md2 of day 2 could not be opened at the open (mode 0000)     */
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec1);   // -1, nothing written
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec2);   // 0: day 2 counted,
                                                                  // A loads whole again

The file counted again gets its cell in the MIDDLE of the key, and every global rowid after it moves up by the rows of the file. The iterators open on the key follow: an unfiltered one takes its segments again at its next page. A FILTERED one (a paging iterator with an index, see tranger2_iterator_get_page()) takes its segments and its index again at once, as a new open builds them now. So its index gains the rows of the file, with their new rowids. Like every append after the open of a filtered iterator, the append that caused the count is not in its index. For example:

/*  key A: v1 (day 1), v2 (day 2, flagged), v3 (day 3), v4 (day 4)          */
it = tranger2_open_iterator(tranger, "topic", "A",
        json_pack("{s:I}", "from_t", (json_int_t)t_day1), NULL, "pager", "me", NULL, NULL);
/*  its page:  total_rows 3, v1 v3 v4                                         */
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec_v9); // counts day 2
/*  its page:  total_rows 4, v1 v2 v3 v4                                     */

tranger2_append_record_text()

Appends a record whose content is already json TEXT, for example a frame as a peer sent it. The text is written as it is: the function does not parse the whole record and does not dump it again. Master-only. For a record of about 1 KB, this removes the parse, the free and the dump that tranger2_append_record() does with a json: about 18 µs per record on a Xeon E-2386G.

The store reads what it needs from the text with json_scan_object_members(). It makes sure that the text is one json object that a read of the record can parse (json_loadb() with JSON_ALLOW_NUL). It takes the pkey and the tkey from the text, as tranger2_append_record() takes them from a json. A text that it refuses is not written. Thus a topic cannot hold a record that nobody can read back.

int tranger2_append_record_text(
    json_t *tranger,
    const char *topic_name,
    uint64_t __t__,
    uint16_t user_flag,
    md2_record_ex_t *md_record_ex,
    const char *content,
    size_t content_len
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic. The topic must be keyed by a string (sf_string_key) or by an integer (sf_int_key).
__t__uint64_tTimestamp of the record. If set to 0, the current time is used.
user_flaguint16_tUser-defined flag associated with the record.
md_record_exmd2_record_ex_t *Required. The function writes the metadata of the new record here.
contentconst char *content_len bytes of json text, followed by a NUL: content[content_len] must be 0. Not owned, not modified.
content_lensize_tThe number of bytes of the text, without the NUL.

Returns

0 on success, -1 on error. Every error is logged.

Example

/*
 *  topic "raw_events": pkey "id" (sf_string_key), tkey "tm"
 */
const char text[] = "{\"id\":\"DEV-1\",\"tm\":1700000001,\"m\":{\"v\":229.0}}";
md2_record_ex_t md;
if(tranger2_append_record_text(tranger, "raw_events", 0, 0, &md, text, sizeof(text) - 1) < 0) {
    // Error already logged: nothing was written
    return -1;
}
/*  md.g_rowid: the key's global rowid; md.__tm__ is 1700000001  */

Notes

Everything else is the same as in tranger2_append_record(): the checks of the key (not empty, at most NAME_MAX, not a path), the write (one content and one md2 row, and the same cut back of a failed write), the cache and the realtime lists. When a realtime list wants the record, the function parses the text one time for all the lists.

These texts are refused with an ERROR, and nothing is written:

The text keeps its spaces and its escapes. A read gets back the same json, not the bytes that json_dumps() writes.


tranger2_record_tkey_time()

The time that tranger2_append_record() keeps as the __tm__ of record in topic, and whether an md2 row keeps it whole. The append and the callers that check a record first share this one computation, so a check cannot disagree with what is written.

uint64_t tranger2_record_tkey_time(
    json_t *topic,      // NOT owned
    json_t *record,     // NOT owned
    BOOL   *fits
);

Parameters

KeyTypeDescription
topicjson_t *The json of an open topic (tranger2_topic()): its tkey and system_flag are read. Not owned.
recordjson_t *The record about to be appended. Not owned, not changed.
fitsBOOL *Set to FALSE when an md2 row cannot keep the time: negative, past TRANGER2_MAX_TIME, a date of an sf_tm_ms topic whose multiplication by 1000 passes the 44 bits, or a string that tranger2_time_of_text() refuses: one that is no date, or one written as a number that is not the text of a JSON integer (the return is then 0, and up to SDK 7.29.3 a text that is no date was NOW and fitted). TRUE otherwise.

Returns

The time the append keeps: the value of the tkey -- an integer as it is, a string read by tranger2_time_of_text() (the text of a JSON integer as that integer, a date multiplied by 1000 on an sf_tm_ms topic) --, or 0 when the topic has no tkey or the record no value of it (a value of another type, and "", are 0 too). A negative integer comes back as a large unsigned value, and *fits is FALSE.

Notes

The append keeps a time that does not fit CUT, and logs it as an ERROR, but it does not refuse it. A caller that takes the record from a peer asks this first and refuses the record: C_TRANGER’s add-record, and the treedb before a create or a saved update of a node (since SDK 7.27.0).

Example

json_t *topic = tranger2_topic(tranger, "events");     // tkey "when", sf_tm_ms
BOOL fits;

json_t *rec = json_pack("{s:s}", "when", "@1700000000 +0000");
uint64_t tm = tranger2_record_tkey_time(topic, rec, &fits); // 1700000000000, fits TRUE
json_decref(rec);

rec = json_pack("{s:s}", "when", "@20000000000 +0000");
tranger2_record_tkey_time(topic, rec, &fits);               // fits FALSE: 2e13 ms is past 44 bits
json_decref(rec);

rec = json_pack("{s:I}", "when", (json_int_t)-1);
tranger2_record_tkey_time(topic, rec, &fits);               // fits FALSE: negative
json_decref(rec);

/*  the check of a peer's record (yours), before the append  */
tranger2_record_tkey_time(topic, record, &fits);
if(!fits) {
    gobj_log_warning(gobj, 0,
        "function",     "%s", __FUNCTION__,
        "msgset",       "%s", MSGSET_PARAMETER,
        "msg",          "%s", "a time in its tkey that a record cannot keep, refused",
        NULL
    );
    JSON_DECREF(record)
    return -1;
}
tranger2_append_record(tranger, "events", 0, 0, &md, record);   // owns record

tranger2_time_of_text()

The time a TEXT says, read by the one rule that timeranger2 applies to a text time bound of a match_cond (from_t, to_t, from_tm, to_tm) and to a text tkey. C_TRANGER reads the before of delete-key-files and archive-key-files with it too. The rule for a number is the command parser’s for an integer.

int tranger2_time_of_text(
    const char *text,
    BOOL in_ms,
    BOOL end_of_second,
    json_int_t *value,
    const char **why
);

Parameters

KeyTypeDescription
textconst char *The text. NULL is read as "".
in_msBOOLThe axis is in milliseconds (t of an sf_t_ms topic, tm of an sf_tm_ms one): a date is multiplied by 1000. An integer is never converted: it is already in the unit of its axis.
end_of_secondBOOLAn end bound: a date on an axis in milliseconds takes the last millisecond of its second (+999).
valuejson_int_t *The time read. 0 when the text is refused.
whyconst char **When not NULL, the cause of a refusal (NULL when the text is read).

Returns

ReturnText
0, *value 0"": no time (no bound, no tkey value).
0, *value the integerThe text of a JSON integer, `-?(0
-1, “a time text longer than 256 bytes”Any other text of more than 256 bytes (also as a json string, in tranger2_time_of_json_text()). No date needs more, and the reader takes time that grows with the length of a text that can be from a peer: a 140 KB text of "at", comments and "the" took it seconds. The text of a JSON integer is not of these: of any length it is an integer, and past INT64_MAX it is -2 (below).
-1, “a number that is not the text of a JSON integer”A text written as another number: " 5", "5 ", "007", "-007", "+5", ".5", "5.", "10.5", "1E5", and with a decimal comma, "1,5", "10,5", ",5", "5,". It is never read as a date.
-1, “a comma between digits, not a date”A comma between two digits among other words: "1,5 days ago", "1,5days ago", "2026,10,01", "10:00,5", "1,5:29", "yesterday 1,2026". approxidate() reads them as September 26 (1 and 5 days ago), January 10 of 2026, the 5th, the 1st at 05:29, September 30. Three commas of a date are kept: (1) the decimal comma of the seconds of a clock time, ISO 8601’s ("2026-10-01T10:00:00,500Z", "10:00:00,5"). (2) A comma that closes a date before its hour, a clock time or an hour with am/pm: "2026-10-01,10:00", "1 Oct 2026,10:00", "01 10 2026,10:00", "Oct 1,3 pm". What is before it must be a date, or the comma is refused: "1 2,10:00", "12 2026,10:00", "2026,10:00", read as February 1, October 12 and the year 2026. (3) A comma between a day (1 or 2 digits) and a year (4 or 2 digits) after the name of a month: "Oct 1,2026", "Oct 1,26". Not when the number after the comma is a count: "Oct 1,50 days ago", as "1,5 days ago" is refused.
-1, “numbers separated by blanks or commas, not a date”Words that are all digits: "5 5", "10 5", "2026 10 01", "2026, 10, 01", ", 5", "10 2026", "5_5", or two or more of them in a row among other words: "5 5 ago", "1 5 days ago", "2026 10 01 10:00", "12 2026 10:00", "yesterday 1 2026", "2026_10_01", "5 the 5". approxidate() reads "5 5" as May 5, "2026 10 01" as January 10 of 2026, ", 5" as the 5th of this month, "1 5 days ago" as September 26, "12 2026 10:00" as October 12 and "yesterday 1 2026" as September 30. The words are parted as approxidate_careful() parts them: by blanks, commas, _, and a . that is not between two digits ("Oct.. 1 2026", "3 pm.", but "01.10.2026" is one word), where letters and digits touch ("Oct1" is "Oct 1", "1 2thu" is "1 2 thu", "5the5" is "5 the 5"), but for an ordinal ("5th") and the T of ISO 8601 ("2026-10-01T10:00", "20261001T1000"), and by the - or / of a date with a month name ("Oct-1-5" is "Oct 1 5", "01/Oct/2026" is "01 Oct 2026"). A comment ("(CEST)") is passed over whole, as the reader passes over it, also between a count and its am/pm or unit ("Oct 1 5 (CEST) pm" is 17:00 CEST, and up to 7.29.3 the reader read 2005-10-01). ago, at, on, the, of and a weekday are passed over, as it passes over them: they part no digits ("5 the 5" is "5 5", May 5) and no month from its day ("march the 5 2026" is March 5). A run of digits is a date only as a person writes one: day month year with a year of 4 digits ("01 10 2026", "01 10 2026 10:00", but with numbers alone a year of 2 digits is refused, because approxidate() reads "1 2 34" as February 1, 2034), or after the name of a month a day, or a day and a year of 4 or 2 digits ("Oct 1", "1 Oct 2026", "Oct 1 2026", "Oct 1, 2026", "Oct 1 26"). A month name is 3 letters or more of its English name, in any case ("Oct", "Sept.", "october"), as approxidate_careful() reads one. Any other word before the run ("yesterday", a weekday) does not make it a date. After a day and a month name, in either order, only a year of 4 or 2 digits can follow ("1 Oct 2026", "Oct 1st 26", "Thu Oct 1 10:00:00 2026"). "1 Oct 5", "01-Oct 5", "Oct 1st 2" and "Oct 1 10:00 5" are refused (read as 2005 and 2002). An ordinal ("1st", "5th") is a day: after it, with no month name, only a month and a year of 4 digits ("1st 10 2026"). "5th 2026" (the 5th of a month the text does not say), "5th 2" (February 5) and "1st 10" are refused. A number alone is no hour: "Oct 29 10" is 2010-10-29, a year of 2 digits as in "Oct 1 26". After at (or at the), a person means a time, and the reader refuses a number alone there, of any value ("Oct 1 at 10", "at 3", "Oct 1 at 24", "Oct 1 at the 10"), a count back ("at 5 days", "at 2 fridays ago", "at -1d", "at last friday") and a number with dots and no year of 4 digits ("at 10.30", "at 10.30.00"), “a text that is no date”. Up to 7.29.3, "Oct 1 at 24" was 2024-10-01, "at 10.30" October 30 and "at 10.30.00" 2000-10-30. "Oct 1 at 10:00", "Oct 1 at 10 pm", "at 15h" and "at 10h30" (15:00 and 10:30, as Spanish and French write them, and up to 7.29.3 15 hours back, and 10 hours back from the 30th), "at 5 Oct 2026" and "at 5 of October" (a day before a month name) are dates. The reader takes ago only right after its count: "1 day 5 ago" is no date (it was October 4). A number that a unit, am/pm or a weekday follows is a count ("5 days", "3 pm", "3pm", "2 fridays"): it ends a run and is not in it, and the run before it must be a date, even of one number. "Oct 1 2026 3 pm", "01 10 2026 3 pm", "Oct 29 3 PM", "01 10 2026 5 days ago" are dates. "12 10 pm" (read as the 12th at 22:00, where 12:10 pm was meant), "5 5 pm", "5 at 3 pm", "2026 3 pm" and "1 5days ago" are refused. After a month, a day or a year, a number before a weekday is that date’s, not a count: "Oct 1 2026 Thu" is October 1. So is a year (4 digits, 1970-9999) before any weekday: "Oct 1 2026 thursdays" is no date, and "2026 Thu", a weekday with no day, is refused by the reader, as "monday 10:00" is (see approxidate_careful()). The weekday is not checked against a date: "Oct 5 2026 friday" is Monday, October 5. Beside a day counted from today, the reader refuses a weekday that is not that day’s ("now Mon" on a Wednesday: the day the text counted, in the zone said or the UTC offset of now), and a second weekday ("Mon Thu now"). A 12 that noon or midnight follows is their hour, right after at or with no date before it, as the reader reads it, and it ends a run too: "Oct 1 2026 at 12 noon" is a date, "2026 12 noon" is refused. "1700000000 +0000" (git’s raw date), "01.10.2026" and epoch seconds alone with a comment ("1761000000 (UTC)", "(EST) 1761000000", 9 digits or more and no leading zero, refused for a while) are not of these shapes, and are dates. Without a comment, a blank or a mark about them leaves a number: ", 1761000000" and "_1761000000" are refused, as " 1761000000" is.
0, *value the dateAny other text that approxidate_careful() reads: "2000-01-02T00:00:00+0000", "2000-01-02 00:00:00 +0000", "yesterday".
-1, “a text that is no date”Any other text: "garbage", "0x10".
-2, “an integer past what a time holds” / “a date past what a time holds”An integer past INT64_MAX, of any length ("9223372036854775808", 300 nines, and a development version answered -1 “a time text longer than 256 bytes” from 257 digits on, and a tkey of them was refused as “a text that is no date”), or a date that does not fit a json_int_t in its unit.

Notes

Nothing is logged: the text is from a peer or a user, and the caller says it. Up to SDK 7.29.3 a bound with no T was cut by strtoll() (" 5" was 5, "10.5" 10, "2026-10-01 00:00:00" 2026) and a tkey was read by approxidate() ("5" the 5th of this month, a text that is no date NOW). Unreleased development versions after it read every text that was not -?[0-9]+ as a date: " 5" was the 5th, "10.5" May 10, in silence. A decimal comma was too (approxidate() reads "1,5" as May 1, "10,5" as May 10), and digits in groups ("5 5" May 5, "2026 10 01" January 10 of 2026). Later a word more kept the misreading ("1,5 days ago", "5 5 ago", "2026 10 01 10:00", ",5"), and "01 10 2026", a date read right, was refused. Then an hour with am/pm after a date ("Oct 1 2026 3 pm"), the decimal comma of the seconds ("2026-10-01T10:00:00,500Z"), a comma before a clock time ("2026-10-01,10:00") and a two-digit year after a month name ("Oct 1 26") were refused, and a day and a year after any word were a date ("yesterday 1 2026" September 30).

A dot between digits cannot be refused by its shape, as a comma is: it is how a day and a month are written ("1.10 2026" and "01.10.2026" are October 1). Only a text that is a number and nothing else ("10.5", ".5") is refused. Any other dot is for approxidate_careful() to judge.

Since the release after SDK 7.29.3 (BREAKING) a comma before an hour is kept only when it closes a date, and a lone number before an hour with am/pm or a unit is refused: up to 7.29.3 "1 2,10:00" was February 1, "12 2026,10:00" October 12, "12 10 pm" the 12th at 22:00 and "1 5days ago" September 26. The words approxidate_careful() passes over ("Oct.. 1 2026", "march on 5 2026", "Oct 1 2026 3 pm.") are dates, as they were in 7.29.3. So is an ordinal day with its year and an hour ("Oct 1st 2026 3 pm", "October 1st, 2026 at 3 pm"). Letters and digits that touch are parted as the reader parts them: up to 7.29.3 "5the5" was May 5, "2026on10on01" January 10 and "Oct-1 5" 2005-10-01, and a development version read "1 2thu" as February 1 and "Oct1 5 friday" as 2005-10-01. A day and a month name take only a year of 4 or 2 digits, and an ordinal with no month name only a month and a year: up to 7.29.3 "1 Oct 5" was 2005-10-01, "5th 2" February 5 and "5th 2026" the 5th of the current month.

approxidate_careful() stays lenient past these shapes: "yesterday 10.30" is NOW, and "1/10/2026" January 10 (the order of a date with slashes is month/day). A time that must be exact is written as an integer, or as ISO 8601 ("2026-10-01T10:30:00+0200").

A local time with no zone does not depend on the text read before it: one that a change of hour repeats is its first one, and one that a change of hour skips is read with the offset in force before the change (in Madrid "2026-10-25 02:30" is 02:30 +0200 and "2026-03-29 02:30" 03:30 +0200, and up to 7.29.3 the repeated hour was +0200 or +0100 according to the text read before it). am/pm with no hour ("today pm"), a day word right after at ("at yesterday", "at now") and "never" with a zone ("never (+0200)") are no date, and an offset after a date or a day is its zone ("yesterday +0000", as "yesterday UTC"): not one glued to the digits of a date with only a day before it ("2026-10-01-10:00" is no date), but one right after the digits of a clock time ("2026-10-01T10:00-05:00") or after a clock time, noon or epoch seconds said before it, where no clock time can follow ("10:00 2026-10-01-0500", "noon 2026-10-01-0500", "@1790000000-0500", "1790000000-0500"). A minus sign, a count and its unit is a duration there, as anywhere ("10:00 2026-10-01-1d" is September 30 at 10:00), but where an offset can stand four digits are that offset, as ISO 8601 writes it: "10:00 -0500 hours", "noon -0500 hours" and "2026-10-01 -0500 hours" are no date. With no clock time an offset outside -12:00..+14:00 is no date ("now-2026", "yesterday -2300"). A run of 12 digits with a sign after it is no date ("202610011000-0500"): it is a date and a time written with no T ("20261001T1000-0500"), and epoch seconds have 11 digits at most there. The same 12 digits with nothing else are the text of an integer, read as that integer. A count right after a month name, with no day in the text, is no date ("Oct 1 pm", "Jan 1 pm", "Oct 5 days ago", and up to 7.29.3 they took the day of today), and nor is a day counted back that the zone skipped whole ("yesterday noon" in Apia on 2011-12-31). See approxidate_careful().

A text in C stops at its first NUL: a json string is read with tranger2_time_of_json_text(), which judges it whole.

Example

json_int_t v;
const char *why;
tranger2_time_of_text("1761000000000", TRUE, FALSE, &v, &why);   // 0, v 1761000000000
tranger2_time_of_text("", FALSE, FALSE, &v, &why);               // 0, v 0: no bound
tranger2_time_of_text(" 5", FALSE, FALSE, &v, &why);             // -1, why "a number that is not the text of a JSON integer"
tranger2_time_of_text("2000-01-02T00:00:00+0000", TRUE, TRUE, &v, &why);   // 0, v 946771200999
tranger2_time_of_text("garbage", FALSE, FALSE, &v, &why);        // -1, why "a text that is no date"
tranger2_time_of_text("1,5", FALSE, FALSE, &v, &why);            // -1, why "a number that is not the text of a JSON integer"
tranger2_time_of_text("Oct 1 2026 3 pm +0000", FALSE, FALSE, &v, &why);    // 0, v 1790866800
tranger2_time_of_text("march the 5 2026 00:00:00 +0000", FALSE, FALSE, &v, &why);  // 0, v 1772668800
tranger2_time_of_text("12 10 pm", FALSE, FALSE, &v, &why);       // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("1 2,10:00", FALSE, FALSE, &v, &why);      // -1, why "a comma between digits, not a date"
tranger2_time_of_text("Oct 1st 2026 3 pm +0000", FALSE, FALSE, &v, &why);  // 0, v 1790866800
tranger2_time_of_text("5th 2026", FALSE, FALSE, &v, &why);       // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct1 5 friday", FALSE, FALSE, &v, &why);  // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct 5 2026 friday 00:00:00 +0000", FALSE, FALSE, &v, &why);  // 0, v 1791158400: Monday, the weekday is not checked
tranger2_time_of_text("Oct 1 2026 5 (UTC) pm", FALSE, FALSE, &v, &why);   // 0, v 1790874000: the comment parts 5 from its pm
tranger2_time_of_text("1761000000 (UTC)", FALSE, FALSE, &v, &why);        // 0, v 1761000000
tranger2_time_of_text(", 1761000000", FALSE, FALSE, &v, &why);            // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct 1 at 10", FALSE, FALSE, &v, &why);             // -1, why "a text that is no date": 10 am or pm?
tranger2_time_of_text("Oct 1 at 24", FALSE, FALSE, &v, &why);             // -1, why "a text that is no date": no hour
tranger2_time_of_text("at 5 Oct 2026 00:00:00 +0000", FALSE, FALSE, &v, &why);  // 0, v 1791158400
tranger2_time_of_text("Oct 1 2026 at 15h +0000", FALSE, FALSE, &v, &why);       // 0, v 1790866800
tranger2_time_of_text("1 day 5 ago", FALSE, FALSE, &v, &why);                   // -1, why "a text that is no date"
tranger2_time_of_text("monday 10:00", FALSE, FALSE, &v, &why);            // -1, why "a text that is no date": which monday?
tranger2_time_of_text(<a text of 300 bytes>, FALSE, FALSE, &v, &why);   // -1, why "a time text longer than 256 bytes"
tranger2_time_of_text(<300 digits>, FALSE, FALSE, &v, &why);            // -2, why "an integer past what a time holds"

tranger2_time_of_json_text()

The time a json STRING says, judged whole: what tranger2_time_of_text() reads in its text, and a refusal for a string with a NUL inside. This is how timeranger2 reads a text time bound of a match_cond and a text tkey, and how C_TRANGER reads the before of delete-key-files and archive-key-files.

int tranger2_time_of_json_text(
    json_t *jn_text,    // NOT owned
    BOOL in_ms,
    BOOL end_of_second,
    json_int_t *value,
    const char **why
);

Parameters

KeyTypeDescription
jn_textjson_t *The json string. Not owned.
in_msBOOLAs in tranger2_time_of_text().
end_of_secondBOOLAs in tranger2_time_of_text().
valuejson_int_t *The time read. 0 when the text is refused.
whyconst char **When not NULL, the cause of a refusal.

Returns

What tranger2_time_of_text() returns for its text, and -1 with “a text with a NUL in it” for a string whose length is not the length of its C text ("5\u0000junk": from json_stringn(), or a json loaded with JSON_ALLOW_NUL, as records are), or -1 with “a time that is no text” when jn_text is not a string.

Notes

json_string_value() gives the text up to its first NUL. Up to 7.29.3 the tranger read a tkey and a bound with it: {"id":1,"tm":"5\u0000junk"} was appended with __tm__ 5, and a before of "2026-10-01T00:00:00Z\u0000garbage" purged the files before that date.

Example

json_int_t v;
const char *why;
json_t *jn = json_stringn("5\0junk", 6);
tranger2_time_of_json_text(jn, FALSE, FALSE, &v, &why);   // -1, why "a text with a NUL in it"
json_decref(jn);
jn = json_string("2000-01-02T00:00:00+0000");
tranger2_time_of_json_text(jn, TRUE, FALSE, &v, &why);    // 0, v 946771200000
json_decref(jn);

tranger2_backup_topic()

Creates a backup of a topic in the TimeRanger database. If backup_path is empty, the topic path is used. If backup_name is empty, the backup file is named topic_name.bak. If overwrite_backup is true and the backup exists, it is overwritten unless tranger_backup_deleting_callback returns true.

json_t *tranger2_backup_topic(
    json_t *tranger,
    const char *topic_name,
    const char *backup_path,
    const char *backup_name,
    BOOL overwrite_backup,
    tranger_backup_deleting_callback_t tranger_backup_deleting_callback
);

Parameters

KeyTypeDescription
trangerjson_t *The TimeRanger database instance.
topic_nameconst char *The name of the topic to back up.
backup_pathconst char *The directory where the backup will be stored. If empty, the topic path is used.
backup_nameconst char *The name of the backup file. If empty, topic_name.bak is used.
overwrite_backupBOOLIf true, an existing backup is overwritten unless tranger_backup_deleting_callback prevents it.
tranger_backup_deleting_callbacktranger_backup_deleting_callback_tA callback function that determines whether an existing backup must be deleted before overwriting.

Returns

The new, empty topic (not yours), created with the description, cols and var of the old one, or NULL (logged) when the backup fails.

Notes

If overwrite_backup is true and the backup exists, tranger_backup_deleting_callback is called. If it returns true, the existing backup is not removed.

Only a master can back up a topic (the backup moves its directory): on a replica it returns NULL with “Only master can back up”. The topic name follows the topic name rule.

The backup CLOSES the topic first, then moves its directory. A failure after that point (the backup exists and overwrite_backup is false, topic_desc.json does not load, or the rename() fails -- the backup name is taken by a file, another filesystem) returns NULL and opens the topic again as it was, with a warning: “Backup of topic failed: the topic is opened again as it was, not backed up”. Every pointer to the old topic is stale all the same, as after a successful backup: take the topic again by name.

So does a failure AFTER the move, when the new topic cannot be created (ENOSPC, a mkdir that fails): what the create left is removed, the backup is moved back where the topic was, and the topic is opened again, with the same warning. Only when the backup cannot be moved back is nothing opened: a CRITICAL, “Backup of topic failed, and the backup cannot be moved back: the data of the topic is in the backup”, with both paths. Up to 7.25.4 the data stayed in the backup and nothing was opened: a queue had no topic until a restart, and, its size read as 0, its backup was never tried again (tests/c/tr_queue/test_tr_queue_backup_failed).

That holds for EVERY tranger, also one opened with on_critical_error LOG_OPT_EXIT_ZERO (the MQTT broker’s queues, C_TRANGER’s default): the create of a backup runs with the exit bits of on_critical_error off (LOG_OPT_EXIT_ZERO, LOG_OPT_EXIT_NEGATIVE, LOG_OPT_ABORT), because its failure is recovered. The CRITICALs are still logged, and the tranger’s value is back when the create returns. If the create exits there, the data of the topic stays in <topic>.bak until moved back by hand.

json_t *topic = tranger2_backup_topic(tranger, "queue", 0, 0, TRUE, 0);
if(!topic) {
    // Error already logged: not backed up; the topic is still the old one
    topic = tranger2_topic(tranger, "queue");
}

Up to 7.25.4 a failed rename() also leaked the topic_var it had loaded, and the topic stayed closed: a queue (trq_check_backup()) was left with no topic.


tranger2_close_all_lists()

Closes the iterators, rt_mem and rt_disk lists of a topic that belong to a given creator. An empty creator closes all of them. A non-empty creator with an empty rt_id closes all of that creator’s. With both set it narrows to a single rt_id.

int tranger2_close_all_lists(
    json_t *tranger,
    const char *topic_name,
    const char *creator,    // if empty, remove all
    const char *rt_id       // if empty, remove all lists of creator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic whose lists must be closed.
creatorconst char *Creator identifier. If empty, all lists are removed.
rt_idconst char *Realtime list identifier. If empty, all lists of the creator are removed.

Returns

Returns 0 on success, or a negative value on failure. No-op (0) if the topic is not opened.

Notes

This closes the iterators, rt_mem and rt_disk lists associated with a specific creator, preventing resource leaks. If creator is empty, every list of the topic is closed regardless of rt_id.


tranger2_close_iterator()

Closes an iterator in the TimeRanger 2 database, releasing associated resources.

int tranger2_close_iterator(
    json_t *tranger,
    json_t *iterator
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger 2 database instance.
iteratorjson_t *A pointer to the iterator object to be closed.

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing an iterator makes sure that any allocated memory or resources are properly released.


tranger2_close_list()

The function tranger2_close_list() closes a previously opened list, which can be a real-time memory list (rt_mem), a real-time disk list (rt_disk), or a non-real-time list.

int tranger2_close_list(
    json_t *tranger,
    json_t *list
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
listjson_t *A pointer to the list object to be closed.

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function makes sure that resources associated with the list are properly released. If the list is a real-time list, it stops receiving updates.


tranger2_close_rt_disk()

The tranger2_close_rt_disk() function closes a previously opened real-time disk stream in the TimeRanger database, releasing associated resources.

int tranger2_close_rt_disk(
    json_t *tranger,
    json_t *disk
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
diskjson_t *A pointer to the real-time disk stream to be closed.

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function must be called when a real-time disk stream is no longer needed to free resources.

The directory of the feed, <topic>/disks/<id>/, is renamed away first (disks/.closing.<pid>-<start>.<seq>) and then removed. The master keeps hard-linking the new records into that directory until it hears the feed is gone. Removed where it was, a key directory the master made in between made the removal fail (“rmdir() FAILED ... Directory not empty”), the directory stayed, and the master went on feeding it for ever (up to 7.25.21, seen at 3000 records/s). Renamed, the master cannot reach it, and its watch of disks/ (FS_FLAG_MOVED_AS_DELETED) hears the rename as the close of the feed. A .closing.* left by a reader that died while removing it is removed by the master the next time it reads disks/ (only when that process is gone). <start> is when the reader’s process started (field 22 of /proc/<pid>/stat): a pid alive with another start time is a pid reused by another process, and the leftover is removed too. Up to 7.25.21 the name had the pid alone, and such a leftover stayed while the unrelated process lived. A name of the old form (.closing.<pid>.<seq>) is judged by its pid alone.


tranger2_close_rt_mem()

The tranger2_close_rt_mem() function closes a real-time memory stream associated with a given TimeRanger instance.

int tranger2_close_rt_mem(
    json_t *tranger,
    json_t *mem
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger instance managing the real-time memory stream.
memjson_t *A pointer to the real-time memory stream to be closed.

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing a real-time memory stream using tranger2_close_rt_mem() makes sure that resources are properly released.


tranger2_close_topic()

The tranger2_close_topic() function closes an open topic in the TimeRanger database, releasing associated resources.

int tranger2_close_topic(
    json_t  *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic to be closed.

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing a topic does not delete its data. It only releases resources associated with the open topic.


tranger2_create_topic()

The tranger2_create_topic() function creates a new topic in the TimeRanger database if it does not already exist. If the topic exists, it returns the existing topic metadata. The function makes sure that the topic is properly initialized with the specified primary key, time key, system flags, and additional metadata.

json_t *tranger2_create_topic(
    json_t              *tranger,
    const char          *topic_name,
    const char          *pkey,
    const char          *tkey,
    json_t              *jn_topic_ext,
    system_flag2_t      system_flag,
    json_t              *jn_cols,
    json_t              *jn_var
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance. If the topic exists, only tranger and topic_name are required.
topic_nameconst char *Name of the topic to be created.
pkeyconst char *Primary key field of the topic. If not specified, the key type defaults to sf_int_key.
tkeyconst char *Time key field of the topic.
jn_topic_extjson_t *JSON object containing additional topic parameters. See topic_json_desc for details. This parameter is owned by the function.
system_flagsystem_flag2_tSystem flags for the topic, defining its behavior and properties.
jn_colsjson_t *JSON object defining the topic’s columns. This parameter is owned by the function.
jn_varjson_t *JSON object containing variable metadata for the topic. This parameter is owned by the function.

Returns

Returns a JSON object representing the topic metadata. The returned JSON object is not owned by the caller and must not be modified or freed.

NULL (logged) for a refused name, a topic that does not exist on a replica, an empty pkey, no key type, a NEW topic that cannot be made whole (below), a topic_cols.json / topic_var.json that cannot be written at a version change, or a topic that cannot then be opened (see tranger2_open_topic()).

A new topic is made whole, or not at all. The master creates the topic directory, topic_desc.json, topic_cols.json, topic_var.json, keys/ and disks/. When any of them cannot be made (no space, a mkdir that fails), the directory it made is removed, nothing is opened or kept in memory, the call answers NULL and logs “Cannot create topic: it is not whole, what was made is removed” after the cause. The next create starts again from nothing. Up to 7.25.4 each failure was logged and the create went on: a topic whose keys/ (or disks/) was not made was opened and returned as created, and a queue backup (see tranger2_backup_topic()) took that half topic as the queue’s new one.

It is whole or not at all under LOG_OPT_EXIT_ZERO too. The failures of the create (the mkdirs, the write of topic_desc.json) are CRITICALs logged without the exit bits of on_critical_error. “Cannot create topic: it is not whole, what was made is removed” is logged after the removal, as a CRITICAL at the tranger’s own on_critical_error. So a tranger that exits on a critical -- LOG_OPT_EXIT_ZERO, the default of C_TRANGER and what C_TREEDB and the MQTT broker’s queues pass -- exits there, with nothing of the topic left on disk. Up to 7.25.4 it exited in the log of the first failure: the topic_desc.json, topic_cols.json and topic_var.json stayed without keys/ or disks/, and the next start opened that half topic (or was not able to open it at all, when the failure was the write of topic_desc.json).

json_t *topic = tranger2_create_topic(tranger, "devices", "id", "tm", NULL,
    sf_string_key, json_pack("{s:s, s:I}", "id", "", "tm", (json_int_t)0), 0);
if(!topic) {
    return -1;  // Error already logged; nothing of "devices" on disk or in memory
}

Notes

This function is idempotent. This means that if the topic already exists, it will return the existing topic metadata instead of creating a new one. If the primary key (pkey) is not specified, the function defaults to sf_string_key if pkey is defined, otherwise it defaults to sf_int_key.

Only the master writes, and it asks first. On a replica, or on a master that lost its lock while it was stopped (see tranger2_stop()), an existing topic is opened read-only and nothing is written, and a new one is refused with “Cannot open TimeRanger topic. Not found and no master”.

Nothing about tm order is written. From 7.25.5 to 7.25.22 a new topic carried "marks_tm_unordered": true and the master marked a file whose tm went back (<file>.tm_unordered). Both are gone, and a store that has them keeps them, ignored. A tm condition is a filter on every row: see tranger2_open_iterator().

{
    "topic_name": "readings",
    "pkey": "id",
    "tkey": "tm",
    "system_flag": 1
}

A new topic_version replaces topic_cols.json, then topic_var.json, each through a temporary file and a rename(), both fsync’ed with their directory, so the new version never survives a power cut its cols did not. The last_rowid_id counter of treedb is carried over. The cols go first because the version is what says they moved. A file that cannot be written refuses the topic: the call answers NULL, logs “Cannot re-create topic_cols.json for a new topic_version: the topic keeps its version, and is not opened” (or the error of topic_var.json), and the next create tries again -- the version on disk has not moved. “Re-Creating topic_var.json” and “Re-Creating topic_cols.json” are logged only for a file written. Up to 7.25.4 topic_cols.json was REMOVED first and a failed write went unnoticed: the topic opened with no cols file under the new version.

Topic names. A topic name is ONE directory under the database, and a segment of the backtick kw paths (topics`<name>`cols). Every call that takes a topic name (create, open, delete, backup, tranger2_topic_path(), tranger2_write_topic_var() / _cols()) refuses, with the error “Invalid topic name (path metacharacters not allowed)”:

A leading . is accepted, unlike a key: MQTT queues are named <client_id>-IN / -OUT, and the broker accepts a client_id such as .foo.

The id of a disk feed follows the directory half of the rule. The id of tranger2_open_rt_disk() (the rt_id of a follower’s open-rt / open-list) becomes the directory <topic>/disks/<id>/, and the follower removes that directory before creating it. An empty id is refused with “Invalid rt id (empty)”. ., .., or an id holding / is refused with “Invalid rt id (path metacharacters not allowed)”, and an id longer than NAME_MAX (255 bytes on Linux) with “Invalid rt id (longer than NAME_MAX)” (its directory cannot exist). A leading . is refused too, with “Invalid rt id (a leading dot is reserved)”: disks/.<name> is where a feed goes while it is removed (see tranger2_close_rt_disk()). A backtick is accepted, since an rt id is not a segment of any kw path (treedb names its own feeds <treedb>`<topic>`<id>). The id can be from a peer, so a refused id is logged as a warning, without a stack.

tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "db_history^7301", "", NULL); // ok
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "a`b`c", "", NULL);           // ok
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, ".hidden", "", NULL);         // NULL: reserved
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "../x", "", NULL);            // NULL: escapes

One id, one feed. The directory is keyed by the id alone, so an id already in use by a live feed of the topic is refused whatever the creator, with a warning: “rt disk id already in use by another creator, refused”, or “rt disk id already in use by the same creator, refused”. A second feed used to take the first one’s directory over, and its close removed it. The guard is per process: it reads the feeds of this tranger, so two processes that follow one store with the same id still take each other’s directory.

tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0);       // OK
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "../../etc", "", 0);       // refused
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "", "", 0);                // refused: empty
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "other", 0);     // refused: in use
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0);       // refused: in use

char long_id[NAME_MAX + 2];
memset(long_id, 'x', sizeof(long_id) - 1);
long_id[sizeof(long_id) - 1] = 0;                                              // 256 bytes
tranger2_open_rt_disk(tranger, "users", "", 0, cb, long_id, "gui", 0);        // refused: > NAME_MAX
tranger2_create_topic(tranger, "users", "id", "tm", 0, sf_string_key, 0, 0);   // OK
tranger2_create_topic(tranger, ".foo-IN", "", "", 0, sf_rowid_key, 0, 0);      // OK
tranger2_create_topic(tranger, "../other_db/users", "id", "tm", 0, 0, 0, 0);   // refused
tranger2_create_topic(tranger, "users/keys", "id", "tm", 0, 0, 0, 0);          // refused

tranger2_delete_key()

The tranger2_delete_key() function deletes a whole record (= a primary key, with every instance stored under it) from the specified topic in the TimeRanger database. It removes the keys/<key>/ directory and drops the key from the topic cache. Irrecoverable. Only the master can delete.

The deletion is propagated to subscribers:

For per-instance delete (one row of a key’s .md2 index — irrecoverable, no resurrection), see tranger2_delete_instance().

int tranger2_delete_key(
    json_t  *tranger,
    const char  *topic_name,
    const char  *key
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic from which the record will be deleted.
keyconst char *The key identifying the record to be deleted.

Returns

Returns 0 on success, or a negative value on failure.

Notes

A key with no directory on disk (its stat() fails with ENOENT) is logged (“key directory not found”, an error) and the call still answers 0: the key is removed from memory all the same. Any OTHER failure of that stat() (EACCES on keys/, EIO) says nothing of the key: the call answers -1 with “Cannot delete key, stat() of its directory FAILED”, and nothing is deleted, dropped from memory or announced. Up to 7.25.4 it was taken as “not found”: the key left the cache, the delete was announced to every feed and follower, and the call answered 0 over files still on disk, which came back at the next open.

key A, 3 rows; stat(keys/A) fails with EIO      -> -1, A keeps its 3 rows, no callback
the same delete once keys/A can be stat'ed      -> 0, one callback

The directory of the key is first moved out of keys/, whole, to <topic>/.deleted_key.<pid>.<start>.<n> (a rename(), atomic: the key is there whole, or its name is gone), and then removed. So a delete never leaves some of a key’s files without the others. Up to 7.27.5 a key without a base was removed in place, file by file: a removal that failed half way left some of its files, and the key, read again from them, numbered its rows anew -- its next records took g_rowids that followers and clients had seen.

A directory that cannot be moved out answers -1 (“Cannot delete key, rename() of its directory out of keys/ FAILED: the key stays as it was”) and announces nothing. The key is on disk as it was. Its cache is read again from it, and its iterators take their segments again from it: an unfiltered one at its next page, a FILTERED one at once, its index built again as an open builds it. A moved directory that cannot be removed is the key deleted all the same: 0, announced, logged (“Key deleted, but its directory moved aside cannot be removed: it is swept at the next open of the topic”), and removed at the master’s next open of the topic:

key A, 3 rows; a filtered pager of A (from_t)    -> total_rows 3
tranger2_delete_key(A), keys/A of mode 0550        -> -1, A keeps its 3 rows, no callback
the same pager, next page                          -> total_rows 3
keys/A holds a directory that cannot be emptied  -> 0, one callback, keys/A gone,
                                                    <topic>/.deleted_key.* left
the next open of the topic by the master         -> the directory left removed (a warning)

The iterators of the key, in this process, lose what they took from it: see tranger2_iterator_get_page().

The legacy name tranger2_delete_record() is kept as a source-level alias in timeranger2.h (#define tranger2_delete_record tranger2_delete_key) so existing callers keep compiling unchanged. New code must use tranger2_delete_key().


tranger2_delete_key_files()

The tranger2_delete_key_files() function deletes the OLD FILES of a key and keeps the g_rowid of every record left. It is the third delete granularity, between a whole key (tranger2_delete_key()) and one instance (tranger2_delete_instance()), and it is what a retention period is made of. Only the master can delete. The design is in kernel/c/timeranger2/DESIGN-retention.md.

It deletes every file of the key whose id sorts before the id of the file that before_t belongs to. The cut is by file and by the time of the append (__t__), never by tm, and no row is read: at least the period asked for is kept, and at most one file more. With filename_mask %Y-%m-%d, “30 days” keeps 30 to 31 days.

The rows deleted go to the base of the key, keys/<key>/.purged:

{"upto": "2026-09-20", "rows": 1823411, "t": 1792573238}

The base is written with an fsync() of the file and of its directory before any file is deleted, so a crash leaves the key as it was or purged, never renumbered. The live rowids of the key are then purged+1 .. purged+rows, and:

json_int_t tranger2_delete_key_files(
    json_t      *tranger,
    const char  *topic_name,
    const char  *key,
    uint64_t    before_t
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance. It must be the master.
topic_nameconst char *The topic of the key. The file ids of its filename_mask must sort as time (tranger2_mask_sorts_as_time(): the year first, numeric fields of a fixed width, and %s, a %G that ends the mask, %Y-%d and the %E/%O modifiers are refused).
keyconst char *The key whose old files are deleted.
before_tuint64_tThe cut, in the topic’s unit of t (milliseconds with sf_t_ms). The files of the records appended before the period of before_t are deleted.

Returns

The rows deleted, 0 when there was nothing to delete, or -1 (logged). A -1 is almost always a refusal, with nothing deleted. One -1 is a purge APPLIED: the base was written but the fsync() of its directory failed, so the cache and the followers took the purge, and the files stay on disk until the next open of the master removes them (see the notes below).

Example

/*
 *  Retention of 30 days on a topic of one file a day (in seconds), a slice
 *  of the keys per tick of the owner's timer
 */
uint64_t cut = time_in_seconds() - 30*86400;
json_t *keys = tranger2_list_keys(tranger, "raw_events");
size_t n = json_array_size(keys);
for(size_t i = priv->purge_cursor; i < n && i < priv->purge_cursor + 100; i++) {
    const char *key = json_string_value(json_array_get(keys, i));
    if(tranger2_delete_key_files(tranger, "raw_events", key, cut) < 0) {
        // Error already logged: refused, the key as it was -- or the purge
        // applied with its base not durable, the files removed at the next open
    }
}
priv->purge_cursor += 100;
if(priv->purge_cursor >= n) {
    priv->purge_cursor = 0;     // the next tick starts the next round
}
JSON_DECREF(keys)
key K: files 2000-01-01 (rows 1-3), 2000-01-02 (4-6), 2000-01-03 (7-9)
tranger2_delete_key_files(K, before_t = 2000-01-02T00:00:00)   -> 3
    files left: 2000-01-02 (4-6), 2000-01-03 (7-9), base {upto 2000-01-01, rows 3}
    tranger2_topic_key_size(K) -> 6, the next append -> g_rowid 10
    from_rowid 1 or 3 -> from 4;  from_rowid -1 -> 10;  to_rowid 3 -> nothing

Notes


tranger2_archive_key_files()

The tranger2_archive_key_files() function is the purge of tranger2_delete_key_files() -- the same cut, the same guards, the same base, the same answer -- and the files it takes out of the topic are kept: each purge of a key hard-links them into a new batch of archive_directory. What is done with a batch (sent to a backup machine, compressed, loaded elsewhere) is the job of another process, which removes the batch when it is done. The design and its crash table are in kernel/c/timeranger2/DESIGN-retention.md §12. Added after 7.28.11: it ships in the next release.

<archive_directory>/<database>/<topic>/<key>/<upto>@<t>/
    <file_id>.json  <file_id>.md2  [<file_id>.unordered] [<file_id>.tm_unordered]
    topic_desc.json  topic_cols.json        the schema, as it is at the purge
    manifest.json

upto is the last file id taken and t the time of the purge (the t of the base). The manifest says what the batch holds and how it is numbered:

{
    "database": "tr_raw_events",
    "topic": "raw_events",
    "key": "dev1",
    "files": ["2026-09-19", "2026-09-20"],
    "rows": 5712,
    "first_g_rowid": 1817700,
    "last_g_rowid": 1823411,
    "upto": "2026-09-20",
    "t": 1792573238
}

Row N of the k-th file of files has the g_rowid first_g_rowid + the rows of the files before it + N - 1.

When the base is written, an empty ready mark is created in <archive_directory>/.ready/. Its name is the path of the batch under the archive, with % written %25 and / written %2F:

<archive_directory>/.ready/tr_raw_events%2Fraw_events%2Fdev1%2F2026-09-20@1792573238

A reader watches that one directory (a rename() gives no fs event, and the tree has a directory per key). The mark is a hint, not a record: a batch without one is found by a scan of the tree. The SDK’s reader is the archive_sender yuno.

json_int_t tranger2_archive_key_files(
    json_t      *tranger,
    const char  *topic_name,
    const char  *key,
    uint64_t    before_t,
    const char  *archive_directory
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance. It must be the master.
topic_nameconst char *The topic of the key, as in tranger2_delete_key_files().
keyconst char *The key whose old files are archived.
before_tuint64_tThe cut, in the topic’s unit of t, as in tranger2_delete_key_files().
archive_directoryconst char *Where the batches go: an absolute path of a directory that EXISTS, on the filesystem of the topic (a hard link cannot cross filesystems), neither inside the directory of the tranger nor holding it. The levels <database>/<topic>/<key> under it are created.

Returns

The rows archived (and taken out of the topic), 0 when there was nothing to take, or -1 (logged). A refusal deletes nothing and leaves no batch. The -1 of a purge APPLIED with its base not durable is as in tranger2_delete_key_files(), and its batch stays.

Example

/*
 *  Retention of 30 days that keeps what it takes: the batches go to
 *  /yuneta/store/archive, where a reader yuno sends them to a backup
 *  machine and removes them
 */
uint64_t cut = time_in_seconds() - 30*86400;
json_int_t rows = tranger2_archive_key_files(
    tranger, "raw_events", "dev1", cut, "/yuneta/store/archive"
);
if(rows < 0) {
    // Error already logged: refused, nothing archived nor deleted
}
key dev1: files 2000-01-01 (rows 1-3), 2000-01-02 (4-6), 2000-01-03 (7-9)
tranger2_archive_key_files(dev1, before_t = 2000-01-02T00:00:00, "/yuneta/store/archive")   -> 3
    topic:   files 2000-01-02 (4-6), 2000-01-03 (7-9), base {upto 2000-01-01, rows 3}
    archive: .../dev1/2000-01-01@<t>/  2000-01-01.json 2000-01-01.md2 topic_desc.json
             topic_cols.json manifest.json {rows 3, first_g_rowid 1, last_g_rowid 3}
tranger2_archive_key_files(dev1, before_t = 2000-01-03T00:00:00, ...)   -> 3
    archive: a second batch .../dev1/2000-01-02@<t>/, first_g_rowid 4

Notes


tranger2_mask_sorts_as_time()

Do the file ids of a filename_mask (a strftime() format) sort as time? tranger2_delete_key_files() deletes the files whose id sorts BEFORE the id of the cut, comparing them as strings, so it refuses a topic whose mask fails this test: with such a mask, it cannot tell which of its files are old.

The ids sort as time when every field is numeric, of a fixed width, the YEAR first, and each field the NEXT one down from the last, with literal text anywhere: a field left out mixes the ones below it (%Y-%d puts the 9th of March before the 10th of February). The fields that can follow each one:

FieldCan be followed by
(first)%Y, %G, %F (%Y-%m-%d)
%Y%m, %j
%m%d
%G (the year of the ISO week)%V (and %G cannot end the mask)
%V%u
%F, %j, %d, %u%H, %R (%H:%M), %T (%H:%M:%S)
%H%M
%M, %R%S

%% is a literal. %s is refused: the epoch has no fixed width (9 digits before 2001-09-09).

BOOL tranger2_mask_sorts_as_time(
    const char *filename_mask
);

Parameters

KeyTypeDescription
filename_maskconst char *The mask, as in the filename_mask of a topic.

Returns

TRUE when the ids of the files it names sort as time.

Example

tranger2_mask_sorts_as_time("tracks-%Y-%m-%d");   // TRUE
tranger2_mask_sorts_as_time("series-1d-%Y");      // TRUE
tranger2_mask_sorts_as_time("%FT%T");             // TRUE
tranger2_mask_sorts_as_time("%G-W%V");            // TRUE (the year of the week, then its week)
tranger2_mask_sorts_as_time("%Y-%d");             // FALSE: the month left out
tranger2_mask_sorts_as_time("%s");                // FALSE: the epoch has no fixed width
tranger2_mask_sorts_as_time("queue");             // FALSE: a tr_queue topic, one file
tranger2_mask_sorts_as_time("%d-%m-%Y");          // FALSE: the year last
tranger2_mask_sorts_as_time("%Y-%d-%m");          // FALSE: the day before the month
tranger2_mask_sorts_as_time("%Y-%b");             // FALSE: "Aug" sorts before "Jun"
tranger2_mask_sorts_as_time("%Y-%e");             // FALSE: " 1" is not "01"

tranger2_record_in_purged_period()

Does a record fall in a purged period of its key? TRUE when tranger2_append_record() refuses it for that (“Cannot append record, it falls in a purged period of its key”): its __t__ names a file at or before the upto of the key’s base (tranger2_delete_key_files()). The append logs that refusal as an ERROR. A caller that must give the cause to a PEER asks first, and answers it. C_TRANGER’s add-record does: “record refused by topic ‘raw_events’: a t in a purged period of its key: its old files were deleted”.

__t__ and record are the ones the append takes. FALSE for __t__ = 0 (now: no purge reaches it), a key with no base, a key not in the topic, and a record whose key or time cannot be read (the append says why). A key flagged unlisted (its .purged, or its directory, was not readable when the topic opened) has no base in its cache: its base is read from the disk, quietly, as the append reads it when it lists the key again. A base that can be read gives the same answer as the append. A base that still cannot be read (a .purged that is corrupt or unreadable, a directory that cannot be listed) gives FALSE, and the append refuses the record for that other cause, with an ERROR “Cannot append record, its key cannot be listed: its row would follow rows no cell counts”: the two agree only that the record is not appended, and only the append says so, in the log. Silent, but for a topic that does not exist (logged).

BOOL tranger2_record_in_purged_period(
    json_t      *tranger,
    const char  *topic_name,
    uint64_t    __t__,
    json_t      *record     // not owned
);

Parameters

KeyTypeDescription
trangerjson_t *The TimeRanger database.
topic_nameconst char *The topic of the record.
__t__uint64_tThe time of the append, in the topic’s unit of t (milliseconds with sf_t_ms). 0 is now.
recordjson_t *The record, with the topic’s pkey. Not owned.

Returns

TRUE when the record falls in a purged period of its key.

Example

/*
 *  Key "dev1" of a topic of one file a day, purged up to 2026-09-22
 *  (tranger2_delete_key_files(tranger, "raw_events", "dev1", <2026-09-23>))
 */
json_t *record = json_pack("{s:s, s:I}", "id", "dev1", "tm", (json_int_t)1790035200);
if(tranger2_record_in_purged_period(tranger, "raw_events", 1790035200, record)) {
    // 2026-09-22: refused, say so to whoever sent it
} else {
    md2_record_ex_t md;
    tranger2_append_record(tranger, "raw_events", 1790035200, 0, &md, json_incref(record));
}
JSON_DECREF(record)

tranger2_delete_instance()

The tranger2_delete_instance() function deletes a single instance (one row of a key’s .md2 index) without touching the surrounding rows. The row’s metadata is mutated in place: sf_deleted_instance (bit 0x0400 of system_flag2_t, on the inherited side of the mask so rt_by_disk followers see the same tombstone) is ORed into the system flag bits. The .json data log stays append-only.

Read paths (tranger2_open_iterator history loop, tranger2_iterator_get_page, and the rt_by_disk follower) skip rows whose tombstone bit is set. Treedb is downstream of those paths and inherits the skip with no tr_treedb change. tranger2_read_record_content() and tranger2_read_user_flag() deliberately do not filter — the caller has already located the row — so they remain available for audit / wipe-verification tooling.

Irrecoverable. Master-only. Second delete of the same row is a silent no-op (returns 0).

int tranger2_delete_instance(
    json_t      *tranger,
    const char  *topic_name,
    const char  *key,
    uint64_t    __t__,
    uint64_t    rowid,          // per-key i_rowid, slot in .md2, based 1
    BOOL        zero_payload
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The topic that owns the key.
keyconst char *The key whose instance is to be deleted.
__t__uint64_tTime of the instance (matches the value passed to tranger2_append_record).
rowiduint64_tPer-key slot index in the .md2 file, based 1.
zero_payloadBOOLIf TRUE, also overwrites the matching __size__ bytes at __offset__ in the .json data file with zeros. Opt-in: after the wipe the JSON is no longer a parseable concatenation — only the .md2 index makes sense of it. Use for sensitive-data wipes.

Returns

Returns 0 on success, a negative value on failure.

Notes

Side effects to be aware of:


tranger2_set_rt_key_deleted_callback()

Registers a key-delete callback on a handle returned by tranger2_open_rt_mem(), tranger2_open_rt_disk() or tranger2_open_iterator(). The callback fires when tranger2_delete_key() runs against a key this subscriber tracks:

The subscriber’s filter is honoured, the same one its records go through: a specific key matches that one, an rkey (in match_cond) the keys its regular expression matches, and neither of them every deletion. Up to 7.25.20 only key was looked at, and a feed opened with an rkey was told every key deleted:

json_t *rt = tranger2_open_rt_mem(
    tranger, "devices", "", json_pack("{s:s}", "rkey", "^DVES_"),
    on_record, "dves-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);
tranger2_delete_key(tranger, "devices", "OTHER_1");     // on_key_deleted not called
tranger2_delete_key(tranger, "devices", "DVES_000000"); // called once

Additive: existing handles default to “no callback”, behavior unchanged. Passing cb=NULL clears any previously registered callback.

typedef int (*tranger2_key_deleted_callback_t)(
    json_t      *tranger,
    json_t      *topic,
    const char  *key,           // the deleted key
    json_t      *list,          // iterator / rt_mem / rt_disk entry, don't own
    void        *user_data
);

int tranger2_set_rt_key_deleted_callback(
    json_t                              *list,
    tranger2_key_deleted_callback_t     cb,
    void                                *user_data
);

Parameters

KeyTypeDescription
listjson_t *A handle previously returned by tranger2_open_rt_mem, tranger2_open_rt_disk or tranger2_open_iterator.
cbtranger2_key_deleted_callback_tFunction to invoke, or NULL to clear.
user_datavoid *Opaque pointer echoed back to the callback.

Returns

Returns 0 on success, -1 if list is NULL.

Example

A follower feed that closes itself when the key it follows is deleted:

PRIVATE int on_key_deleted(
    json_t *tranger, json_t *topic, const char *key, json_t *list, void *user_data)
{
    tranger2_close_rt_disk(tranger, list);  // allowed: see the notes
    return 0;
}

/*  The loop is the PARAMETER: a "yev_loop" key in the config is overwritten  */
json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop);
json_t *rt = tranger2_open_rt_disk(
    tranger, "devices", "DVES_000000", NULL, on_record, "live-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);

Notes

The callback fires once per feed for each delete. An rt_disk feed that owns an inotify watcher hears the delete only from its own directory, and the master-side fan-out skips it. Until 2026-09-15 a feed was fired once for every inotify event of every feed of the topic (six times in a three-feed follower), and not at all for a key that had no record since the feed opened.

The callback can close its own feed (tranger2_close_rt_disk()). The watcher is stopped at the end of the batch it is handling, and the rest of that batch is dropped.

The feed’s watcher exists only when the tranger was started with a loop: pass it as the third argument of tranger2_startup(). Without a loop an rt_disk has no watcher, and it is fired directly by the master’s delete path, like an rt_mem.


tranger2_delete_topic()

The tranger2_delete_topic() function deletes a topic from the TimeRanger database, effectively removing all associated records and metadata.

int tranger2_delete_topic(
    json_t  *tranger,
    const char  *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic to be deleted.

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

Deleting a topic is irreversible. Make sure that the topic is no longer needed before calling tranger2_delete_topic().

Only a master can delete: on a replica (master=0) it returns -1 with “Only master can delete”, and the directory stays. The topic name follows the topic name rule: tranger2_delete_topic(tranger, "../other_db/users") returns -1 and deletes nothing.


tranger2_dict_topic_desc_cols()

tranger2_dict_topic_desc_cols() retrieves the column descriptions of a specified topic in dictionary format.

json_t *tranger2_dict_topic_desc_cols(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A reference to the TimeRanger database instance.
topic_nameconst char *The name of the topic whose column descriptions are to be retrieved.

Returns

A JSON object containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object.

Notes

This function is similar to tranger2_list_topic_desc_cols(), but returns the data in dictionary format instead of a list.


tranger2_get_iterator_by_id()

Retrieve an iterator by its identifier. If the iterator exists, it is returned. Otherwise, NULL is returned.

json_t *tranger2_get_iterator_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *iterator_id,
    const char  *creator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic associated with the iterator.
iterator_idconst char *Unique identifier of the iterator to retrieve.
creatorconst char *Identifier of the entity that created the iterator. NULL or empty matches only creatorless iterators. To reach one registered with a creator, pass that same creator.

Returns

Returns a pointer to the iterator JSON object if found. Otherwise, returns NULL.

Notes

This function does not produce any error messages if the iterator is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.

It is one hash lookup (since 7.25.0): the topic keeps iterators_by_id — {creator: {id: iterator}} — beside its iterators array. It used to walk the array, and tranger2_open_iterator() calls it to refuse a duplicate, so opening N iterators on a topic was O(N²).

It also opens the topic when it is closed (it goes through tranger2_topic()). To ask whether a handle you registered is still alive, check tranger2_topic_is_open() first, then look the handle up by its id: a topic closed and opened again has the same name and none of the old handles.

// Is my iterator "it1" still there? Never trust a pointer kept from the open.
json_t *it = tranger2_topic_is_open(tranger, "frames")?
    tranger2_get_iterator_by_id(tranger, "frames", "it1", "my_service") : NULL;

tranger2_get_rt_disk_by_id()

Retrieve a real-time disk instance by its identifier. If the specified real-time disk exists, it returns the corresponding JSON object. Otherwise, it returns NULL.

json_t *tranger2_get_rt_disk_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *rt_id,
    const char  *creator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic associated with the real-time disk.
rt_idconst char *Identifier of the real-time disk to retrieve.
creatorconst char *Creator identifier used to filter the real-time disk instances. NULL or empty matches only creatorless feeds. To reach one registered with a creator, pass that same creator.

Returns

Returns a JSON object representing the real-time disk instance if found. Otherwise, returns NULL.

Notes

This function does not produce any error messages if the real-time disk is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.


tranger2_get_rt_mem_by_id()

Retrieve a real-time memory instance by its identifier. If the specified real-time memory instance exists, it is returned. Otherwise, NULL is returned.

json_t *tranger2_get_rt_mem_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *rt_id,
    const char  *creator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic associated with the real-time memory instance.
rt_idconst char *Identifier of the real-time memory instance to retrieve.
creatorconst char *Creator identifier used to filter the real-time memory instance. NULL or empty matches only creatorless feeds. To reach one registered with a creator, pass that same creator.

Returns

Returns a pointer to the real-time memory instance as a json_t * if found, otherwise returns NULL.

Notes

This function does not produce any internal logging or error messages if the requested real-time memory instance is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.


tranger2_iterator_get_page()

Retrieves a page of records from an iterator in the TimeRanger database. The function returns a JSON object containing the total number of rows, the number of pages based on the specified limit, and the list of retrieved records.

json_t *tranger2_iterator_get_page(
    json_t *tranger,
    json_t *iterator,
    json_int_t from_rowid,    // based 1
    size_t limit,
    BOOL backward
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
iteratorjson_t *A pointer to the iterator from which records will be retrieved.
from_rowidjson_int_tThe starting position of the page, 1-based. See the note below: it is a position among the rows the iterator RETURNS, which is a global rowid only when the iterator does not filter.
limitsize_tThe maximum number of records to retrieve in the page.
backwardBOOLIf TRUE, records are retrieved in reverse order. Otherwise, they are retrieved in forward order.

Returns

A JSON object containing the retrieved records, total row count, and pagination details. The caller owns the returned JSON object and must manage its memory.

Notes

This function is useful for paginating through records in an iterator. The from_rowid parameter determines the starting point, and the limit parameter controls the number of records retrieved per page.

A filtered iterator (one opened with any match_cond condition) pages over its row index, so total_rows, pages and the page contents count only the records that MATCH — and from_rowid is a position among those, not a global rowid. An unfiltered iterator has no index, and its positions are the global rowids. See tranger2_open_iterator().

An unfiltered iterator pages the key as it is: its segments are taken from the cache again when the key’s cache moved since they were taken (its rows, its number of files, its last file), so the rows appended since the open are readable and total_rows / pages (the live count) can be paged to the end. A filtered iterator keeps the index and the segments of its open.

A delete of the key drops them: an unfiltered iterator takes its segments again at its next page (a key written again with the same number of rows and files, spread another way, is not the same key), and a filtered one pages an empty index -- the rows it indexed are gone. A page that finds the key gone from disk behind its back (a replica whose master deleted it) logs “Key gone from disk while its iterator was open” and stops.

json_t *it = tranger2_open_iterator(tranger, "topic", "K", NULL, NULL, "pager", "", NULL, NULL);
// K: 2 rows on day 1, 1 on day 2
tranger2_delete_key(tranger, "topic", "K");
// K written again: 1 row on day 1, 2 on day 2
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
// page["data"]: the 3 new rows (7.25.4: 1 row, and a CRITICAL)
json_t *it = tranger2_open_iterator(tranger, "topic", "key", NULL, NULL, "it1", "", NULL, NULL);
/*  ... 3 records appended to "key" since ...  */
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
/*  page["data"] holds the 3 new rows too; page["total_rows"] counts them  */

A purge of the old files of the key (tranger2_delete_key_files()) moves no position. The positions of an unfiltered iterator count from the base the key had when it opened (position N is g_rowid = base + N), and the ones of the rows purged since are returned empty. A filtered iterator keeps its index, which holds g_rowids, and skips the rows purged. Counted from the base of NOW, every position moved by the rows purged, and a client paging forward skipped live rows.

// K: 2 rows a day, days 1-3 (g_rowids 1-6), pages of 2
json_t *it = tranger2_open_iterator(tranger, "topic", "K", json_object(), NULL, "pager", "", NULL, NULL);
tranger2_iterator_get_page(tranger, it, 1, 2, FALSE);  // A1:1 A2:2
tranger2_delete_key_files(tranger, "topic", "K", day2); // the 2 rows of day 1
tranger2_iterator_get_page(tranger, it, 3, 2, FALSE);  // B1:3 B2:4 -- the next page, nothing skipped
tranger2_iterator_get_page(tranger, it, 1, 2, FALSE);  // empty: purged
tranger2_iterator_size(it);                             // 6: its positions

tranger2_iterator_size()

tranger2_iterator_size() returns the number of records in the specified iterator. For a filtered iterator, the size of its index. For an unfiltered one, its page positions: on a key purged since the open (tranger2_delete_key_files()) they include the positions of the rows purged, which are returned empty.

size_t tranger2_iterator_size(
    json_t *iterator
);

Parameters

KeyTypeDescription
iteratorjson_t *The iterator whose size is to be determined.

Returns

The number of records the iterator will return.

Notes

A filtered iterator answers with its index: the exact number of records matching its match_cond. An unfiltered one sums its segments — the same thing when nothing is filtered out.


tranger2_list_keys()

Returns a JSON array with the key names of a topic, read from its in-memory cache.

json_t *tranger2_list_keys(
    json_t *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic from which to retrieve the keys.

Returns

A JSON array of key-name strings. The caller owns the returned array and must free it when no longer needed. Never NULL — a missing topic yields an empty array.

Notes

The keys are from the topic’s in-memory cache, not a disk scan. Slow for thousands of keys (allocates one string per key).


tranger2_list_topic_desc_cols()

Returns a JSON array containing the column descriptions of a topic in the TimeRanger database. The returned JSON object must be decremented after use.

json_t *tranger2_list_topic_desc_cols(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A JSON object representing the TimeRanger database instance.
topic_nameconst char *The name of the topic whose column descriptions are to be retrieved.

Returns

A JSON array containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object after use.

Notes

This function was previously known as tranger_list_topic_desc().


tranger2_list_topics()

Returns a JSON array with the names of the topics currently opened in memory (the tranger["topics"] registry) — not a disk scan.

json_t *tranger2_list_topics(
    json_t *tranger
);

Parameters

KeyTypeDescription
trangerjson_t *The tranger database instance.

Returns

A JSON array with the names of the currently-opened topics. The caller owns the returned array and must free it. Never NULL — an empty array when no topic is opened.

Notes

This lists only topics already opened in memory. To list every topic present on disk — including ones not yet opened — use tranger2_list_topic_names().


tranger2_open_iterator()

Opens an iterator for traversing records in a topic within the TimeRanger database. The iterator allows filtering records based on specified conditions and supports real-time data loading.

json_t *tranger2_open_iterator(
    json_t *tranger,
    const char *topic_name,
    const char *key,    // required
    json_t *match_cond, // owned
    tranger2_load_record_callback_t load_record_callback, // called on LOADING and APPENDING, optional
    const char *iterator_id,     // iterator id, optional, if empty will be the key
    const char *creator,     // creator
    json_t *data,       // JSON array, if not empty, fills it with the LOADING data, not owned
    json_t *extra       // owned, user data, this json will be added to the return iterator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic to iterate over.
keyconst char *Specific key to filter records. Required.
match_condjson_t *JSON object containing filtering conditions. Owned by the function.
load_record_callbacktranger2_load_record_callback_tCallback function invoked during data loading and appending. Optional.
iterator_idconst char *Unique identifier for the iterator. If empty, the key is used.
creatorconst char *Identifier of the entity creating the iterator.
datajson_t *JSON array to store loaded data. Not owned by the function.
extrajson_t *Additional user data attached to the iterator. Owned by the function.

Returns

Returns a JSON object representing the iterator. The topic owns it: close it with tranger2_close_iterator().

NULL (logged, and match_cond and extra are consumed all the same) when the topic cannot be opened, key is empty, an iterator with this iterator_id and creator is already open, or the index of a filtered paging iterator cannot be built.

Notes

The iterator supports real-time data loading and filtering based on various conditions. Use tranger2_close_iterator() to release resources when done.

A load that stops half way says so. When the LOADING (a callback, or data) meets a row whose metadata or whose CONTENT cannot be read, it stops there, logs it, and leaves "load_failed": true in the iterator. The rows before it, in the load’s direction, were handed (forward: the oldest, backward: the newest). What the callback saw, and what data holds, is then NOT the whole history: a caller that asks the history a question (is any record of this key frozen by a snapshot?) must read the flag before it takes a “no”. Up to 7.25.4 a record whose content was not readable was handed to the callback as NULL and the load went on. An only_md load reads no content, and no content fails it.

After a restart too. The cache of a topic is built from disk when it is opened. A .md2 file of a key that it cannot count flags the key: one that cannot be opened or read. It logs “md2 file of the key unreadable when its cache was built: every load of the key says load_failed” once, at the open, and then every iterator of the key -- paging ones too -- logs “The history of the key is not whole: a md2 file of it could not be read when its cache was built” and says load_failed. A loading stops where the first flagged file is in its direction, the place a running tranger stops when the damage happens behind its back. Key A with the files of days 1, 2 and 3, and the .md2 of day 2 made unreadable (mode 0000) while the yuno was down:

forward load   -> the rows of day 1, then load_failed
backward load  -> the rows of day 3, then load_failed

A key directory that cannot be LISTED is not an empty key. When the cache build cannot open keys/<key>/ (EMFILE, EACCES, ENOMEM), cannot READ it (readdir() fails: EIO, ESTALE, and 7.25.4 took that as the end of the directory, and the key was read without the files not listed yet), or has no memory for an entry of the listing, none of the key’s files is known, so the key is flagged whole: it logs “key directory cannot be listed when its cache was built: every load of the key says load_failed” once, and every iterator of the key logs “The history of the key is not whole: its directory could not be listed when its cache was built” and says load_failed with no row handed. A keyless tranger2_open_list() names the key in load_failed_keys, so treedb refuses a create of that id. An append into the key lists it again first: while the directory still cannot be listed the append is refused (“Cannot append record, its key cannot be listed: its row would follow rows no cell counts”). Once it can, the key’s cache is built from the disk, the flag goes (“key directory listed again”, an INFO) and the append goes on. Up to 7.25.4 the key loaded as an EMPTY key with load_failed false, and an append counted its row as the first of its file.

keys/A of mode 0000 at the open:
iterator of A              -> no row, load_failed
keyless list               -> the rows of B, load_failed, load_failed_keys ["A"]
append into A              -> -1 (refused)
keys/A of mode 02770 again:
append into A              -> 0, the key listed again
iterator of A              -> every row of A, the new one last, load_failed false

When keys/ itself cannot be listed, no key can be flagged by name: the topic is not opened (see tranger2_open_topic()).

A flag goes when its cause goes, on a replica too. A flag is not kept until the topic is opened again:

replica opens, keys/A of mode 0000     -> A flagged unlisted, feed "rtALL" open
chmod 02770 keys/A; the master appends A (its 4th row)
replica, at the notification           -> INFO key directory listed again,
                                          rtALL gets the row, rowid 4
replica: iterator of A                 -> A@1 A@2 A@3 A@4, load_failed false

In 7.25.4 a key whose directory listing failed loaded as an empty key (see above), and the notification counted the notified file alone as the whole key (the feed got rowid 1 for the key’s 4th row).

A .md2 of 0 rows gets no cell and flags nothing. With an EMPTY .json it loses nothing. With a .json that is NOT empty it is what an append that was never acknowledged leaves (the content is written first, the md2 row after, and the row was never written: the md2 failed, or the process died between the two). That is not damage: the file is ignored, as until 7.25.4, with a warning that names it, and the key loads from its other files:

WARNING: md2 file of the key with no rows and a content file that is not empty:
         an append that was never acknowledged, the file is ignored
         topic_directory=<store>/devices key=A file_id=2000-01-02 content_size=32
forward load   -> the rows of days 1 and 3, load_failed false

The next append into that day’s file names its content with a row of its own, and the warning goes. A .md2 cut to 0 bytes behind the yuno’s back has the same shape, and loses the rows of its file the same way it did until 7.25.4: a .json much larger than one record is the sign. Put the pair back from a backup.

A .md2 whose last row is torn is not damage either. Its size is not a whole number of 32-byte rows: a power cut came during the write of a row. The md2 row is the commit point of an append, and an append is acknowledged only after the whole row is written, so a torn row is an append that was never acknowledged. A MASTER cuts the .md2 back to its whole rows (to floor(size / 32) * 32 bytes) when it first reads the file, logs one warning, and the key loads whole. The next append goes into the same file:

WARNING: md2 file of the key ends in a part of a row: an append that was never
         acknowledged was cut back
         topic=devices key=A file_id=2000-01-02
         path=<store>/devices/keys/A/2000-01-02.md2 old_size=109 new_size=96
forward load   -> the rows of days 1, 2 (3 rows) and 3, load_failed false

The cut removes fewer than 32 bytes, all after the last whole row. The .json is left as it is: its bytes after the last row belong to no row, and the next append writes at its end. If the cut itself fails, that is damage: the file is flagged as above (“Cannot cut back a md2 file that ends in a part of a row: the file is damaged”). A torn first row (a .md2 of fewer than 32 bytes) is cut to 0 bytes, and the file is then a .md2 of 0 rows, ignored with the warning above.

Not every tail on no row boundary is a torn row. 7.25.4 did not cut a torn row back: the next appends wrote their rows after the torn bytes, on no row boundary, and those rows were acknowledged. In such a file the last 32 bytes are a whole row, and a cut removes the end of it. So the tail is cut only when it is a torn row after good rows. A row is good when its content is WHOLE: inside the .json, one json value and one NUL byte after it, as an append writes it (or only zero bytes, for an instance deleted with its content zeroed, see tranger2_delete_instance()). A NUL inside a string is written as the escape \u0000, so it is part of the value and is not a NUL byte. A content that the process that checks it has not the memory to parse is not a content that is not whole: the largest memory block (gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and the yuno that wrote it can have a larger block. A content larger than that block is read in parts: a NUL before its last byte says that it is not whole, with no block of its size, and when its only NUL is its last byte the check cannot tell. A content smaller than the block can still need more to parse (a long string, a long array or object: jansson doubles its buffers and tables). When its parse fails for memory, the check cannot tell either. The tail is not cut on a content that the check cannot tell. The two rules, which must both hold:

  1. The last 32 bytes of the .md2, read as a row, are NOT a row of their own: their content does not end exactly at the end of the .json (__offset__ + __size__ equal to its size), and it is not whole.

  2. The last whole row is good, and so is the whole row before it, whose content ends at or before the start of the content of the last one.

Rule 1 stops the cut of every file that 7.25.4 left whose last row still names its content as 7.25.4 wrote it, also when the .json has content after the last row (an append killed between its content and its md2 row, and 7.25.4 did not cut that content back): the last row is a real row, and its content is whole wherever the .json ends. When the process that checks has not the memory to parse that content, the check cannot tell, and that is enough to not cut. Only a content damaged since (its bytes changed) lets such a file reach rule 2. For a torn row, the last 32 bytes are the end of the last whole row and the start of the torn one: read as a row, their __offset__ and __size__ are bytes moved out of their fields. They name a whole record of the .json, to the byte, only by a coincidence of 64-bit values, and then the file is flagged, not cut. Rule 2 stops a cut when the rows before the tail are not good. Example: a topic with no tkey (its __tm__ is 0, as in a treedb topic) and records of 256 bytes, so every __offset__ is a multiple of 256. There, a row moved by one byte names a range INSIDE the .json (the real one divided by 256). A test of the range only takes it for a good row. It is not whole: it does not start where a record starts.

When a rule fails, the file is NOT cut: a CRITICAL names the shape and the cause, the file is flagged as above, and the key says load_failed until the file is repaired by hand (see the repair in treedb). The bytes of the file do not change:

CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
          written by 7.25.4 after a torn row; not cut, repair it by hand
          cause="its content is a whole record of the content file"
          topic=devices key=A file_id=2000-01-02
          path=<store>/devices/keys/A/2000-01-02.md2 md2_size=173
          content_size=242 row_at=141 __offset__=160 __size__=32
CRITICAL: md2 file of the key ends in a part of a row after a last whole row
          that is not valid: not cut, repair it by hand
          cause="its content is not a record"
          topic=devices key=A file_id=2000-01-02 ... row_at=64

The cause of the first is “its content ends the content file”, “its content is a whole record of the content file” or “its content has no NUL but the one at its end, and this process has not the memory to parse it (MEM_MAX_BLOCK): it can be a record written by a yuno with a larger block”. The cause of the second is one of “its content is not inside the content file”, “its content is not a record”, “this process has not the memory to parse its content (MEM_MAX_BLOCK): it cannot be checked”, “the whole row before it is not a good row” or “this process has not the memory to parse the content of the whole row before it (MEM_MAX_BLOCK): it cannot be checked” (for these two, row_at is that row), and “its content starts before the end of the content of the row before it”.

Example: a master with a block of 64 KiB opens a file that 7.25.4 left, whose last row names a record of 100 KiB, written by a yuno with a larger block. The file is flagged, and no row is lost. The same when the record is a string of 40 000 bytes, or an array in 20 000 bytes: it is smaller than the block, but its parse needs more (the log also has the ERROR “SIZE GREATER THAN MAX_BLOCK” of the allocation that failed):

CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
          written by 7.25.4 after a torn row; not cut, repair it by hand
          cause="its content has no NUL but the one at its end, and this process
          has not the memory to parse it (MEM_MAX_BLOCK): it can be a record
          written by a yuno with a larger block"
          topic=devices key=A file_id=2000-01-01 md2_size=159 row_at=127
          __offset__=1024 __size__=102400

When the check cannot RUN, a CRITICAL says so, and the file is not cut. It is one of three:

CRITICAL: Cannot check the torn tail of a md2 file, its content file cannot be
          read: not cut
          topic=devices key=A file_id=2000-01-02
          path=<store>/devices/keys/A/2000-01-02.json errno=13 serrno="Permission denied"

At an open the file is flagged too: without the check, the whole rows can be the moved rows of the shape 7.25.4 left. A master’s next append into the file counts it again, and the check runs then. At an append the file is not flagged (see tranger2_append_record()).

A replica makes the same check, and flags the file the same way: it does not read rows that are not on a row boundary.

A REPLICA never writes. It reads only the whole rows of such a file, and logs nothing for the torn row: a replica also sees a torn row while a live master is writing it, and counts the row when the master’s next notification of the file comes. A file with no whole row (fewer than 32 bytes) is a .md2 of 0 rows for the replica, so when its .json is not empty the replica logs the 0-rows warning above, and does not cut. When the master opens the store, it cuts the file back. Up to 7.25.4 the cache build logged a CRITICAL (“Cannot read last record, md2 file corrupted”) and left the whole file out of the key, on a master and on a replica: its acknowledged rows were missing from every load, and nothing failed.

The flag of a file goes once a cell counts it again: an append into the file that finds it readable (see tranger2_append_record()), on a follower the file read whole from the disk after the master wrote it, or a load of the key once the file changed or opens again (see A flag goes when its cause goes below). tranger2_delete_key() clears every flag with the key. Up to 7.25.4 the cache build dropped an unreadable file and nothing failed: the key read as a shorter key.

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "devices", "dev-1", NULL, NULL, "", "me", data, NULL
);
if(!it) {
    // not opened, the cause is in the log
} else if(json_is_true(json_object_get(it, "load_failed"))) {
    // data holds only a part of the history: do not decide on it
}
if(it) {
    tranger2_close_iterator(tranger, it);
}
JSON_DECREF(data)

A deleted key. A delete of the key (tranger2_delete_key(), or a replica’s key-deleted notice) drops what its iterators took from it: an unfiltered iterator takes its segments again at its next page, a filtered one keeps an empty index (its rows are gone, and an index is built only at the open).

match_cond — the conditions an iterator honors

backward
only_md                 (don't load jn_record on calling callbacks; honored by
                        the historical load AND the realtime rt_mem/rt_disk
                        feeds, which hand the callback NULL jn_record)

rkey                    (str) PCRE2 regex over keys ("" == ".*"), for a KEYLESS
                        list (no `key`) = the whole topic, narrowed to the keys
                        matching it. Honored by BOTH halves: the disk load only
                        visits matching keys, and the realtime feed only
                        dispatches appends of matching keys.

from_rowid / to_rowid
from_t   / to_t         t:  PERSISTENCE time (when the record was appended)
from_tm  / to_tm        tm: MESSAGE time (the record's tkey field)

user_flag / not_user_flag / user_flag_mask_set / user_flag_mask_notset

t and tm are two independent axes and their ranges combine (AND). Both are expressed in the topic’s own unit: seconds, or milliseconds when the topic sets sf_t_ms / sf_tm_ms.

A time bound can be given as text, read once when the iterator, the list or the feed (rt_mem, rt_disk) opens, and written back into its match_cond as an integer, by tranger2_time_of_json_text() -- the same rule as a text tkey:

A text that is no date, or a number refused, is refused with the WARNING “match_cond refused: a time bound is no date” (why, topic_name, bound, and the first 64 bytes of the text in value), and the open answers NULL. Up to SDK 7.29.3 a text with a T that was no date was NOW, a text without one was cut by strtoll() ("garbage" was no bound, "2026-10-01 00:00:00 +0000" the time 2026, " 5" 5, "10.5" 10), and a feed did not read a text bound at all.

The open writes the bounds it read into its match_cond, as integers in the unit of ITS topic, and so the negative bounds it resolves for the key. A match_cond is the open’s own: a caller that opens several topics or keys with one condition hands each open a copy, never the same json increfed. Only the library shares one, where it is the same topic and key: tranger2_open_list() with a key hands its history iterator and then its feed the same match_cond, so the feed gets the bounds the iterator resolved for that key (a keyless list gives each key’s iterator a copy, and its feed the negative bounds unresolved). Up to 7.29.3 tr2search -r did, and every topic read a date in the unit of the first one.

json_t *cond = json_pack("{s:s, s:I}",
    "from_t", "2026-10-01T00:00:00+0000",
    "load_record_callback", (json_int_t)(uintptr_t)load_cb
);
for(int i = 0; topics[i]; i++) {
    json_t *list = tranger2_open_list(tranger, topics[i],
        json_deep_copy(cond),   /* each topic reads the date in its own unit */
        NULL, NULL, FALSE, NULL);
    /* ... */
}
JSON_DECREF(cond)
json_t *it = tranger2_open_iterator(tranger, "raw_t", "r1",
    json_pack("{s:s}", "from_t", "2026-10-01 00:00:00 +0000"),   /* the date */
    NULL, "day", "", data, NULL);
it = tranger2_open_iterator(tranger, "raw_t", "r1",
    json_pack("{s:s}", "from_t", "garbage"),
    NULL, "bad", "", data, NULL);
/*  NULL; WARNING "match_cond refused: a time bound is no date", why "a text that is no date"  */
it = tranger2_open_iterator(tranger, "raw_t", "r1",
    json_pack("{s:s}", "from_t", "10.5"),
    NULL, "bad", "", data, NULL);
/*  NULL; the same WARNING, why "a number that is not the text of a JSON integer"  */
it = tranger2_open_iterator(tranger, "raw_t", "r1",
    json_pack("{s:s}", "from_t", ""),
    NULL, "all", "", data, NULL);
/*  every record: "" is no bound  */

A negative t / tm bound is relative to the key’s last record: its t, and its tm -- the tm of the last row of the file that holds the key’s highest t, not the highest tm of the key (which a replica and a reloaded master do not know: they read the first and last rows of each file). from_t = -N takes the records after last_t - N (that time excluded), and to_t = -N takes the ones up to last_t - N. The bound is resolved once, when the iterator opens, and written back into its match_cond. A list of several keys gives each key’s iterator its own copy, so each key is bounded by its own last record (before v7 the bound was relative to the TOPIC’s last record, of any key). The realtime half of a list of several keys keeps the bound unresolved -- there is no one last record for all of them -- so a negative from bounds nothing there and a negative to takes no new record. On a key with no record yet a negative from bounds nothing and a negative to takes no row. A db_history that starts with the last day of its key:

json_t *match_cond = json_pack("{s:I}",
    "from_t", (json_int_t)-86400        // the last 24 h before the last record
);

Up to 7.25.22 a negative bound reached the comparison raw, against an unsigned __t__, and no record matched -- without a word. (Before v7 it already meant “relative to the last record”, and that is the meaning restored.)

A negative from_rowid / to_rowid counts rows from the key’s end: -1 is its last live row. It is resolved the same way, once, when the iterator opens, and written back into its match_cond as the absolute g_rowid. So a filtered iterator opened with from_rowid = -4 indexes the last 4 rows of the key at its open, and pages those 4 rows when the key grows and when its old files are purged later (tranger2_delete_key_files()). A new open takes the last 4 rows of the key as it is then:

json_t *it = tranger2_open_iterator(tranger, "tracks", key,
    json_pack("{s:I}", "from_rowid", (json_int_t)-4),  // rows 6..9 of a key of 9 rows
    NULL, "last4", "", NULL, NULL
);
/*  4 more rows appended, then the old files purged: its pages still give 6..9  */
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 10, FALSE);

In 7.27.5 and 7.27.6 the bound was resolved again when a purge took the segments again: with rows appended since the open, the pages came back empty with “indexed rowid without segment”.

A replica whose numbering of the key moves to the master’s (the master purged rows it had not counted, a md2 file it flagged unreadable: the WARNING “The numbering of the key moved to the master’s”) moves by the same delta the index of its filtered iterators AND the absolute rowid bounds of their match_cond, positive ones included: a pager opened with from_rowid=4, to_rowid=6 over rows C2 D1 D2 of the replica pages C2 D1 D2 after the purge, numbered 6..8. In 7.27.5 and 7.27.6 a positive to_rowid stayed behind the index, and the page held one row, with “indexed rowid without segment”.

The two axes are not ordered the same way, and the scan knows it:

Both hold in both directions, and for a paged iterator too. With the rows t=100 E1, t=50000 E2, t=200 E3 (late) in one file:

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "readings", "0000000000000000001",
    json_pack("{s:I, s:I, s:b}",
        "from_t", (json_int_t)150,
        "to_t", (json_int_t)250,
        "backward", 0
    ),
    NULL, "range", "", data, NULL
);
// data holds E3 (up to 7.25.3: nothing, the scan stopped at E2)
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)

And with three day files of one key, tm 100, 5000 and 150, the middle one is out of to_tm = 300 and the scan steps over it:

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "readings", "K",
    json_pack("{s:I}", "to_tm", (json_int_t)300),
    NULL, "range", "", data, NULL
);
// data holds D1 D3 (7.25.4: D1, and "next rowids not consecutive")
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)

Every condition is honored per record, in both modes of the iterator:


tranger2_open_list()

tranger2_open_list() opens a list of records in memory, optionally enabling real-time updates via memory or disk.

json_t *tranger2_open_list(
    json_t *tranger,
    const char *topic_name,
    json_t *match_cond, // owned
    json_t *extra,      // owned, will be added to the returned rt
    const char *rt_id,
    BOOL rt_by_disk,
    const char *creator
);

Parameters

KeyTypeDescription
trangerjson_t *The TimeRanger database instance.
topic_nameconst char *The name of the topic to open.
match_condjson_t *A JSON object specifying filtering conditions for records. Owned by the function.
extrajson_t *Additional user data to be added to the returned real-time object. Owned by the function.
rt_idconst char *The real-time list identifier.
rt_by_diskBOOLIf TRUE, enables real-time updates via disk. Otherwise, uses memory.
creatorconst char *The identifier of the entity creating the list.

Returns

Returns a JSON object representing the real-time list (rt_mem or rt_disk), or, when real-time is disabled (to_rowid set), the extra given, tagged "list_type": "no_rt". Close either with tranger2_close_list().

NULL (logged, and match_cond and extra are consumed) on error: no topic, no load_record_callback, a bad rkey, a real-time feed that cannot be opened, no extra for a no_rt list, and when the history of THE key of a list of one key (key set) was not loaded whole (the load_failed of tranger2_open_iterator()).

A keyless list with a key that cannot be loaded logs the key (“Cannot load the whole history of a key of the list: the records read before the failure were handed, the list goes on with the next key”), loads every other key, opens its real-time feed, and says what it lacks in the handle it returns:

{
    "list_type": "rt_mem",
    "load_failed": true,
    "load_failed_keys": ["B"]
}

The records already handed to the callback stay handed: of a failed key, the ones read before the failure (forward: its oldest, backward: its newest). A caller that answers a question from the list must read the flag, or it takes “not in the list” for “not on disk”:

json_t *list = tranger2_open_list(tranger, "devices", match_cond, extra, "", FALSE, "");
if(list && json_is_true(json_object_get(list, "load_failed"))) {
    json_t *keys = json_object_get(list, "load_failed_keys");
    // these keys are on disk and not in what the callback saw
}

What the callers in the SDK do with it:

CallerWith load_failed
treedb (every topic, __snaps__, __graphs__)remembers the keys, refuses a create of such an id, refuses shoot/activate of a snap when __snaps__ has any, and its asset guards fail closed (see TreeDB).
treedb’s snapshot guard of the assetsa walk that did not load everything fails closed: the gc and the delete of an asset refuse.
tr_queue / tr2q_mqtt (trq_load(), tr2q_load())the pending messages read are in memory, but those of a row that cannot be read are not, and are not delivered. The load returns -1 and does NOT move nor save the queue’s first_rowid, so the next load, once the store is repaired, finds them (up to 7.25.4 it saved the size of the topic, and they were skipped for ever).
msg2dbreloads the key newest first, up to the damage. The id is marked incomplete (msg2db_id_incomplete()) and an ERROR names it with served=N. See tr_msg2db.

Until 7.25.4 such a list was handed over silently, with the key missing.

Notes

Loading all records can introduce delays in application startup. Use filtering conditions in match_cond to optimize performance.

The history is loaded with one-shot iterators of their own creator (__tranger2_open_list__): an iterator the caller keeps open on a key does not make the load “already exist”.

json_t *list = tranger2_open_list(
    tranger, "devices",
    json_pack("{s:I, s:I}",
        "to_rowid", (json_int_t)1000000,    // one-shot, no realtime
        "load_record_callback", (json_int_t)(uintptr_t)load_cb
    ),
    json_object(),      // extra: IS the returned handle of a no_rt list
    "", FALSE, "me"
);
if(!list) {
    // refused (logged)
} else {
    if(json_is_true(json_object_get(list, "load_failed"))) {
        // some key's history could not be read: what load_cb got is not the
        // whole topic, and "load_failed_keys" names those keys
    }
    tranger2_close_list(tranger, list);
}

tranger2_open_rt_disk()

Opens a real-time disk-based iterator for monitoring changes in a topic. The function allows tracking new records appended to the topic by monitoring disk events.

json_t *tranger2_open_rt_disk(
    json_t *tranger,
    const char *topic_name,
    const char *key,        // if empty receives all keys, else only this key
    json_t *match_cond,     // owned
    tranger2_load_record_callback_t load_record_callback,   // called on append new record on disk
    const char *rt_id,      // rt disk id, REQUIRED
    const char *creator,
    json_t *extra           // owned, user data, this json will be added to the return iterator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic to monitor.
keyconst char *Specific key to monitor. If empty, all keys are monitored.
match_condjson_t *JSON object containing conditions for filtering records. Owned by the function.
load_record_callbacktranger2_load_record_callback_tCallback function invoked when a new record is appended to the disk.
rt_idconst char *Unique identifier for the real-time disk iterator. Required.
creatorconst char *Identifier of the entity creating the iterator.
extrajson_t *Additional user data to be associated with the iterator. Owned by the function.

Returns

Returns a JSON object representing the real-time disk iterator. The caller does not own the returned object.

Notes

This function is used when monitoring real-time changes in a topic via disk events. The iterator remains active until explicitly closed using tranger2_close_rt_disk().

The rowid the callback receives is the key’s global g_rowid — the one that never resets — not the row’s position inside the .md2 file it lives in (that one is i_rowid, and it restarts at 1 with every new file). A consumer can therefore dedupe or page by it across a file rotation.

Several feeds can be open on the same key (a per-key one and a whole-topic one, say). Each is served exactly once per record: the feed keeps a watermark of the last row it was given, per key and per file -- one mark per FILE, so a batch that touches two files of a key (a late record and a current one, or a rotation while the follower is busy) serves each feed both (before 7.25.0, the second feed lost both) -- and the master hard-links each new .md2 into the directory of every feed that wants the key. A feed’s watermark dies with its key, so a key re-created later does not inherit it.

The path of that link, disks/<rt_id>/<key>/<file>, is longer than the master’s keys/<key>/<file>, so a key the master stores can be too long for a feed. Its records do not reach that feed, and the master logs it ONCE for the key and the feed, at its first record (“Path too long, the records of this key do not reach this feed (logged once)”). An unreleased development version logged an ERROR with a stack for every record and every feed. Up to 7.29.3, the path was cut in silence and the link was made from the cut path. For example, with a feed directory of 3962 bytes, a key of 200 bytes (no room for <key>) or of 125 bytes (no room for /2000-01-01.md2 after it): five records log one ERROR for each key.


tranger2_open_rt_mem()

Opens a real-time memory stream for a given topic in tranger. This function enables real-time message processing for the specified key and applies filtering conditions from match_cond. The callback tranger2_load_record_callback_t is invoked when new records are appended.

json_t *tranger2_open_rt_mem(
    json_t *tranger,
    const char *topic_name,
    const char *key,        // if empty receives all keys, else only this key
    json_t *match_cond,     // owned
    tranger2_load_record_callback_t load_record_callback,   // called on append new record on mem
    const char *list_id,    // rt list id, optional (internally will use the pointer of rt)
    const char *creator,
    json_t *extra           // owned, user data, this json will be added to the return iterator
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic for which real-time memory streaming is enabled.
keyconst char *Specific key to monitor. If empty, all keys are monitored.
match_condjson_t *JSON object containing filtering conditions. Owned by the function.
load_record_callbacktranger2_load_record_callback_tCallback function invoked when a new record is appended in memory.
list_idconst char *Optional identifier for the real-time list. If empty, the internal pointer is used.
creatorconst char *Identifier of the entity creating the real-time memory stream.
extrajson_t *Additional user data stored in the returned iterator. Owned by the function.

Returns

Returns a JSON object representing the real-time memory stream. The caller does not own the returned object.

Notes

This function is valid when the Yuno instance is the master writing real-time messages from tranger2_append_record().


tranger2_open_topic()

The tranger2_open_topic() function opens a topic in the TimeRanger database. If the topic is already open, it returns the existing topic JSON object.

json_t *tranger2_open_topic(
    json_t  *tranger,
    const char  *topic_name,
    BOOL  verbose
);

Parameters

KeyTypeDescription
trangerjson_t *A JSON object representing the TimeRanger database.
topic_nameconst char *The name of the topic to open.
verboseBOOLIf TRUE, additional debug information is printed.

Returns

A JSON object representing the opened topic. The returned object is not owned by the caller and must not be modified or freed.

Notes

This function is idempotent. This means that calling it multiple times with the same topic_name will return the same JSON object without creating a new instance.

It returns NULL, never a critical, when the name is refused by the topic name rule, and when the directory exists but is not a topic (it has no topic_desc.json): “Not a topic: topic_desc.json not found”, logged only with verbose. A peer can send any name, and with on_critical_error=2 a critical is an exit(0) of the yuno.

It also returns NULL when the keys/ directory of the topic exists and cannot be listed (EMFILE, EACCES, ENOMEM, or a readdir() that fails): “Cannot list the keys of the topic” (with errno), then “Cannot open topic: its keys cannot be listed”. On a filesystem whose readdir() gives no d_type (XFS with ftype=0, NFS, FUSE, overlay) each key is asked with lstat(), as the entry type tells it: a symbolic link is not a key (up to 7.25.4 stat() followed it, and made a key of another key). An lstat() that fails with EACCES (keys/ can be read and not searched) takes the entry as a key, as the entry type does, and the listing of that key says the EACCES and flags it. One that fails with anything else but ENOENT (a key removed meanwhile) fails the listing the same way: “Cannot list the keys of the topic, lstat() FAILED”. Up to 7.25.4 that key was taken as “not a directory” and left out with no log, and the topic opened without it. The topic is not left open: opened with no keys, a key nobody read looked like a key that does not exist, and a treedb accepted a create of its id (so up to 7.25.4). The next open, or the next tranger2_topic(), tries again:

json_t *topic = tranger2_open_topic(tranger, "devices", TRUE);
if(!topic) {
    // Error already logged: not a topic, a refused name, or keys/ cannot be listed.
    // Nothing of the topic is in memory; try again later.
    return -1;
}

A keys/ that does not exist (ENOENT) is still a topic with no key, as before.


tranger2_print_md0_record()

Prints metadata of a record, including row ID, time, message time, and key, into a buffer.

void tranger2_print_md0_record(
    char *bf,
    int bfsize,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

KeyTypeDescription
bfchar *Buffer to store the formatted metadata string.
bfsizeintSize of the buffer to prevent overflow.
keyconst char *Key associated with the record.
md_record_exconst md2_record_ex_t *Pointer to the record metadata structure.
print_local_timeBOOLFlag indicating whether to print time in local format.

Returns

This function does not return a value.

Notes

The function formats the metadata into the provided buffer. This makes sure of it does not exceed bfsize. It prints row ID, time, message time, and key.


tranger2_print_md1_record()

Prints metadata information of a record, including row ID, user flag, system flag, timestamps, and key, into a buffer.

void tranger2_print_md1_record(
    char *bf,
    int bfsize,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

KeyTypeDescription
bfchar *Buffer where the formatted metadata string will be stored.
bfsizeintSize of the buffer to make sure that safe writing.
keyconst char *Key associated with the record.
md_record_exconst md2_record_ex_t *Pointer to the record metadata structure containing the information to be printed.
print_local_timeBOOLFlag indicating whether to print timestamps in local time (if TRUE) or UTC (if FALSE).

Returns

This function does not return a value.

Notes

The function formats and writes metadata details into the provided buffer. This makes sure that the output does not exceed bfsize bytes.


tranger2_print_md2_record()

Prints detailed metadata of a record, including row ID, offset, size, timestamp, and file path, into the provided buffer.

void tranger2_print_md2_record(
    char *bf,
    int bfsize,
    json_t *tranger,
    json_t *topic,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

KeyTypeDescription
bfchar *Buffer where the formatted metadata string will be stored.
bfsizeintSize of the buffer to make sure that safe writing.
trangerjson_t *Reference to the TimeRanger database instance.
topicjson_t *JSON object representing the topic associated with the record.
keyconst char *Key identifying the record within the topic.
md_record_exconst md2_record_ex_t *Pointer to the extended metadata structure of the record.
print_local_timeBOOLFlag indicating whether to print timestamps in local time.

Returns

This function does not return a value.

Notes

The function formats metadata details into the provided buffer. This makes sure that the output remains within the specified buffer size.


tranger2_print_record_filename()

Formats and stores the filename of a record in a buffer, using metadata from tranger2_print_record_filename().

void tranger2_print_record_filename(
    char *bf,
    int bfsize,
    json_t *tranger,
    json_t *topic,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

KeyTypeDescription
bfchar *Pointer to the buffer where the formatted filename will be stored.
bfsizeintSize of the buffer in bytes.
trangerjson_t *Pointer to the TimeRanger database instance.
topicjson_t *Pointer to the topic associated with the record.
md_record_exconst md2_record_ex_t *Pointer to the record metadata structure.
print_local_timeBOOLFlag indicating whether to format timestamps in local time.

Returns

This function does not return a value.

Notes

The buffer bf must be large enough to store the formatted filename. The function retrieves metadata from tranger2_print_record_filename() to construct the filename.


tranger2_read_record_content()

Reads the content of a record from a given topic in the TimeRanger database. The function retrieves the record’s data based on its metadata.

json_t *tranger2_read_record_content(
    json_t            *tranger,
    json_t            *topic,
    const char        *key,
    md2_record_ex_t   *md_record_ex
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topicjson_t *Pointer to the topic from which the record will be read.
keyconst char *The key identifying the record to be read.
md_record_exmd2_record_ex_t *Pointer to the metadata structure of the record to be retrieved.

Returns

Returns a json_t * object containing the record’s content. The caller owns the returned JSON object and must free it when no longer needed.

Notes

This function is useful when only metadata was loaded and the full record content needs to be retrieved.


tranger2_read_user_flag()

The tranger2_read_user_flag() function retrieves the user flag associated with a specific record in a given topic.

uint16_t tranger2_read_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic containing the record.
rowiduint64_tUnique identifier of the record whose user flag is to be retrieved.

Returns

Returns the uint16_t user flag associated with the specified record.

Notes

The function is used in writing mode to check the user flag of a record before modifying it.


tranger2_set_trace_level()

Sets the trace level of the tranger instance, controlling the verbosity of logging and debugging output.

void tranger2_set_trace_level(
    json_t *tranger,
    int      trace_level
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the tranger instance whose trace level is to be modified.
trace_levelintThe new trace level to be set, determining the verbosity of logging.

Returns

This function does not return a value.

Notes

Higher trace levels typically enable more detailed logging, which can be useful for debugging.


tranger2_set_system_flag()

Sets or clears specific bits in the system flag of a record — the metadata band reserved for the framework (for example the immutable-record bit), distinct from the user flag set by tranger2_set_user_flag().

int tranger2_set_system_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t mask,
    BOOL set
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic containing the record.
keyconst char *Record key — with __t__ and rowid it identifies the record metadata.
__t__uint64_tRecord time.
rowiduint64_tReal rowid in the file (not the topic-global rowid).
maskuint16_tBitmask specifying which bits in the system flag to modify.
setBOOLIf TRUE, the bits in mask are set. If FALSE, they are cleared.

Returns

Returns 0 on success, or a negative error code on failure.

Notes

Only the bits in mask are touched. The system flag is framework-owned. User code normally goes through higher-level treedb helpers rather than setting system bits directly.

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).


tranger2_set_user_flag()

Sets or clears specific bits in the user flag of a record in a topic within the TimeRanger database.

int tranger2_set_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t mask,
    BOOL set
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic containing the record.
rowiduint64_tUnique identifier of the record whose user flag is modified.
maskuint32_tBitmask specifying which bits in the user flag must be modified.
setBOOLIf TRUE, the bits specified in mask are set. If FALSE, they are cleared.

Returns

Returns 0 on success, or a negative error code on failure.

Notes

This function modifies only the bits specified in mask. This leaves other bits in the user flag unchanged.

// Mark row 3 of key "dev-1" (its __t__ is 1731601280) as pending, keep the other bits
if(tranger2_set_user_flag(tranger, "messages", "dev-1", 1731601280, 3, 0x0001, TRUE) < 0) {
    // not the master, or the row is not there: logged
}

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).


tranger2_shutdown()

The tranger2_shutdown() function stops the TimeRanger database, releasing all allocated memory.

int tranger2_shutdown(
    json_t *tranger
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance to be stop.

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

This function must be called when the database is no longer needed to free resources.


tranger2_startup()

Initializes the TimeRanger 2 database, setting up its internal structures and preparing it for use. The function requires a hgobj instance and a JSON configuration object that defines database parameters.

json_t *tranger2_startup(
    hgobj      gobj,
    json_t    *jn_tranger,
    yev_loop_h yev_loop
);

Parameters

KeyTypeDescription
gobjhgobjA handle to the GObj framework instance managing the database.
jn_trangerjson_t *A JSON object containing database configuration parameters. See tranger2_json_desc for details.
yev_loopyev_loop_hAn event loop handle for asynchronous operations.

Returns

A JSON object representing the initialized TimeRanger 2 database instance, or NULL on failure (logged).

Notes

The returned JSON object must be properly managed and eventually passed to tranger2_stop() or tranger2_shutdown() to release resources.

A path or a path/database that does not fit in PATH_MAX is refused: NULL, and the ERROR of build_path(). Up to 7.29.3 the store was made in the directory that fitted, the one above the store asked for.

A master that finds __timeranger2__.json EMPTY makes it again, as if it were not there: the write of a first open did not end (the disk was full, or the process died between the create and the write). It says it with a WARNING, “timeranger2.json is empty (its first write did not end): made again”, then the INFO “Creating timeranger2.json”. A replica ("master": false) still refuses the empty file (“Cannot load json file, bad json”): it cannot write. Up to 7.30.18 a master refused it too, at every try, and the store never opened again, even when the disk had space again. To see it on a node:

ls -l /yuneta/store/myapp/events/__timeranger2__.json
# -rw-rw---- 1 yuneta yuneta 0 ...   <- empty: the next open of a master makes it again
# -r--r----- 1 yuneta yuneta 0 ...   <- empty in mode 0440: made again too

The file is saved and then set to mode 0440. On a filesystem with delayed allocation (ext4), a power cut can keep the mode and lose the data, so an empty file can be 0440. The master makes it writable again (mode rpermission) before it writes it. Up to 7.30.19 that write failed with “Cannot create json file” (EACCES) for any user but root, and the yuno exited.

json_t *tranger = tranger2_startup(gobj, json_pack("{s:s, s:s, s:b}",
    "path", "/yuneta/store/myapp",
    "database", "events",
    "master", 1
), yev_loop);
if(!tranger) {
    return -1;  // Error already logged
}

tranger2_stop()

The tranger2_stop() function closes the TimeRanger database. This makes sure that all topics and file descriptors are properly closed.

int tranger2_stop(
    json_t *tranger
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance to be closed.

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

tranger2_shutdown() calls it when nobody did. Call it yourself when the event loop needs a turn between the two, to finish the watchers the topics close.

What the stop does:

This is the life of the tranger of a C_TRANGER that is stopped and started again: the tranger is built in mt_create, stopped in mt_stop, and shut down only in mt_destroy.

json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop);   // master, lock taken
tranger2_open_topic(tranger, "devices", TRUE);

tranger2_stop(tranger);         // topics closed, lock given back
                                // (C_TRANGER mt_stop)

tranger2_topic(tranger, "devices");  // opens it again: revived, lock retaken
                                     // (C_TRANGER started again)

tranger2_shutdown(tranger);     // closes what the revival opened

And when another process took the store while it was stopped:

tranger2_stop(a);                                   // a gives the lock back
json_t *b = tranger2_startup(0, jn_tranger_b, 0);   // b takes the store

tranger2_create_topic(a, "devices", "id", "tm", 0, 0, 0, 0);
// a cannot take the lock: nothing written, "devices" opened read-only
kw_get_bool(0, a, "master", 0, 0);        // FALSE
kw_get_bool(0, a, "master_lost", 0, 0);   // TRUE

tranger2_str2system_flag()

Converts a formatted string containing system flag representations into a system_flag2_t integer. The input string can use delimiters such as ‘|’, ’ ', or ‘,’.

system_flag2_t tranger2_str2system_flag(
    const char *system_flag
);

Parameters

KeyTypeDescription
system_flagconst char *A string containing system flag values separated by ‘|’, ’ ', or ‘,’.

Returns

Returns the corresponding system_flag2_t integer representing the parsed system flags.

Notes

This function is useful for converting human-readable flag representations into their corresponding bitmask values.


tranger2_topic()

Retrieve a topic by its name from the TimeRanger database. If the topic is not already opened, tranger2_open_topic() is called to open it.

json_t *tranger2_topic(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic to retrieve.

Returns

A JSON object representing the topic. The returned object is not owned by the caller and must not be modified or freed.

Notes

If the topic exists on disk but has not been opened yet, tranger2_open_topic() is automatically invoked.


tranger2_topic_desc()

tranger2_topic_desc() retrieves the description of a specified topic from the TimeRanger database.

json_t *tranger2_topic_desc(
    json_t *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *A reference to the TimeRanger database instance.
topic_nameconst char *The name of the topic whose description is to be retrieved.

Returns

A JSON object containing the topic description. The caller must decrement the reference count of the returned JSON object.

The description is a projection of the topic, not the topic itself:

KeyDescription
topic_nameThe name of the topic.
pkeyThe column that is the primary key.
pkey2sThe columns that are secondary keys, if the topic declares any.
tkeyThe column that carries the time, if the topic declares one.
system_flagThe system flag the topic was created with.
topic_versionThe version of the topic schema.
system_topictrue when the topic cannot be deleted.
main_topictrue on the topic the tree of a treedb hangs from, as the schema marks it (a topic hooked to itself, one per treedb, stamped by treedb_open_db()).
colsThe column descriptors, as a list.

A key the topic does not carry is omitted rather than returned empty, so a topic with no secondary key has no pkey2s at all.

pkey2s matters to any reader that must NAME a record and not only identify it. A topic whose id column is flagged rowid, uuid or qualified keys its records by a value that is not the name: a rowid and a UUID mean nothing to a person, and a qualified id (<parent id>.<name>) names every ancestor together with the record. treedb_system_schema keys topics and cols with qualified ids. In all three the name lives in the secondary key. A viewer given only pkey can do nothing but print the address.

Notes

The returned JSON object must be properly decremented using json_decref() to avoid memory leaks.


tranger2_topic_key_size()

Retrieves the number of records associated with a specific key in a given topic within the TimeRanger database. If key is empty, the whole-topic size is returned instead.

uint64_t tranger2_topic_key_size(
    json_t  *tranger,
    const char *topic_name,
    const char *key
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic containing the key.
keyconst char *The key whose record count is to be retrieved. If empty, the whole-topic size is returned (sum over every key).

Returns

Returns the number of records associated with the specified key, or the whole-topic total when key is empty.

Notes

Counts are from the in-memory cache. A missing topic returns 0. When key is empty the call delegates to tranger2_topic_size().


tranger2_topic_key_range()

Returns the time span of one key of a topic, on both axes, read from the in-memory cache totals (maintained on load and on every append) — so a client can bound a time picker to what the key holds without reading a single record.

json_t *tranger2_topic_key_range( // return is yours
    json_t *tranger,
    const char *topic_name,
    const char *key
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic containing the key.
keyconst char *The key whose span is to be retrieved.

Returns

A JSON object, yours to decref:

{"fr_t": t, "to_t": t, "fr_tm": tm, "to_tm": tm, "rows": n}

rows is the LIVE count, the rows that can be read. A key whose old files were deleted (tranger2_delete_key_files()) also has "purged": n, its base: the rows of the files deleted. Its live g_rowids are purged+1 .. purged+rows, so its LAST g_rowid is purged + rows, not rows. A key never purged has no purged. On a replica, "renumbered": n says how far the g_rowids of the key moved when it took a base of the master that counted rows it had not (a file it had flagged unreadable): whoever keeps positions of the key outside its iterators moves them by it.

NULL if the topic (logged) or the key (silent) is unknown.

Example

/*
 *  The last g_rowid of a key, to page it by g_rowid: a key of 1823411 rows
 *  purged and 4000 live answers {"rows": 4000, "purged": 1823411, ...}
 */
json_t *range = tranger2_topic_key_range(tranger, "raw_events", "dev1");
if(range) {
    json_int_t last_rowid = kw_get_int(gobj, range, "purged", 0, 0) +
                            kw_get_int(gobj, range, "rows", 0, 0);   // 1827411
    JSON_DECREF(range)
}

Notes

t (persistence) and tm (message time) are independent, so the two spans differ whenever data was backfilled. Both are in the topic’s own unit: seconds, or milliseconds when the topic sets sf_t_ms / sf_tm_ms (read system_flag from the topic desc).


tranger2_name_is_identifier()

TRUE if name can name a topic, a treedb or a msg2db. It is the name rule of everything that is a DIRECTORY, kept in one place (since SDK 7.26.9). Such a name is a directory (<database>/<topic>/), a key of the in-memory json and a segment of the kw paths built from it, so it must not be empty, "." or "..", nor hold / or the path delimiter `. A leading . is allowed. Silent: the caller says what the name was for. A column, a hook or an fkey is only a key, and follows the looser treedb_col_name_is_identifier().

BOOL tranger2_name_is_identifier(
    const char *name
);

Parameters

KeyTypeDescription
nameconst char *The name to check.

Returns

TRUE if the name is usable, FALSE otherwise.

Example

tranger2_name_is_identifier("users");       // TRUE
tranger2_name_is_identifier(".hidden");     // TRUE: a leading '.' is allowed
tranger2_name_is_identifier("users`id");    // FALSE: the path delimiter
tranger2_name_is_identifier("kids/all");    // FALSE: a path separator
tranger2_name_is_identifier("..");          // FALSE

tranger2_reserved_name_fault()

Why name cannot name a NEW topic from outside (a command), or NULL. Beside its topics, the directory of a store holds what the libraries keep there. A topic with such a name is a directory over that file or folder, now or later, when the library writes it (it is possible that the store does not exist yet). So these names are refused:

NameWhyWhat the store keeps there
a leading .“a name starting with ‘.’ is a hidden file of the store”.blobs (the files of a treedb), .<treedb>.treedb_schema.json.tmp (apply-schema)
a leading __“a name starting with ‘__’ is of the system”__timeranger2__.json, the system topics of a treedb (__snaps__, __graphs__, __assets__, __icons__)
a .json suffix“a name ending in ‘.json’ is a file of the store”__timeranger2__.json, <treedb>.treedb_schema.json, <msg2db>.msg2db_schema.json
a .bak suffix“a name ending in ‘.bak’ is the backup of a topic”the backup of tranger2_backup_topic()

The name rule (tranger2_name_is_identifier()) is not asked here: ask it too. The function logs nothing. The caller says what the name was for. A topic that a C caller creates is not asked this: the system topics of a treedb, and the queues of the MQTT broker, named <client_id>-IN / <client_id>-OUT after a client id the client chose (a client .foo has the queues .foo-IN and .foo-OUT), are made by C code. C_TRANGER and C_TREEDB ask it of the new topics of a peer (since SDK 7.27.0).

const char *tranger2_reserved_name_fault(
    const char *name
);

Parameters

KeyTypeDescription
nameconst char *The name of the new topic. An empty or NULL name answers NULL: the name rule refuses it.

Returns

A static string with the cause, or NULL when a topic can take the name.

Example

tranger2_reserved_name_fault("users");          // NULL
tranger2_reserved_name_fault(".blobs");         // "a name starting with '.' is a hidden file of the store"
tranger2_reserved_name_fault("__snaps__");      // "a name starting with '__' is of the system"
tranger2_reserved_name_fault("x.json");         // "a name ending in '.json' is a file of the store"
tranger2_reserved_name_fault("users.bak");      // "a name ending in '.bak' is the backup of a topic"
tranger2_reserved_name_fault("a/b");            // NULL: tranger2_name_is_identifier() refuses it

/*  a new topic that a peer names  */
const char *why = tranger2_reserved_name_fault(topic_name);
if(why) {
    gobj_log_warning(gobj, 0,
        "function",     "%s", __FUNCTION__,
        "msgset",       "%s", MSGSET_PARAMETER,
        "msg",          "%s", "topic name of the store, refused",
        "why",          "%s", why,
        "topic_name",   "%s", topic_name,
        NULL
    );
    return -1;
}

tranger2_topic_var_fault()

Why jn_var cannot be the topic_var.json of a topic from outside (a command), or NULL. tranger2_create_topic() merges jn_var into the topic, and a new topic_version in it writes the topic’s topic_cols.json and topic_var.json again. A NULL or a json null is no var. Anything else must be a dict that:

The function logs nothing. The caller says it. C_TRANGER’s create-topic asks it of a peer’s jn_var (since SDK 7.27.0).

const char *tranger2_topic_var_fault(
    json_t *jn_var
);

Parameters

KeyTypeDescription
jn_varjson_t *The var of the topic, not owned. NULL or a json null: no var.

Returns

A static string with the cause, or NULL when the var can be used.

Example

tranger2_topic_var_fault(NULL);                                     // NULL: no var
json_t *v = json_pack("{s:i}", "topic_version", 2);
tranger2_topic_var_fault(v);                                        // NULL
json_decref(v);

v = json_pack("{s:s}", "topic_version", "2");
tranger2_topic_var_fault(v);    // "the topic_version of jn_var is not an integer from 0 to 2147483647"
json_decref(v);
v = json_pack("{s:b}", "system_topic", 1);
tranger2_topic_var_fault(v);    // "jn_var marks a system topic, which nothing deletes"
json_decref(v);
v = json_pack("{s:s}", "directory", "/tmp");
tranger2_topic_var_fault(v);    // "jn_var has a field the tranger keeps itself"
json_decref(v);
v = json_pack("{s:i}", "last_rowid_id", -1);
tranger2_topic_var_fault(v);    // "the last_rowid_id of jn_var is not an integer, 0 or more"
json_decref(v);
v = json_pack("[i]", 2);
tranger2_topic_var_fault(v);    // "jn_var is not a json object"
json_decref(v);

From a peer, through C_TRANGER:

command-yuno id=<id> service=tranger command=create-topic topic_name=pp jn_var='{"system_topic":true}'
# -> -1 <role^name>: cannot create topic 'pp': jn_var marks a system topic, which nothing deletes

tranger2_topic_is_open()

TRUE if the topic is currently open in this tranger. Silent — a closed topic is a legitimate answer, not an error.

BOOL tranger2_topic_is_open(
    json_t *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic to check.

Returns

TRUE if the topic is open, FALSE otherwise.

Notes


tranger2_topic_name()

Retrieves the topic name from the given json_t * topic object.

const char *tranger2_topic_name(
    json_t *topic
);

Parameters

KeyTypeDescription
topicjson_t *A JSON object representing the topic.

Returns

A pointer to a string containing the topic name. The returned string must not be modified or freed by the caller.

Notes

If the topic parameter is NULL or invalid, the behavior is undefined.


tranger2_topic_size()

tranger2_topic_size() retrieves the total number of records present in a specified topic within the TimeRanger database.

uint64_t tranger2_topic_size(
    json_t *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
trangerjson_t *Pointer to the TimeRanger database instance.
topic_nameconst char *Name of the topic whose record count is to be retrieved.

Returns

Returns the total number of records in the specified topic as a uint64_t value.

Notes

If the topic does not exist, the function can return 0.


tranger2_write_topic_cols()

The tranger2_write_topic_cols() function updates the column definitions of a specified topic in the TimeRanger database.

int tranger2_write_topic_cols(
    json_t  *tranger,
    const char  *topic_name,
    json_t  *jn_cols  // owned
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic whose column definitions are updated.
jn_colsjson_t *A JSON object containing the new column definitions. The ownership of this object is transferred to tranger2_write_topic_cols().

Returns

Returns 0 on success, or a negative error code on failure.

Notes

Master-only. Replaces the topic’s columns wholesale — both the in-memory topic["cols"] and the on-disk topic_cols.json (not a merge). A columns change must bump the topic’s topic_version (and schema_version for structural changes), otherwise the persisted topic_cols.json masks the new schema on the next reload.

The file is replaced, never written in place: topic_cols.json.new is removed if a process left it, created O_EXCL|O_NOFOLLOW with the tranger’s rpermission, written, fsync’ed, renamed over the old file, and the directory fsync’ed: it survives a power cut too (it is written when a topic is created, re-versioned or re-ordered, never per record, and up to 7.25.4 it was not fsync’ed: a topic_version change was able to leave the new version on disk over cols lost with the power). The memory takes the new columns only when the file did. Otherwise, the call logs (“Cannot replace topic_cols.json, ...”) and returns -1 with the old file and the old columns in place. Up to 7.25.4 it wrote in place, put the columns in memory before writing, and returned 0 whatever happened. tranger2_write_topic_var() replaces topic_var.json the same way.

json_t *cols = json_pack("{s:s, s:s, s:s}", "id", "", "content", "", "extra", "");
if(tranger2_write_topic_cols(tranger, "readings", cols) < 0) {  // cols consumed either way
    // not written: topic_cols.json and topic["cols"] are the old ones
}

One change does not wait for the bump. When the incoming schema holds the same columns saying the same things and only their order differs, tranger2_create_topic() rewrites the file: the freeze is there so that a change to what a column declares cannot arrive unannounced, and an order announces nothing new.


tranger2_write_topic_var()

The tranger2_write_topic_var() function updates the variable metadata of a specified topic in the TimeRanger database.

int tranger2_write_topic_var(
    json_t  *tranger,
    const char  *topic_name,
    json_t  *jn_topic_var
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic whose variable metadata is to be updated.
jn_topic_varjson_t *A JSON object containing the new variable metadata for the topic. The ownership of this object is transferred to tranger2_write_topic_var().

Returns

Returns 0 on success, or a negative value on failure.

Notes

Master-only. Merges jn_topic_var into the existing topic_var.json (update, not replace) and persists it. The in-memory topic is updated too, except the immutable descriptor fields, and only when the file was written.

The file is never rewritten in place: the merge is written to topic_var.json.new and renamed over topic_var.json, so a process that dies half way leaves the old file or the new one, never an empty one. treedb calls this on every create of a rowid-key node (the last_rowid_id counter lives here), so it does not fsync(): it is safe against the death of the process, not against a power cut. A topic_version change (tranger2_create_topic()) does fsync.

A write that fails leaves the old topic_var.json as it was and answers -1. A write cut part way (the file size limit, a full disk) is logged as “Cannot replace topic_var.json, short write: the file size limit or the disk is full”, without an errno: a short write does not set one.

tranger2_write_topic_var(tranger, "items", json_pack("{s:I}", "last_rowid_id", (json_int_t)42));
// topic_var.json: {"topic_version": 3, "last_rowid_id": 42}  (a new inode)

tranger2_write_user_flag()

The tranger2_write_user_flag() function updates the user flag of a specific record identified by rowid in the given topic_name within the TimeRanger database.

int tranger2_write_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t user_flag
);

Parameters

KeyTypeDescription
trangerjson_t *A pointer to the TimeRanger database instance.
topic_nameconst char *The name of the topic containing the record.
rowiduint64_tThe unique row identifier of the record to update.
user_flaguint32_tThe new user flag value to be assigned to the record.

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function modifies the user flag of an existing record but does not alter other record attributes. Use tranger2_set_user_flag() if you need to update the flag using a mask.

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).


tranger2_list_topic_names()

tranger2_list_topic_names() returns a JSON array of topic names by scanning the tranger database directory on disk. Unlike tranger2_list_topics(), which reads from the in-memory topic registry, this function reads subdirectory names from the filesystem.

json_t *tranger2_list_topic_names(
    json_t *tranger
);

Parameters

KeyTypeDescription
trangerjson_t *The tranger database handle (JSON object).

Returns

A new JSON array of strings, each being a topic name found as a subdirectory in the tranger database directory. The caller owns the returned array and must call json_decref() on it. Hidden entries (names starting with .) are excluded.

NULL when the directory cannot be listed (opendir() or readdir() fails): logged, “Cannot list the topics of the store” with errno, and the cause is left in gobj_log_last_message(). A store that cannot be listed is not a store with no topic: up to 7.25.4 it answered [] with no log, and list-queues and clean-queues of the MQTT broker answered an empty list with 0. They answer -1 now.

Notes

This function operates on disk, not in memory. It can return topic names that are not currently open, or miss topics that exist only in memory. Use tranger2_list_topics() to get the in-memory list instead.

Example

json_t *names = tranger2_list_topic_names(tranger);
if(!names) {
    // Error already logged: the store cannot be listed, NOT "no topic"
    return -1;
}
size_t idx; json_t *jn_name;
json_array_foreach(names, idx, jn_name) {
    printf("%s\n", json_string_value(jn_name));
}
JSON_DECREF(names)

tranger2_topic_path()

tranger2_topic_path() writes the filesystem path of a topic into the provided buffer. The path is constructed by appending the topic name to the tranger database directory.

int tranger2_topic_path(
    char *bf,
    size_t bfsize,
    json_t *tranger,
    const char *topic_name
);

Parameters

KeyTypeDescription
bfchar *Output buffer where the topic path will be written.
bfsizesize_tSize of the output buffer in bytes.
trangerjson_t *The tranger database handle (JSON object).
topic_nameconst char *Name of the topic.

Returns

Returns 0 on success.

Notes

The resulting path has the format <directory>/<topic_name>, where <directory> is the tranger database directory stored in the tranger handle.