← All posts

Self-hosted analytics data loss: 12 tools tested under a kill signal

Which tools keep acknowledged events after a component dies. 20 kills, counts reconciled against each tool's own database, dataset and run records included · Published September 23, 2026 · Runs performed September 14, 2026

A successful tracking response does not necessarily mean an event will survive a process failure.

We abruptly stopped components in eleven self-hosted analytics and error-tracking tools, then compared successful responses with database counts after recovery. Several configurations showed a deficit. Others showed none, and two tools exposed single-event losses in their processing paths.

The scope is twelve tools: eleven external ones and our own stack, with Matomo tested in two configurations (its default install and its QueuedTracking plugin on Redis). These are individual observations from two test protocols, both run on September 14: a fixed-rate kill test (kill-test/2) with the load generator on the tool's own box, and the kill at the end of our load test (ramp/2), at each tool's soak rate with the generator on a separate machine and, where the tool stores an event id, per-event reconciliation; those rows are marked. Each is a single run on a 2-vCPU box, except PostHog on the 4-vCPU box its vendor requires. Method, definitions and the dataset are below the results.

Three things we did not expect

A write-behind queue in front of a safe store moves the loss to the queue, and can hide it. The configuration that writes to its database before it answers lost nothing. Putting a memory-backed queue in front of that same database to raise throughput created two failure modes the store never had: with the queue down, the tracker logged the connection error and still acknowledged every request, so 220 acknowledged hits went nowhere while the client saw no failures; and when the queue restarted from a snapshot taken two minutes earlier, it replayed 24 hits the store already held. Matomo showed both: its default install had no deficit, and with its QueuedTracking plugin on the Redis it ships with it had the largest loss in the table as a share of acknowledged requests (253 of 1,800 verified lost, 24 duplicated). This is also the one configuration where the count deficit (229) could be separated from the verified loss and the duplicates, because the duplicates were found by grouping rows in the database. A queue that only snapshots is a durability downgrade for every event it holds, whatever the store behind it does.

A container restart policy is not a crash-recovery plan. Every compose file that shipped a restart: value (always, unless-stopped, on-failure) left its container dead through the ten-second watch after docker kill, eleven times out of eleven. Docker records an API kill as a manual stop and skips the policy; a host-side kill -9 of the same container's process, tried once on the Matomo variant, restarted it within a second. So the policies cover a process that crashes on its own, not the stop that deployment tooling, orchestrators and operators actually send. In every install tested the accept path stayed down until the driver started it after the ten-second watch, and the recovery times in the tables include that wait. In production the wait is however long it takes someone, or something, to notice.

When session state shares the buffer with the events, a kill corrupts the metrics as well as losing rows. A tool that batches hits in memory before persisting loses the batch on a kill, which its documentation predicts; here that cost 373 page views. What the documentation does not say is that the visitor identity lived in the same memory: after the restart every one of the 500 pool visitors was counted again as a new unique. GoatCounter's dashboard shows a dip in page views and a spike in visitors in the same minute, which a reader would attribute to traffic, not to the restart. Anything derived from in-memory state, session ids, deduplication keys or rate counters, goes with the buffer, and that loss shows up as wrong numbers rather than missing ones.

Results at a glance

Tool and outcome only; counts, rates and recovery are in the tables. "No count deficit" means successful responses and stored rows matched; it does not rule out losses offset by duplicates, except in the rows reconciled by event id (Umami, Rybbit, ClickStack and our own stack under ramp/2, and the Matomo variant's duplicate check).

Results

One row per kill. A configuration with a queue or worker between the acknowledgement and the database was killed twice, once on the accept path and once on the consumer. "Lost" is the number of acknowledged events that were not in the database after recovery: matched by event id where the tool stores one (marked), otherwise acknowledged minus stored. Our own stack shows its strictest run here, the app killed with the store already stopped; its other three kills are in the section below and in the dataset. Versions, protocol, box and every qualification per row are in the dataset.

† Verified by event id. Other rows compare counts, so a zero means no count deficit, not a verified absence of loss. Versions, protocol, box, qualifications and every kill are in the dataset (CSV) and the run records.

Error trackers ingest Sentry-style envelopes rather than page views, at 10 envelopes per second, so they sit in their own table.

Versions, protocol, box, qualifications and every kill are in the dataset (CSV) and the run records.

Operational results

What the client saw while the component was down, how long recovery took and by which definition, and whether the compose file's restart policy brought the container back on its own. What each vendor's browser tracker does with a failed send is read from its code, not measured, so it stays in the dataset (tracker_retry) and the acknowledgement-path section below.

* Seconds from the kill to a healthy probe, which may precede successful ingestion.

Acknowledgement paths, and what the kills found

Every tool answers success at a different point, and that point predicts the shape of what a kill can lose. The path is an architectural description read from each tool's source; the observed result is in the tables.

Synchronous write. Umami, Matomo's default install, and Traceway write the row before they answer. None showed a deficit after its kill (Umami reconciled by event id: 0 missing of 4,992), and the client absorbed the whole outage as failed requests. These were also the slowest tools in the load test; the two facts have the same cause.

In-memory batch. Plausible, GoatCounter, Rybbit, the ClickStack collector, and GlitchTip answer success, then hold the event in process memory until a batch flushes. What was in memory when the process died was gone, in every case, and the deficit is bounded by the flush interval times the rate: where the kill lands in the cycle decides the number, which is why Plausible lost 3.3 seconds of its 5-second window, GoatCounter 7.5 of 10, GlitchTip 15 events on one kill and 2 on the other, Rybbit 359 events at 605 per second (about 0.6 s of its 1-second flush, counted by event id), and the ClickStack collector 4,154 at 2,513 rows per second. Matomo with QueuedTracking belongs here in practice: its queue is a Redis list that only snapshots to disk, so a Redis kill loses the list.

Durable queue or log. PostHog, Openpanel, Bugsink, and our own stack write somewhere that survives the process before they answer: Kafka on disk, a Redis list plus a job queue, an on-disk task queue, an append-only log. Killing the consumer left no count deficit for PostHog and ours, and the accept path kept answering. Openpanel and Bugsink each ended one event short of the acknowledged total, through hand-off windows their own code documents: Openpanel's worker gathers up to ten milliseconds of events in memory before pushing them to Redis, and Bugsink's on-disk task queue hands a task to its worker with delete-before-run semantics. The count-based method cannot rule out offsetting duplicates in any of these rows; only the Matomo row was checked.

We wrote a prediction for each September tool before running it (the brief is in the evidence bundle). Three were wrong: PostHog's capture kill showed no deficit rather than a small one, because its capture service answers only after producing to Kafka; Bugsink's acknowledgement is not synchronous but is written to disk; GlitchTip 6 no longer runs a separate Celery worker, and its in-process batch queue is what dies.

Method

• Box. One fresh e2-standard-2 (2 vCPU, 8 GB) per tool, torn down after the run, with two exceptions stated in the rows: PostHog ran on e2-standard-4 because that is the vendor's stated minimum, and its stack was revived from containers materialised a week earlier rather than installed fresh.
• Install. The vendor's own compose file, defaults untouched, image digests recorded on the run day and listed in the dataset. The only changes were the ones needed for synthetic traffic to be counted at all, such as a trusted-proxy header or a bot filter; each is listed per row under deviations.
• Load and kill, kill-test/2 (September 14). A fixed 50 page views per second (10 envelopes per second for error trackers; 20 per second for the Matomo variant, which sits under the default install's measured ceiling of about 26) for 90 s, one event per request in the shape the vendor's own tracker sends, 500 distinct visitors. A 60 s pre-check at the same rate had to hold p95 under 1 s with no errors; every tool did. docker kill on the accept process 20 s in, a 10 s wait for the vendor's restart policy, then docker start if nothing happened, and the time to the first successful request recorded. Where a queue or worker sits between acknowledgement and storage, a second 90 s run killed that instead while the accept path stayed up. Reconcile once the row count had been stable for 30 s.
• Load and kill, ramp/2 (September 14). The kill at the end of our load test: one docker kill of the accept process at 20 s of a 90 s run at each tool's soak rate (Umami 99, Matomo 22, Rybbit 605, Traceway 95 and our stack 366 events per second; ClickStack 2,513 rows per second), with k6 on a separate machine, a 10 s restart-policy watch, docker start by the driver if the policy did not fire, recovery timed to a healthy probe, and reconciliation by event id where the tool stores one (Umami, Rybbit, ClickStack, ours) or by count (Matomo, Traceway). These rows are marked "other rate".
• Reconciliation. By counts for the kill-test/2 rows and for Matomo and Traceway under ramp/2: acknowledged requests minus stored rows, where duplicates and stored rows whose response failed net against each other. By event id for Umami, Rybbit, ClickStack and our stack under ramp/2, where "missing" is an exact count. A zero in a count-based row means "no count deficit", not "verified absence of missing events"; the Matomo variant was additionally checked for duplicates by grouping rows.
• Restart policies. Containers were stopped with docker kill, which Docker records as a manual stop; see "No vendor restart policy brought a container back" above. This test measures data survival after abrupt termination; it does not establish whether a deployment recovers automatically from a real crash, which Docker's policies do cover.
• Load generator. k6 never retries, so hard failures are unretried requests; what a real browser would do with them is the tracker column, read from each vendor's code. For the kill-test/2 rows k6 ran on the same box as the tool; for the ramp/2 rows on a separate machine.
• Single trials. Each kill happened once, at one point in the batch cycle. The deficits are single observations, not typical loss rates.

What a self-hoster can do today

Each item names the vendor's own control. None of these was tested here; each is inferred from documentation or source, and the marker says which.

• Plausible CE: CLICKHOUSE_FLUSH_INTERVAL_MS (default 5000) shortens the in-memory window; it is read from the environment (config/runtime.exs) but not listed on the CE configuration page. Stop with SIGTERM rather than SIGKILL: the write buffer flushes on an orderly shutdown (WriteBuffer). From source, not tested.
• GoatCounter: -store-every 1 (GOATCOUNTER_STORE_EVERY=1) reduces the exposure to about one second of hits; the vendor documents the performance trade-off in the same sentence (goatcounter help serve). The compose file ships no restart policy; add one. From documentation, not tested.
• Rybbit: the batch flushes every second and there is no persisted queue to turn on; a restart policy and a graceful stop reduce the window. From source, not tested.
• ClickStack / HyperDX: the OpenTelemetry collector's persistent sending queue is off in the shipped compose and covers the exporter stage only. From documentation, not tested.
• Matomo + QueuedTracking: redis-server --appendonly yes bounds the queue loss to about one second and, because AOF restores the exact list, removes the stale-snapshot replay; the plugin's backend = mysql option keeps the queue in the database you already run, which also avoids the 204-with-Redis-down case; the plugin FAQ recommends a queuedtracking:process cron so the queue stays short (plugin README). From documentation and source, not tested.
• GlitchTip: VTASKS_INGEST_BATCH_COUNT (default 25) is an environment knob; the 2 s batch timeout is set in the settings file, not the environment. From source, not tested.
• Openpanel: EVENT_BUFFER_MICRO_BATCH_SIZE=1 makes every event an immediate Redis push; both durable tiers live in a Redis that only snapshots by default, so turn on AOF. From source, not tested.
• Bugsink: TASK_ALWAYS_EAGER = True makes the 200 wait for the stored row; the Docker image needs a settings-file override for it (Bugsink settings docs). From documentation, not tested.
• PostHog: no durability change indicated at this rate. Its tracker retries a failed send up to ten times in memory and flushes on page hide, per its code; how much of a capture outage that recovers in a real browser was not measured.
• Everyone: docker kill is not a crash. Test the restart path with the kill your tooling actually sends, and if a tool holds acknowledged events in memory, put a durable queue in front of it or shorten its flush window.

Our own stack

The pipeline we run for customers appends each accepted event to a durable log and waits for that write before answering 202; a separate process drains the log into ClickHouse. When the store is unavailable the accept path keeps answering and the log grows; when the process is killed, the log replays on restart. Its browser tracker keeps a persisted retry queue, so a request that failed during an outage is resent when the page returns; that behaviour is read from the code like every other tracker here, not measured.

The measurements: with the store up, the shape every competitor row uses, the plain kill showed no count deficit (3,927 acknowledged, 3,927 stored, 0 duplicates by event id) and recovered in 11 s; with the store stopped 10 s before the kill, the strict shape no other row was subjected to, no count deficit (3,691 and 3,691) and 17 s to the first acknowledgement with the store still down; killing ClickHouse while the app kept accepting, no count deficit (4,501 and 4,501) with zero failed requests. Under the load-test protocol with the generator on a separate machine, the kill at 366 events per second showed 0 missing of 27,073 by event id. One run each, on the same box class as the other rows.

Caveats

• Default installs only. Every tool here can be configured to lose less; the controls above are the vendors' own, and none was tested.
• k6, not the vendor's tracker, generated the traffic, on the tool's own box for the fixed-rate rows. Failed requests are therefore unretried.
• One kill per configuration and component. A memory-buffer deficit depends on where in the flush cycle the kill lands.
• Two protocols. The ramp/2 rows were killed at each tool's soak rate rather than the fixed 50 per second, so their windows differ in size; the tables mark them.
• PostHog on twice the box, and revived rather than freshly installed.
• Reconciliation by counts except where the tables say "by id".
• All of these products move quickly. The dataset carries the run dates and a last-verified date; the revalidation policy is in the appendix.

Evidence

The dataset (results.json, results.csv) has one row per kill with its configuration, digests, method, qualifications and the sha256 of its run record. The run records are published as plain files under records/, and the kits (compose files with pinned digests, setup, load script, kill driver, reconciliation queries) with the same records and the dataset are in durability-kill-test-2026-09-14.tar.gz. The ramp/2 rows' kits, harness and records are in the load-test post's bundle. If you run one of these tools with a different traffic shape, we will run the same kill against it and publish the row.

Appendix: dataset, citation and licence

Published

September 23, 2026

Runs performed

September 14, 2026 (kill-test/2 and ramp/2 rows)

Last verified

September 14, 2026, against the run records linked below

Revalidation

re-run the week of any launch that cites this table, or quarterly; a revision that changes copy but not measurements keeps the run dates and updates only the revised date

Protocols

kill-test/2, ramp/2 (defined below)

Dataset

results.json · results.csv · schema 0.2.0, dataset 1.1.0 · schema

Run records

records/ (one file per configuration, sha256 in the dataset) · evidence bundle

Citation

Jetboat Analytics, "Self-hosted analytics data loss: 12 tools tested under a kill signal", https://jetboatanalytics.com/blog/self-hosted-analytics-kill-test/, published 2026-09-23, dataset 1.0.0, runs 2026-09-14.

Licence

CC BY 4.0. Reuse is welcome; attribution must link to this page. Attribution string: Jetboat Analytics, "Self-hosted analytics data loss: 12 tools tested under a kill signal", https://jetboatanalytics.com/blog/self-hosted-analytics-kill-test/, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Attribution must include this link.

Appendix: definitions

acknowledged

requests that received the tool's success status (2xx; Plausible 202; Matomo 204; GoatCounter 200) inside the window

stored_rows

rows in the tool's own database attributable to the window, counted after the settle rule

stored_distinct

stored rows after removing duplicates, when duplicates were checked; otherwise a typed null

acknowledged_minus_stored_rows

signed difference of the two measured totals; positive = fewer rows than acknowledgements. A count deficit, not an identified set of events

acknowledged_missing_count

acknowledged events shown to be absent by event-id matching or a documented correction; typed null 'count_based' when only totals were compared

duplicate_count

stored rows that duplicate an already-stored event, when checked

stored_without_ack_count

stored events whose success response never reached the client, when established

hard_failures

requests that got an error status or no response; lost only if the client does not resend them (see tracker_retry)

recovery

seconds from the kill to the named event; each entry carries its definition

restart_policy_fired

whether the container came back on its own within the 10 s watch after an API-initiated docker kill; this does not test a real crash, which Docker's policies do restart

acknowledgement_strategy

architecture, from source reading: what the tool does before it answers success

tracker_retry

what the vendor's browser tracker does with a failed send, from reading its code; not measured in any row

axis

pageview = one page view per request; error_envelope = one Sentry-style envelope per request; telemetry_row = one OpenTelemetry log row per request

class_display

S = synchronous_write, M = in_memory_batch, D = durable_queue or durable_log; a display label for the acknowledgement strategy, not an outcome

count-based count_based_reconciliation

stored rows were counted, not matched by event id; a net deficit cannot separate missing, duplicate and stored-without-ack events

1 trial single_trial

one run, one kill; a memory-buffer loss depends on where in the flush cycle the kill lands

k6 on box generator_colocated

k6 ran on the same box as the tool

generator-limited generator_saturated

k6 dropped iterations; the achieved rate is a lower bound on the tool

2× box nonstandard_box

not the e2-standard-2 floor box

other rate protocol_variant

not the fixed-rate kill-test/2 window: a ramp/2 kill at the tool's soak rate with the generator on a separate machine, or the 20/s Matomo variant rate

store stopped first store_stopped_before_kill

the store was stopped before the process was killed, so every acknowledged event was provably only in the buffer

retry from code retry_inferred_from_code

client retry behaviour read from the tracker's source, not measured

acked with dependency down acknowledged_while_dependency_down

the server returned its success status while a dependency it needed was unavailable

probe-defined recovery healthy_probe_recovery

recovery is timed to the health endpoint answering, not to a successfully ingested event

sheds under overload overload_discard

this stack deliberately discards low-value events after acknowledging them under sustained overload (overload_behaviour); not exercised here, recorded so a zero-loss kill is not read as a general guarantee

kill-test/2

Fixed offered rate (50 requests/s pageview, 10/s error envelopes, 20/s for the Matomo variant) for 90 s, one event per request in the vendor tracker's shape, 500 distinct visitors; a 60 s pre-check at the same rate must hold p95 < 1 s with no errors; docker kill of the accept process at t=20 s; 10 s watch for the vendor's restart policy; docker start at t≈31 s if it did not fire; recovery timed to the first successful request; a second 90 s run kills the consumer or worker where a queue sits between acknowledgement and storage; reconcile once the row count has been stable for 30 s. Run 2026-09-14.

ramp/2

120 s constant-arrival stages at offered rates 25→1,600 requests/s (ClickStack →3,200; Traceway 10→400), one event per request, 500 distinct visitors, k6 v2.2.0 on a separate e2-standard-4 with generator CPU, achieved rate and dropped iterations recorded; reconcile after every stage once the stage's row count is unchanged across two consecutive 5 s readings (cap 180 s); a 600 s soak at 80 % of the ceiling; one docker kill (SIGKILL) of the accept process at t=20 s of a 90 s run at the soak rate, 10 s restart-policy watch, docker start by the driver if the policy did not fire, recovery = first healthy probe over ssh. Per-event id reconciliation where the tool stores a client-supplied id (Umami url_query, Rybbit querystring, ClickStack log attribute, ours event_id); count comparison for Matomo (a unique URL per event would change its workload) and Traceway (attributes not queryable per key). Run 2026-09-14. Ceiling rule: highest observed throughput = the highest achieved acknowledged rate across the ramp stages where acknowledged ids missing from storage are at most 2 % of acknowledged (id matching) or stored rows are at least 98 % of 2xx (count comparison); the stage that produced it is the ceiling stage; no latency or error threshold enters the rule

Appendix: notes referenced by the tables

n-posthog-box PostHog hobby

PostHog's hobby deploy ran on e2-standard-4 (4 vCPU / 16 GB), the vendor's stated minimum, twice the standard box. The stack had been materialised on 2026-09-07 and was revived from the same containers on 2026-09-14 with no re-pull.

n-posthog-mode PostHog hobby

k6 sent the captured browser request with its own gzip (adds a Content-Encoding: gzip header the setup smoke proved capture accepts and stores) because gzip in k6's JavaScript starved the generator on this box.

n-openpanel-batch Openpanel

The worker accumulates up to 100 events or 10 ms in process memory before pushing to Redis (event-buffer.ts, confirmed in the shipped image); the one-event deficit is consistent with that window but was not matched by id.

n-plausible Plausible CE

Events are held in an in-process write buffer flushed every 5 s (CLICKHOUSE_FLUSH_INTERVAL_MS, default 5000); the last flush was 3.3 s before the SIGKILL. A graceful stop flushes the buffer; the loss class is crashes and kills.

n-goatcounter GoatCounter

Hits are held in memory and persisted every 10 s (-store-every); the kill landed about 7.5 s after the last persist. Session ids live in the same memory, so all 500 visitors reappeared as new uniques after the restart.

n-matomo-dup Matomo + QueuedTracking (Redis)

Redis was SIGKILLed with no shutdown save; the last RDB snapshot was 112–123 s old. On restart Redis replayed 24 request sets from the earlier app-kill window that had already been processed, and Matomo inserted them a second time with their original server_time (verified in MariaDB by idvisitor + server_time + idaction_url grouping).

n-matomo-204 Matomo + QueuedTracking (Redis)

220 of the 1,800 acknowledged requests were answered 204 by Matomo core (Tracker/Response.php) after logging a Redis connection error; 33 more were the queue contents at the kill instant. 220 + 33 = 253 missing; 1,571 stored rows minus 24 duplicates = 1,547 of this run's hits.

n-ours-strict Jetboat lite stack (ours)

Strict shape: ClickHouse stopped at t=10 s, app SIGKILLed at t=20 s, docker start at t=31 s, ClickHouse started at t=45 s. Recovery is the first 202 for a real event while ClickHouse was still stopped.

n-bugsink-handoff Bugsink

The 200 is returned after the envelope is written to an on-disk SQLite task queue inside the container; the queue hands a task to its worker with delete-before-run semantics, so a kill mid-task can lose the task in flight. A supplementary probe on a separate project (50 acknowledged, killed, restarted) stored all 50.

n-glitchtip-batch GlitchTip

GlitchTip 6 runs its task worker in the web process by default; the ingest queue is a batch queue (count 25 / timeout 2 s) whose fetch has no processing list, so the batch being accumulated dies with the process. Everything still on the Valkey list survived and drained after restart.