← All posts

We load-tested five open-source tools. Which accepted events survived a process kill?

Umami, Matomo, Rybbit, ClickStack and Traceway, plus our own stack, on 2-vCPU boxes: highest observed throughput, latency, and whether acknowledged events survived a SIGKILL · Published September 11, 2026 · Revised September 23, 2026 · Runs performed September 14 and 17, 2026

We tested five open-source analytics and telemetry tools on small cloud servers. Alongside throughput and latency, we checked whether accepted events survived an abrupt process termination.

In two tested configurations, some acknowledged events were missing after recovery. Others preserved every acknowledged event but became slow under load. The results show why throughput alone is not enough to evaluate an analytics pipeline.

Scope: one 2-vCPU, 8 GB VM per tool, default installs, one k6 load script on its own 4-vCPU machine, reconciliation against each tool's own database after every stage (by event id for four tools, by count for two), and one docker kill per tool at the soak rate. Our own stack ran the identical protocol on the identical box size; its row is in the pageview table, on the same terms. Every number traces to a row in the dataset and a run record linked from it.

Why these five

Umami, Matomo and Rybbit are the self-hosted pageview analytics tools a team reaches for first: the incumbent, the popular lightweight option, and a newer ClickHouse-backed one. ClickStack (HyperDX) stands in for "build it on an OpenTelemetry collector and ClickHouse", which is what many teams do instead of adopting an analytics product. Traceway is an error tracker, the third kind of workload we run alongside analytics and session replay. Together they cover the three jobs a product-data pipeline does.

Results at a glance

Tool and outcome only, from the September 14 run (our stack: September 17); rates, latency and recovery are in the tables. "By id" means acknowledged and stored event ids were matched as sets. "Count-based" means totals were compared, which cannot separate a loss from an offsetting duplicate.

Pageview results

Three tools accept the same kind of work, a pageview or custom event with visitor stitching, so their numbers sit in one table with ours. The "highest observed throughput" is the achieved acknowledged rate of the highest 120 s stage where at most 2% of acknowledged events were missing from storage; it is a reconciliation limit, not a latency target, which is why its p95 sits under it and the 80% soak stage has its own column. The kill column is one SIGKILL of the accept process, twenty seconds into a 90 s run at 80% of that ceiling. The full latency ladder per stage is in the dataset.

* Seconds from the kill to a healthy probe, which may precede successful ingestion. Versions, protocol, box, qualifications and every kill are in the dataset (CSV) and the run records.

At the ceiling, latency is unusable for every tool that queues. At 80% of each ceiling, p95 was 63 ms for Umami at 99 events/s, 768 ms for Rybbit at 605 events/s, 2.7 s for ours at 495 events/s and 5.9 s for Matomo at 22 events/s.

Telemetry and error-tracking results

Two more tools ran under the same protocol but do different work per request, so they get their own table. ClickStack's row is a lower bound, not a ceiling: it stored every acknowledged id at every stage, and at 3,200 requests per second the k6 machine could no longer produce the offered rate (6,925 dropped iterations, 72% busy). We ran out of generator before ClickStack ran out; finding its ceiling needs a bigger generator or several.

* Seconds from the kill to a healthy probe, which may precede successful ingestion. Versions, protocol, box, qualifications and every kill are in the dataset (CSV) and the run records.

What this means if you are choosing a tool

• An acknowledgement is not a stored event. The two fastest tools were the two with acknowledged events missing after a process kill, and the September run matched individual event ids, so the losses are counted, not inferred, and no duplicates were found anywhere. In the configurations tested, Rybbit and the ClickStack collector hold accepted events in process memory before writing them in batches; that buffer died with the process. OpenTelemetry collectors can be configured with a persistent sending queue; we did not test it, and a persistent queue covers the exporter stage only, not every point where an accepted event can be lost.
• Umami and Matomo showed no acknowledged-event loss in this process-kill test. Umami's result is verified by id; Matomo's is a count with no deficit, which does not rule out a loss offset by a duplicate. Both write synchronously, which is also why they are slow. This covers a killed process; it says nothing about host or disk failure, which we did not test.
• Slow is the common failure, not errors, but only near the ceiling. None of the three pageview tools returned an HTTP error response at any stage. What they did at their limits was queue: p95 reached 13 s for Umami and 60 s client timeouts for Matomo. At 80% of each tool's ceiling, Umami sat at 63 ms and Rybbit at 768 ms. Traceway is the exception to queueing: it answers overload with an explicit 503.
• Compare against your own numbers. Take your peak event rate, your payload size, and the latency you can accept at the tracking endpoint, and read the tables at that rate rather than at the ceiling. A tool that is fine at 100 events/s and unusable at 200 is a fine tool for a site that peaks at 50.
• What this benchmark does not evaluate: dashboard and query performance, feature completeness, and long-term operating cost. Those matter at least as much and need a different test.

Method, briefly

One fresh GCP e2-standard-2 per tool (our own documented floor for a small deployment, not a vendor's sizing), Docker 29.1.3, torn down afterwards. k6 v2.2.0 on a separate e2-standard-4 in the same zone, with the generator's CPU, achieved rate and dropped iterations recorded per stage; it stayed under a third busy except in ClickStack's top stage. Stages of 120 s at offered rates from 25 up to 1,600 requests per second (3,200 for ClickStack; 10 to 400 for Traceway), one event per request, then a 600 s soak at 80% of the ceiling, then the kill. Rybbit and ClickStack ran from the vendors' own compose files; Umami, Matomo and Traceway from compose files written around the vendors' images; ours from its shipped compose file at a pinned commit. Every deviation from defaults is listed per row in the dataset; the two that exist only because the generator is a separate machine are Rybbit's backend port and our app port bound to all interfaces.

Each tool was killed once, at one point in its batch cycle, so the loss figures are single observations, not a typical loss rate. We stopped containers with docker kill, which affects Docker's automatic restart behaviour; no tool's restart policy fired, so the driver started the container about fifteen seconds after the kill. "Recovery" is the first healthy probe after the kill, which can precede successful ingestion. Full definitions, the ceiling rule, protocol text and per-row notes are in the appendices; the complete method is in the campaign record.

How our pipeline handles the same question

The pipeline we run for customers writes each accepted event to a durable, append-only log on disk and waits for that write before returning the acknowledgement; a separate process drains the log into the store. Under sustained overload past a soft cap it acknowledges, then deliberately drops low-value events and counts what it dropped. That is a separate behaviour from crash survival and the dataset records it as one: an acknowledgement is a durability promise against process and store failure, not a promise that every event survives overload.

Under this protocol, from a clean docker compose up at a pinned commit, every stage reconciled by event id with nothing missing: 25 through 1,600 offered events/s, a ten-minute soak at 495 events/s (293,518 events), and the kill. When the accept process was killed with SIGKILL twenty seconds into the 495 events/s window, 36,549 events had been acknowledged; after restart, all 36,549 were in the store, plus 17 whose acknowledgement never reached the client. Once started, the app answered its health probe 16 s after the kill. The costs are in the table: the acknowledgement waits for the disk, so p95 is 5 ms at 200 events/s and 19 ms at 400 but 2.7 s at 800, and at 800 and 1,600 offered the drain fell behind and took about 70 s to catch up. The 620 events/s ceiling is on the application's side, but not its CPU: the app container was at 58% and ClickHouse at 23% while the generator machine sat under a fifth busy. The limit is one fsync per request on the durable log, and this disk class fsyncs in about 1.5 ms.

This is the second trial of the protocol for our stack; the first, on September 14, measured 459 events/s at the same stage with the same kill outcome. One kill per trial: the same limits as the other rows. The kit and the complete record are in the bundle alongside the others.

Evidence

The dataset (results.json, results.csv) holds every stage, soak and kill as a row with its configuration, reconciliation method, generator load, qualifications and the sha256 of its run record. Run records are plain files under records/. The evidence bundle carries the harness, the kits, the records and the raw per-stage pickups (gzipped k6 logs with every acknowledged id, stored-id dumps, generator and container CPU samples). Traceway was measured in ClickHouse mode only. Matomo's QueuedTracking plugin with Redis would change its acknowledge-versus-store behaviour; we tested the default install. All five products move quickly; treat anything older than a quarter as history.

Download the scripts and run records to test your own workload, or contact us for a workload-specific benchmark.

Appendix: kill tests in full

Tool	Killed	Acknowledgedsuccess responses	Storedrows after settle	Lostacknowledged, not stored
Page views
Umami	app	4,992	4,994	0 †
Matomo	app	1,411	1,411	0
Rybbit	backend	19,310	18,951	359 †
Jetboat (ours)	app	36,549	36,566	0 †
Telemetry rows
ClickStack	otel-collector	177,246	173,092	4,154 †
Error envelopes
Traceway	all-in-one container	3,424	3,424	0

† Verified by event id. Other rows compare counts, so a zero means no count deficit, not a verified absence of loss. Versions, protocol, box, qualifications and every kill are in the dataset (CSV) and the run records.

Tool	Killed	Failed requestsclient saw the error	Recovery *seconds from the kill	Restart policyfired on docker kill?
Page views
Umami	app	3,194	63 s	no did not fire
Matomo	app	337	80 s	no did not fire
Rybbit	backend	31,028	75 s	unless-stopped did not fire
Jetboat (ours)	app	7,523	16 s	unless-stopped did not fire
Telemetry rows
ClickStack	otel-collector	48,173	23 s	always did not fire
Error envelopes
Traceway	all-in-one container	4,834	26 s	no did not fire

* Seconds from the kill to a healthy probe, which may precede successful ingestion.

Appendix: dataset, citation and licence

Published

September 11, 2026 · revised September 23, 2026

Runs performed

September 14, 2026 (ramp/2: k6 on a separate machine, per-event reconciliation where the tool stores an id) and September 17, 2026 (our stack, second trial of the same protocol)

Last verified

September 17, 2026, against the run records linked below

Revalidation

re-run the week of any launch that cites this table, or quarterly; a revision that changes copy but not measurements keeps the run dates and updates only the revised date

Protocols

ramp/2 (defined below)

Dataset

results.json · results.csv · schema 0.3.0, dataset 2.2.0 · schema

Run records

records/ (one file per configuration, sha256 in the dataset) · evidence bundle

Citation

Jetboat Analytics, "We load-tested five open-source tools. Which accepted events survived a process kill?", https://jetboatanalytics.com/blog/five-open-source-analytics-tools-load-tested/, published 2026-09-11, revised 2026-09-23, dataset 2.2.0, runs 2026-09-14 and 2026-09-17.

Licence

CC BY 4.0. Reuse is welcome; attribution must link to this page. Attribution string: Jetboat Analytics, "We load-tested five open-source tools. Which accepted events survived a process kill?", https://jetboatanalytics.com/blog/five-open-source-analytics-tools-load-tested/, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Attribution must include this link.

Appendix: definitions

acknowledged

requests that received the tool's success status (2xx; Matomo 204; Traceway 2xx) inside the window; in id mode, the number of distinct acknowledged ids

stored_rows

rows for the stage's run in the tool's own database after the settle rule was satisfied

stored_distinct

distinct stored event ids (id mode only)

acknowledged_minus_stored_rows

acknowledged − stored_rows, signed; a net count deficit, not an identified set

acknowledged_missing_count

acknowledged ids absent from storage after settle (id mode); typed null with reason count_based otherwise

duplicate_count

stored rows beyond distinct ids (id mode); the ramp/2 id-matched rows found none anywhere

stored_without_ack_count

stored ids whose success response never reached the client (id mode)

hard_failures

requests that ended in a client-visible error (connection refused/EOF, client timeout) inside the window; HTTP 503 sheds are counted here for Traceway

achieved_rate

acknowledged ÷ stage duration; the rate the tool actually accepted, not the offered rate

rate_bound

exact when the application was the limit; lower when the generator machine was near capacity (see generator_busy_max_pct)

is_ceiling_stage

the stage the protocol's ceiling rule selected; its achieved_rate is the row's headline throughput

p95_ms

k6 http_req_duration p95 over the whole stage window

recovery

seconds from the kill to the named condition; healthy_probe = the tool's health endpoint answered, which may precede successful ingestion

restart_policy_fired

whether the container's restart policy restarted it within a 10 s watch after docker kill (API SIGKILL does not trigger Docker restart policies)

overload_behaviour

what the accept path does under sustained overload, from source or documentation; distinct from crash survival

superseded_by

the observation that replaces this row for headline use, when one exists

axis

pageview rows are directly comparable with each other; telemetry_row (ClickStack) and error_envelope (Traceway) are annotated axes and not comparable with pageview rows

acknowledgement_strategy

S synchronous write; M in-memory batch; D durable queue or durable log — classified from source, evidence in classification_evidence

count-based count_based_reconciliation

stored rows were counted, not matched by event id; a net deficit cannot separate missing, duplicate and stored-without-ack events

1 trial single_trial

one run, one kill; a memory-buffer loss depends on where in the flush cycle the kill lands

generator near capacity generator_near_capacity

the separate generator machine reached 63–72 % CPU during the stage; the achieved rate is a lower bound

application-bound application_bound

the limit was on the tool's side (its containers, or a per-request disk barrier) while the generator machine stayed under a third busy; the ceiling is the application's

probe-defined recovery healthy_probe_recovery

recovery is timed to the health endpoint answering, not to a successfully ingested event

sheds under overload overload_discard

this stack deliberately discards low-value events after acknowledging them under sustained overload (overload_behaviour); not exercised here, recorded so a zero-loss kill is not read as a general guarantee

ramp/2

120 s constant-arrival stages at offered rates 25→1,600 requests/s (ClickStack →3,200; Traceway 10→400), one event per request, 500 distinct visitors, k6 v2.2.0 on a separate e2-standard-4 with generator CPU, achieved rate and dropped iterations recorded; reconcile after every stage once the stage's row count is unchanged across two consecutive 5 s readings (cap 180 s); a 600 s soak at 80 % of the ceiling; one docker kill (SIGKILL) of the accept process at t=20 s of a 90 s run at the soak rate, 10 s restart-policy watch, docker start by the driver if the policy did not fire, recovery = first healthy probe over ssh. Per-event id reconciliation where the tool stores a client-supplied id (Umami url_query, Rybbit querystring, ClickStack log attribute, ours event_id); count comparison for Matomo (a unique URL per event would change its workload) and Traceway (attributes not queryable per key). Run 2026-09-14; our stack re-run 2026-09-17 under the same protocol (second trial). Ceiling rule: highest observed throughput = the highest achieved acknowledged rate across the ramp stages where acknowledged ids missing from storage are at most 2 % of acknowledged (id matching) or stored rows are at least 98 % of 2xx (count comparison); the stage that produced it is the ceiling stage; no latency or error threshold enters the rule

Appendix: notes referenced by the tables

n-ours-second-trial Jetboat lite stack (ours)

Second trial of ramp/2 for this configuration, run 2026-09-17 on the same box class and build class. The first trial (2026-09-14) measured 459 events/s at 800 offered with p95 3.8 s and the same kill outcome (0 of 27,073 acknowledged missing, healthy 17 s after the kill); this stack's accept path serialises one fsync per request and its stall tail varies from run to run (analytics-platform stress/results/2026-09-15-ours-causes.md).