Benchmarks: peer to peer across Europe

One device in Paris holds the files. Devices in Amsterdam and Warsaw join the vault with an empty folder and download everything: once directly from the Paris device and from each other (peer to peer), once from an S3 bucket in Paris. Measured on 9 and 10 October 2026 on rented cloud machines.

This page reports what was measured, how, what limited it, and what changed in Varsto because of it. Every planned case now has numbers, from two runs each unless a table says otherwise. The numbers belong to an alpha: they describe the code as it was on those days. Since 0.0.1-alpha.11 large files are cut into larger blocks (1 MiB on average from 32 MiB, 4 MiB from 1 GiB), so the block counts here no longer apply to them. The driver is in the repository under scripts/bench/, and the summarised results are published as JSON, so every number on this page can be checked and the run repeated.

Summary

• From a home connection behind NAT (100 Mbit/s): every one of 14 transfers between a home PC and servers in Amsterdam and Warsaw arrived from peers only, through the home router's NAT, NAT to NAT, and through a symmetric NAT by relay. Large files ran at 8.0 to 9.3 MB/s, 1.3 to 1.8 times as fast as the same download from the bucket on the same line. Details.
• Many small files, Paris to Amsterdam (9 ms apart): 10 000 files arrived peer to peer in 111 s (90 files/s), from the bucket in 299 s (34 files/s): peer to peer was 2.7 times faster. To Warsaw (33 ms apart) the two were close: 260 s against 294 s. The bucket's speed changed with the hour, though: three hours later, with two readers at once, it delivered 64 files/s to Amsterdam and 53 to Warsaw, close to peer to peer's 70 and 39 with two readers.
• Large files: to Amsterdam, 4 GiB peer to peer in 159 s (27.0 MB/s) against 180 s (23.9 MB/s) from the bucket, and much steadier. To Warsaw the bucket was faster: 190 s (23.4 MB/s) against 423 s (10.2 MB/s). The peer-to-peer path opens a TCP connection for every block, which costs round trips; the longer the distance, the more.
• Two readers at once, peer to peer: the readers help each other. Warsaw took 69 % of its blocks from Amsterdam, which had them already, and only 31 % from Paris, so the Paris device sent 5.6 GB instead of 8.6 GB. Neither reader slowed down.
• Compressible text: the bucket was a little faster to Amsterdam (18.1 against 16.7 MB/s) and more than twice as fast to Warsaw (16.7 against 7.3 MB/s). A device that serves text compresses every block again for every request, at a level chosen for small storage, not speed.
• Every block of the peer-to-peer runs came from a peer: 202 544 of 202 544 blocks in 24 measured downloads, none from the bucket, whose blocks had been deleted; every file matched its SHA-256 afterwards.
• The runs found and fixed nine problems, among them downloads that fetched one block at a time (now eight in flight, and small files fetched ahead across files) and a peer-to-peer server that answered one request at a time. Making QUIC the preferred path across the internet was tried, measured slower here, and reverted. Details and numbers.

From a home behind NAT (10 October)

The runs above are between data centres with public addresses. These are from a home: a desktop PC on a 100 Mbit/s consumer connection behind its router's NAT, with Varsto installed the way a user installs it (the Linux package, a background service of the user). Two cloud machines joined it: one in Amsterdam with a public address, and one in Warsaw hosting two devices in network namespaces behind NAT, one behind a cone NAT (Linux MASQUERADE keeps the port) and one behind a symmetric NAT (MASQUERADE with random ports, which defeats hole punching). As above, the bucket in Paris held the ledger and the file lists only: before every pull its blocks were deleted, so every block had to come from a peer, and every file was checked against its SHA-256. The driver is scripts/nat-test/home-nat.py.

The line

Home PC and	Round trip	TCP, one stream, to the home	TCP, one stream, from the home
Amsterdam	31 ms	9.1 MB/s	12.1 MB/s
Warsaw	21 ms	10.6 MB/s	11.9 MB/s
The bucket (Paris), rclone, 256 MiB	–	7.5 MB/s	4.3 MB/s

100 Mbit/s is 12.5 MB/s at most, so that is the ceiling for anything on this line.

Through the NATs

STUN classified the home router as a cone NAT. Each transfer moved one 64 MiB file of random bytes; at this size the fixed cost of a sync (ledger and file list in the bucket) is a large part of the time.

From	To	Path the receiver used	Time	Speed
Home	Amsterdam (public)	direct, into the home NAT	12.2 s	5.5 MB/s
Amsterdam	Home	direct	14.3 s	4.7 MB/s
Warsaw, cone NAT	Home	direct, NAT to NAT (hole punching)	13.3 s	5.0 MB/s
Home	Warsaw, cone NAT	direct, NAT to NAT	14.5 s	4.6 MB/s
Warsaw, symmetric NAT	Home	relayed via Amsterdam	65.7 s	1.0 MB/s
Home	Warsaw, symmetric NAT	relayed via Amsterdam	65.3 s	1.0 MB/s

Every path worked the way the design says: direct where a NAT keeps its mapping, a relay through one of the user's own reachable devices where it does not. No vendor server was involved; the devices found each other through records in the bucket and public STUN servers.

Large files

One 512 MiB file of random bytes per transfer, peer to peer only. The first column is version 0.0.1-alpha.9 with the relay fix below; the second adds the download changes of the same day (connections kept open per peer, objects sent without compressing them again, a window that adapts between 8 and 32 blocks).

Transfer	Before the changes	With the changes	Share of one TCP stream	Share of 100 Mbit/s
Amsterdam → home	4.7 MB/s	8.0 MB/s	88 %	64 %
Warsaw (cone NAT) → home	6.9 MB/s	9.3 MB/s	88 %	74 %
Home → Amsterdam	8.7 MB/s	9.3 MB/s	77 %	74 %
Home → Warsaw (cone NAT)	–	8.6 MB/s	72 %	69 %

The same download from the bucket

A second device on the home PC, with peer to peer off, pulled a 512 MiB file that a server had just pushed to the bucket, through the same line.

Written by	From the bucket	Peer to peer (above)	Peer to peer was
Amsterdam	6.1 MB/s	8.0 MB/s	1.3 times as fast
Warsaw (cone NAT)	5.2 MB/s	9.3 MB/s	1.8 times as fast

Between data centres the bucket can keep up, because the bucket and the machines sit on the same fast backbone. From a home, the bucket is one more distant hop with its own per-request cost, while a peer sends straight down the line.

Found and fixed

• The relay could not reach the device it relays for. A device behind a symmetric NAT keeps a connection open with a reachable device of the vault, and other devices reach it through that relay. The relay itself had no route to it: it would not relay through itself and kept dialling addresses nothing can dial. It now asks such a device over the connection the device keeps with it.
• Peer-to-peer downloads near the line's speed. The changes above roughly doubled the download from Amsterdam to the home (4.7 to 8.0 MB/s).

Still slow

• The relayed path carried only 1 to 1.6 MB/s: every block travels as its own request over two hops, home to Amsterdam to Warsaw and back. It is the fallback for the hardest NATs, but it should still fill more of the line.
• Small transfers pay a fixed cost for the ledger and the file list in the bucket: 64 MiB took 12 to 15 s where the bytes alone need about 7.
• The service's interface could wait for minutes for an answer while a sync ran and kept trying a peer it could not reach; the status calls should not wait for the sync.

Limits: one home connection, one afternoon, one run per case (a repeat of Warsaw to home during the screenshot ran at 5.2 MB/s, with Amsterdam briefly unreachable); the NATs in Warsaw are Linux's, not consumer routers'; the servers were small (Scaleway DEV1-M). The download changes are in review and not yet released.

What was measured

Data set	From the bucket (S3)	Peer to peer
Large files (2 × 2 GiB)	Amsterdam, Warsaw, both at once (9 Oct)	Amsterdam, Warsaw (9 Oct); both at once (10 Oct)
Many small files (10 000)	Amsterdam, Warsaw, both at once (10 Oct; the 9 Oct runs are kept as “before fetching ahead”)	Amsterdam, Warsaw, both at once (10 Oct)
Compressible text (512 MiB)	Amsterdam, Warsaw (9 Oct); both at once (10 Oct)	Amsterdam, Warsaw, both at once (10 Oct)
The writer's push	2 runs per set, one each day	RAM disk: 2 runs per set; local disk: 1 run per set

Two runs per case; times in the tables are their median (the mean of two), with both runs next to it. Single runs say so. Between the two days the download code changed only in ways that do not touch the compared case: the 9 October large-file and text runs fetch blocks inside a file exactly as the 10 October build does, and the many-small-files bucket runs were repeated on 10 October because fetching ahead across files is new. The last two cases, compressible text peer to peer and many small files from the bucket, each with both readers at once, were run three hours after the others on fresh machines with the merged build; the bucket was clearly faster in that session than in the earlier one.

Setup and baseline

• Machines: three Scaleway POP2-4C-16G instances (4 vCPU AMD EPYC 7543, 16 GB RAM, 80 GB network block volume, Ubuntu 24.04), created fresh each day: the writer in Paris (fr-par-1), readers in Amsterdam (nl-ams-1) and Warsaw (pl-waw-1). Each machine had a public IPv4 address on its own interface, so there was no NAT between them.
• Bucket: a temporary Scaleway Object Storage bucket in Paris (fr-par), the writer's region. The readers in Amsterdam and Warsaw read it across borders.
• Varsto: 0.0.1-alpha.8 with the fixes listed below, a static Linux build (musl). 9 October: commits up to 37bee19 (bucket runs: 0180ca8, which differs only in the peer-to-peer server and the file watcher). 10 October: commit 9b93263 for the peer-to-peer runs; that day's earlier bucket runs used a3ba1c0, which differs only in the order of peer-to-peer routes; the two cases run last used the merged build dc09721, the same code as 9b93263 for these runs.
• A run: a new device joins the vault with the vault key, starts the background service, pauses it, attaches an empty folder and asks the service for one sync of that folder (POST /api/sync). The time is from that request to its answer: the file list is read, every block is fetched, checked against its hash, decrypted, checked against the chunk id, and written. Afterwards the driver compares a SHA-256 list of the folder with the writer's. Then the device is removed again. Automatic verification was turned off on all benchmark devices, because it downloads and repairs blocks in the background.

Network baseline

Between	Round trip (ping)	TCP, one stream (iperf3, Paris sends), 9 / 10 Oct	TCP, four streams
Paris – Amsterdam	9.4 ms	925 / 958 Mbit/s	875 / 836 Mbit/s
Paris – Warsaw	33.4 ms	738 / 777 Mbit/s	949 / 909 Mbit/s
Amsterdam – Warsaw	20.6 ms	–	–

Reader	S3: one 256 MiB object, one stream (rclone), 9 / 10 Oct	S3: 4 KiB objects one after another, 9 / 10 Oct
Paris	91 / 59 MB/s	7 / 45 ms each
Amsterdam	26 / 21 MB/s	16 / 107 ms each
Warsaw	52 / 15 MB/s	52 / 122 ms each

The links between the machines carried 700 to 950 Mbit/s (90 to 120 MB/s) in one TCP stream, and none of the Varsto transfers came close: all were limited by round trips and per-request work. The bucket's own speed varied a lot from one day to the next (the second measurement was taken in the early morning); its runs varied more than the peer-to-peer runs throughout.

Data sets

• Large files: two files of 2 GiB of random (incompressible) bytes, 4 GiB in all, about 13 400 blocks.
• Many small files: 10 000 files in 500 directories, sizes drawn from a log-normal distribution with a median of 16 KiB, cut to 1 KiB…256 KiB, one file in twenty a copy of another; 281 MiB in all. A 50 000-file set of the same shape was pushed, not downloaded.
• Compressible text: 128 files of 4 MiB of generated web-server log lines, 512 MiB in all; zstd stores them in 138 MiB.

What “peer to peer only” means

Varsto asks peers first and storages second, so a peer-to-peer measurement must make sure the bucket cannot help. After the writer had pushed and the bucket runs were done, every object under chunks/ was deleted from the bucket. The bucket kept what devices need to find each other and to know what exists (signed ledger, file lists, device and peer records), but no file content: a block that no peer could deliver would have left the file unavailable. Then:

• each run's pull report counted blocks from peers and blocks in all, and the driver failed any run where the two differed;
• the readers' traffic counters (/api/p2p/traffic) and the writer's agreed: with one reader, the reader received from writer-paris exactly the stored size of the data set and the writer sent the same; with two readers, what each received from Paris and from the other reader added up to the stored size, and the Paris device sent the rest;
• the path was direct in every run, over TCP to the writer's public address on port 17893 (the service log names the route it settled on), and between the two readers in the both-at-once runs. No relay was involved; a LAN path was impossible between three countries.

Run	Chunks downloaded	From peers	From S3	Path
files, Amsterdam, alone (×2)	20 240	20 240	0	writer-paris: direct
files, Amsterdam, both at once (×2)	20 240	20 240	0	waw-p2p-files-ams+waw-1: direct, writer-paris: direct
files, Warsaw, both at once (×2)	20 240	20 240	0	: direct, ams-p2p-files-ams+waw-2: , writer-paris: direct
files, Warsaw, alone (×2)	20 240	20 240	0	writer-paris: direct
large, Amsterdam, alone (×2)	26 946	26 946	0	writer-paris: direct
large, Amsterdam, both at once (×2)	26 758	26 758	0	: direct, writer-paris: direct
large, Warsaw, both at once (×2)	26 758	26 758	0	ams-p2p-large-ams+waw-1: direct, ams-p2p-large-ams+waw-2: direct, writer-paris: direct
large, Warsaw, alone (×2)	26 946	26 946	0	writer-paris: direct
text, Amsterdam, alone (×2)	3 544	3 544	0	writer-paris: direct
text, Amsterdam, both at once (×2)	3 544	3 544	0	: direct, writer-paris: direct
text, Warsaw, both at once (×2)	3 544	3 544	0	ams-p2p-text-ams+waw-1: direct, writer-paris: direct
text, Warsaw, alone (×2)	3 544	3 544	0	writer-paris: direct

Large files

4 GiB in two files, random content, so zstd stores it as it is.

Peer to peer (from the writer in Paris) S3 bucket in Paris (MB/s, more is better)

Reader	Peer to peer: time	MB/s	S3: time	MB/s	P2P vs S3
Amsterdam, alone	159 s (158 s / 161 s)	27.0	180 s (171 s / 190 s)	23.9	1.1× faster
Warsaw, alone	423 s (420 s / 426 s)	10.2	190 s (155 s / 226 s)	23.4	2.2× slower
Amsterdam, both at once	162 s (162 s / 161 s)	26.6	95 s (90 s / 101 s)	45.2	1.7× slower
Warsaw, both at once	365 s (367 s / 362 s)	11.8	208 s (185 s / 232 s)	20.9	1.8× slower

Build	Reader	Time	MB/s
Peer to peer with QUIC tried before TCP (one connection per peer), 10 October	Amsterdam, alone	230 s (229 s / 230 s)	18.7
Peer to peer with QUIC tried before TCP (one connection per peer), 10 October	Warsaw, alone	466 s (466 s)	9.2

To Amsterdam, nine milliseconds away, peer to peer beat the bucket by a little and was much steadier: every peer-to-peer run took 158 s to 162 s, the bucket's 171 s and 190 s. To Warsaw the bucket was more than twice as fast. Each block of about 300 KiB is one HTTP request on a new TCP connection: a handshake, then TCP slow start from ten packets, so a block needs several round trips however fast the link is. At 33 ms that caps eight requests in flight at about 10 MB/s, which is what was measured. The bucket's HTTPS connections are kept open between requests.

With both readers at once, Warsaw took 2.97 GB of its 4.30 GB from Amsterdam (20 ms away, closer than Paris) and only 1.3 GB from Paris; Amsterdam, which had the blocks first, served them while it downloaded. Warsaw was even a little faster than alone (11.8 against 10.2 MB/s) and Amsterdam as fast as alone. The bucket served two readers at once without slowing either, too (45 and 21 MB/s); it is shared infrastructure built for many readers.

The reader's own work: about 40 s of CPU time for 4 GiB (decrypting, two hashes per block, writing), at most 490 MB of memory. The Paris device used 40 s of CPU time to serve each 4 GiB, one core about a quarter busy.

Many small files

10 000 files of 1 KiB to 256 KiB. Each file is one block. Since 10 October a pull fetches the blocks of up to 32 coming files together, eight requests at a time, before writing them one by one.

Peer to peer (from the writer in Paris) S3 bucket in Paris (files/s, more is better)

Reader	Peer to peer: time	MB/s	files/s	S3: time	MB/s	files/s	P2P vs S3
Amsterdam, alone	111 s (111 s / 111 s)	2.6	90.2	299 s (323 s / 274 s)	1.0	33.7	2.7× faster
Warsaw, alone	260 s (258 s / 263 s)	1.1	38.4	294 s (269 s / 320 s)	1.0	34.2	1.1× faster
Amsterdam, both at once	156 s (202 s / 110 s)	2.1	70.3	156 s (142 s / 170 s)	1.9	64.7	1.0× faster
Warsaw, both at once	259 s (265 s / 253 s)	1.1	38.6	192 s (161 s / 223 s)	1.6	53.5	1.3× slower

Build	Reader	Time	MB/s	files/s
S3 before fetching ahead across files (one file at a time), 9 October	Amsterdam, alone	404 s (313 s / 495 s)	0.8	26.1
S3 before fetching ahead across files (one file at a time), 9 October	Amsterdam, both at once	381 s (381 s / 380 s)	0.8	26.2
S3 before fetching ahead across files (one file at a time), 9 October	Warsaw, both at once	528 s (494 s / 563 s)	0.6	19.1
S3 before fetching ahead across files (one file at a time), 9 October	Warsaw, alone	596 s (536 s / 655 s)	0.5	16.9

To Amsterdam, peer to peer delivered 90 files a second, 2.7 times the bucket's 34. To Warsaw, three and a half times farther away, both came to the same 34 to 38 files a second: the round trip per request dominates there, on both paths. Fetching ahead across files doubled the bucket's speed to Warsaw (from 17 to 34 files a second) and helped to Amsterdam (26 to 34). Once the blocks are there, writing each file (data synced to disk, a rename, ledger events) costs a few milliseconds on these network block volumes, which caps any source at a few hundred files a second.

With both readers at once, Warsaw was as fast as alone and Amsterdam's two runs differed (202 s and 110 s): the readers' blocks are fetched from whichever peer answers first, and one run waited on the other reader more than the other run did. The bucket, measured with both readers three hours later, delivered 64 files a second to Amsterdam and 53 to Warsaw: nearly twice what it gave single readers in the earlier session. Its speed depends on the hour more than on the number of readers.

Compressible text

512 MiB of log lines, stored as 138 MiB. Fewer bytes travel, so a reader gets more file per second than with random data.

Peer to peer (from the writer in Paris) S3 bucket in Paris (MB/s, more is better)

Reader	Peer to peer: time	MB/s	S3: time	MB/s	P2P vs S3
Amsterdam, alone	32 s (33 s / 31 s)	16.7	36 s (21 s / 52 s)	18.1	1.1× faster
Warsaw, alone	74 s (70 s / 79 s)	7.2	32 s (32 s / 32 s)	16.7	2.3× slower
Amsterdam, both at once	32 s (32 s / 31 s)	16.9	31 s (23 s / 39 s)	18.4	1.0× slower
Warsaw, both at once	66 s (63 s / 69 s)	8.1	63 s (28 s / 98 s)	12.3	1.0× slower

Here the bucket was a little faster to Amsterdam and more than twice as fast to Warsaw. With both readers at once, peer to peer gave Amsterdam the same 16.9 MB/s as alone and Warsaw 8.1 MB/s, slightly more than alone; in the first of the two runs Warsaw took two thirds of its blocks from Amsterdam, in the second all from Paris. A device that serves a block does not keep the compressed copy: it reads the plain file and compresses the block again, at zstd level 12, for every request; the Paris device spent 31 s of CPU time per 512 MiB served, against 40 s for 4 GiB of random data. The bucket returns the stored object as it is.

The writer's side

Before anything can be downloaded, the writer cuts its files into blocks, compresses, encrypts and stores them. Three destinations show where the time goes: a RAM disk (Varsto's own work only), the machine's disk (a network block volume, where every object is written with a sync to disk), and the bucket.

Data set	Destination	Build	Time	MB/s	CPU (user + system)	Peak memory
files: 10 000 files, 0.27 GiB	RAM disk (tmpfs)	measured build	50 s (median of 2)	5.9	12 s (5 + 7)	68 MiB
files: 10 000 files, 0.27 GiB	S3 bucket, same region	measured build	652 s (median of 2)	0.5	21 s (8 + 13)	69 MiB
small: 50 000 files, 1.36 GiB	RAM disk (tmpfs)	measured build	255 s (median of 2)	5.7	62 s (27 + 35)	258 MiB
small: 50 000 files, 1.36 GiB	Local disk (network block volume)	measured build	468 s	3.1	67 s (27 + 41)	258 MiB
small: 50 000 files, 1.36 GiB	Local disk (network block volume)	alpha.8 (before the zstd fix)	515 s	2.8	109 s (31 + 78)	253 MiB
large: 2 files, 4.00 GiB	S3 bucket, same region	measured build	932 s (median of 2)	4.6	80 s (37 + 43)	108 MiB
big: 5 files, 12.00 GiB	Local disk (network block volume)	measured build	418 s	30.9	197 s (85 + 112)	147 MiB
big: 5 files, 12.00 GiB	Local disk (network block volume)	alpha.8 (before the zstd fix)	694 s	18.6	483 s (110 + 373)	137 MiB
text: 128 files, 0.50 GiB	RAM disk (tmpfs)	measured build	45 s (median of 2)	12.0	44 s (41 + 2)	68 MiB
text: 128 files, 0.50 GiB	Local disk (network block volume)	measured build	56 s	9.6	45 s (42 + 3)	68 MiB
text: 128 files, 0.50 GiB	Local disk (network block volume)	alpha.8 (before the zstd fix)	68 s	7.9	56 s (43 + 13)	68 MiB
text: 128 files, 0.50 GiB	S3 bucket, same region	measured build	145 s (median of 2)	3.7	45 s (41 + 4)	77 MiB

• Random data costs about 16 s of CPU per GiB on these cores after the zstd fix (12 GiB: 197 s), against 40 s per GiB before it.
• Text costs about 85 s of CPU per GiB: zstd at level 12, a format constant, is slow on compressible data; it buys the 3.7 : 1 ratio every later transfer profits from.
• Small files are not CPU-bound even on a RAM disk: 10 000 files took 50 s, of which 12 s CPU. The device's logical clock is written to disk, with two syncs, for every file pushed. Not fixed yet.
• The bucket is slowest by far: every block is a HEAD and a PUT, one after another. 10 000 files took 622 s and 681 s, 4 GiB 578 s and 1 287 s on the two days. Uploads are not parallel yet.

Fixed along the way

The benchmark was also a hunt for bottlenecks. These changes are in the measured builds, each with a test:

Problem	Fix	Before	After
A download fetched one block at a time: every block waited a full round trip before the next was asked for.	Up to eight blocks of a file in flight at once, from peers or storages, in a sliding window; they are still checked and written in order. The S3 client keeps enough idle connections for them.	Text from the bucket to Warsaw: 122 s (4.4 MB/s)	27.8 s (19.4 MB/s), 4.4 times faster (single runs; steps on the way: fixed batches of eight 104 s, with idle connections kept 61 s)
Small files went one at a time: each file is one block, so the window above never applied.	A pull fetches the blocks of up to 32 coming single-block files together (10 Oct).	10 000 files from the bucket to Warsaw: 596 s (17 files/s)	294 s (34 files/s)
The peer-to-peer HTTP server answered one request at a time, so eight requests in flight queued at the server.	Eight requests answered at once.	Large files peer to peer to Amsterdam: 363 s (11.8 MB/s), one run	159 s (27.0 MB/s)
A fresh zstd context was created for every block; its tables were allocated and cleared each time, more work than compressing the block.	One context per thread, reused (the output is byte-identical, which a test checks).	12 GiB to local disk: 694 s, 483 s CPU	418 s, 197 s CPU (single runs)
The S3 client refused any response over 10 MiB. After a push of a few thousand files the ledger batch was 13 MB, and no other device could join or sync.	Responses up to 2 GiB are read.	Joining failed	Joining works; test with a 12 MiB object
The background service treated every opened file as a change: serving blocks to peers, a backup tool or a file viewer started a full sync every few seconds.	Opening and reading are not changes; closing a file opened for writing still is.	90 syncs in 11 minutes while files were read	Reading all 10 000 files: no extra sync
Requests in flight together each opened their own QUIC connection to a peer they had no connection to yet.	One connection attempt per address; the others wait and share it (a test checks one connection for a whole pull).	Found while testing QUIC (below)
A kept QUIC connection to a peer that had restarted failed every request on it, and the peer dropped to the next route for good.	Reconnect once and ask again.	Found while testing QUIC (below)
Requests that finished on an old route could undo a switch to a better route that another request had just made.	A late answer keeps the current route.	Found while testing QUIC (below)

Tried and reverted: QUIC first

Because the TCP path opens a connection per block, QUIC looked like the cure: Varsto already speaks QUIC to peers behind NAT, with one connection per peer and one stream per request. On 10 October the route order was changed to try QUIC before TCP for addresses beyond the LAN. It was slower on these machines, and used more than twice the CPU time:

Large files, peer to peer	TCP first (kept)	QUIC first (reverted)
Amsterdam	159 s, 27.0 MB/s; reader 38 s CPU, Paris 40 s	230 s, 18.7 MB/s (229 s / 230 s); reader 98 s CPU, Paris 88 s
Warsaw	423 s, 10.2 MB/s	466 s, 9.2 MB/s (one run); reader 130 s CPU

So TCP stays first and QUIC remains the way through NATs. Where the QUIC path loses its time was not found in the time available; the host dropped no UDP packets. The experiment also turned up a gap that is not fixed yet: a device's peer table is built before a sync reads the device registry, so a device that has just joined is refused over QUIC by its peers until their next sync (over TCP it is served at once).

Still slow, and why

• One TCP connection per block on the peer-to-peer path (the builds measured here). Each request paid a handshake and a fresh slow start, which is what limited Warsaw. Changed since: connections to a peer are now kept open between requests, up to 32 requests are in flight, the window follows the measured throughput, and blocks that were stored uncompressed are no longer compressed again when served. On one machine with a network namespace emulating a 32 ms round trip (Paris–Warsaw) at 800 Mbit/s, which carried 62 MB/s in one TCP stream and 81 MB/s in 16, peer to peer went from 10.4 to 44.7 MB/s for incompressible data and from 11.3 to 38.6 MB/s for text, and the bucket (served by rclone serve s3 over the same link) from 41.5 to 45.5 and from 39.1 to 52.8 MB/s (means of two runs). These are local measurements, not a repeat of the runs above; the driver is scripts/bench/netem-local.sh.
• Serving compressible data costs CPU, because each request compresses the block again; that is why text peer to peer stays below the bucket in the emulation. Blocks compressed for one peer are now kept in memory (64 MiB by default) for the next device that asks, as when several devices catch up at once.
• A download with peer-to-peer on now uses the bucket too when it answers faster, in proportion to its speed. In the emulation both shared one link, so using both was not faster than either alone (44.4 MB/s for both data sets); across real networks the two paths differ.
• Writing many small files costs a disk sync per file on the reader, and on the writer two syncs per file for the logical clock.
• Uploads to a bucket go one block at a time, with a HEAD before every PUT.
• The ledger grows by about 1 KB per block: pushing the 4 GiB set wrote a 13 MB ledger batch, which every device reads when it joins.

When peer to peer wins, and when the bucket does

Varsto uses both on purpose, and the measurements show why.

• Peer to peer needs a peer online. The readers could fetch only because a device that had the blocks was running. The bucket is always there: a laptop that is closed, a phone in a drawer, devices that are never online at the same time all still sync through it. That is why a vault always has at least one storage, and why the file lists and the ledger stay in it.
• Near devices, peer to peer wins: 2.7 times faster for many small files to Amsterdam than the bucket in the same session (in a later session the bucket came close), a little faster and much steadier for large files. On a LAN the round trip is a fraction of a millisecond.
• Devices that already have the data share it. With two readers, Warsaw took most of its blocks from Amsterdam; the first device to sync becomes a source for the next, and the writer's upload is spared.
• Far devices and compressible data, the bucket won on these links: large files and text to Warsaw, text to Amsterdam by a little. The causes are known (a connection per block, compressing again when serving) and are not in the network: the link to Warsaw carried 777 Mbit/s in one stream.
• The bucket scales to many readers without anyone's upload; a home device serving many others is limited by its upload, unless the readers help each other as above.
• Peer to peer costs nothing per gigabyte and leaves no copy with a provider. The data goes from your device to your device; the bucket needs the blocks only for the devices that are offline when others want them.

Cost

Scaleway's public price list (read on 9 October 2026, Paris region, before tax): Object Storage Standard Multi-AZ €0.000022 per GB-hour (about €0.016 per GB-month), requests included, ingress included, egress the first 75 GB a month free and then €0.01 per GB. Transfers between regions (Paris – Amsterdam, Paris – Warsaw) count as egress; within a region they are free.

So one reader in Amsterdam downloading the 4 GiB set from the Paris bucket costs about €0.04 once the free 75 GB are used, and a household that syncs 500 GB a month to devices outside the bucket's region pays about €4 a month in egress. Peer to peer moves the same bytes over the devices' own connections and adds nothing to the storage bill. Other providers charge more for egress or per request; the method above applies to them unchanged. The machines for this benchmark (three instances at €0.147 an hour, about seven and a half hours over the two days) cost about €3.50.

Limits of this benchmark

• Two days, two builds. The large-file runs from single readers and part of the bucket runs are from 9 October, the rest from 10 October; the changes between them do not touch the compared cases (see above). The bucket's speed differed between the days.
• Cloud machines, not homes. Data-centre links, public addresses on the interface, no NAT, no Wi-Fi. At home the peer-to-peer path across the internet is usually QUIC through a hole-punched NAT, which these runs measured only in the reverted experiment; home upload speed would often be the limit.
• Three regions of one provider, in Europe. The bucket and the machines share a provider and its backbone; other providers and longer distances will differ.
• Network block volumes make every disk sync slow (milliseconds), which weighs on small files; a laptop's SSD syncs faster.
• Two runs per case are few; the bucket's results varied by up to 58 % between repetitions for small files and by a factor of 3.5 for text, and the bucket's small-file speed nearly doubled between two sessions on the same day.
• Alpha software. Formats and defaults may still change, and so will these numbers.

Reproduce: scripts/bench/p2p-bench.py creates the machines and the bucket with the Scaleway command line, runs each step, writes results.jsonl, and deletes everything again; report.py and page.py turn the results into the summary and the tables on this page.