--- title: Benchmarks nav: How it works output: benchmarks/index.html description: Varsto measured between Paris, Amsterdam and Warsaw on 9 and 10 October 2026, and from a home connection behind NAT: downloads peer to peer from another device compared with the same downloads from an S3 bucket, with the method, the raw numbers, the fixes the runs led to and their limits. ---

Benchmarks: peer to peer across Europe

One device in Paris holds the files. Devices in Amsterdam and Warsaw join the vault with an empty folder and download everything: once directly from the Paris device and from each other (peer to peer), once from an S3 bucket in Paris. Measured on 9 and 10 October 2026 on rented cloud machines.

This page reports what was measured, how, what limited it, and what changed in Varsto because of it. Every planned case now has numbers, from two runs each unless a table says otherwise. The numbers belong to an alpha: they describe the code as it was on those days. Since 0.0.1-alpha.11 large files are cut into larger blocks (1 MiB on average from 32 MiB, 4 MiB from 1 GiB), so the block counts here no longer apply to them. The driver is in the repository under scripts/bench/, and the summarised results are published as JSON, so every number on this page can be checked and the run repeated.

Summary

From a home behind NAT (10 October)

The runs above are between data centres with public addresses. These are from a home: a desktop PC on a 100 Mbit/s consumer connection behind its router's NAT, with Varsto installed the way a user installs it (the Linux package, a background service of the user). Two cloud machines joined it: one in Amsterdam with a public address, and one in Warsaw hosting two devices in network namespaces behind NAT, one behind a cone NAT (Linux MASQUERADE keeps the port) and one behind a symmetric NAT (MASQUERADE with random ports, which defeats hole punching). As above, the bucket in Paris held the ledger and the file lists only: before every pull its blocks were deleted, so every block had to come from a peer, and every file was checked against its SHA-256. The driver is scripts/nat-test/home-nat.py.

The line

Home PC andRound tripTCP, one stream, to the homeTCP, one stream, from the home
Amsterdam31 ms9.1 MB/s12.1 MB/s
Warsaw21 ms10.6 MB/s11.9 MB/s
The bucket (Paris), rclone, 256 MiB–7.5 MB/s4.3 MB/s

100 Mbit/s is 12.5 MB/s at most, so that is the ceiling for anything on this line.

Through the NATs

STUN classified the home router as a cone NAT. Each transfer moved one 64 MiB file of random bytes; at this size the fixed cost of a sync (ledger and file list in the bucket) is a large part of the time.

FromToPath the receiver usedTimeSpeed
HomeAmsterdam (public)direct, into the home NAT12.2 s5.5 MB/s
AmsterdamHomedirect14.3 s4.7 MB/s
Warsaw, cone NATHomedirect, NAT to NAT (hole punching)13.3 s5.0 MB/s
HomeWarsaw, cone NATdirect, NAT to NAT14.5 s4.6 MB/s
Warsaw, symmetric NATHomerelayed via Amsterdam65.7 s1.0 MB/s
HomeWarsaw, symmetric NATrelayed via Amsterdam65.3 s1.0 MB/s

Every path worked the way the design says: direct where a NAT keeps its mapping, a relay through one of the user's own reachable devices where it does not. No vendor server was involved; the devices found each other through records in the bucket and public STUN servers.

Varsto's Traffic card on the home PC: lines from this device to waw-cone (direct), ams (direct) and waw-sym (via ams), a table with path, address, speeds and totals per device, and at the top right the curve of a download that just finished.The same Traffic card in dark mode.
The home PC's Traffic card seconds after a 512 MiB download from the cone-NAT device in Warsaw (the curve at the top right): Warsaw and Amsterdam reached directly, the symmetric-NAT device through Amsterdam.

Large files

One 512 MiB file of random bytes per transfer, peer to peer only. The first column is version 0.0.1-alpha.9 with the relay fix below; the second adds the download changes of the same day (connections kept open per peer, objects sent without compressing them again, a window that adapts between 8 and 32 blocks).

TransferBefore the changesWith the changesShare of one TCP streamShare of 100 Mbit/s
Amsterdam → home4.7 MB/s8.0 MB/s88 %64 %
Warsaw (cone NAT) → home6.9 MB/s9.3 MB/s88 %74 %
Home → Amsterdam8.7 MB/s9.3 MB/s77 %74 %
Home → Warsaw (cone NAT)–8.6 MB/s72 %69 %

The same download from the bucket

A second device on the home PC, with peer to peer off, pulled a 512 MiB file that a server had just pushed to the bucket, through the same line.

Written byFrom the bucketPeer to peer (above)Peer to peer was
Amsterdam6.1 MB/s8.0 MB/s1.3 times as fast
Warsaw (cone NAT)5.2 MB/s9.3 MB/s1.8 times as fast

Between data centres the bucket can keep up, because the bucket and the machines sit on the same fast backbone. From a home, the bucket is one more distant hop with its own per-request cost, while a peer sends straight down the line.

Found and fixed

Still slow

Limits: one home connection, one afternoon, one run per case (a repeat of Warsaw to home during the screenshot ran at 5.2 MB/s, with Amsterdam briefly unreachable); the NATs in Warsaw are Linux's, not consumer routers'; the servers were small (Scaleway DEV1-M). The download changes are in review and not yet released.

What was measured

Data setFrom the bucket (S3)Peer to peer
Large files (2 × 2 GiB)Amsterdam, Warsaw, both at once (9 Oct)Amsterdam, Warsaw (9 Oct); both at once (10 Oct)
Many small files (10 000)Amsterdam, Warsaw, both at once (10 Oct; the 9 Oct runs are kept as “before fetching ahead”)Amsterdam, Warsaw, both at once (10 Oct)
Compressible text (512 MiB)Amsterdam, Warsaw (9 Oct); both at once (10 Oct)Amsterdam, Warsaw, both at once (10 Oct)
The writer's push2 runs per set, one each dayRAM disk: 2 runs per set; local disk: 1 run per set

Two runs per case; times in the tables are their median (the mean of two), with both runs next to it. Single runs say so. Between the two days the download code changed only in ways that do not touch the compared case: the 9 October large-file and text runs fetch blocks inside a file exactly as the 10 October build does, and the many-small-files bucket runs were repeated on 10 October because fetching ahead across files is new. The last two cases, compressible text peer to peer and many small files from the bucket, each with both readers at once, were run three hours after the others on fresh machines with the merged build; the bucket was clearly faster in that session than in the earlier one.

Setup and baseline

Network baseline

BetweenRound trip (ping)TCP, one stream (iperf3, Paris sends), 9 / 10 OctTCP, four streams
Paris – Amsterdam9.4 ms925 / 958 Mbit/s875 / 836 Mbit/s
Paris – Warsaw33.4 ms738 / 777 Mbit/s949 / 909 Mbit/s
Amsterdam – Warsaw20.6 ms––
ReaderS3: one 256 MiB object, one stream (rclone), 9 / 10 OctS3: 4 KiB objects one after another, 9 / 10 Oct
Paris91 / 59 MB/s7 / 45 ms each
Amsterdam26 / 21 MB/s16 / 107 ms each
Warsaw52 / 15 MB/s52 / 122 ms each

The links between the machines carried 700 to 950 Mbit/s (90 to 120 MB/s) in one TCP stream, and none of the Varsto transfers came close: all were limited by round trips and per-request work. The bucket's own speed varied a lot from one day to the next (the second measurement was taken in the early morning); its runs varied more than the peer-to-peer runs throughout.

Data sets

What “peer to peer only” means

Varsto asks peers first and storages second, so a peer-to-peer measurement must make sure the bucket cannot help. After the writer had pushed and the bucket runs were done, every object under chunks/ was deleted from the bucket. The bucket kept what devices need to find each other and to know what exists (signed ledger, file lists, device and peer records), but no file content: a block that no peer could deliver would have left the file unavailable. Then:

RunChunks downloadedFrom peersFrom S3Path
files, Amsterdam, alone (×2)20 24020 2400writer-paris: direct
files, Amsterdam, both at once (×2)20 24020 2400waw-p2p-files-ams+waw-1: direct, writer-paris: direct
files, Warsaw, both at once (×2)20 24020 2400: direct, ams-p2p-files-ams+waw-2: , writer-paris: direct
files, Warsaw, alone (×2)20 24020 2400writer-paris: direct
large, Amsterdam, alone (×2)26 94626 9460writer-paris: direct
large, Amsterdam, both at once (×2)26 75826 7580: direct, writer-paris: direct
large, Warsaw, both at once (×2)26 75826 7580ams-p2p-large-ams+waw-1: direct, ams-p2p-large-ams+waw-2: direct, writer-paris: direct
large, Warsaw, alone (×2)26 94626 9460writer-paris: direct
text, Amsterdam, alone (×2)3 5443 5440writer-paris: direct
text, Amsterdam, both at once (×2)3 5443 5440: direct, writer-paris: direct
text, Warsaw, both at once (×2)3 5443 5440ams-p2p-text-ams+waw-1: direct, writer-paris: direct
text, Warsaw, alone (×2)3 5443 5440writer-paris: direct

Large files

4 GiB in two files, random content, so zstd stores it as it is.

Peer to peer (from the writer in Paris) S3 bucket in Paris (MB/s, more is better)

Amsterdam, alone Amsterdam, alone, peer to peer: 27.0 MB/s 27.0 Amsterdam, alone, S3 bucket: 23.9 MB/s 23.9 Warsaw, alone Warsaw, alone, peer to peer: 10.2 MB/s 10.2 Warsaw, alone, S3 bucket: 23.4 MB/s 23.4 Amsterdam, both at once Amsterdam, both at once, peer to peer: 26.6 MB/s 26.6 Amsterdam, both at once, S3 bucket: 45.2 MB/s 45.2 Warsaw, both at once Warsaw, both at once, peer to peer: 11.8 MB/s 11.8 Warsaw, both at once, S3 bucket: 20.9 MB/s 20.9
ReaderPeer to peer: timeMB/sS3: timeMB/sP2P vs S3
Amsterdam, alone159 s (158 s / 161 s)27.0180 s (171 s / 190 s)23.91.1× faster
Warsaw, alone423 s (420 s / 426 s)10.2190 s (155 s / 226 s)23.42.2× slower
Amsterdam, both at once162 s (162 s / 161 s)26.695 s (90 s / 101 s)45.21.7× slower
Warsaw, both at once365 s (367 s / 362 s)11.8208 s (185 s / 232 s)20.91.8× slower
BuildReaderTimeMB/s
Peer to peer with QUIC tried before TCP (one connection per peer), 10 OctoberAmsterdam, alone230 s (229 s / 230 s)18.7
Peer to peer with QUIC tried before TCP (one connection per peer), 10 OctoberWarsaw, alone466 s (466 s)9.2

To Amsterdam, nine milliseconds away, peer to peer beat the bucket by a little and was much steadier: every peer-to-peer run took 158 s to 162 s, the bucket's 171 s and 190 s. To Warsaw the bucket was more than twice as fast. Each block of about 300 KiB is one HTTP request on a new TCP connection: a handshake, then TCP slow start from ten packets, so a block needs several round trips however fast the link is. At 33 ms that caps eight requests in flight at about 10 MB/s, which is what was measured. The bucket's HTTPS connections are kept open between requests.

With both readers at once, Warsaw took 2.97 GB of its 4.30 GB from Amsterdam (20 ms away, closer than Paris) and only 1.3 GB from Paris; Amsterdam, which had the blocks first, served them while it downloaded. Warsaw was even a little faster than alone (11.8 against 10.2 MB/s) and Amsterdam as fast as alone. The bucket served two readers at once without slowing either, too (45 and 21 MB/s); it is shared infrastructure built for many readers.

The reader's own work: about 40 s of CPU time for 4 GiB (decrypting, two hashes per block, writing), at most 490 MB of memory. The Paris device used 40 s of CPU time to serve each 4 GiB, one core about a quarter busy.

Many small files

10 000 files of 1 KiB to 256 KiB. Each file is one block. Since 10 October a pull fetches the blocks of up to 32 coming files together, eight requests at a time, before writing them one by one.

Peer to peer (from the writer in Paris) S3 bucket in Paris (files/s, more is better)

Amsterdam, alone Amsterdam, alone, peer to peer: 90.2 files/s 90.2 Amsterdam, alone, S3 bucket: 33.7 files/s 33.7 Warsaw, alone Warsaw, alone, peer to peer: 38.4 files/s 38.4 Warsaw, alone, S3 bucket: 34.2 files/s 34.2 Amsterdam, both at once Amsterdam, both at once, peer to peer: 70.3 files/s 70.3 Amsterdam, both at once, S3 bucket: 64.7 files/s 64.7 Warsaw, both at once Warsaw, both at once, peer to peer: 38.6 files/s 38.6 Warsaw, both at once, S3 bucket: 53.5 files/s 53.5
ReaderPeer to peer: timeMB/sfiles/sS3: timeMB/sfiles/sP2P vs S3
Amsterdam, alone111 s (111 s / 111 s)2.690.2299 s (323 s / 274 s)1.033.72.7× faster
Warsaw, alone260 s (258 s / 263 s)1.138.4294 s (269 s / 320 s)1.034.21.1× faster
Amsterdam, both at once156 s (202 s / 110 s)2.170.3156 s (142 s / 170 s)1.964.71.0× faster
Warsaw, both at once259 s (265 s / 253 s)1.138.6192 s (161 s / 223 s)1.653.51.3× slower
BuildReaderTimeMB/sfiles/s
S3 before fetching ahead across files (one file at a time), 9 OctoberAmsterdam, alone404 s (313 s / 495 s)0.826.1
S3 before fetching ahead across files (one file at a time), 9 OctoberAmsterdam, both at once381 s (381 s / 380 s)0.826.2
S3 before fetching ahead across files (one file at a time), 9 OctoberWarsaw, both at once528 s (494 s / 563 s)0.619.1
S3 before fetching ahead across files (one file at a time), 9 OctoberWarsaw, alone596 s (536 s / 655 s)0.516.9

To Amsterdam, peer to peer delivered 90 files a second, 2.7 times the bucket's 34. To Warsaw, three and a half times farther away, both came to the same 34 to 38 files a second: the round trip per request dominates there, on both paths. Fetching ahead across files doubled the bucket's speed to Warsaw (from 17 to 34 files a second) and helped to Amsterdam (26 to 34). Once the blocks are there, writing each file (data synced to disk, a rename, ledger events) costs a few milliseconds on these network block volumes, which caps any source at a few hundred files a second.

With both readers at once, Warsaw was as fast as alone and Amsterdam's two runs differed (202 s and 110 s): the readers' blocks are fetched from whichever peer answers first, and one run waited on the other reader more than the other run did. The bucket, measured with both readers three hours later, delivered 64 files a second to Amsterdam and 53 to Warsaw: nearly twice what it gave single readers in the earlier session. Its speed depends on the hour more than on the number of readers.

Compressible text

512 MiB of log lines, stored as 138 MiB. Fewer bytes travel, so a reader gets more file per second than with random data.

Peer to peer (from the writer in Paris) S3 bucket in Paris (MB/s, more is better)

Amsterdam, alone Amsterdam, alone, peer to peer: 16.7 MB/s 16.7 Amsterdam, alone, S3 bucket: 18.1 MB/s 18.1 Warsaw, alone Warsaw, alone, peer to peer: 7.2 MB/s 7.2 Warsaw, alone, S3 bucket: 16.7 MB/s 16.7 Amsterdam, both at once Amsterdam, both at once, peer to peer: 16.9 MB/s 16.9 Amsterdam, both at once, S3 bucket: 18.4 MB/s 18.4 Warsaw, both at once Warsaw, both at once, peer to peer: 8.1 MB/s 8.1 Warsaw, both at once, S3 bucket: 12.3 MB/s 12.3
ReaderPeer to peer: timeMB/sS3: timeMB/sP2P vs S3
Amsterdam, alone32 s (33 s / 31 s)16.736 s (21 s / 52 s)18.11.1× faster
Warsaw, alone74 s (70 s / 79 s)7.232 s (32 s / 32 s)16.72.3× slower
Amsterdam, both at once32 s (32 s / 31 s)16.931 s (23 s / 39 s)18.41.0× slower
Warsaw, both at once66 s (63 s / 69 s)8.163 s (28 s / 98 s)12.31.0× slower

Here the bucket was a little faster to Amsterdam and more than twice as fast to Warsaw. With both readers at once, peer to peer gave Amsterdam the same 16.9 MB/s as alone and Warsaw 8.1 MB/s, slightly more than alone; in the first of the two runs Warsaw took two thirds of its blocks from Amsterdam, in the second all from Paris. A device that serves a block does not keep the compressed copy: it reads the plain file and compresses the block again, at zstd level 12, for every request; the Paris device spent 31 s of CPU time per 512 MiB served, against 40 s for 4 GiB of random data. The bucket returns the stored object as it is.

The writer's side

Before anything can be downloaded, the writer cuts its files into blocks, compresses, encrypts and stores them. Three destinations show where the time goes: a RAM disk (Varsto's own work only), the machine's disk (a network block volume, where every object is written with a sync to disk), and the bucket.

Data setDestinationBuildTimeMB/sCPU (user + system)Peak memory
files: 10 000 files, 0.27 GiBRAM disk (tmpfs)measured build50 s (median of 2)5.912 s (5 + 7)68 MiB
files: 10 000 files, 0.27 GiBS3 bucket, same regionmeasured build652 s (median of 2)0.521 s (8 + 13)69 MiB
small: 50 000 files, 1.36 GiBRAM disk (tmpfs)measured build255 s (median of 2)5.762 s (27 + 35)258 MiB
small: 50 000 files, 1.36 GiBLocal disk (network block volume)measured build468 s3.167 s (27 + 41)258 MiB
small: 50 000 files, 1.36 GiBLocal disk (network block volume)alpha.8 (before the zstd fix)515 s2.8109 s (31 + 78)253 MiB
large: 2 files, 4.00 GiBS3 bucket, same regionmeasured build932 s (median of 2)4.680 s (37 + 43)108 MiB
big: 5 files, 12.00 GiBLocal disk (network block volume)measured build418 s30.9197 s (85 + 112)147 MiB
big: 5 files, 12.00 GiBLocal disk (network block volume)alpha.8 (before the zstd fix)694 s18.6483 s (110 + 373)137 MiB
text: 128 files, 0.50 GiBRAM disk (tmpfs)measured build45 s (median of 2)12.044 s (41 + 2)68 MiB
text: 128 files, 0.50 GiBLocal disk (network block volume)measured build56 s9.645 s (42 + 3)68 MiB
text: 128 files, 0.50 GiBLocal disk (network block volume)alpha.8 (before the zstd fix)68 s7.956 s (43 + 13)68 MiB
text: 128 files, 0.50 GiBS3 bucket, same regionmeasured build145 s (median of 2)3.745 s (41 + 4)77 MiB

Fixed along the way

The benchmark was also a hunt for bottlenecks. These changes are in the measured builds, each with a test:

ProblemFixBeforeAfter
A download fetched one block at a time: every block waited a full round trip before the next was asked for.Up to eight blocks of a file in flight at once, from peers or storages, in a sliding window; they are still checked and written in order. The S3 client keeps enough idle connections for them.Text from the bucket to Warsaw: 122 s (4.4 MB/s)27.8 s (19.4 MB/s), 4.4 times faster (single runs; steps on the way: fixed batches of eight 104 s, with idle connections kept 61 s)
Small files went one at a time: each file is one block, so the window above never applied.A pull fetches the blocks of up to 32 coming single-block files together (10 Oct).10 000 files from the bucket to Warsaw: 596 s (17 files/s)294 s (34 files/s)
The peer-to-peer HTTP server answered one request at a time, so eight requests in flight queued at the server.Eight requests answered at once.Large files peer to peer to Amsterdam: 363 s (11.8 MB/s), one run159 s (27.0 MB/s)
A fresh zstd context was created for every block; its tables were allocated and cleared each time, more work than compressing the block.One context per thread, reused (the output is byte-identical, which a test checks).12 GiB to local disk: 694 s, 483 s CPU418 s, 197 s CPU (single runs)
The S3 client refused any response over 10 MiB. After a push of a few thousand files the ledger batch was 13 MB, and no other device could join or sync.Responses up to 2 GiB are read.Joining failedJoining works; test with a 12 MiB object
The background service treated every opened file as a change: serving blocks to peers, a backup tool or a file viewer started a full sync every few seconds.Opening and reading are not changes; closing a file opened for writing still is.90 syncs in 11 minutes while files were readReading all 10 000 files: no extra sync
Requests in flight together each opened their own QUIC connection to a peer they had no connection to yet.One connection attempt per address; the others wait and share it (a test checks one connection for a whole pull).Found while testing QUIC (below)
A kept QUIC connection to a peer that had restarted failed every request on it, and the peer dropped to the next route for good.Reconnect once and ask again.Found while testing QUIC (below)
Requests that finished on an old route could undo a switch to a better route that another request had just made.A late answer keeps the current route.Found while testing QUIC (below)

Tried and reverted: QUIC first

Because the TCP path opens a connection per block, QUIC looked like the cure: Varsto already speaks QUIC to peers behind NAT, with one connection per peer and one stream per request. On 10 October the route order was changed to try QUIC before TCP for addresses beyond the LAN. It was slower on these machines, and used more than twice the CPU time:

Large files, peer to peerTCP first (kept)QUIC first (reverted)
Amsterdam159 s, 27.0 MB/s; reader 38 s CPU, Paris 40 s230 s, 18.7 MB/s (229 s / 230 s); reader 98 s CPU, Paris 88 s
Warsaw423 s, 10.2 MB/s466 s, 9.2 MB/s (one run); reader 130 s CPU

So TCP stays first and QUIC remains the way through NATs. Where the QUIC path loses its time was not found in the time available; the host dropped no UDP packets. The experiment also turned up a gap that is not fixed yet: a device's peer table is built before a sync reads the device registry, so a device that has just joined is refused over QUIC by its peers until their next sync (over TCP it is served at once).

Still slow, and why

When peer to peer wins, and when the bucket does

Varsto uses both on purpose, and the measurements show why.

Cost

Scaleway's public price list (read on 9 October 2026, Paris region, before tax): Object Storage Standard Multi-AZ €0.000022 per GB-hour (about €0.016 per GB-month), requests included, ingress included, egress the first 75 GB a month free and then €0.01 per GB. Transfers between regions (Paris – Amsterdam, Paris – Warsaw) count as egress; within a region they are free.

So one reader in Amsterdam downloading the 4 GiB set from the Paris bucket costs about €0.04 once the free 75 GB are used, and a household that syncs 500 GB a month to devices outside the bucket's region pays about €4 a month in egress. Peer to peer moves the same bytes over the devices' own connections and adds nothing to the storage bill. Other providers charge more for egress or per request; the method above applies to them unchanged. The machines for this benchmark (three instances at €0.147 an hour, about seven and a half hours over the two days) cost about €3.50.

Limits of this benchmark

Reproduce: scripts/bench/p2p-bench.py creates the machines and the bucket with the Scaleway command line, runs each step, writes results.jsonl, and deletes everything again; report.py and page.py turn the results into the summary and the tables on this page.