Alpha software, not for production data. Bugs can cause data loss, formats may change without migration, and the cryptography has not been independently audited. Keep your own backups and check that you can restore from them.
Varsto

Testing

From TESTING.md in the repository, rendered 2026-10-12.

Varsto is tested in layers. The goal is that cheaper AI systems and scripts can run most tests from written instructions, while a human covers what cannot be automated (Windows, macOS, Android and iOS on real devices).

Alpha (0.0.1-alpha.11): the Rust core, the CLI, the background service with tray and web interface, and the Android, macOS and iOS (Simulator) shells exist (crates/, apps/). CI builds and tests all of them on GitHub-hosted runners; the VM levels below are prepared for what CI does not cover.

Principles

  • Machine-decidable results. Every test returns an exit code and a machine-readable result (JUnit XML plus JSON). A tester should never have to judge "did it work?" by reading prose.
  • Reproducible. Randomised and simulation tests print their seed; any failure can be replayed with the seed.
  • Disposable environments. Linux tests run in throw-away virtual machines or containers that can be started, broken and destroyed without limit.
  • Synthetic data only. Never use real user data or real credentials. Test data is generated by scripts.
  • Safe for AI agents. Agents follow docs/testing/ai-runbook.md. They run tests and write reports; they do not change production code unless explicitly asked.

Test levels

  1. Unit and property tests (Rust core).
  2. Integration tests against real storage back ends: MinIO (S3), an SFTP server, a local folder, rclone remotes, an S3 server emulating storage classes and restore delays (cold storage).
  3. Deterministic simulation of the sync and bookkeeping protocol with fault injection (network partition, crashes, clock skew, partial writes).
  4. Multi-device scenarios in virtual networks (LAN-only sync without internet, NAT types, packet loss).
  5. Removable-disk scenarios (hot-plugged virtual disks, unplug during write, corruption, full disk, cloned disk).
  6. Upgrade and migration tests, long soak tests, performance tests (very large files, millions of small files).
  7. CLI contract tests (JSON schema, exit codes).
  8. Security tests: fuzzing, dependency audit and licence checks, secret scanning, reproducible-build comparison.
  9. Mobile tests: see docs/testing/mobile.md. Devices: docs/testing/devices.md.
  10. Manual tests on Windows, macOS, Android and iOS: see docs/testing/manual.

The matrix is defined in tests/matrix.yaml.

Running tests (Linux)

Rust (levels 1 and 2, no VM needed)

cargo test --workspace                 # unit tests, two-device sync scenarios, CLI smoke test
cargo clippy --workspace --all-targets
cargo fmt --all --check

The integration tests in crates/varsto-core/tests/ simulate two devices that are never online together, a concurrent edit, and a device restored from an old backup (which must be fenced). The CLI test in crates/varsto-cli/tests/ runs the built binary end to end with --json.

Virtual machines (levels 3 and up)

scripts/test/provision-vm.sh <distro>      # create a disposable VM, prints its name
scripts/test/run-suite.sh <suite> <vm>     # run a suite in that VM
scripts/test/collect-logs.sh <vm>          # fetch logs and results
scripts/test/destroy-vm.sh <vm>            # always destroy the VM afterwards

Every script accepts --dry-run, which prints the commands without executing them.

Only the smoke suite and the debian-stable (x86_64) image exist so far.

VM lab (scripts/test/lab)

scripts/test/lab/lab.py up builds a small internet on one Linux host with libvirt: a WAN in a documentation range with an S3 server (rclone serve s3) and STUN, a home network behind a cone NAT router, an office network behind a symmetric NAT router, and a public device. lab.py build compiles a static binary in a build VM, install-bin puts it on every device, and scenarios.py plays 27 scenarios against them: setup and pairing, sync and conflicts, selective sync, repair, policies, Strongrooms, organizations and removal, revocation, peer-to-peer on the LAN, through cone and symmetric NAT and during a slow upload, sharing, disk pools, mixed versions, storage outage, a black-holed network, a crash during a push, clock skew and scale (20 000 files and 1 GiB). lab.py release builds the Linux, Windows and Android release packages in the build VM.

The rest of this section describes the smoke suite. The smoke suite checks that the VM boots, accepts SSH, finished cloud-init, has a sane clock and has no route out of the test network.

Prerequisites

  • Linux with libvirt and QEMU/KVM (virsh, qemu-img, /dev/kvm), ssh, ssh-keygen, curl, sha512sum.
  • One of xorriso, genisoimage, mkisofs or cloud-localds to build the cloud-init seed.
  • Your user must be in the libvirt group; no root is needed. The scripts use qemu:///system and the storage pool default.
  • shellcheck if you change the scripts (shellcheck -x scripts/test/*.sh scripts/test/lib/common.sh).

What the scripts create and where

  • Base images are downloaded once from the distribution's official cloud-image site, verified against the published SHA-512 checksum, and cached in ~/.cache/varsto-test/images/ and in the libvirt pool.
  • Per-VM state (SSH key generated for that run, cloud-init files, generated XML) lives in ~/.cache/varsto-test/vms/<vm>/ and is removed by destroy-vm.sh.
  • Each VM gets a copy-on-write disk, a seed ISO and its own isolated libvirt network (no <forward>: no route to the LAN or the internet). All of these are named varsto-test-<distro>-<id>; the scripts refuse to touch anything else.
  • Results are written to test-results/<vm>/ (smoke.junit.xml, smoke.json, var-log.tar.gz); the directory is git-ignored.

Environment variables: VARSTO_LIBVIRT_URI, VARSTO_POOL, VARSTO_CACHE_DIR, VARSTO_RESULTS_DIR, VARSTO_VM_CPUS, VARSTO_VM_MEM_MIB, VARSTO_VM_DISK_SIZE, and VARSTO_KEEP_ON_FAIL=1 (keep a half-created VM for debugging when provisioning fails).

Exit codes

Code Meaning
0 Success; for run-suite.sh, the suite passed
1 The suite ran and at least one test failed
2 Usage error, unknown or not yet implemented distro or suite
3 Environment problem: missing tool, failed download or checksum, VM did not accept SSH

destroy-vm.sh exits 0 even if the VM is already gone. If provision-vm.sh fails it cleans up after itself.

Result format

smoke.json has the fields suite, vm, commit, result (pass or fail), started_utc, finished_utc and tests (a list of name, status, seconds, message). smoke.junit.xml is standard JUnit XML.

Results and reports

Use docs/testing/report-template.md. Automated and manual runs use the same fields so reports can be combined.

Release gates

  • Alpha to beta: all automated Linux suites green, and signed manual checklists for Windows, macOS, Android and iOS.
  • Beta to 1.0: independent security review published, reproducible builds verified.