Sclone, for "Storage Clone", is a program to sync files to and from different cloud storage providers supporting S3 and OpenStack SWIFT.
It offers fast speed thanks to parallelized workloads and file list caching. If you would like to know more about performance, refer to the benchmarks section.
- Unidirectional mode: mode to just copy new/changed/deleted files from a source to a destination.
- Bidirectional mode: mode to make the source and target buckets identical.
- Job scheduler: Use the Cron syntax to start the process every minute, every 30 minutes, or anytime you want.
- Optional deletion: by default deletion is disabled: missing files are added on the source/target bucket. When enabled, files are deleted on the source/target bucket.
- High speed: File transfers are split into parallel queues. At the end of each process, the list of files is cached for better performance on the next execution.
- Optional integrity check: MD5 hashes checked for file integrity. Disabled by default.
- Metadata preserved: from S3 to SWIFT or SWIFT to S3, metadata is preserved/converted automatically.
- Production ready: Battle tested with terabytes of buckets.
- Dry run: Output what operations will be performed without actually carrying out those operations.
- Support any S3 and SWIFT provider: AWS S3, OVHCloud, Scaleway, Ceph.io, DigitalOcean Spaces, Cloudflare R2, Seagate Lyve Cloud, Tencent Cloud, Alibaba Cloud OSS, IBM COS S3, Dreamhost S3, GCS, IDrive e2, Synology C2, IONOS Cloud, Minio, Petabox, and more...
The transfer benchmarks below were measured with sclone 1.0.0 and have not been run again with 2.0. With 2.0, a synchronisation without any change of 100 000 files takes about 3.9 s on a local MinIO, and the plan of 350 000 files is computed in about 50 ms.
- Environment: VPS OVH - 2 vCores - 4GB Ram - Bandwidth 500Mbit/s - Debian 12 - Node 20.5.1 - Strasbourg (France)
- Default options were used for sclone (1.0.0) / rclone (v1.63.1) / s3sync (2.61)
- OVH S3 bucket type: normal (and not performance)
Unidirectional sync from a source storage to a target storage located in different regions. Every synchronization used the same 10 GB dataset of 1624 files.
| 10GB from S3 OVH Gra to S3 OVH Sbg | 10GB from S3 OVH Gra to S3 Scaleway Paris | 10GB from S3 OVH Gra to SWIFT OVH Gra | |
|---|---|---|---|
| sclone | 3.30 Min | 4.10 Min | 3.27 Min |
| rclone | 5.45 Min | 10.51 Min | 6.32 Min |
| s3sync | 3.10 Min | 4.09 Min | ❌ |
Bidirectional sync between two storages located in different regions. Every synchronization used the same two 8 GB datasets (1354 files) with 1/3 common files, 1/3 new files and 1/3 edited files.
| 8GB S3 OVH Gra <> 8GB S3 OVH Sbg | 8GB S3 OVH Gra <> 8GB S3 Scaleway Paris | 8GB from S3 OVH Gra <> 8GB SWIFT OVH Gra | |
|---|---|---|---|
| sclone | 3.59 Min | 4.11 Min | 3.42 Min |
| rclone | 8.4 Min | 13.57 Min | 10.40 Min |
- Download the latest binary for Linux (
sclone-<version>-linux) or macOS (sclone-<version>-macos) from the Release page and make it executable:
chmod +x ./sclone-2.0.0-linux- Create a file
config.jsonin the directory you will run the binary from to define the source/target storage credentials, and options. You can copy theconfig.default.jsonfile as an example. Read the configuration section for details. - Finally start the synchronisation:
./sclone-2.0.0-linux- Install the npm package globally:
npm install -g @carboneio/sclone- Create a
config.jsonfile in the directory you will run sclone from (see the configuration section). - Start the synchronisation:
sclone🟢 Tip: Set the option
"dryRun": truein theconfig.json: it writes the operations that would be performed into a JSON file of thelogsfolder, without carrying them out.
Copy the config.default.json (from the repository or the npm package) to a config.json file in the working directory, or set its path with SCLONE_CONFIG (see Environment variables). The file must be plain JSON: remove the comments of the examples below.
| Options | Default value | Description |
|---|---|---|
| source | Option required Storage credentials (S3 Example / SWIFT example) |
|
| target | Option required Storage credentials (S3 Example / SWIFT example) |
|
| mode | Option required Synchronisation mode: ⏺ unidirectional: One way synchronization from source to destination without modifying any of the source files and deleting any of the destination files (unless delete option is enabled).⏺ bidirectional: Two way synchronisation between a source and target storage, without deleting any files (unless delete option is enabled). |
|
| cron | Define the period of the program execution, and must follow the CRON syntax, for instance every minute: "*/1 * * * *". A new process is not started until the current synchronisation is finished. If the option is not defined, the synchronisation is executed immediately, one time. |
|
| delete | false |
If true, files are deleted according to the synchronization mode logic. |
| integrityCheck | false |
If true, MD5 hashes are checked for file integrity. |
| logSync | false |
If true, at the end of each synchronisation, a JSON file including all file operations is created in the logs folder of the working directory (sync-<date>.json). |
| cacheFilename | "sclone-<mode>-<hash>.cache.json" |
File name of the cache: the state of each file after the last synchronisation, written in both modes, unidirectional included (see Cache). By default the name is derived from the mode and the two storages (and their accounts), so several instances can share a working directory. The file is created in the current working directory (the directory the program is started from). An absolute path can also be provided. The lock file is created in the same directory: when the same pair of storages may run from several working directories, use an absolute path, otherwise each directory has its own cache and lock and the runs are not excluded. |
| transfers | 15 |
Number of file operations to run in parallel: upload/deletion. If a storage responds with many errors (status 500, socket or authentication error), consider reducing the number of transfers. |
| retry | 1 |
Max number of retries to sync a file. 0 is read as 1. |
| listRetry | 5 |
Max number of retries of a listing page on a temporary error (network, status 5xx, 408, 429, page cut or invalid). Only the failed page is requested again, not the whole listing. A permanent error (access denied, bucket not found) is not retried. If a page still fails, the synchronisation is stopped before any operation. |
| listRetryDelay | 1000 |
Delay in milliseconds before the first retry of a listing page, doubled on each retry: 1s, 2s, 4s, 8s, 16s by default. |
| dryRun | false |
Do a trial run with no permanent changes: a JSON file including all file operations is created, as with logSync. The cache is not saved. |
| maxDeletion | Max number of deletions allowed on each storage during a synchronisation: up to maxDeletion files are deleted. Above it, no file is deleted on either storage, the uploads still run, and the synchronisation ends as failed (deletion_refused reason=max_deletion). It is a security measure to avoid propagation of unexpected bulk deletions. Not set or 0: no limit. |
|
| allowEmptySide | false |
By default, no file is deleted when a storage is empty while files would be deleted on the other one: listed empty, or none of its listed files was synchronised before (emptied, then refilled with new files). Nothing is uploaded to the empty side and the cache is not saved (it stays empty, so the next runs keep refusing). Checked in both modes, together with maxDeletion. The other uploads still run, and the synchronisation ends as failed (deletion_refused reason=empty_side). An emptied, wrong or unreadable bucket must not wipe the other side. Set true to allow it. |
| name | name of the working directory | Name of the instance, written in the job field of every log line. Give a distinct name to each instance, without spaces. Overridden by the SCLONE_NAME environment variable. |
| logLevel | "info" |
Minimum level of the logs: debug, info, warn or error (case-insensitive). Overridden by the SCLONE_LOG_LEVEL environment variable. An invalid value logs a warning and uses info. |
| logMaxFailures | 100 |
Max number of transfer_failed lines per queue and per synchronisation (one line per failed attempt). The other failures are counted in one transfer_failed_summary line. |
| retryDelay | 1000 |
Delay in milliseconds before retrying the failed transfers (retry), doubled on each retry, capped to 60 seconds. 0 retries immediately. |
| errorFile | false |
Write the failed transfers into a JSON file of the logs folder of the working directory: true after each failed attempt, "final" only for the failures remaining after the last retry. The causes are already in the logs (transfer_failed). |
| logRetentionDays | 7 |
When errorFile is enabled, the error files older than this number of days are deleted. 0 keeps them all. |
{
"name" : "swift", /** Required by Sclone **/
"authUrl" : "https://auth.cloud.ovh.net/v3", /** Insert your own auth URL **/
"username": "", /** Your username **/
"password": "", /** Your password **/
"region" : "", /** Bucket region **/
"bucket" : "", /** Bucket name that will be synchronised **/
"tenantName": "", /** Optional: project (tenant) name to scope the authentication **/
"interface" : "" /** Optional: endpoint interface of the catalog ("public", "internal"...), the first endpoint of the region otherwise **/
}{
"name" : "s3", /** Required by Sclone **/
"url" : "s3.gra.first.cloud.test", /** S3 URL, without the bucket name ("https://" is optional) **/
"accessKeyId" : "", /** Your access key ID **/
"secretAccessKey": "", /** Your secret key **/
"region" : "", /** Bucket region **/
"bucket" : "" /** Bucket name that will be synchronised **/
}| Variable | Description |
|---|---|
| SCLONE_CONFIG | Path to the configuration file, relative to the working directory or absolute. Defaults to config.json. |
| SCLONE_CRON | Cron expression, overrides the cron option of the configuration file. |
| SCLONE_NAME | Name of the instance in the logs (job field), overrides the name option. |
| SCLONE_LOG_LEVEL | Minimum level of the logs (debug, info, warn, error), overrides the logLevel option. |
Sclone writes one line per event on the standard output, in the logfmt format, ready to be parsed by Loki (| logfmt), Vector, Datadog...
ts=2026-11-19T09:57:25.071Z level=info job=backup-invoices run=089a9f5b event=sync_start msg="Synchronisation started" mode=unidirectional source=s3:invoices target=s3:invoices-copy delete=true dry_run=false cron=true
ts=2026-11-19T09:57:25.087Z level=info job=backup-invoices run=089a9f5b event=listing_done msg="target listed" side=target storage=s3 bucket=invoices-copy objects=5 bytes=15000 duration_ms=4
ts=2026-11-19T09:57:25.088Z level=info job=backup-invoices run=089a9f5b event=listing_done msg="source listed" side=source storage=s3 bucket=invoices objects=5 bytes=15000 duration_ms=4
ts=2026-11-19T09:57:25.089Z level=info job=backup-invoices run=089a9f5b event=sync_end msg="Synchronisation success" status=success duration_ms=16 uploaded=0 uploaded_bytes=0 deleted=0 failed=0 source_objects=5 target_objects=5 source_bytes=15000 target_bytes=15000
- Each line starts with
ts(UTC),level,job(thenameoption),run(a random id shared by all the lines of one synchronisation) andevent(a stable name, such assync_start,listing_done,sync_plan,transfer_failed,sync_end). - The
sync_endline ends every synchronisation with itsstatus(success,failed,dry_run,interrupted), its duration, the number of files and bytes transferred, the number of failures and the size of both buckets. Thetransfer_failedlines give the cause of each failed operation: key, HTTP status and error. - Levels:
infoby default. A synchronisation without any change writes 4 lines with acron. Use"logLevel": "debug"to see the cache and queue details,"warn"to only keep the problems. - Credentials are never logged. Object keys are.
- The process exits with code
0on success, and1on a configuration error, a storage connection failure at startup or a failed synchronisation (withoutcron). - On
SIGTERM(systemctl stoporrestart) orSIGINT(Ctrl+C) during a run, sclone writessync_end status=interruptedwith the transfers done so far, releases its lock and exits with code143(SIGTERM) or130(SIGINT); the cache of the interrupted run is not saved, the previous one is kept. Between twocronruns it writes a singleshutdownline. A second signal exits at once.
The list of events and fields, LogQL queries and alerts are available in docs/grafana.md, with a ready-to-import Grafana dashboard: docs/grafana-dashboard.json.
Several sclone processes can share the same server and the same log stream (journald):
- give each instance its own
name(orSCLONE_NAME): the dashboards and alerts are grouped byjob, - each pair of storages gets its own cache and lock file by default, so instances can share a working directory (a
cacheFilenameset by hand must be distinct for each instance). Two configurations of the same pair (unidirectional and bidirectional, or another cache name) whose caches are in the same directory never run at the same time, - stagger the crons (
1-59/3 * * * *,2-59/3 * * * *...) to spread the listings over time.
Sclone adds, updates, and deletes (if enabled) files from a source to a target storage. The source always wins: a file edited or deleted on the target only is copied again from the source.
If the delete option is false, files on the target storage are not deleted even if they do not exist on the source storage. In other words, the target will accumulate all files.
If the delete option is true, files on the target storage that do not exist on the source are deleted. The target will be an exact copy of the source storage.
Sclone propagates the changes made on each side since the last synchronisation, recorded in the cache. It never decides from the modification dates alone: a copy made by sclone is always more recent than its original.
| Source | Target | Operation |
|---|---|---|
| unchanged | unchanged | nothing |
| new or modified | unchanged | copied to the target |
| unchanged | new or modified | copied to the source |
| deleted | unchanged | deleted from the target with delete: true, copied back to the source otherwise |
| unchanged | deleted | deleted from the source with delete: true, copied back to the target otherwise |
| deleted | deleted | nothing |
| deleted | modified | conflict: the modified file is copied back. A modified file is never deleted |
| modified | deleted | conflict: the modified file is copied back |
| modified | modified | nothing if both have the same content, otherwise conflict: the most recent version wins (the source on a tie) |
Without cache (first synchronisation, or a file never synchronised), nothing is deleted: a file present on one side is copied to the other one, and a file different on both sides is a conflict, the most recent version wins. Each conflict is logged as a sync_conflict warning with the key, the winner, and the etag, size and modification date of both versions.
Modification dates are compared to the millisecond.
At the end of each synchronisation, in both modes, the cache file records for each file the token of each side as listed (etag and size), one line per file. A side changed if its etag or size differs from the recorded token. The etag is only used as a token: any rewrite of the file changes it.
- A copied file records the listed token of its origin and the etag returned by the upload. A deleted file is removed from the cache. A failed or refused operation keeps the previous state: it is done again by the next run.
- The cache is saved at the end of every run (except a dry run), also when some operations failed or deletions were refused, atomically (temporary file, flushed, then renamed, directory flushed): a crash never leaves a truncated cache. Temporary files left by a crash are removed after an hour.
- The cache file starts with the mode and the identity of both storages:
s3:<url>:<region>/<bucket>@<account>, where<account>is a hash of the access key id (S3) or of the username and project (Swift), never of a secret. A cache written for another pair of storages or another account, of an unknown version or corrupted is ignored with acache_errorwarning: the run is a first synchronisation, nothing is deleted because of it. Changing the access key id therefore starts a first synchronisation again. - A cache written by sclone 1.x in bidirectional mode is migrated when found under the configured
cacheFilename(cache_migrated): its entries are kept for the files listed on one side only (to propagate the deletions), the files listed on both sides are compared as on a first synchronisation. The previous default namelistFiles.cache.jsonis not read: it may be shared by several instances. - Without cache, both files are identical when their etags are the same MD5 (single upload) or the same multipart etag. An etag which is neither (OVH lists the encrypted objects copied server-side with an opaque 64 hex etag) is resolved once with a
HEADrequest, only when both sizes are equal (etag_resolvedinsync_plan). On the first synchronisation only, an S3 multipart file and a copy of the same size are identical if the multipart version is not more recent (in unidirectional mode, only a multipart source and its plain copy on the target).
A synchronisation without any change sends the listing requests only, plus the connection requests: one ListObjectsV2 of one key (max-keys=1) on the bucket of each S3 storage, and the authentication of each Swift storage (with delete, also one HEAD of the container). With a cron, the storages are connected once at startup, otherwise at the start of each run.
- Lock: a lock file is created in the directory of the cache (
sclone-<hash>.lock, derived from the pair of storages only, with the pid, hostname and start time). Another process synchronising the same storages, in any mode, is refused: a run withoutcronfails, acronrun is skipped (sync_skipped reason=locked). The lock is refreshed every 30 seconds while held. It is stale, and taken over with alock_stalewarning, if it was not refreshed for 90 seconds and its process is no longer running on this host (a recent lock is always alive: a process of another container with the same hostname cannot be checked), or if it was not refreshed for 10 minutes (a process of another host sharing the directory, or a container restarted with a new hostname: the next run may wait up to 10 minutes). The lock is checked again before the first change of a run: if another process took it over in between, the run changes nothing and fails (Lock lost). Acronrun skipped because the lock cannot be created (lock_error), or held by a process running for more than 10 minutes, is logged aterrorlevel. - Deletion guards:
maxDeletionandallowEmptySide, checked before any operation. A refused deletion stops only the deletions: the uploads still run, the refused files keep their cache (the deletion is planned again by the next run), and the run ends as failed with adeletion_refusederror, so that the alerts fire. - Deletions confirmed: a file missing from the listing of one side is deleted from the other side only if a
HEADrequest on the side it is missing from returns404. A listing can miss files (eventually consistent Swift listings...): a file still present is not deleted and keeps its cache (deletion_skipped reason=still_present); aHEADfailing after the retries skips the deletion and fails the run (reason=not_confirmed). OneHEADper deletion,transfersin parallel. - Swift versioning: a container with
X-Versions-Location("stack" mode) restores the previous version of a file onDELETE: a deleted file would come back. Withdelete, such a container is detected at connection (swift_versions_stackerror) and the deletions on this side are refused (deletion_refused reason=swift_versions_stack), the uploads still run.X-History-Location("history" mode) is supported. - Listings: an incomplete or invalid listing page stops the synchronisation before any operation. A Swift listing ends only on an empty page (one more request per listing): a page shorter than requested is not the end on servers capping the pages.
- Errors which do not change by retrying (
400,403such as an archived object or an access denied,404on download,413) are not retried: they are reported once by run astransfer_failed reason=permanentwith the S3 errorcode. Network errors,5xxand429are retried.
- A change of metadata only (headers, without content change) is not detected.
- Files larger than 5 GiB are not copied (
transfer_failed reason=too_large): sclone uploads with a single request, limited to 5 GiB by S3 and by default by Swift. - Memory: about 64 MB, plus 2.2 times the size of the files transferred in parallel (
transfers), plus about 0.5 KB per file for both listings and the cache (measured: 143 MB of heap for 2 × 350 000 files). Each file is buffered in memory during its transfer (up to 5 GiB): size the memory for the largest files, abouttransfers× 2.2 × the largest file size in the worst case (15 files of 1 GB: about 33 GB). Lowertransfersfor buckets of large files. - With
integrityCheck, the downloaded content of an S3 multipart file or a Swift large object (SLO, DLO) cannot be checked against its etag: the check is skipped for these files. The upload is still verified by the storage with theContent-MD5header (S3) or theETagheader (SWIFT).
Requires Node.js 26 or higher.
npm install # install dependencies
npm test # run the test suite (mocha)
npm run lint # lint and auto-fix (eslint)
npm run build # build the Linux and macOS binaries (@yao-pkg/pkg, installed as a dev dependency)Tests and lint run automatically on GitHub Actions for every push and pull request. Pushing a v* tag (matching the package.json version) builds the binaries, attaches them to a GitHub release, and publishes the package to npm.
This package is maintained by Carbone:

{ "mode" : "bidirectional", "delete" : false, "logSync" : false, "cacheFilename" : "sync.cache.json", "cron" : "*/5 * * * *", "integrityCheck": false, "source" : { "name" : "swift", "authUrl" : "https://auth.cloud.ovh.net/v3", "username": "", "password": "", "region" : "", "bucket" : "" }, "target" : { "name" : "s3", "accessKeyId" : "access-key-id", "secretAccessKey": "secret-access-key", "url" : "s3.gra.first.cloud.test", "region" : "gra", "bucket" : "" } }