SFTP Automation: A Cron-Safe Script, and the Way That Needs No Script

SFTP automation almost always starts the same way: someone types a put into an sftp prompt every morning, gets tired of it, and writes a script. The script grows a cron entry, then a retry, then a second partner, then an owner who left the company two years ago.

There are two sensible ways to handle it. Build the script carefully, with key authentication, host verification, locking, logging, and failure reporting, because most of them are built without any of that. Or stop maintaining transfer scripts and let Files.com make the SFTP connections and run the schedule.

This guide covers both, with OpenSSH, bash, and cron as the examples and the Windows equivalent noted where it differs.

The batch file: sftp -b

The OpenSSH sftp client ships on Linux, macOS, and current Windows, and its automation mode reads commands from a batch file passed with -b:

cat > /var/lib/transfer/upload.sftp <<'SFTP'
put /data/exports/orders.csv /inbound/orders.csv.tmp
rename /inbound/orders.csv.tmp /inbound/orders.csv
bye
SFTP

sftp -b /var/lib/transfer/upload.sftp \
     -i /etc/sftp/keys/brightway_ed25519 \
     -o BatchMode=yes \
     -o StrictHostKeyChecking=yes \
     alice@sftp.example.com

Each flag earns its place. -b reads the commands from the file. -i names the private key, because an unattended job authenticates with a key and never a typed password; the walkthrough on setting up SSH keys covers generating one.

BatchMode=yes stops the client from ever pausing for a prompt, so a missing key fails the job instead of hanging until the next cron run stacks on top. StrictHostKeyChecking=yes makes an unknown or changed host key a hard failure, which is what you want at 2 a.m.

A nonstandard port is -P 2222 with a capital P; lowercase -p means something else and is a classic five-minute mistake.

The upload-then-rename pair is the detail most scripts skip. A partner polling /inbound can see orders.csv while it is still half written and process a truncated file. Uploading to a temporary name and renaming makes the file appear all at once, on servers where a same-directory rename is atomic, which is nearly all of them. Confirm how the partner's server handles a rename onto an existing file if you replace files in place.

In a batch file, a failing command aborts the rest and sftp exits nonzero. Prefix a command with - when its failure is acceptable, for example -rm /inbound/old-orders.csv when the file may already be gone.

A script that is safe to run from cron

The sftp command is the easy part. Most production problems come from the wrapper around it. An unattended job has to fail without prompting, refuse to overlap itself, preserve the transfer's exit status, write a log someone can read, tell a person when it fails, and run under a dedicated service account. Here is a version that does all of that:

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

LOG=/var/log/transfer/brightway.log
LOCK=/var/lock/brightway-upload.lock
KEY=/etc/sftp/keys/brightway_ed25519
BATCH=/var/lib/transfer/upload.sftp
KNOWN_HOSTS=/etc/sftp/known_hosts

log() {
  printf '%s %s\n' "$(date -Is)" "$*" >>"$LOG"
}

exec 9>"$LOCK"
if ! flock -n 9; then
  log "previous upload is still running, exiting"
  echo "brightway upload: previous run still active" >&2
  exit 75
fi

log "starting upload"

if sftp -b "$BATCH" -i "$KEY" \
        -o BatchMode=yes \
        -o StrictHostKeyChecking=yes \
        -o UserKnownHostsFile="$KNOWN_HOSTS" \
        -o ConnectTimeout=30 \
        alice@sftp.example.com >>"$LOG" 2>&1
then
  log "upload complete"
else
  status=$?
  log "upload FAILED with exit status $status"
  tail -n 100 "$LOG" >&2
  exit "$status"
fi

set -Eeuo pipefail catches the common shell mistakes: -e stops on an unhandled failure, -u treats an unset variable as an error, pipefail surfaces a failure hidden inside a pipeline, and -E keeps any error trap working inside functions. It does not replace the explicit if sftp block, which is what records and returns the transfer's real exit status.

flock takes a lock so two runs never overlap. Without it, a slow transfer can still be running when the next scheduled run starts, and two processes upload and rename the same files at once.

This script treats an overlap as an error and exits with status 75, so a job that has been stuck for hours gets noticed instead of hidden.

flock is standard on Linux and absent from macOS by default, where a launchd job or another locking mechanism does the same work.

The dedicated known_hosts file, named with UserKnownHostsFile, holds the partner's verified host key and belongs to the transfer account. Get the fingerprint from the partner through a trusted channel and check it before saving the key. ssh-keyscan retrieves a server's key but proves nothing about whose server it is; treat it as a collection tool, not verification.

Every log line carries a timestamp, and the script's exit code is the transfer's exit code, which is what every scheduler uses to decide whether something went wrong.

Failure notification

The cron entry:

MAILTO=transfers@example.com
15 2 * * * /usr/local/bin/brightway-upload.sh

Cron sends mail when a job produces output, not because it exits nonzero. That is why the script writes routine detail to its log and writes the tail of the log to standard error only when a run fails: a successful night is silent and a failed one lands in someone's inbox. On a systemd host, a timer unit gives the same schedule with journal logging, dependencies, and better failure handling than cron.

Test it as the account that will run it

Do not test the script as yourself and assume cron will behave the same way. Run it as the service account, with sudo -u transfer /usr/local/bin/brightway-upload.sh, because SSH keys, file permissions, home directories, environment variables, and known_hosts are all per user, and "it works from my terminal" is the most common way an SFTP job fails in production.

While you are there, check that the private key is readable only by that account, that the log and lock locations are writable by it, that a changed host key produces a hard failure and an alert, and that a failed upload never reaches the final rename.

Mirroring a whole folder

sftp is built for known file operations, not synchronization. When the requirement is "upload everything that changed," lftp speaks SFTP and has a mirror command that copies only the differences:

lftp -e "mirror -R /data/exports /inbound; bye" -u alice, sftp://sftp.example.com

-R reverses the direction to upload. Configure key authentication and host verification explicitly before running it unattended, and never turn on automatic host key acceptance to stop a prompt. rsync is the better synchronization tool when both sides support it, but it needs a shell on the far end, and the SFTP-only account most partners hand out does not have one.

Windows

The same job on Windows is a scheduled task running the built-in OpenSSH sftp.exe with the same -b batch file, or a PowerShell script using the Posh-SSH module. The PowerShell SFTP walkthrough covers both, with key authentication and the Task Scheduler configuration. The principles do not change: keys not passwords, a verified host key, no concurrent runs, a failing exit status, logs and alerts, and testing under the task's service account.

The way that needs no script

A well-built script is reasonable for one partner, one direction, and one machine you already monitor.

The second partner is a copy of the script with a different key and hostname. The tenth is a folder of nearly identical scripts, each with slightly different behavior, a known_hosts entry, a log nobody reads, and a cron line that fires whether or not the partner's server is up.

There is also a blind spot no script can fix: scripts report failed commands, and they cannot report missing activity. If the partner owes you a file by 6 a.m. and never sends it, no command fails because no command runs.

The alternative is to move the SFTP connections and the schedule off the machine and onto Files.com.

Files.com makes the SFTP connections

A Remote Server Mount connects a folder on your Files.com site to any SFTP server. You give Files.com the partner's hostname, port, and credentials, and the partner's server becomes a folder, say /partners/brightway, that passes every operation through in real time.

Files.com authenticates with a password, a private key, or both, and it can generate the key pair itself (RSA, 4096-bit) so you hand the partner a public key and the private half never leaves Files.com.

It detects and stores the partner's host key on the first connection and disables the connection if that key ever changes, which is StrictHostKeyChecking=yes enforced by default and managed in one place. The partner allowlists the Files.com IP addresses once.

Ten partners are ten mounts, each with encrypted credentials, and no keys on any machine.

A Sync is the cron job

A Files.com Sync moves files between any two locations on the site on a schedule: push from /outbound/brightway to /partners/brightway/inbound, pull acknowledgments back the other way, with include and exclude filters and a start time you choose.

It retries failures, logs every run and every file, emails administrators when a run fails, and has a dry run that shows what it would do before it does it.

Because both ends are folders on one site, a Sync also runs between two remote systems, say a partner's SFTP server and your cloud storage, with none of your own machines as the middleman.

The upload-then-rename problem is handled for you: a Sync does not expose a partially transferred file as complete.

Automations react to what arrives

A Sync is a schedule. An Automation is a reaction. When a file lands in a folder, a Move Files or Copy Files automation routes it, renames it to your convention, or starts another Sync with Run Sync.

The documented pattern is a partner that deposits invoices on its own SFTP server: Files.com pulls them every ten minutes, and a Move Files automation routes each one to its processing folder.

Turn on the Remote Metadata Index for a mount and Files.com scans it on the interval you set, as often as every few minutes on the higher plans, so files the partner drops on their own server become events your automations trigger on.

The Sync moves the file; the Automation decides what happens next.

Expectations catch the file that never came

Files.com Expectations declare when a file is due in a folder, and when the deadline passes without it, an alert fires. "The downstream system processed yesterday's data again" becomes "the expected file did not arrive by 6:05 a.m.," which is the difference between a successful transfer and a complete business process.

When you still want a command line

Moving the schedule to Files.com does not mean giving up scripting. The Files.com CLI runs anywhere your scripts run and talks to Files.com over HTTPS on port 443 with an API key:

files-cli upload /data/exports/ /partners/brightway/inbound/ --sync

--sync sends only what changed. The CLI runs transfers in parallel, resumes interrupted ones, and retries on its own, so the bash wrapper above shrinks to one line, and the same command targets a native Files.com folder or a folder mounted to a partner's server.

Existing sftp -b scripts keep working too: point them at yourcompany.files.com, upload into the mounted folder, and Files.com proxies the transfer out to the partner over SFTP. The script holds one key for one server that never changes.

A partner whose server sits inside a private network is reachable through the Files.com Agent, which connects outbound with no inbound firewall rule.

Script or platform

A bash or PowerShell script when there is one partner and one straightforward workflow, on a machine that is already monitored, with someone who owns the script and answers its alerts, and no need to detect a file that never arrived.

Files.com when several partners or systems are involved, when files move in both directions, when credentials are spread across machines, when transfers need central retries, logs, and auditing, when someone has to know that a file did not arrive, or when the current setup depends on one person understanding a folder of scripts.

The same decision, written for a specific language, is in the Python, Java, C#, and Node.js walkthroughs.

When the folder of scripts stops being the right answer

Nobody sets out to run a fleet of transfer scripts. It accumulates one partner at a time until it is infrastructure with no owner. The question is not whether shell scripts can move files. They can. The question is how much transfer infrastructure you want to build and operate yourself.

Files.com is the cloud-native File Orchestration Platform: one platform that replaces the stack of legacy tools IT teams run to move files, including the SFTP servers, the scripts, and the schedulers holding it all together. It speaks every protocol, connects 50+ cloud and on-prem systems, automates every transfer, and keeps a complete audit trail.

For the job in this post, that means each partner is a mount, each schedule is a Sync with retries and a log, arrivals trigger Automations, Expectations watch for the file that does not come, and every transfer is recorded in one audit trail.

If the SFTP server has to stay in your own data center, the ExaVault appliance is a free SFTP server for up to 50 users, and Files.com mounts it like any other.

Start with one mount and one Sync, and move the rest of the folder when it makes sense.

Start a free Files.com trial, add a mount, and schedule a Sync. No credit card required, and it is live in minutes.

Frequently asked questions

How do I automate SFTP uploads?

With OpenSSH, write the commands to a batch file and run sftp -b batchfile -i keyfile -o BatchMode=yes -o StrictHostKeyChecking=yes user@host from cron, a systemd timer, or Windows Task Scheduler, uploading to a temporary name and renaming so the partner never sees a partial file. Without a script, a Files.com Sync pushes to the partner's SFTP server on a schedule with retries, logs, and alerts.

How do I run sftp without a password prompt?

Authenticate with an SSH key using -i, and pass -o BatchMode=yes so the client fails instead of prompting. Give the partner your public key and keep the private key readable only by the account that runs the job.

What does sftp -b do?

-b tells the OpenSSH sftp client to read its commands from a batch file instead of an interactive prompt. A failing command aborts the rest of the batch and the client exits nonzero, unless the command is prefixed with -.

How do I keep two SFTP jobs from overlapping?

Take a nonblocking lock with flock -n on a lock file at the top of the script and exit if it is already held. In Files.com, a Sync is one scheduled job that does not overlap itself.

How do I know when a partner's file does not arrive?

A script cannot tell you, because nothing fails when nothing runs. A Files.com Expectation declares when a file is due in a folder and alerts when the deadline passes without it.

Can Files.com connect to my partner's SFTP server for me?

Yes. A Remote Server Mount connects a folder on your Files.com site to any SFTP server in real time. Files.com holds the credentials, can generate the key pair, pins the partner's host key, and passes every operation through. A scheduled Sync moves the files with no code, or the Files.com CLI uploads over HTTPS from any script.