File Backups with Yandex Disk & Rclone

I’ve been using Yandex Disk as my cloud file storage because it is cheap. As of right now, they’re offering 1TB for $1.90 a month. I think I paid around $12 for 1 year of 1TB storage.

That’s about all it has going for it. The clients are… ok. It’s missing many features of mega.nz which I liked and it has no official GNU/Linux GUI client. I don’t trust them with personal info, in fact I don’t trust any cloud service with really personal info besides iCloud.

Luckily, we can backup to it with rclone and encrypt private info on the fly. We won’t be able to view encrypted data on the web, but that’s a tradeoff as again, this is for backups.

There are many rclone tutorials out there so I won’t be covering how to install or configure other than to tell you what my setup is.

I have yandex configured as a remote, and then a crypt which points to the remote. I then have a systemd “yandex-media-sync.timer” which will run the backup script every so often.

I was having issues with random files not syncing and 404s/409s. I think I’ve finally figured out that the problem is not characters in filenames but rather long filenames which are extended further when encrypted.

“The crypt remote (with default filename_encryption = standard and filename_encoding = base32) encrypts filenames to protect them, but this encoding expands the length significantly…”

“Base32 encoding has an expansion factor of about 1.6x (every 5 bytes of input become 8 characters of output). Your filename is 198 characters long (all ASCII, so 198 bytes). After encryption and base32, it becomes approximately 320 characters long (ceil(198/5) * 8 = 320).”

“Yandex Disk has a filename length limit of 255 characters (or bytes, depending on encoding—it’s a common filesystem-style restriction).”

You can use a simple script to find and trim the names of these files. I trimmed to around 130 characters.

grayson@libre  ~/.local/bin  cat charlimit.sh
#!/bin/bash

# Default to interactive mode
interactive=1
limit=0

# Parse arguments
while [[ $# -gt 0 ]]; do
case $1 in
-n)
if [[ $2 =~ ^[0-9]+$ ]]; then
limit=$2
else
echo "Error: -n must be followed by a positive integer."
exit 1
fi
shift 2
;;
--no-interactive)
interactive=0
shift
;;
*)
echo "Unknown option: $1"
echo "Usage: $0 -n <limit> [--no-interactive]"
exit 1
;;
esac
done

if [ $limit -eq 0 ]; then
echo "Error: Must provide -n <limit> where <limit> is a positive integer."
exit 1
fi

# Find all files recursively
files=()
while IFS= read -r -d '' file; do
files+=("$file")
done < <(find . -type f -print0)

# Arrays to store files to change and their new paths
to_change=()
new_names=()

for file in "${files[@]}"; do
base=$(basename "$file")
if [ ${#base} -gt "$limit" ]; then
dir=$(dirname "$file")
if [[ $base == *.* ]]; then
ext="${base##*.}"
name="${base%.*}"
max_len=$((limit - ${#ext} - 1))
if [ $max_len -lt 1 ]; then
continue  # Skip if can't trim reasonably
fi
new_base="${name:0:$max_len}.${ext}"
else
new_base="${base:0:$limit}"
fi
new_path="$dir/$new_base"
to_change+=("$file")
new_names+=("$new_path")
fi
done

count=${#to_change[@]}

if [ $count -eq 0 ]; then
echo "No files found with filename length over $limit characters."
exit 0
fi

if [ $interactive -eq 1 ]; then
# List potential changes
for i in "${!to_change[@]}"; do
old_base=$(basename "${to_change[i]}")
new_base=$(basename "${new_names[i]}")
echo "$old_base > $new_base"
done
echo "$count files would be changed."
read -p "Confirm changes? (y/n) " confirm
if [[ ! $confirm =~ ^[yY]$ ]]; then
echo "Aborted."
exit 0
fi
fi

# Perform renames
for i in "${!to_change[@]}"; do
if [ -e "${new_names[i]}" ]; then
echo "Skipping $(basename "${to_change[i]}") because $(basename "${new_names[i]}") already exists."
continue
fi
mv "${to_change[i]}" "${new_names[i]}"
if [ $interactive -eq 0 ]; then
old_base=$(basename "${to_change[i]}")
new_base=$(basename "${new_names[i]}")
echo "$old_base > $new_base"
fi
done

cat ~/.config/systemd/user/yandex-media-sync.service
[Unit]
Description=Sync files and folders to Yandex Disk with rclone
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/grayson/.local/bin/yandex-sync.sh

[Install]
WantedBy=default.target

grayson@libre  ~/Documents  cat ~/.local/bin/yandex-sync.sh
#!/bin/bash

# Set up environment for notifications
export DISPLAY=:0
export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus"

RCLONE_OPTS=(
--stats=1m
-v
--create-empty-src-dirs
--fast-list              # Use faster recursive listing
--ignore-errors          # Don't stop on delete errors
--delete-during    # Delete files as it syncs, not after
)

BACKUPS=(
"$HOME/Documents|yandexremotecryptcompress:documents2"
"$HOME/Music|yandexremote:music2"
"$HOME/Pictures|yandexremote:pictures2"
"$HOME/Videos|yandexremote:videos2"
"$HOME/.config|yandexremote:.config"
"$HOME/.ssh|yandexremote:.ssh"

)

# Internet wait settings
timeout=60   # total seconds to wait
interval=5   # seconds between checks
elapsed=0

notify-send -a Rclone -h "string:desktop-entry:org.kde.konsole" "Starting Yandex backup..."

# Wait for internet (DNS check)
while ! getent hosts google.com &>/dev/null; do
sleep "$interval"
elapsed=$((elapsed + interval))
if [ "$elapsed" -ge "$timeout" ]; then
notify-send -a Rclone -h "string:desktop-entry:org.kde.konsole" "Yandex backup failed" "No internet connection after $timeout seconds."
echo "Error: No internet connection after $timeout seconds. Aborting sync."
exit 1
fi
done

for entry in "${BACKUPS[@]}"; do
IFS='|' read -r src dest <<< "$entry"

name="$(basename "$src")"
log="/tmp/rclone-${name,,}.log"

echo "Syncing $name..."
/usr/bin/rclone sync "${RCLONE_OPTS[@]}" "$src" "$dest" \
2>&1 | tee "$log"

STATS=$(tail -n 20 "$log" |
grep -i "Transferred:\|Errors:\|Checks:\|Elapsed time" |
tr "\n" " ")

if [ -n "$STATS" ]; then
notify-send -a Rclone \
-h "string:desktop-entry:org.kde.konsole" \
"$name synced" "$STATS"
fi
done

notify-send -a Rclone -h "string:desktop-entry:org.kde.konsole" "Sync attempt finished."

There is a little nasty gotcha with rclone and trying to sync individual files. You can’t run sync like ~/.config/file1.txt and ~/.config/file2.txt as separate commands, the path overlap will cause the first ones files to be removed. Alternatives include rclone copy or syncing the entire directory and adding excludes for the files you don’t want. Because of this, and rclone’s nature of sending an entire file if there’s any small change to it, forces us to consider rclone as purely a backup/remote mounting solution instead of day-to-day software to keep files between machines the same.

Unfortunately, files can get stuck. If they are removed during a transfer(?) or if their filenames are too long, sometimes you end up with rclone not being able to delete files on the remote. In this case, files in this “Monotype Fonts” folder were acting up, so I ran purge to fix this issue.

rclone lsd "yandexremotecryptcompress:documents2/Random/"
rclone purge "yandexremotecryptcompress:documents2/Random/Monotype Fonts"

That’s about it. I want “directional” syncing, so if files are deleted in Yandex, I don’t want them deleted off my laptop. I became aware and concerned of this with Mega and how much of a risk it is if something goes wrong and your account gets terminated or something.

Oh, I did play with yandex-cli which is the only official option on Linux. I found that it wouldn’t resume after device sleep properly, I’m sure it could be mitigated but I didn’t have a ton of success with it. In case you do use it, and in tandem with the above rclone stuff, I recommend excluding all the same directories from being downloaded locally as this would cause duplication.

cat ~/.config/yandex-disk/config.cfg
auth="/home/grayson/.config/yandex-disk/passwd"
dir="/home/grayson/Yandex.Disk"
proxy="no"
exclude-dirs=Music,Pictures,encrypted,Thinkpad,Shared

In this case, it’s useful to move a file from one device to another by dragging it into the root of yandex disk instead of one of the directories that serves as backup.