Cloud Backup Is Not Necessarily Safe: Test Restore Before Data Is Really Needed

Many teams feel secure after seeing backup files successfully enter a cloud bucket or backup server. However, the size of a backup is not just about "data stored," but rather "data that can be used again when needed." This difference becomes apparent when the database is corrupted, the admin account is compromised, or someone accidentally deletes an important folder.

A backup that has never been tested is like an emergency exit that has never been opened. It exists, but it may not be usable when the situation worsens.

Backup, versioning, and replication are not the same thing

Several cloud terms are often used as if they have the same function, but their protections differ.

• Backup is a copy of data stored so that it can be restored at a specific point in time.
• Versioning saves old versions when an object is overwritten or deleted. This is useful for reverting a file to a previous version.
• Replication copies data to another location, such as a different region or bucket, to help withstand infrastructure disruptions.
• Immutability means data is made difficult to change or delete for a certain period.

Versioning can help restore overwritten files, but it does not automatically protect the entire bucket from deletion. Google Cloud documentation, for example, explains that Object Versioning does not protect data if the entire bucket is deleted. AWS also distinguishes between restoring old versions and protection using Object Lock. ([docs.cloud.google.com](https://docs.cloud.google.com/storage/docs/object-versioning?utm_source=openai))

The biggest risks often come from overly broad access

Ransomware attacks are not the only threat. Command errors, leaked credentials, compromised admin accounts, and incorrect lifecycle rules can also lead to mass data deletion.

The problem becomes more serious when the backup system uses the same accounts and credentials as the production system. If an attacker gains access to these, they may be able to delete both the primary data and its copy.

Therefore, backups should be treated as a separate zone. Use dedicated accounts or roles, limit write and delete permissions, enable layered authentication if available, and avoid granting full admin rights to applications that only need to upload files.

Use layered protection, not a single feature

There is no single button that can solve all backup problems. A more sensible approach is to combine several layers of protection.

1. Primary copy: data used by daily applications.
2. Quick recovery copy: backup in easily accessible storage to address user errors or application failures.
3. Isolated copy: data in different accounts, buckets, regions, or providers that cannot be deleted with production credentials.
4. Retention and immutability: rules that prevent data from being deleted before the protection period ends.

In object storage services, features like soft delete, retention policy, bucket lock, or Object Lock can help prevent data from being deleted directly. However, such features need to be carefully designed because overly long retention can increase costs, while overly short retention may not be sufficient to handle incidents that are only discovered after several days.

Google Cloud notes that Object Versioning can increase storage costs and usually needs to be paired with lifecycle management. AWS explains that Object Lock can prevent objects from being deleted or overwritten for a certain period, but this configuration has consequences that are not always easy to reverse. ([docs.cloud.google.com](https://docs.cloud.google.com/storage/docs/control-data-lifecycles?hl=en&utm_source=openai))

Test restore with small scenarios first

Testing restore does not have to be an immediate recovery of the entire server. Start with small, measurable scenarios.

1. Choose one important file, one folder, and one database dump.
2. Restore everything to a separate environment, not directly overwriting production data.
3. Check if the file can be opened, checksum matches, and access permissions have not changed unintentionally.
4. For databases, run a test application and ensure tables, indexes, relationships, and transaction data are readable.
5. Record the start time, end time, failed steps, and who is responsible.

The test results should answer two practical questions: how long can the system take to get back up and running? and how much data might be lost? In recovery terms, the first question relates to Recovery Time Objective or RTO, while the second question relates to Recovery Point Objective or RPO.

Don’t just test files—test how to find them too

Restores can fail not because the data is missing, but because no one knows which backup is correct. File names like backup-final-2-correct.sql may seem sufficient when created, but can be confusing months later.

Use consistent naming, clear timestamps, and brief notes about the contents of each backup. Also, store important information such as application versions, database versions, network configurations, environment variables, and the location of secrets needed to restart services.

If backups are encrypted, ensure that the recovery keys are not only stored on the server being backed up. A backup without a key is data that technically exists but is practically unusable.

Checklist to do now

• Ensure backups have a clear schedule and retention policy.
• Use dedicated backup credentials, not production admin accounts.
• Keep at least one copy that is not easily deleted from the production account.
• Enable versioning or soft delete as needed, then calculate the cost impact.
• Document the restore process in steps that others can follow.
• Conduct small restore tests every month or whenever there are significant architectural changes.
• Test scenarios of credential loss, bucket deletion, and database corruption.

What does this mean for us?

A good backup is not the most expensive or the one with the most features. A good backup is one that is separate from the main system, has a reasonable protection period, and has been proven to be recoverable.

Start with the one service that is most critical to the business. Restore a small copy, measure the time, document the issues, and then fix them. Only then should you expand testing to the entire system. This way, backups shift from being mere passive archives to a real part of the recovery plan.

Sources & further reading

• Google Cloud Documentation — Object Versioning
• Google Cloud Documentation — Options for controlling data lifecycles
• Amazon S3 Documentation — Restoring previous versions
• Amazon S3 Documentation — Locking objects with Object Lock

– Rio Yotto @rioyotto