Updated on July 7, 2026 in #linux (https://nickjanetakis.com/blog/tag/linux-tips-tricks-and-tutorials) There's a number of subtle things going on when you redirect output to a file, here's how to do it safely and verify it with strace. • *Quick Jump:** • * Showing the Problem (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#showing-the-problem) • Using strace to Pry Deeper (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#using-strace-to-pry-deeper) • Writing Files Safely (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#writing-files-safely) • Demo Video (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#demo-video) • *Prefer video? Here it is on YouTube (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#demo-video).** I’ve written 10,000+ lines of shell scripts over the years and I can’t believe I only learned about this recently. Let’s say you have a normal shell script. I pretty much always set these options to protect myself. They’ll make your script exit if any commands you run encounter an error, if a pipeline fails or if a referenced variable is unset: Easy right? The expected output is “Hello” which is what we get. What about echo "Hello" > results? Also easy, it will create a new results file in the directory where you ran this script with the contents of “Hello”. This is shell syntax to redirect output to a file. # (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#showing-the-problem) Showing the Problem Without looking it up or trying it, take a guess what happens if you put this line at the bottom of the script. Don’t just run it in your terminal, since the same options aren’t set: To give you a hint, the awk command is just sleeping for 5 seconds. We’re using that instead of sleep 5 because the sleep command doesn’t accept STDIN where as awk does. I don’t know about you but initially I thought this would happen: • “Hello” would be printed to STDOUT • It would get piped to awk which accepts STDIN • awk would sleep for 5 seconds and print “Hello” to STDOUT • A results file would be created with the contents “Hello” The above mental model is how most pipelines are read. It flows from left to right. What Really Happens • The file descriptor for the results file is created immediately and a 0 byte file is created • You can verify this by running the command and catting the file in terminal #2 • echo prints “Hello” and pipes it into awk • awk receives this input and begins executing its system command • awk sleeps for 5 seconds • awk prints “Hello” to STDOUT which is already connected to the open results file, this yields our expected text in the file The important takeaway is as soon as this pipeline starts to run the file you’re redirecting to can be considered truncated to a 0 byte file and there’s 3 possible states this file can be in after the script is called: • All commands finish successfully and it’s what you expect • One of the commands fails and the scripts exits with a partially written file • The script exits before processing has occurred (power outage) and the file is empty What If You Use Tee Instead? If you’re comfortable writing shell scripts, you might think about trying: This has the same problem as > since the results file will get immediately created. Technically when you run the above, echo, awk and tee are being run in parallel. Then tee is like “oh, I need to write to the results file, let me open a file descriptor for that”. At this point awk can start streaming data into the file at its discretion. Why This Is a Problem It comes down to potential unexpected data loss. Here’s 1 of many examples: Imagine you’re doing something like a database backup. It’s not unreasonable to think you’d run your database dump command (pg_dumpall, etc.), maybe pipe it into gzip and then redirect it to a file. If you’re keeping 1 day’s worth of backups by overwriting the existing file, if your current day’s backup command fails to run for some reasons, you could end up with a partial backup that overwrote yesterday’s valid backup. In a disaster scenario where you have to restore, you suddenly have data loss. # (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#using-strace-to-pry-deeper) Using strace to Pry Deeper The more I use native Linux on the desktop, the more I find myself interested in lower level details. I wish I had unlimited time: If you run the above with either version you’ll see this line early on which demonstrates a truncated 0 byte file is being created: Also, while no timestamps are shown, when you run the above command with the tee version you’ll see something like this pop out at about the same time: This demonstrates all parts of the pipeline are being running in parallel. Your shell forked them, that’s also why we set --follow-forks with strace, we want to be informed of what each command is doing. # (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#writing-files-safely) Writing Files Safely What I like to do is write the output to a temporary file and in a separate command use mv to move that to the final destination: As long as set -o errexit and set -o pipefail are set up top we can be confident our pipeline finished successfully by the time mv is called. As long as the temp file and the real destination file exist on the same filesystem then this move command will be atomic. The easiest way to be sure of that is putting it in the same directory as the real file. For example if you have 2 partitions or separate disks, mv will copy the file instead of doing an atomic rename. This is why I avoid /tmp because sometimes that can be mounted on a different filesystem (often tmpfs, etc.). You can verify an atomic rename with strace with this 1 liner: Here’s an example from my machine where /tmp is mounted in RAM (different filesystem): The above makes sense because it didn’t do a renameat2 call. If you wanted to see some of the calls taken for a cross device copy, here you go: In the above case it wrote a 5 byte file (“cool\n”) because a 0 byte file from touch wouldn’t get written in the same way a file with > 0 bytes would be. Considerations The upside here is it lets your pipeline operate normally. Data will be streamed into the file without needing to hold everything in memory. The amount of memory used will depend on the tools you’re using to process your data. The downside or consideration is you temporarily need double the storage in our database backup example. You need the old / original backup and the new temp file backup to exist in parallel for a short period of time. Personally this redirect to temp file + mv combo is what I use in most cases. Funny enough I’ve been using this approach to backup databases for a long time. I was recently doing client work and revisited some backup and restore scripts and wanted to justify my thought process so I did a deeper dive. It’s always fun when you accidentally land on a safe solution based on intuition. With backups, you could of course consider using S3 instead (or another object store) and that’s good too. This example focuses on backing up to a mounted drive. This drive could exist on a different filesystem than your database but the mv is still an atomic rename since that’s related to the temporary backup vs real backup file, not your database itself. The demo video below shows running these commands. # (https://nickjanetakis.com/blog/watch-out-for-data-loss-when-redirecting-to-a-file-in-a-shell-script#demo-video) Demo Video Video 3 (https://www.youtube.com/watch?v=V2AX6aUBXtM) Timestamps • 0:24 – Redirecting files unsafely • 2:12 – Potential data loss • 3:05 – What about using tee? • 5:04 – Verifying it with strace • 6:55 – Redirecting files safely • 9:48 – Verifying atomic renames with mv • 10:55 – Considerations of this temp file + mv approach • *How do you safely redirect files? Let me know below.** Like you, I'm super protective of my inbox, so don't worry about getting spammed. You can expect a few emails per year (at most), and you can 1-click unsubscribe at any time. See what else you'll get (https://nickjanetakis.com/newsletter) too.