Useful docker commands | Menu | The three rules of asking questions

Correctly working with files and directories

If you work with files and directories, you can do a lot of things wrong. Here's a set of things you should consider when implementing file system routines.

Code samples are given in C#

Don't check if a file exsist before opening it

Checking if a file exists is something everyone does at first, feels right, but is wrong.

Consider this piece of code:

public ConfigData GetConfigOrDefault(string configName)
{
    ArgumentException.ThrowIfNullOrEmpty(configName);
    if (File.Exists(configName))
    {
        return ConfigData.Parse(File.ReadAllText(configName));
    }
    return GetDefaultConfig();
}

This seems like a reasonable function:

However, there are a few problems with this:

The fix for this is simple. Instead of trying to avoid errors by checking if a file exists, you should catch and deal with possible errors correctly:

public ConfigData GetConfigOrDefault(string configName)
{
    ArgumentException.ThrowIfNullOrEmpty(configName);
    try
    {
        return ConfigData.Parse(File.ReadAllText(configName));
    }
    catch (FileNotFoundException) //File not found (Maybe deleted by user)
    {
        return GetDefaultConfig();
    }
    catch (DirectoryNotFoundException) //Path segment not found (Maybe first run)
    {
        return GetDefaultConfig();
    }
    catch (IOException) //Other file system error (for example file in use)
    {
        //TODO: Report error to user, offer retry
        throw;
    }
    catch(InvalidDataException)
    {
        //TODO: Config is corrupt and failed in ConfigData.Parse
        throw;
    }
    //All other errors are thrown as-is
}

The level of error handling is obviously up to you, and not all languages provide the same level of detail in their error handling capabilities.

Checking file size before reading

This is essentially the same as in the chapter above and should be avoided. Between you checking the size and you opening the file, it may get changed.

Open the file and use the provided functions/properties to check the length of the stream/handle. In C# this would be via Stream.Length property.

In some languages you might need to seek to the end of the file for this to work. The size value might be faulty if the file is opened for appending. In ASCII mode, the reported size might be wrong.

Note that this number is only reliable if you also impose a readonly lock on the file.

Trying to be smart when fixing errors

Don't try to be overly smart when you encounter problems with file operations. Sure, trying to find out what application holds a lock on a file you want to edit might be an interesting thing to implement, but most things you will be doing are going to potentially introduce other problems, and are usually not worth the time to implement.

Buffering

Try to avoid writing individual bytes, and instead write them in chunks because system calls are expensive. Do not try to outsmart file system buffers either.

If possible, you should try to write a file in valid chunks, so that when your application dies for some reason, the file you have been writing to is in a valid state, or at least valid enough to be recovered.

For example, for a log file this would mean to construct the entire line in memory and writing it at once.

Seeking

Seeking a file stream will likely destroy the read and write buffer. Any pending writes will still occur properly, but you may experience seeking operations taking a long time if a lot is pending to be written.

Take this into account when you create a custom file format. The biggest problem is alternating between seek and read/write operations.

So instead of <prop> <data> <prop> <data> ... it's better to do <prop> <prop> ... <data> <data> ... or <data> <data> ... <prop> <prop> ...

Either one will allow you to read all properties in one go, and then seek to the data you need instead of alternating between reading properties and seeking over data until you reach the segment you're interested in. This is the reason why opening a zip file feels instant even if it contains hundreds of thousands of entries, where a tar file with the same number of files will take a while to process because the properties and file data is interleaved.

Sparse files

Set the file size if you know in advance how big your file is going to be. Some programming languages have a way to set file sizes. In C#, you can use Stream.SetLength(long). This avoids fragmentation and ensures that there is enough space for your file. These functions usually alow you to trim the end off a file too if you specify a size that's smaller than the current file.

There is no universal guarantee that this operation is fast. Windows for example guarantees that the contents of sparse files consists of nullbytes. The NTFS file system has the ability to declare sparse files and thus this will be very fast. FAT and FAT32 lack this ability. If you try to seek 10 million bytes beyond the file end, Windows will write 10 million nullbytes.

Sync vs Async

If you perform file system operations, there's always a chance that the file system is busy and your call needs to wait. This can be caused by something as simple as a disk needing to spin up.

In general you want to use asynchronous file I/O if it's available. This can either be done with the async/await model, or by using synchronous IO inside of a thread. Threads can be suspended which gives you an easy mechanism to pause and resume operations.

The time to spawn a task is generally less than for a thread, but it's not zero. If you do a large number of async file system operations you will run into the situation, where synchronous calls inside of a thread would be faster than the repeated invocation of the async/await pattern.

File handle lifetime

Don't repeatedly use functions that open a file, write a line and close the file. Instead open the file handle, write all lines, then close it.

Opening files is a quite expensive operation compared to writing to them.

Binary vs. ASCII mode

You almost always want to open the file in binary mode, unless you don't mind that the linebreaks are not necessarily exactly like you wrote them.

Additionally on Windows, if it encounters a CTRL+Z character (byte 0x1A) it will pretend it reached the end of the file.

About CTRL+Z

CTRL+Z being a file content terminator comes from a time where file systems lacked the concept of a file size.

Data is stored in blocks on a file system, and when your data doesn't exactly fits to a block boundary, the remaining data of that block becomes part of it. File systems now have a file size indicator so the operating system knows how many bytes to trim off at the end when reading the data, but that was not always the case.

For binary files, having extra data is generally not a problem, since most binary file types of that era can handle extra data at the end.

Text files however would display that extra data as garbage on the screen, so it was decided that any extra data be filled with byte 0x1A, and that this byte would terminate the output when operating in ASCII mode.

This is the reason why trying wo write a PNG file to the console just prints ëPNG and then aborts, because PNG files have an 0x1A in the header for this exact reason.

Linux uses a different byte than CTRL+Z

File locking

When writing files, it is usually a good idea to acquire an exclusive lock on it. Applications that try to read the file might be confused if the file grows as they're reading it, or the read might leave them with partial data.

In general, this means to do the things below unless your application is prepared to deal with the consequences.

There are legitimate reasons to allow certain actions, for example, allowing other applications to read a file while you write to it is useful for log files.

Copying file properties

When copying a file verbatim, you should copy the modification time too.

Timestamps are as follows (and not available on all file systems and operating systems):

Note that when you copy files, you do not copy the time of creation, only the time of modification.

This means that the creation time of a file can be after the modification time. This is no mistake.

Most web browsers will change the modification time of downloaded files if the server sends a Last-Modified header.

Moving files

In most file systems, moving a file is technically the same as renaming it. This is an atomic operation. Moving a file from one file system to another on the other hand, is not.

Moving a file between file systems involves:

  1. Creating the destination file
  2. Copying the contents from the source to the destination
  3. Setting file properties of the destination
  4. Deleting the source

Thers's a lot that can go wrong here (not exhaustive):

You don't have to deal with every problem individually, but be sure you can correctly clean up a failed move. If step 4 fails, you may offer the user to keep the destination file.

Permissions

Permissions should not be copied from one file to another unless there are good reasons (Backups for example). A file should inherit permissions according to what is defined at the destination.

In general you don't want to change permissions unless you know what you do, or the user explicitly requested copying of the permissions.

Path strings and file names

Not all file systems and operating systems can accomodate the same path and file name strings. Unless really needed, you should avoid these:

There is no general consensus as to how to handle these kind of files. Methods I encountered most often include replacing unsupported characters with _ or a space, and cutting off excessive number of characters if the name is too long.

Custom file extensions

Custom file extensions should generally be avoided. Only use custom extensions if you actually invented a new file format and want your application to open the file when the user double clicks it, otherwise it's probably fine to use a generic extension like .conf, .bin ,etc.

Custom extensions are no security measure. Tools that can identify the real file type based on the contents exist. (Example)

File system structure

If you recursively search through a file system, you should ignore links and junctions. These can be used to construct cyclic structures or create duplicates.

Unless you know how to handle these problems, it's best to ignore anything other than real files and directories.

Many files

Every entry in a directory is going to slightly slow down some operations. For example, every time the system has to check for name conflicts, or if you want to list the directory contents and insist on the result being sorted.

If you can avoid it, don't put too many files into a single directory. Good strategies to avoid this are to combine small files into one, or to create more directory levels. You can for example use the SHA1 function to create file names, and then sort them into two levels of directories, so instead of test\FC35B10C78D44D4A3E61A5B3326CCC33B7189087 you do test\F\C\FC35B10C78D44D4A3E61A5B3326CCC33B7189087

If your application is likely to create a lot of files, consider storing them in an SQLite database instead, as it will likely be faster than individual files.

Files as temporary data exchange

Files are not the ideal mechanism to pass data from one application to another. For that, you should use another method. Methods available on pretty much all systems include piping data via command line, network sockets, and pipe streams.

Files as scratch space

Do not create temporary files for the sole purpose of passing data from one library to another, unless you want to pass so much data that you risk running out of ram. Use stream based APIs and something like System.IO.MemoryStream instead. Some functions in libraries that accept file names as arguments will have an alternative version that accept stream resources/handles as arguments. You usually want to use those functions to pass data around and avoid using the file system as a temporary buffer.

Temporary file location

Temporary files should only be created below the users temp directory. To avoid conflicts, your application should not create files directly in the temp directory, but create its own folder where files are kept, or use system provided functions that generate unique file names.

Finally, make no assumption that a file is still in the temp directory between two launches of your application.

Ephemeral files

It's possible to create ephemeral files which automatically get cleaned up by the system, even if your application or system crashes and is not running any exit handler.

Here's a library I wrote to handle this in .NET

The basic trick is to delete the file after it has been opened. The tricky part is undoing this in a transparent manner.

/ vs \

It's no secret that Windows uses \ while most other systems use / as directory separator. While support for / has increased in Windows, it's still not the official separator and some functions will fail if you use it. If your language is portable across systems, it should provide functions or constants with the proper slash, or it might silently replace / with \ for you in its file system functions.

In general, do not use string concatenation functions to build path strings. Always use the file system functions provided by your language.