Goodbye os.path: 15 Pathlib Tricks to Quickly Master The File System in Python | by Bex T. | Apr, 2023

No complications and unreadable code from os.path

Pathlib could also be my favourite library (after Sklearn, clearly). And given there are over 130 thousand libraries, that’s saying one thing. Pathlib helps me flip code like this written in os.path:

import os
dir_path = "/dwelling/consumer/paperwork"
# Discover all textual content information inside a listing
information = [os.path.join(dir_path, f) for f in os.listdir(dir_path) 
if os.path.isfile(os.path.join(dir_path, f)) and f.endswith(".txt")]

into this:

from pathlib import Path
# Discover all textual content information inside a listing
information = listing(dir_path.glob("*.txt"))

Pathlib got here out in Python 3.4 as a substitute for the nightmare that was os.path. It additionally marked an vital milestone for Python language on the entire: they lastly turned each single factor into an object (even nothing).

The most important disadvantage of os.path was treating system paths as strings, which led to unreadable, messy code and a steep studying curve.

By representing paths as fully-fledged objects, Pathlib solves all these points and introduces class, consistency, and a breath of recent air into path dealing with.

And this long-overdue article of mine will define among the greatest features/options and methods of pathlib to carry out duties that will have been actually horrible experiences in os.path.

Studying these options of Pathlib will make all the pieces associated to paths and information simpler for you as a knowledge skilled, particularly throughout knowledge processing workflows the place you must transfer round hundreds of photos, CSVs, or audio information.

Let’s get began!

Working with paths

1. Creating paths

Nearly all options of pathlib is accessible by way of its Path class, which you should utilize to create paths to information and directories.

There are just a few methods you possibly can create paths with Path. First, there are class strategies like cwd and dwelling for the present working and the house consumer directories:

from pathlib import Path
Path.cwd()

PosixPath('/dwelling/bexgboost/articles/2023/4_april/1_pathlib')

Path.dwelling()

PosixPath('/dwelling/bexgboost')

You may as well create paths from string paths:

p = Path("paperwork")
p

PosixPath('paperwork')

Becoming a member of paths is a breeze in Pathlib with the ahead slash operator:

data_dir = Path(".") / "knowledge"
csv_file = data_dir / "file.csv"
print(data_dir)
print(csv_file)

knowledge
knowledge/file.csv

Please, do not let anybody ever catch you utilizing os.path.be part of after this.

To test whether or not a path, you should utilize the boolean perform exists:

data_dir.exists()

True

csv_file.exists()

True

Typically, your complete Path object received’t be seen, and you must test whether or not it’s a listing or a file. So, you should utilize is_dir or is_file features to do it:

data_dir.is_dir()

True

csv_file.is_file()

True

Most paths you’re employed with can be relative to your present listing. However, there are instances the place you must present the precise location of a file or a listing to make it accessible from any Python script. That is while you use absolute paths:

csv_file.absolute()

PosixPath('/dwelling/bexgboost/articles/2023/4_april/1_pathlib/knowledge/file.csv')

Lastly, when you’ve got the misfortune of working with libraries that also require string paths, you possibly can name str(path):

str(Path.dwelling())

'/dwelling/bexgboost'

Most libraries within the knowledge stack have lengthy supported Path objects, together with sklearn, pandas, matplotlib, seaborn, and so on.

2. Path attributes

Path objects have many helpful attributes. Let’s see some examples utilizing this path object that factors to a picture file.

image_file = Path("photos/midjourney.png").absolute()
image_file

PosixPath('/dwelling/bexgboost/articles/2023/4_april/1_pathlib/photos/midjourney.png')

Let’s begin with the father or mother. It returns a path object that’s one stage up the present working listing.

image_file.father or mother

PosixPath('/dwelling/bexgboost/articles/2023/4_april/1_pathlib/photos')

Typically, it’s your decision solely the file identify as an alternative of the entire path. There may be an attribute for that:

image_file.identify

'midjourney.png'

which returns solely the file identify with the extension.

There may be additionally stem for the file identify with out the suffix:

image_file.stem

'midjourney'

Or the suffix itself with the dot for the file extension:

image_file.suffix

'.png'

If you wish to divide a path into its parts, you should utilize elements as an alternative of str.cut up('/'):

image_file.elements

('/',
'dwelling',
'bexgboost',
'articles',
'2023',
'4_april',
'1_pathlib',
'photos',
'midjourney.png')

If you need these parts to be Path objects in themselves, you should utilize mother and father attribute, which creates a generator:

for i in image_file.mother and father:
print(i)

/dwelling/bexgboost/articles/2023/4_april/1_pathlib/photos
/dwelling/bexgboost/articles/2023/4_april/1_pathlib
/dwelling/bexgboost/articles/2023/4_april
/dwelling/bexgboost/articles/2023
/dwelling/bexgboost/articles
/dwelling/bexgboost
/dwelling
/

Working with information

To create information and write to them, you do not have to make use of open perform anymore. Simply create a Path object and write_text or write_btyes to them:

markdown = data_dir / "file.md"
# Create (override) and write textual content
markdown.write_text("# It is a take a look at markdown")

Or, if you have already got a file, you possibly can read_text or read_bytes:

markdown.read_text()

'# It is a take a look at markdown'

len(image_file.read_bytes())

1962148

Nevertheless, word that write_text or write_bytes overrides present contents of a file.

# Write new textual content to present file
markdown.write_text("## It is a new line")

# The file is overridden
markdown.read_text()

'## It is a new line'

To append new data to present information, it’s best to use open methodology of Path objects in a (append) mode:

# Append textual content
with markdown.open(mode="a") as file:
file.write("n### That is the second line")
markdown.read_text()

'## It is a new linen### That is the second line'

It is usually widespread to rename information. rename methodology accepts the vacation spot path for the renamed file.

To create the vacation spot path within the present listing, i. e. rename the file, you should utilize with_stem on the present path, which replaces the stem of the unique file:

renamed_md = markdown.with_stem("new_markdown")
markdown.rename(renamed_md)

PosixPath('knowledge/new_markdown.md')

Above, file.md is was new_markdown.md.

Let’s examine the file measurement by way of stat().st_size:

# Show file measurement
renamed_md.stat().st_size

49 # in bytes

or the final time the file was modified, which was just a few seconds in the past:

from datetime import datetime
modified_timestamp = renamed_md.stat().st_mtime
datetime.fromtimestamp(modified_timestamp)

datetime.datetime(2023, 4, 3, 13, 32, 45, 542693)

st_mtime returns a timestamp, which is the depend of seconds since January 1, 1970. To make it readable, you should utilize use the fromtimestamp perform of datatime.

To take away undesirable information, you possibly can unlink them:

renamed_md.unlink(missing_ok=True)

Setting missing_ok to True will not elevate any alarms if the file would not exist.

Working with directories

There are just a few neat methods to work with directories in Pathlib. First, let’s have a look at the right way to create directories recursively.

new_dir = (
Path.cwd()
/ "new_dir"
/ "child_dir"
/ "grandchild_dir"
)
new_dir.exists()

False

The new_dir would not exist, so let’s create it with all its kids:

new_dir.mkdir(mother and father=True, exist_ok=True)

By default, mkdir creates the final baby of the given path. If the intermediate mother and father do not exist, you must set mother and father to True.

To take away empty directories, you should utilize rmdir. If the given path object is nested, solely the final baby listing is deleted:

# Removes the final baby listing
new_dir.rmdir()

To listing the contents of a listing like ls on the terminal, you should utilize iterdir. Once more, the outcome can be a generator object, yielding listing contents as separate path objects one by one:

for p in Path.dwelling().iterdir():
print(p)

/dwelling/bexgboost/.python_history
/dwelling/bexgboost/word_counter.py
/dwelling/bexgboost/.azure
/dwelling/bexgboost/.npm
/dwelling/bexgboost/.nv
/dwelling/bexgboost/.julia
...

To seize all information with a selected extension or a reputation sample, you should utilize the glob perform with a daily expression.

For instance, under, we’ll discover all textual content information inside my dwelling listing with glob("*.txt"):

dwelling = Path.dwelling()
text_files = listing(dwelling.glob("*.txt"))
len(text_files)

3 # Solely three

To seek for textual content information recursively, which means inside all baby directories as effectively, you should utilize recursive glob with rglob:

all_text_files = [p for p in home.rglob("*.txt")]
len(all_text_files)

5116 # Now way more

Study common expressions here.

You may as well use rglob('*') to listing listing contents recursively. It’s just like the supercharged model of iterdir().

One of many use instances of that is counting the variety of file codecs that seem inside a listing.

To do that, we import the Counter class from collections and supply all file suffixes to it inside the articles folder of dwelling:

from collections import Counter
file_counts = Counter(
path.suffix for path in (dwelling / "articles").rglob("*")
)
file_counts

Counter({'.py': 12,
'': 1293,
'.md': 1,
'.txt': 7,
'.ipynb': 222,
'.png': 90,
'.mp4': 39})

Working system variations

Sorry, however we now have to speak about this nightmare of a problem.

Up till now, we now have been coping with PosixPath objects, that are the default for UNIX-like methods:

sort(Path.dwelling())

pathlib.PosixPath

When you have been on Home windows, you’ll get a WindowsPath object:

from pathlib import WindowsPath
# Consumer uncooked strings that begin with r to write down home windows paths
path = WindowsPath(r"C:customers")
path

NotImplementedError: can not instantiate 'WindowsPath' in your system

Instantiating one other system’s path raises an error just like the above.

However what should you have been pressured to work with paths from one other system, like code written by coworkers who use Home windows?

As an answer, pathlib affords pure path objects like PureWindowsPath or PurePosixPath:

from pathlib import PurePosixPath, PureWindowsPath
path = PureWindowsPath(r"C:customers")
path

PureWindowsPath('C:/customers')

These are primitive path objects. You have entry to some path strategies and attributes, however basically, the trail object stays a string:

path / "bexgboost"

PureWindowsPath('C:/customers/bexgboost')

path.father or mother

PureWindowsPath('C:/')

path.stem

'customers'

path.rename(r"C:losers") # Unsupported

AttributeError: 'PureWindowsPath' object has no attribute 'rename'

Conclusion

When you have observed, I lied within the title of the article. As a substitute of 15, I consider the depend of recent methods and features was 30ish.

I did not need to scare you off.

However I hope I’ve satisfied you adequate to ditch os.path and begin utilizing pathlib for a lot simpler and extra readable path operations.

Forge a brand new path, if you’ll 🙂

When you loved this text and, let’s face it, its weird writing model, think about supporting me by signing as much as grow to be a Medium member. Membership prices 4.99$ a month and provides you limitless entry to all my tales and tons of of hundreds of articles written by extra skilled folks. When you join by way of this link, I’ll earn a small fee with no additional value to your pocket.