Linux Fundamentals for Data Engineering

When you are starting out in Data Engineering, it is easy to focus entirely on writing pristine Python code, designing SQL schemas, or learning complex tools like Apache Spark. But sooner rather than later, you’ll realize that data engineering isn't done in a vacuum. Most of our day-to-day work involves wrestling with data infrastructure.

We are constantly moving files around, orchestrating automated tasks and debugging code running inside containers.
At the heart of almost all modern data infrastructure is Linux.

For many beginners, stepping away from a visual desktop and into the Linux terminal can feel like stepping into a sci-fi movie. A blinking cursor on a blank screen isn't exactly welcoming.

But here is the secret: the command line isn't harder than a graphical interface; it’s just more direct. Instead of hunting through layers of menus to click a button, you simply tell the computer exactly what you want it to do.

This guide will walk you through the essential Linux commands you need to navigate, manage files, and feel right at home in the terminal.

1. Navigating the File System

When you open a terminal, you are always "inside" a specific folder (called a directory). Think of navigation commands as your digital map and compass.

pwd (Print Working Directory)
If you ever get lost, type pwd and hit Enter. This tells you exactly where you are in the system's folder hierarchy.
E.g
pwd
The output will be the equivalent of
Output: /home/username/Documents

ls (List)
To see what files and folders are inside your current directory, use ls.

Pro Tip: You can add flags (extra options) to modify how commands work.

ls -l shows a detailed list with file sizes and permissions.

ls -a reveals hidden files (files starting with a dot, like .bashrc).
E.g ls -l

cd (Change Directory)
This is how you move between folders.

To move into a folder named "Downloads": cd Downloads

To go back up one level (to the parent folder): cd ..

To jump straight back to your personal home folder: just type cd and hit Enter.
E.g cd ..

2. Managing Files and Folders

Now that you can move around, let's look at how to create, copy, move, and destroy files.

mkdir (Make Directory)
Need a new folder? mkdir followed by the folder name will create it instantly.

E.g: mkdir Projects

touch
The quickest way to create a brand-new, empty file.

E.g: touch notes.txt

cp (Copy)
To copy a file, specify the original file first, and then the name/location of the duplicate.

E.g:cp notes.txt backup_notes.txt

Note: To copy an entire folder, you need to use the recursive flag: cp -r folder1 folder2.

mv (Move / Rename)
The mv command does double duty. It can move a file to a different folder, or rename the file if you keep it in the same place.

To move: mv notes.txt Documents/

To rename: mv notes.txt journal.txt

rm (Remove)
This deletes files. Be careful: Linux doesn't have a "Trash Bin" for the command line. Once you use rm, the file is permanently gone.

To delete a file: rm notes.txt

To delete a folder and everything inside it: rm -r Projects

3. Viewing and Inspecting Files

You don’t always need to open a heavy text editor just to see what’s inside a file.

cat (Concatenate)
This prints the entire contents of a file directly onto your screen. It’s great for short files.

E.g: cat notes.txt

less
If a file is hundreds of lines long, cat will flood your screen. less opens the file in a scrollable view. Use the arrow keys to read through it, and press q to quit and go back to the command prompt.

E.g: less large_log_file.txt

4. The Linux "Superpowers"

As you get comfortable, these three tools will make you feel like a true power user.

sudo (SuperUser Do)
Some actions—like installing software or changing system settings—require administrator privileges. Prepend sudo to a command to run it as the administrator (root user). It will ask for your password.

E.g: sudo apt update

grep (Global Regular Expression Print)
grep is a powerful search tool. It searches through files to find specific words or phrases.

E.g: grep "password" notes.txt (This will print every line in notes.txt that contains the word "password").

The Pipe (|)The pipe key (usually above the Enter key) allows you to send the output of one command directly into another command as an input.

E.g: ls -l | grep "June"(This lists all files, but immediately filters the results to only show files modified in June).

---

QUICK REFERENCE SUMMARY

Click here for Quick Reference:
(https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jjqi8q2iwwuktpe3kf3j.png)

A Final Piece of Advice
Don't worry about memorizing all of these right away. Keep a cheat sheet handy, open up your terminal, and practice making a few folders and files. Within a week, typing these commands will feel as natural as moving a mouse.

If you ever get stuck on how to use a specific command, just type man (short for manual) followed by the command name—for example, man ls—to read its official instruction guide!