But here's the thing: file extensions are just part of the filename. They're a label, not a guarantee.

Consider this scenario:

malware.exe  →  renamed  →  totally-safe-image.png

The filename changed. The bytes inside the file? Not a single one. If your system relied purely on the extension to decide how to handle that file, it just got fooled. This is a real technique used to sneak malicious files past unsuspecting users.

So how do operating systems and tools actually detect the real file type? They don't look at the name at all — they look inside the file itself.

Magic Numbers: The Fingerprints of Files

Every common file format has a sequence of bytes baked into it that acts as its identity. These bytes — usually sitting right at the start of the file — are called file signatures, or magic numbers.

When a program wants to know what a file really is, it opens the file and reads those first few bytes, then compares them against a table of known signatures.

Take a PNG image. No matter what you name it, the first four bytes will always be:

89 50 4E 47

The ASCII representation for that is ‰PNG.

The same is true for PDFs (%PDF-), SQLite databases (literally SQLite format 3), and dozens of other formats.

In our Go detector, we represent each of these as a struct:

type FileMagicNumberMapping struct {
    Name        string
    Signature   []byte
    Offset      int
    CustomCheck func(filePath string) bool
}

And the known signatures are defined in a lookup table:

var FileType = []FileMagicNumberMapping{
    {"PNG Image",        []byte{0x89, 0x50, 0x4E, 0x47},             0, nil},
    {"PDF Document",     []byte{0x25, 0x50, 0x44, 0x46, 0x2D},       0, nil},
    {"SQLite Database",  []byte{0x53, 0x51, 0x4C, 0x69, 0x74, 0x65, 0x20, 0x66}, 0, nil},
    // ...
}

When we run the detector, we open the file and walk through this list, reading just enough bytes at the right position to check for a match:

for _, s := range FileType {
    header := make([]byte, len(s.Signature))
    _, err := file.ReadAt(header, int64(s.Offset))
    if err != nil {
        continue
    }

    if bytes.HasPrefix(header, s.Signature) {
        fmt.Printf("Match found: %s\n", s.Name)
        found = true
        break
    }
}

The key function here is file.ReadAt — it lets us read bytes from any position in the file. Now this is just a base case.

Not Every Signature Starts at Zero

It would be convenient if every file format stored its magic bytes at the very beginning. Most do. But some don't.

ISO disk images are the classic example. Their identifying bytes (43 44 30 30 31, or CD001) sit at byte offset 32769 — deep into the file, not at the start. This is a deliberate part of the ISO 9660 spec; the first 32KB is reserved for boot code.

This is exactly why the struct has an Offset field, and why ReadAt is the right tool for the job:

go

{"ISO Image", []byte{0x43, 0x44, 0x30, 0x30, 0x31}, 32769, nil},

Instead of reading from position 0, we jump directly to byte 32769 and check for the signature there.

When Bytes Aren't Enough

Here's where things get interesting. Not every file format has a reliable magic number. Scripts, config files, and plain text files have no standardized binary header — they're just... text.

Many scripts start with a shebang line: a special first line that tells the shell which interpreter to use.

#!/usr/bin/python3

The #! bytes (0x23 0x21) do act as a kind of signature, but they're shared by every type of script — Python, Bash, Ruby, Perl. The bytes alone only tell you "this is a script." To know which script, you have to actually read the line.

This is called content sniffing — inspecting the file's actual content rather than relying on fixed byte patterns. In scripts_check.go, we handle this case:

func ScriptCheck(fileName string) bool {
    file, err := os.Open(fileName)
    if err != nil {
        return false
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    if scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, "#!") && strings.Contains(line, "python") {
            fmt.Printf("Match found: Python Script\n")
            return true
        }
        // Fallback: look for common Python patterns
        if strings.HasPrefix(line, "import ") || strings.HasPrefix(line, "def ") || strings.Contains(line, "print(") {
            fmt.Printf("Match found: Python Script\n")
            return true
        }
    }
    return false
}

Notice the fallback: if there's no shebang, we look for patterns like import, def, or print( — things that strongly suggest Python code. This is a rougher heuristic, but it's how you handle formats that were never designed to be identified by a fixed header.

This is why the CustomCheck field exists in our struct. For file types where raw byte comparison isn't sufficient, we plug in a function that can do deeper inspection. The byte signature gives us a quick first filter (#!), and if that matches, the custom check takes over to figure out the specifics.

Your browser does exactly the same thing. When a server sends a file with the wrong MIME type — or no type at all — the browser reads the content and makes its best guess. That's content sniffing in the wild.

The Tricky Case: Container Formats

There's one more scenario worth knowing about.

Many modern file formats — DOCX, XLSX, PPTX, JAR, APK — aren't really unique formats at all. They're ZIP archives with a specific internal structure. And they all start with the same signature:

50 4B 03 04

That's the ZIP magic number. So if you only check the magic bytes, every Word document, every Excel spreadsheet, and every Android app looks identical — just "a ZIP file."

To tell them apart, you have to open the archive and inspect what's inside. A DOCX will contain word/document.xml. An XLSX will have xl/workbook.xml. The container tells you the encoding; the contents tell you the actual format.

This is a known limitation of pure signature-based detection, and it's a good example of where content sniffing and deeper inspection become necessary — the magic number is just the starting point.

The Limits of Magic Numbers

Magic numbers are remarkably useful, but they're not a perfect solution.

Some formats have no signature at all. Plain text files, CSVs, JSON, log files — none of these have defined magic bytes. You can't fingerprint a .csv the way you fingerprint a .png. Detection has to rely on content analysis, file extension as a hint, or context.

Some formats share a signature. As we just saw with ZIP-based formats, a shared magic number means you need additional inspection to disambiguate. The signature gets you to the right neighborhood; you still have to find the right house.

The file command on Linux, and MIME detection in browsers, both combine these techniques — magic numbers, content sniffing, and context — to make their best determination.

Wrapping Up

Next time you rename a file, remember: the extension is just a label. The truth is in the bytes.

File signatures are one of those concepts that seem small until you realize how much of your operating system, your browser, and your security tooling depends on them quietly doing their job. Understanding how they work — and where they fall short — is useful any time you're building something that needs to handle files reliably.

The full source for the detector used in this post is linked below. It's a small codebase but covers the major cases: fixed-offset signatures, content sniffing via custom checks, and the BOM-based detection used to identify Unicode text encoding variants.

Github: https://github.com/kiranmurali93/learning/tree/main/file-signature

Wikipedia: https://en.wikipedia.org/wiki/Magic_number_(programming)

For the purpose of this blog, let’s consider a business use case: a job posting platform for freelance recruiters.

Companies post jobs on the platform, and recruiters can claim the jobs they want to work on. However, to avoid overwhelming companies with too many applications, a single job can only be claimed by a limited number of recruiters—say, four. This means we need to enforce a concurrency-safe mechanism to ensure that no more than the allowed number of claims are made per job.

Now, some of you might think: “That sounds simple—we just need to query the number of claims for a job, and if it’s greater than or equal to 4, return an error, right?”

And if I ask you about the code would it be like this?

count := db.Query(`SELECT COUNT(*) FROM job_claims WHERE job_id = ?`)
if count >= 4 {
    return error("Job fully claimed")
}
// proceed to insert claim

If so then you’re almost right—but not completely.

The problem with this approach lies in how web servers handle requests and how we think they do.

As developers, we often imagine our logic runs sequentially: a request comes in, it checks the count, inserts the claim, and we're done. But in reality, multiple requests can hit the server at the same time, and each of them can execute that same SELECT COUNT(*) query before any of them inserts their claim.

This means two or more recruiters could read the same count (say, 3), believe the job is still available, and go on to insert their own claim—resulting in more than four claims being stored. This is a classic race condition.

Before we test this out, let’s set up a basic project. For this demo, I’ll be using Golang and PostgreSQL.

Here’s what the folder structure looks like:

|- db/
   |- db.go
|- handlers/
   |- jobs.go
|-.env
|- main.go
|- test.go

Now let's break down what each file will do:

• db/db.go : Handles postgresql connection setup.
• handlers/jobs.go: Contains the handler/business logic for claiming the job.
• .env: Stores the secrets which you dont want to expose in the code.
• main.go: The entry file, which starts the http server and sets up routes.
• test.go: Simulates multiple concurrent recruiters trying to claim a job.

We also need to define the database schema and relationships.
Let’s start with the basic tables required for this system:

• companies: Stores information about all the companies registered on the platform.
• jobs: Contains job postings created by companies.
• recruiters: Holds the list of all recruiters who have signed up.
• job_claims: Records when a recruiter claims a job, forming a many-to-many relationship between recruiters and jobs.

Here’s a quick sql query for setting up the tables companies table

CREATE TABLE public.companies (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  name TEXT NOT NULL
);

jobs table

CREATE TABLE public.jobs (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  role TEXT,
  company_id BIGINT,
  CONSTRAINT jobs_company_id_fkey FOREIGN KEY (company_id) REFERENCES companies (id)
);

recruiter table

CREATE TABLE public.recruiter (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  name TEXT
);

job_claims table

CREATE TABLE public.job_claims (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  created_at TIMESTAMPTZ DEFAULT now(),
  job_id BIGINT NOT NULL,
  recruiter_id BIGINT NOT NULL,
  CONSTRAINT job_claims_job_id_recruiter_id_key UNIQUE (job_id, recruiter_id),
  CONSTRAINT job_claims_job_id_fkey FOREIGN KEY (job_id) REFERENCES jobs (id),
  CONSTRAINT job_claims_recruiter_id_fkey FOREIGN KEY (recruiter_id) REFERENCES recruiter (id)
);

Now since the setup is done let's move a head with implementation:

Let’s now implement the logic where we simply check how many claims a job already has, and only proceed if it’s less than 4.

This is how our handler/jobs.go will look like

func ClaimJob(w http.ResponseWriter, r *http.Request) {
    jobId := r.URL.Query().Get("jobId")
    recruiterId := r.URL.Query().Get("recruiterId")

    if jobId == "" || recruiterId == "" {
        http.Error(w, "Missing jobId or recruiterId", http.StatusBadRequest)
        return
    }

    // Count how many claims exist for this job
    var count int
    err := db.DB.QueryRow(`SELECT COUNT(*) FROM job_claims WHERE job_id = $1`, jobId).Scan(&count)
    if err != nil {
        http.Error(w, "Could not fetch claim count", http.StatusInternalServerError)
        return
    }

    if count >= 4 {
        http.Error(w, "Job fully claimed", http.StatusForbidden)
        return
    }

    // Insert the claim
    _, err = db.DB.Exec(`INSERT INTO job_claims (job_id, recruiter_id) VALUES ($1, $2)`, jobId, recruiterId)
    if err != nil {
        http.Error(w, "Failed to claim the job", http.StatusInternalServerError)
        return
    }

    json.NewEncoder(w).Encode(map[string]string{
        "message": "Job claimed successfully",
    })
}

And to test this out we have a test.go file. Note: This file would remain the same during entire processes

func main() {
    var wg sync.WaitGroup

    for i := 1; i <= 5; i++ {
        wg.Add(1)
        go func(rid int) {
            defer wg.Done()
            url := fmt.Sprintf("http://localhost:3000/claim-job?jobId=1&recruiterId=%d", rid)
            resp, err := http.Get(url)
            if err != nil {
                fmt.Printf("Recruiter %d error: %v\n", rid, err)
                return
            }
            defer resp.Body.Close()
            fmt.Printf("Recruiter %d: %s\n", rid, resp.Status)
        }(i)
    }

    wg.Wait()
}

If you run this test, you will get something like this

Recruiter 4: 200 OK
Recruiter 1: 200 OK
Recruiter 2: 200 OK
Recruiter 5: 200 OK
Recruiter 3: 200 OK

Note: This can be in any order since we are running it concurrently.

If you notice the output you will be able to see there are a total of 5 claims rather than just 4.

So what went wrong? Each request checked the count before inserting, but none of them had visibility into the others that were happening at the same time.

At this point, you might be thinking:
“Why not just wrap the whole thing in a transaction so that it's atomic?”

Let's try that out too. The updated section of handler/jobs.go would look like this

tx, err := db.DB.Begin()
if err != nil {
    http.Error(w, "Could not start transaction", http.StatusInternalServerError)
    return
}
defer tx.Rollback()

var count int
err = tx.QueryRow(`SELECT COUNT(*) FROM job_claims WHERE job_id = $1`, jobId).Scan(&count)
if err != nil {
    http.Error(w, "Could not fetch claim count", http.StatusInternalServerError)
    return
}

if count >= 4 {
    http.Error(w, "Job fully claimed", http.StatusForbidden)
    return
}

_, err = tx.Exec(`INSERT INTO job_claims (job_id, recruiter_id) VALUES ($1, $2)`, jobId, recruiterId)
if err != nil {
    http.Error(w, "Failed to claim the job", http.StatusInternalServerError)
    return
}

err = tx.Commit()
if err != nil {
    http.Error(w, "Failed to commit transaction", http.StatusInternalServerError)
    return
}

This feels like this would solve the problem. To check this out let's try running the test.go again and we would get something like this

Recruiter 3: 200 OK
Recruiter 4: 200 OK
Recruiter 2: 200 OK
Recruiter 5: 200 OK
Recruiter 1: 200 OK

Unfortunately, this too doesn’t seem to help.

If you ask why, here's the reason: Even though the request is running a transaction, there is no locking happening. Each transaction still sees the same initial count before others have committed their transaction.

Transactions can only isolates their own change. Without explicitly locking in rows. So multiple transactions can read the same state and make decision based on that, leading to same race condition.

So now what would be the fix? The solution to this problem is to lock the job row that recruiters are trying to claim. This ensures that only one transaction can access and modify the relevant data at a time.

In postgres we have a command "SELECT FOR UPDATE" In our case it will be something like this

SELECT * FROM jobs WHERE id = $1 FOR UPDATE;

What does it do?

• It locks on to the corresponding job.
• Other transactions which tries to access the same row for update, will be blocked until the lock is released.

Let's update the handler logic

tx, err := db.DB.Begin()
if err != nil {
    http.Error(w, "Could not start transaction", http.StatusInternalServerError)
    return
}
defer tx.Rollback()

// Lock the job row
_, err = tx.Exec(`SELECT * FROM jobs WHERE id = $1 FOR UPDATE`, jobId)
if err != nil {
    http.Error(w, "Failed to lock job", http.StatusInternalServerError)
    return
}

// Check if this recruiter already claimed the job
var exists bool
err = tx.QueryRow(`SELECT EXISTS(SELECT 1 FROM job_claims WHERE job_id = $1 AND recruiter_id = $2)`, jobId, recruiterId).Scan(&exists)
if err != nil {
    http.Error(w, "Could not check claim status", http.StatusInternalServerError)
    return
}

if exists {
    http.Error(w, "Recruiter already claimed this job", http.StatusConflict)
    return
}

// Count existing claims
var count int
err = tx.QueryRow(`SELECT COUNT(*) FROM job_claims WHERE job_id = $1`, jobId).Scan(&count)
if err != nil {
    http.Error(w, "Could not fetch claim count", http.StatusInternalServerError)
    return
}

if count >= 4 {
    http.Error(w, "Job fully claimed", http.StatusForbidden)
    return
}

// Insert the claim
_, err = tx.Exec(`INSERT INTO job_claims (job_id, recruiter_id) VALUES ($1, $2)`, jobId, recruiterId)
if err != nil {
    http.Error(w, "Failed to claim job", http.StatusInternalServerError)
    return
}

err = tx.Commit()
if err != nil {
    http.Error(w, "Failed to commit transaction", http.StatusInternalServerError)
    return
}

Now when you run the test.go you will notice something like this

Recruiter 3: 200 OK
Recruiter 5: 200 OK
Recruiter 2: 200 OK
Recruiter 4: 200 OK
Recruiter 1: 403 Forbidden

As you can see only 4 claims are successful, the 5th one fails. This solves the race condition.

Now there might be some confusion that I would like to clear You might notice that we're locking a row in the jobs table, but performing operations on the job_claims table. This is intentional and highlights an important concept in database locking:

When managing concurrency, you need to lock the "decision-making resource," not just the tables being modified. In our case:

1. The job itself (in the jobs table) is the logical resource we're contending for
2. The claims (in the job_claims table) are just the implementation detail of how we track that resource allocation

By locking the job row, we ensure that all decisions about whether a particular job can accept more claims are serialised—meaning they happen one after another, not simultaneously. This prevents the race condition where multiple transactions might all see "3 claims" at the same time and all decide they can proceed.

Another approach would be to create a separate "job_claims_count" table that tracks the current count per job, and lock rows in that table instead. This would achieve the same goal while potentially reducing lock contention on the main jobs table.

For checking the implementation code check out github