Pandas DataFrames are one of the most important tools for working with structured data in Python. If you are learning data analysis, machine learning, or AI development, understanding DataFrames should be one of your first priorities after basic Python, NumPy, and introductory Pandas concepts.

A DataFrame allows you to organize information into rows and columns, similar to an Excel spreadsheet or database table.

For example, a DataFrame can store employee data like this:

Name    Age    Salary
Aman    25     45000
Riya    28     52000
Rahul   23     40000

Once the data is inside a Pandas DataFrame, you can filter rows, select columns, handle missing values, calculate statistics, sort records, create new columns, combine datasets, and prepare information for machine learning models.

This guide explains Pandas DataFrames step by step with beginner-friendly examples.

What Is a Pandas DataFrame?

A Pandas DataFrame is a two-dimensional labeled data structure provided by the Pandas library.

Two-dimensional means the data is organized using:

• Rows
• Columns

Each column normally represents a type of information, while each row represents one record.

For example:

import pandas as pd

data = {
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, 28, 23],
    "city": ["Jaipur", "Delhi", "Mumbai"]
}

df = pd.DataFrame(data)

print(df)

Output:

    name  age    city
0   Aman   25  Jaipur
1   Riya   28   Delhi
2  Rahul   23  Mumbai

Here:

• name, age, and city are columns.
• Each numbered line is a row.
• 0, 1, and 2 are index labels.

The variable df is a DataFrame.

Why Are DataFrames Useful?

Real-world applications often work with tabular data.

Examples include:

• Customer records
• Product catalogs
• Student marks
• Sales reports
• Employee information
• Website analytics
• Machine learning datasets
• Financial transactions

A DataFrame provides convenient tools for working with this data.

Instead of manually processing every record using loops, you can perform operations directly on entire columns or filtered groups of rows.

For example:

df["salary"].mean()

can calculate the average salary of all employees in one line.

How to Install Pandas

Before using DataFrames, install Pandas.

Run:

pip install pandas

You can also use:

python -m pip install pandas

Then test the installation:

import pandas as pd

print(pd.__version__)

If a version number appears, Pandas is installed correctly.

How to Import Pandas

The standard Pandas import is:

import pandas as pd

pd is the commonly used alias for Pandas.

You will therefore create a DataFrame using:

pd.DataFrame()

How to Create a Pandas DataFrame

There are several ways to create a DataFrame.

Create a DataFrame from a Dictionary

This is one of the easiest methods.

import pandas as pd

data = {
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, 28, 23]
}

df = pd.DataFrame(data)

print(df)

Output:

    name  age
0   Aman   25
1   Riya   28
2  Rahul   23

Each dictionary key becomes a column name.

The corresponding list becomes the data for that column.

Create a DataFrame from a List of Dictionaries

You can also create records individually:

data = [
    {"name": "Aman", "age": 25},
    {"name": "Riya", "age": 28},
    {"name": "Rahul", "age": 23}
]

df = pd.DataFrame(data)

print(df)

This format is useful when data already exists as JSON-like records.

Create a DataFrame from Lists

You can provide rows and define the column names separately:

data = [
    ["Aman", 25],
    ["Riya", 28],
    ["Rahul", 23]
]

df = pd.DataFrame(
    data,
    columns=["name", "age"]
)

print(df)

Understanding DataFrame Rows and Columns

Consider:

df = pd.DataFrame({
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, 28, 23],
    "salary": [45000, 52000, 40000]
})

The columns are:

name
age
salary

The rows represent individual records.

You can see the column names using:

print(df.columns)

You can inspect the index using:

print(df.index)

Understanding DataFrame Shape

The shape attribute tells you the number of rows and columns.

print(df.shape)

Output:

(3, 3)

This means:

• 3 rows
• 3 columns

For machine learning, shape is especially important because models expect input data in particular dimensions.

Finding the Number of DataFrame Elements

Use:

print(df.size)

For a DataFrame containing 3 rows and 3 columns:

9

The result represents the total number of cells.

Checking Data Types

Different columns can have different data types.

Use:

print(df.dtypes)

Example output:

name      object
age        int64
salary     int64
dtype: object

Data types matter because numerical calculations usually require numerical columns.

A column accidentally stored as text may need conversion before analysis.

Viewing the First Rows

Large datasets may contain thousands or millions of rows.

Instead of printing everything, use:

print(df.head())

By default, head() shows the first five rows.

You can specify another number:

print(df.head(10))

Viewing the Last Rows

Use:

print(df.tail())

Or:

print(df.tail(3))

This is useful for checking how a dataset ends.

Getting DataFrame Information

Use:

df.info()

This displays useful information such as:

• Number of rows
• Column names
• Non-null values
• Data types
• Approximate memory usage

This should usually be one of the first commands you run after loading a new dataset.

Getting Summary Statistics

Use:

print(df.describe())

For numerical columns, Pandas can show:

• Count
• Mean
• Standard deviation
• Minimum
• 25th percentile
• Median
• 75th percentile
• Maximum

This gives you a quick overview of your numerical data.

Selecting Columns from a DataFrame

Selecting columns is one of the most common DataFrame operations.

Suppose:

df = pd.DataFrame({
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, 28, 23],
    "salary": [45000, 52000, 40000]
})

Select One Column

print(df["name"])

Output:

0     Aman
1     Riya
2    Rahul
Name: name, dtype: object

Selecting one column normally returns a Pandas Series.

Select Multiple Columns

print(df[["name", "salary"]])

Output:

    name  salary
0   Aman   45000
1   Riya   52000
2  Rahul   40000

Selecting multiple columns returns another DataFrame.

Selecting Rows with iloc

iloc selects rows and columns by integer position.

For example:

print(df.iloc[0])

This selects the first row.

You can select multiple rows:

print(df.iloc[0:2])

This returns the first two rows.

You can also select a specific row and column:

print(df.iloc[0, 1])

This selects:

• First row
• Second column

In this example, the result is:

25

Selecting Rows with loc

loc selects data using labels.

With the default index:

print(df.loc[0])

This returns the row labeled 0.

You can also select a specific value:

print(df.loc[0, "name"])

Output:

Aman

A useful beginner rule is:

loc  → labels
iloc → integer positions

Filtering DataFrame Rows

Filtering allows you to select rows that match a condition.

For example:

df = pd.DataFrame({
    "name": ["Aman", "Riya", "Rahul", "Neha"],
    "age": [25, 30, 22, 27]
})

Select people older than 25:

result = df[df["age"] > 25]

print(result)

Output:

   name  age
1  Riya   30
3  Neha   27

The expression:

df["age"] > 25

produces Boolean values such as:

False
True
False
True

Pandas returns only rows where the condition is True.

Filtering with Multiple Conditions

You can combine conditions.

Example:

result = df[
    (df["age"] >= 25) &
    (df["age"] <= 28)
]

print(result)

Use:

&

for AND.

Use:

|

for OR.

Remember to place individual conditions inside parentheses.

Adding a New Column

Adding a new column is straightforward.

df["salary"] = [45000, 52000, 40000, 60000]

You can also calculate a column from existing data.

df["bonus"] = df["salary"] * 0.10

Pandas calculates the value for every row.

Updating DataFrame Values

You can modify values using loc.

For example:

df.loc[0, "age"] = 26

This changes the age value in row 0.

You can also update rows conditionally:

df.loc[df["age"] < 25, "category"] = "Young"

This sets category to Young where the person’s age is below 25.

Removing Columns

Use drop():

df = df.drop(columns=["salary"])

Remove multiple columns:

df = df.drop(
    columns=["salary", "bonus"]
)

Assigning the result back to df makes the change explicit.

Removing Rows

Remove one row:

df = df.drop(index=0)

Remove several rows:

df = df.drop(index=[0, 2])

Renaming DataFrame Columns

You can rename columns with:

df = df.rename(
    columns={
        "name": "employee_name",
        "age": "employee_age"
    }
)

Clear column names make datasets easier to understand and maintain.

Sorting DataFrames

Sort a DataFrame using:

df = df.sort_values("age")

This sorts age in ascending order.

For descending order:

df = df.sort_values(
    "age",
    ascending=False
)

You can also sort using multiple columns:

df = df.sort_values(
    ["department", "salary"]
)

Handling Missing Values in DataFrames

Missing data is common in real datasets.

Consider:

import pandas as pd

data = {
    "name": ["Aman", "Riya", "Rahul"],
    "age": [25, None, 23]
}

df = pd.DataFrame(data)

Find Missing Values

Use:

print(df.isnull())

Count missing values:

print(df.isnull().sum())

Example output:

name    0
age     1
dtype: int64

Remove Missing Values

Use:

clean_df = df.dropna()

This removes rows containing missing values.

However, removing rows is not always appropriate.

You should understand why data is missing before deciding what to do.

Fill Missing Values

You can replace missing values:

df["age"] = df["age"].fillna(0)

Or replace them using the mean:

df["age"] = df["age"].fillna(
    df["age"].mean()
)

This approach is sometimes useful for numerical data, but whether it is appropriate depends on the dataset and problem.

Working with Duplicate Rows

Check for duplicates:

print(df.duplicated())

Count duplicates:

print(df.duplicated().sum())

Remove them:

df = df.drop_duplicates()

Do not automatically remove duplicates without verifying whether repeated records are actually incorrect.

Working with Text Columns

Pandas provides string functions through .str.

Suppose:

df = pd.DataFrame({
    "name": [" aman ", "RIYA", "rahul"]
})

Remove extra spaces:

df["name"] = df["name"].str.strip()

Convert to lowercase:

df["name"] = df["name"].str.lower()

Convert to uppercase:

df["name"] = df["name"].str.upper()

You can also search for text:

result = df[
    df["name"].str.contains(
        "aman",
        case=False
    )
]

These operations are useful for cleaning messy text data.

Working with Dates in a DataFrame

Dates are frequently stored as text when datasets are loaded.

Example:

df = pd.DataFrame({
    "date": [
        "2025-01-10",
        "2025-02-12"
    ]
})

Convert the column to datetime:

df["date"] = pd.to_datetime(
    df["date"]
)

Then extract values:

df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["day"] = df["date"].dt.day

This is useful for:

• Sales analysis
• User activity tracking
• Financial analysis
• Time-series datasets

Grouping DataFrames with groupby()

groupby() allows you to divide data into groups and calculate values for each group.

Example:

df = pd.DataFrame({
    "department": [
        "IT",
        "HR",
        "IT",
        "HR"
    ],
    "salary": [
        50000,
        40000,
        60000,
        45000
    ]
})

Calculate the average salary by department:

result = (
    df.groupby("department")["salary"]
      .mean()
)

print(result)

Output:

department
HR    42500.0
IT    55000.0
Name: salary, dtype: float64

Conceptually, Pandas:

1. Groups rows by department.
2. Selects salary values.
3. Calculates the average for each group.

Aggregating Data

You can calculate multiple statistics at once.

result = df.groupby(
    "department"
)["salary"].agg(
    ["mean", "min", "max"]
)

print(result)

This is useful when creating reports and summaries.

Merging DataFrames

In real projects, related information often exists in different datasets.

Suppose:

customers = pd.DataFrame({
    "customer_id": [1, 2, 3],
    "name": ["Aman", "Riya", "Rahul"]
})

And:

orders = pd.DataFrame({
    "customer_id": [1, 2, 3],
    "amount": [500, 800, 300]
})

Merge them using:

result = pd.merge(
    customers,
    orders,
    on="customer_id"
)

print(result)

Output:

   customer_id   name  amount
0            1   Aman     500
1            2   Riya     800
2            3  Rahul     300

This is similar to a database join.

Concatenating DataFrames

If two DataFrames contain similar columns, you can combine them vertically.

df1 = pd.DataFrame({
    "name": ["Aman", "Riya"]
})

df2 = pd.DataFrame({
    "name": ["Rahul", "Neha"]
})

result = pd.concat(
    [df1, df2],
    ignore_index=True
)

print(result)

Output:

    name
0   Aman
1   Riya
2  Rahul
3   Neha

Reading a CSV into a DataFrame

One of the most common Pandas operations is loading a CSV file.

Suppose students.csv contains:

name,age,score
Aman,20,85
Riya,21,92
Rahul,22,78

Load it:

import pandas as pd

df = pd.read_csv(
    "students.csv"
)

print(df.head())

Pandas automatically converts the CSV table into a DataFrame.

Reading Excel Data

Use:

df = pd.read_excel(
    "students.xlsx"
)

You may need:

pip install openpyxl

depending on your Excel format and environment.

Reading JSON into a DataFrame

You can load compatible JSON data using:

df = pd.read_json(
    "data.json"
)

This is particularly useful when working with data exported from APIs or applications.

Saving a DataFrame to CSV

Save a processed DataFrame:

df.to_csv(
    "clean_data.csv",
    index=False
)

Using:

index=False

prevents the DataFrame index from being written as an extra CSV column.

Saving a DataFrame to Excel

Use:

df.to_excel(
    "clean_data.xlsx",
    index=False
)

Practical Example: Cleaning Employee Data

Let’s combine several DataFrame operations.

import pandas as pd

data = {
    "name": [
        "Aman",
        "Riya",
        "Rahul",
        "Neha",
        "Aman"
    ],
    "age": [
        25,
        28,
        None,
        30,
        25
    ],
    "salary": [
        45000,
        55000,
        40000,
        65000,
        45000
    ]
}

df = pd.DataFrame(data)

First inspect the dataset:

print(df.head())
print(df.shape)
df.info()

Check missing values:

print(df.isnull().sum())

Remove duplicates:

df = df.drop_duplicates()

Fill the missing age:

df["age"] = df["age"].fillna(
    df["age"].mean()
)

Create a bonus column:

df["bonus"] = (
    df["salary"] * 0.10
)

Filter higher salaries:

high_salary = df[
    df["salary"] > 45000
]

print(high_salary)

Calculate the average salary:

average_salary = (
    df["salary"].mean()
)

print(average_salary)

Finally, export the cleaned data:

df.to_csv(
    "clean_employees.csv",
    index=False
)

This demonstrates a typical data workflow:

Load Data
   ↓
Inspect Data
   ↓
Clean Data
   ↓
Transform Data
   ↓
Filter Data
   ↓
Analyze Data
   ↓
Export Data

Pandas DataFrames in Machine Learning

DataFrames are extremely useful when preparing machine learning datasets.

Suppose:

df = pd.DataFrame({
    "age": [22, 25, 30, 35],
    "income": [
        30000,
        45000,
        60000,
        80000
    ],
    "purchased": [0, 0, 1, 1]
})

You can separate input features:

X = df[
    ["age", "income"]
]

And target values:

y = df["purchased"]

Here:

• X contains the input features.
• y contains the value you want the model to predict.

This structure is commonly used with machine learning libraries such as scikit-learn.

Converting a DataFrame to a NumPy Array

Many AI and numerical libraries work with arrays.

You can convert a DataFrame using:

arr = df.to_numpy()

print(arr)

This creates a NumPy array containing the DataFrame values.

You can also convert one column:

ages = df["age"].to_numpy()

Understanding the relationship between Pandas DataFrames and NumPy arrays is very useful for AI development.

DataFrame Index Explained

Every DataFrame has an index.

For example:

df = pd.DataFrame({
    "name": ["Aman", "Riya"]
})

print(df.index)

By default, Pandas creates an integer index starting from zero.

You can set another column as the index:

df = pd.DataFrame({
    "student_id": [101, 102, 103],
    "name": [
        "Aman",
        "Riya",
        "Rahul"
    ]
})

df = df.set_index(
    "student_id"
)

print(df)

Now the student ID becomes the row label.

Resetting the Index

Use:

df = df.reset_index()

To discard the old index:

df = df.reset_index(
    drop=True
)

Applying Functions to Columns

Suppose:

df = pd.DataFrame({
    "salary": [
        40000,
        50000,
        60000
    ]
})

You can create a function:

def calculate_bonus(salary):
    return salary * 0.10

Then apply it:

df["bonus"] = (
    df["salary"]
    .apply(calculate_bonus)
)

For simple vectorized arithmetic, however, this is usually clearer:

df["bonus"] = (
    df["salary"] * 0.10
)

Prefer direct vectorized operations when possible.

Why DataFrames Matter for AI Developers

AI models do not begin with model training.

Real projects usually begin with raw data.

That data often needs to be:

• Loaded
• Inspected
• Cleaned
• Filtered
• Converted
• Encoded
• Reshaped
• Split into features and targets

Pandas DataFrames make these tasks easier.

A simplified AI workflow looks like:

Raw Data
   ↓
Pandas DataFrame
   ↓
Data Cleaning
   ↓
Exploratory Data Analysis
   ↓
Feature Engineering
   ↓
Train/Test Split
   ↓
Machine Learning Model
   ↓
Evaluation

If your goal is to become an AI Developer, becoming comfortable with DataFrames will make later machine learning concepts much easier.

Advantages of Pandas DataFrames

Pandas DataFrames provide several important benefits.

Easy-to-Understand Structure

Rows and columns make tabular data intuitive for beginners.

Labeled Data

Columns and indexes can have meaningful names.

Powerful Filtering

You can select records using conditions without writing complicated loops.

Data Cleaning Tools

Pandas includes convenient tools for missing values, duplicates, text, dates, and data types.

Integration with Python Libraries

DataFrames work well with:

• NumPy
• Matplotlib
• scikit-learn
• Jupyter
• PyTorch
• TensorFlow

Easy File Handling

You can read and write common formats such as CSV, Excel, and JSON.

Limitations of Pandas DataFrames

DataFrames are powerful, but they are not ideal for every problem.

Memory Usage

Traditional Pandas operations generally work in memory.

Very large datasets may exceed the available RAM.

Performance for Extremely Large Data

Distributed or very large workloads may require other tools and approaches.

Indexing Can Be Confusing

Beginners often confuse:

loc
iloc

Data Types Require Attention

Messy real-world datasets may load columns using unexpected data types.

DataFrames Are Not Machine Learning Models

Pandas is mainly used for manipulating and preparing data.

Model training requires libraries such as scikit-learn, PyTorch, or TensorFlow.

Common Mistakes to Avoid

1. Confusing Series and DataFrames

This:

df["name"]

usually returns a Series.

This:

df[["name"]]

returns a DataFrame.

2. Confusing loc and iloc

Remember:

loc  → label-based
iloc → position-based

3. Ignoring Missing Values

Always inspect:

df.isnull().sum()

before important analysis.

4. Ignoring Data Types

Check:

df.dtypes

Unexpected data types can cause incorrect calculations or errors.

5. Modifying Data Without Keeping the Result

For example:

df.drop(columns=["age"])

may return a modified DataFrame without replacing df.

Prefer:

df = df.drop(
    columns=["age"]
)

6. Removing Data Too Quickly

Do not automatically remove every row containing a missing value or duplicate.

Understand the dataset first.

7. Using Loops for Simple Column Calculations

Avoid unnecessarily doing:

for value in df["salary"]:
    ...

when you can use:

df["salary"] * 1.10

Vectorized operations are usually clearer for column-based numerical work.

Best Practices for Pandas DataFrames

When you load a new DataFrame, start by checking:

df.head()
df.shape
df.info()
df.describe()
df.isnull().sum()

Keep column names clear and consistent.

For example:

customer_name
total_price
order_date

is usually easier to work with than inconsistent names such as:

Customer Name
Total-Price
ORDER DATE

Keep the original source data unchanged whenever possible.

Create a cleaned copy or export a processed dataset instead.

Use vectorized DataFrame operations when appropriate rather than unnecessary Python loops.

Check your dataset after major transformations to make sure the result still contains the rows and columns you expect.

What to Learn Next

If you want to become an AI Developer, learning DataFrames should lead naturally into deeper data analysis.

A useful learning path is:

Python → NumPy → Pandas → Data Cleaning → Data Visualization → Statistics → scikit-learn → Machine Learning → Deep Learning

After mastering DataFrames, learn:

1. Pandas indexing and slicing
2. Pandas data cleaning
3. Missing value handling
4. Pandas groupby()
5. DataFrame merging and joining
6. Exploratory data analysis
7. NumPy mathematical operations
8. Matplotlib
9. Statistics for machine learning
10. scikit-learn basics

You should eventually be able to take a raw CSV dataset and independently clean, inspect, transform, and prepare it for machine learning.

Frequently Asked Questions

1. What is a Pandas DataFrame in simple terms?

A Pandas DataFrame is a two-dimensional table in Python containing rows and columns. It is similar to a spreadsheet or database table.

2. What is the difference between a Series and DataFrame?

A Series is one-dimensional labeled data, while a DataFrame is two-dimensional and contains rows and multiple columns.

3. How do I create a DataFrame in Pandas?

One common method is:

df = pd.DataFrame({
    "name": ["Aman", "Riya"],
    "age": [25, 28]
})

4. How do I select a column in a DataFrame?

Use:

df["column_name"]

For multiple columns:

df[
    ["column1", "column2"]
]

5. What is the difference between loc and iloc?

loc selects data using labels, while iloc selects data using integer positions.

6. How do I remove missing values?

Use:

df.dropna()

However, you should understand why values are missing before removing rows.

7. Are DataFrames used in machine learning?

Yes. DataFrames are commonly used to load, clean, transform, analyze, and prepare tabular datasets before training machine learning models.

8. Can I convert a DataFrame to a NumPy array?

Yes.

Use:

df.to_numpy()

This converts DataFrame values into a NumPy array.

Conclusion

Pandas DataFrames provide one of the most convenient ways to work with structured data in Python.

They allow you to create tables, select rows and columns, filter records, handle missing values, remove duplicates, calculate statistics, sort data, group information, merge datasets, and export processed results.

For beginners learning AI and machine learning, DataFrames are especially important because most real projects require significant data preparation before model training begins.

Once you are comfortable creating, inspecting, filtering, cleaning, grouping, and merging DataFrames, your next step should be learning Pandas data cleaning and exploratory data analysis in greater depth.

---