Topic 19030 min readStructured Tutorial

DataFrames in Python

Quick Answer

DataFrames explains dataFrames are a fundamental data structure in Python for data analysis and manipulation.

Learning Objectives

Explain the purpose of DataFrames in a practical learning context.
Identify the main ideas, terms, and decisions involved in DataFrames.
Apply DataFrames in a simple real-world scenario or practice task.

Introduction

DataFrames are a fundamental data structure in Python for data analysis and manipulation.

They provide a tabular, spreadsheet-like format that is easy to work with and powerful for handling complex datasets.

DataFrames make data analysis in Python intuitive and efficient.

What is a DataFrame?

A DataFrame is a two-dimensional labeled data structure with columns of potentially different types.

It is similar to a spreadsheet or SQL table and is provided by the pandas library in Python.

Rows and columns are both labeled.
Supports heterogeneous data types.
Allows easy data filtering, aggregation, and transformation.

Creating DataFrames

You can create DataFrames from various data sources such as dictionaries, lists, CSV files, or SQL queries.

The most common way is to use the pandas.DataFrame constructor.

From a dictionary of lists or arrays.
From a list of dictionaries.
From CSV or Excel files using pandas read functions.

Example: Creating a DataFrame from a Dictionary

Here is a simple example of creating a DataFrame from a dictionary.

Basic DataFrame Operations

Once you have a DataFrame, you can perform many operations to explore and manipulate your data.

These include selecting columns, filtering rows, adding new columns, and summarizing data.

Selecting columns with df['column_name'] or df.column_name.
Filtering rows using boolean indexing.
Adding new columns by assignment.
Using methods like df.describe() for summary statistics.

Common DataFrame Methods

Pandas provides many built-in methods to work efficiently with DataFrames.

head() - view first rows.
tail() - view last rows.
info() - summary of DataFrame structure.
describe() - statistical summary.
groupby() - group data for aggregation.
merge() - join DataFrames.

Handling Missing Data

Missing data is common in real-world datasets and pandas offers tools to handle it effectively.

isnull() and notnull() to detect missing values.
dropna() to remove missing data.
fillna() to replace missing values with a specified value.

Practical Example

This example creates a DataFrame with three columns: Name, Age, and City, and prints it.

This example filters the DataFrame to include only rows where the Age is greater than 28.

Examples

Creating a DataFrame from a Dictionary

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35], 'City': ['New York', 'Paris', 'London']}
df = pd.DataFrame(data)
print(df)

This example creates a DataFrame with three columns: Name, Age, and City, and prints it.

Filtering Rows in a DataFrame

filtered_df = df[df['Age'] > 28]
print(filtered_df)

This example filters the DataFrame to include only rows where the Age is greater than 28.

Best Practices

Always inspect your DataFrame using head() and info() before analysis.
Use vectorized operations instead of loops for better performance.
Handle missing data early to avoid errors in analysis.
Use descriptive column names for clarity.
Leverage pandas built-in functions for common tasks.

Common Mistakes

Modifying DataFrames without creating copies can lead to unexpected side effects.
Ignoring missing data can cause errors or misleading results.
Using loops instead of vectorized operations reduces performance.
Not setting an appropriate index when needed.
Confusing loc and iloc for label-based vs position-based indexing.

Hands-on Exercise

Create and Inspect a DataFrame

Create a DataFrame from a dictionary with at least three columns and five rows. Use head() and info() to inspect it.

Expected output: Printed output showing the first rows and summary info of the DataFrame.

Hint: Use pandas.DataFrame constructor and call df.head() and df.info().

Filter DataFrame Rows

Filter the DataFrame to show only rows where a numeric column value is greater than a threshold.

Expected output: A subset of the DataFrame with rows meeting the condition.

Hint: Use boolean indexing with df[df['column'] > value].

Interview Questions

What is a DataFrame in pandas?

Interview

A DataFrame is a two-dimensional labeled data structure with columns of potentially different types, similar to a spreadsheet or SQL table.

How do you handle missing data in a DataFrame?

Interview

You can detect missing data with isnull(), remove it with dropna(), or fill it with fillna() using a specified value.

What is the difference between loc and iloc in pandas?

Interview

loc is label-based indexing, selecting data by row and column labels, while iloc is position-based indexing, selecting data by integer position.