In the world of data science and scientific computing, two Python libraries stand out as foundational tools: NumPy and Pandas. Both libraries serve distinct but complementary purposes. NumPy (Numerical Python) provides support for large, multi-dimensional arrays and matrices, along with a vast collection of mathematical functions to operate on these arrays efficiently. Pandas, on the other hand, is designed for data manipulation and analysis, offering data structures like Series and DataFrames that make handling labeled and heterogeneous data intuitive and powerful.
This lesson dives deep into the basics of NumPy and Pandas, exploring their core data structures, essential functions, and typical operations you’ll frequently use in data processing workflows. By mastering these, you’ll be well-equipped to handle complex datasets, perform vectorized computations, and prepare your data for machine learning, visualization, or reporting.
We will start by understanding NumPy arrays, their creation, indexing, and broadcasting capabilities, then transition to Pandas Series and DataFrames, focusing on data selection, filtering, aggregation, and merging. Throughout, we emphasize best practices and common pitfalls to avoid, ensuring your code is both efficient and readable.
💡 A Simple Analogy: NumPy Arrays as Excel Worksheets, Pandas DataFrames as Excel Tables
Think of NumPy arrays as the underlying Excel worksheet grid — a grid of cells where every cell is the same type and can be processed very fast. Meanwhile, Pandas DataFrames are like Excel tables where you have labeled rows and columns, with different data types in each column, and powerful tools to filter, sort, and aggregate the data. NumPy provides the computational engine, and Pandas provides the user-friendly interface for data analysis.
🎯 Real-World Use Case: Financial Time Series Analysis
Imagine you need to analyze stock prices over time. NumPy can efficiently handle numerical computations such as calculating moving averages or volatility over large arrays of price data. Pandas complements this by allowing you to handle timestamps, label your data by date, easily slice time periods, handle missing data, and compute statistics grouped by different labels such as sectors or stock exchanges.
⚠️ Common Pitfall: Confusing NumPy Arrays and Pandas DataFrames
While both libraries use similar terminology (arrays, indexing), they behave differently. NumPy arrays are homogeneous and support vectorized operations but lack built-in labeling. Pandas DataFrames support heterogeneous data types and indexing by labels but sometimes sacrifice raw performance. Mixing these up can lead to bugs or inefficient code. Always choose the right tool based on the task: NumPy for numerical computations, Pandas for data manipulation.
Understanding NumPy Arrays — Learn how to create arrays, check their dimensions, and manipulate them using slicing, reshaping, and broadcasting.
Exploring Pandas Series — Discover the one-dimensional labeled array, its construction from lists or NumPy arrays, and how to access and manipulate data with labels.
Diving into Pandas DataFrames — Understand multi-dimensional tabular data, creation from dictionaries or CSV files, indexing with loc and iloc, and basic data cleaning operations.
Data Aggregation and Grouping — Master groupby operations, aggregation functions, and pivot tables to summarize data effectively.
Combining and Merging DataFrames — Learn how to join, concatenate, and merge datasets based on keys or indices to build comprehensive data views.

📌 Deep Dive: Creating and Manipulating NumPy Arrays
# Import NumPy library
import numpy as np
# Create a 1D NumPy array from a Python list
arr = np.array([10, 20, 30, 40, 50])
print("Original array:", arr)
# Create a 2D array (matrix) of shape (3, 3)
matrix = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
print("2D Matrix:
", matrix)
# Access elements: element at row 1, column 2
elem = matrix[1, 2]
print("Element at (1, 2):", elem)
# Slicing: get first two rows and last two columns
slice_sub = matrix[:2, 1:]
print("Sliced submatrix:
", slice_sub)
# Reshape array: flatten matrix into 1D array
flat = matrix.reshape(-1)
print("Flattened array:", flat)
# Broadcasting: add a scalar to entire array
broadcasted = matrix + 10
print("After adding 10 to each element:
", broadcasted)
# Element-wise multiplication
multiplied = matrix * matrix
print("Element-wise squared matrix:
", multiplied)
2D Matrix:
[[1 2 3]
[4 5 6]
[7 8 9]]
Element at (1, 2): 6
Sliced submatrix:
[[2 3]
[5 6]]
Flattened array: [1 2 3 4 5 6 7 8 9]
After adding 10 to each element:
[[11 12 13]
[14 15 16]
[17 18 19]]
Element-wise squared matrix:
[[ 1 4 9]
[16 25 36]
[49 64 81]]
📌 Deep Dive: Pandas Series and DataFrame Basics
import pandas as pd
# Create a Pandas Series from a list with custom indices
data = [100, 200, 300, 400]
index_labels = ['a', 'b', 'c', 'd']
series = pd.Series(data, index=index_labels)
print("Series:
", series)
# Access elements by label and position
print("Element with label 'b':", series['b'])
print("Element at position 2:", series.iloc[2])
# Create a DataFrame from a dictionary of lists
data_dict = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'Salary': [70000, 80000, 90000]
}
df = pd.DataFrame(data_dict)
print("
DataFrame:
", df)
# Select a column (returns a Series)
ages = df['Age']
print("
Ages column:
", ages)
# Select multiple columns (returns a DataFrame)
subset = df[['Name', 'Salary']]
print("
Subset DataFrame:
", subset)
# Filter rows where Age > 28
filtered = df[df['Age'] > 28]
print("
Filtered DataFrame (Age > 28):
", filtered)
# Using loc and iloc for selection
print("
Select row with label 1 using loc:
", df.loc[1])
print("
Select first row using iloc:
", df.iloc[0])
a 100
b 200
c 300
d 400
dtype: int64
Element with label 'b': 200
Element at position 2: 300
DataFrame:
Name Age Salary
0 Alice 25 70000
1 Bob 30 80000
2 Charlie 35 90000
Ages column:
0 25
1 30
2 35
Name: Age, dtype: int64
Subset DataFrame:
Name Salary
0 Alice 70000
1 Bob 80000
2 Charlie 90000
Filtered DataFrame (Age > 28):
Name Age Salary
1 Bob 30 80000
2 Charlie 35 90000
Select row with label 1 using loc:
Name Bob
Age 30
Salary 80000
Name: 1, dtype: object
Select first row using iloc:
Name Alice
Age 25
Salary 70000
Name: 0, dtype: object
📌 Deep Dive: Grouping and Aggregating Data in Pandas
import pandas as pd
# Sample sales data
sales_data = {
'Region': ['North', 'South', 'East', 'West', 'North', 'South'],
'Salesperson': ['Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Frank'],
'Sales': [250, 150, 200, 300, 400, 100]
}
df = pd.DataFrame(sales_data)
# Group by 'Region' and sum the sales
region_sales = df.groupby('Region')['Sales'].sum()
print("Total sales per region:
", region_sales)
# Group by 'Region' and calculate multiple aggregations
agg_funcs = df.groupby('Region')['Sales'].agg(['sum', 'mean', 'max'])
print("
Aggregated sales stats by region:
", agg_funcs)
# Create a pivot table for Sales by Region and Salesperson
pivot = pd.pivot_table(df, values='Sales', index='Region', columns='Salesperson', fill_value=0)
print("
Pivot table:
", pivot)
Region
East 200
North 650
South 250
West 300
Name: Sales, dtype: int64
Aggregated sales stats by region:
sum mean max
Region
East 200 200.0 200
North 650 325.0 400
South 250 125.0 150
West 300 300.0 300
Pivot table:
Salesperson Alice Bob Charlie David Eva Frank
Region
East 0 0 200 0 0 0
North 250 0 0 0 400 0
South 0 150 0 0 0 100
West 0 0 0 300 0 0
📌 Deep Dive: Combining DataFrames with Merge and Concat
import pandas as pd
# DataFrame 1: Employee info
df1 = pd.DataFrame({
'EmployeeID': [1, 2, 3],
'Name': ['Alice', 'Bob', 'Charlie']
})
# DataFrame 2: Employee department
df2 = pd.DataFrame({
'EmployeeID': [2, 3, 4],
'Department': ['HR', 'IT', 'Finance']
})
# Merge on EmployeeID (inner join)
merged_inner = pd.merge(df1, df2, on='EmployeeID')
print("Inner Merge:
", merged_inner)
# Merge with outer join to keep all records
merged_outer = pd.merge(df1, df2, on='EmployeeID', how='outer')
print("
Outer Merge:
", merged_outer)
# Concatenate DataFrames vertically
df3 = pd.DataFrame({
'EmployeeID': [5],
'Name': ['David']
})
concatenated = pd.concat([df1, df3], ignore_index=True)
print("
Concatenated DataFrame:
", concatenated)
EmployeeID Name Department
0 2 Bob HR
1 3 Charlie IT
Outer Merge:
EmployeeID Name Department
0 1 Alice NaN
1 2 Bob HR
2 3 Charlie IT
3 4 NaN Finance
Concatenated DataFrame:
EmployeeID Name
0 1 Alice
1 2 Bob
2 3 Charlie
3 5 David
Quick Knowledge Check
Test what you just learned
Question 1 of 2
Which of the following statements about NumPy arrays and Pandas DataFrames is true?
Question 2 of 2
What does the groupby() function in Pandas primarily help you do?
Loading results...