NumPy & Pandas Basics

In the world of data science and scientific computing, two Python libraries stand out as foundational tools: NumPy and Pandas. Both libraries serve distinct but complementary purposes. NumPy (Numerical Python) provides support for large, multi-dimensional arrays and matrices, along with a vast collection of mathematical functions to operate on these arrays efficiently. Pandas, on the other hand, is designed for data manipulation and analysis, offering data structures like Series and DataFrames that make handling labeled and heterogeneous data intuitive and powerful.

This lesson dives deep into the basics of NumPy and Pandas, exploring their core data structures, essential functions, and typical operations you’ll frequently use in data processing workflows. By mastering these, you’ll be well-equipped to handle complex datasets, perform vectorized computations, and prepare your data for machine learning, visualization, or reporting.

We will start by understanding NumPy arrays, their creation, indexing, and broadcasting capabilities, then transition to Pandas Series and DataFrames, focusing on data selection, filtering, aggregation, and merging. Throughout, we emphasize best practices and common pitfalls to avoid, ensuring your code is both efficient and readable.

💡 A Simple Analogy: NumPy Arrays as Excel Worksheets, Pandas DataFrames as Excel Tables

Think of NumPy arrays as the underlying Excel worksheet grid — a grid of cells where every cell is the same type and can be processed very fast. Meanwhile, Pandas DataFrames are like Excel tables where you have labeled rows and columns, with different data types in each column, and powerful tools to filter, sort, and aggregate the data. NumPy provides the computational engine, and Pandas provides the user-friendly interface for data analysis.

🎯 Real-World Use Case: Financial Time Series Analysis

Imagine you need to analyze stock prices over time. NumPy can efficiently handle numerical computations such as calculating moving averages or volatility over large arrays of price data. Pandas complements this by allowing you to handle timestamps, label your data by date, easily slice time periods, handle missing data, and compute statistics grouped by different labels such as sectors or stock exchanges.

⚠️ Common Pitfall: Confusing NumPy Arrays and Pandas DataFrames

While both libraries use similar terminology (arrays, indexing), they behave differently. NumPy arrays are homogeneous and support vectorized operations but lack built-in labeling. Pandas DataFrames support heterogeneous data types and indexing by labels but sometimes sacrifice raw performance. Mixing these up can lead to bugs or inefficient code. Always choose the right tool based on the task: NumPy for numerical computations, Pandas for data manipulation.

1

Understanding NumPy Arrays — Learn how to create arrays, check their dimensions, and manipulate them using slicing, reshaping, and broadcasting.

2

Exploring Pandas Series — Discover the one-dimensional labeled array, its construction from lists or NumPy arrays, and how to access and manipulate data with labels.

3

Diving into Pandas DataFrames — Understand multi-dimensional tabular data, creation from dictionaries or CSV files, indexing with loc and iloc, and basic data cleaning operations.

4

Data Aggregation and Grouping — Master groupby operations, aggregation functions, and pivot tables to summarize data effectively.

5

Combining and Merging DataFrames — Learn how to join, concatenate, and merge datasets based on keys or indices to build comprehensive data views.

Architecture of NumPy & Pandas Basics
Architecture of NumPy & Pandas Basics

📌 Deep Dive: Creating and Manipulating NumPy Arrays

PYTHON

# Import NumPy library
import numpy as np

# Create a 1D NumPy array from a Python list
arr = np.array([10, 20, 30, 40, 50])
print("Original array:", arr)

# Create a 2D array (matrix) of shape (3, 3)
matrix = np.array([[1, 2, 3],
                   [4, 5, 6],
                   [7, 8, 9]])
print("2D Matrix:
", matrix)

# Access elements: element at row 1, column 2
elem = matrix[1, 2]
print("Element at (1, 2):", elem)

# Slicing: get first two rows and last two columns
slice_sub = matrix[:2, 1:]
print("Sliced submatrix:
", slice_sub)

# Reshape array: flatten matrix into 1D array
flat = matrix.reshape(-1)
print("Flattened array:", flat)

# Broadcasting: add a scalar to entire array
broadcasted = matrix + 10
print("After adding 10 to each element:
", broadcasted)

# Element-wise multiplication
multiplied = matrix * matrix
print("Element-wise squared matrix:
", multiplied)
    
Output
Original array: [10 20 30 40 50]
2D Matrix:
[[1 2 3]
[4 5 6]
[7 8 9]]
Element at (1, 2): 6
Sliced submatrix:
[[2 3]
[5 6]]
Flattened array: [1 2 3 4 5 6 7 8 9]
After adding 10 to each element:
[[11 12 13]
[14 15 16]
[17 18 19]]
Element-wise squared matrix:
[[ 1 4 9]
[16 25 36]
[49 64 81]]

📌 Deep Dive: Pandas Series and DataFrame Basics

PYTHON

import pandas as pd

# Create a Pandas Series from a list with custom indices
data = [100, 200, 300, 400]
index_labels = ['a', 'b', 'c', 'd']
series = pd.Series(data, index=index_labels)
print("Series:
", series)

# Access elements by label and position
print("Element with label 'b':", series['b'])
print("Element at position 2:", series.iloc[2])

# Create a DataFrame from a dictionary of lists
data_dict = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'Salary': [70000, 80000, 90000]
}
df = pd.DataFrame(data_dict)
print("
DataFrame:
", df)

# Select a column (returns a Series)
ages = df['Age']
print("
Ages column:
", ages)

# Select multiple columns (returns a DataFrame)
subset = df[['Name', 'Salary']]
print("
Subset DataFrame:
", subset)

# Filter rows where Age > 28
filtered = df[df['Age'] > 28]
print("
Filtered DataFrame (Age > 28):
", filtered)

# Using loc and iloc for selection
print("
Select row with label 1 using loc:
", df.loc[1])
print("
Select first row using iloc:
", df.iloc[0])
    
Output
Series:
a 100
b 200
c 300
d 400
dtype: int64
Element with label 'b': 200
Element at position 2: 300

DataFrame:
Name Age Salary
0 Alice 25 70000
1 Bob 30 80000
2 Charlie 35 90000

Ages column:
0 25
1 30
2 35
Name: Age, dtype: int64

Subset DataFrame:
Name Salary
0 Alice 70000
1 Bob 80000
2 Charlie 90000

Filtered DataFrame (Age > 28):
Name Age Salary
1 Bob 30 80000
2 Charlie 35 90000

Select row with label 1 using loc:
Name Bob
Age 30
Salary 80000
Name: 1, dtype: object

Select first row using iloc:
Name Alice
Age 25
Salary 70000
Name: 0, dtype: object

📌 Deep Dive: Grouping and Aggregating Data in Pandas

PYTHON

import pandas as pd

# Sample sales data
sales_data = {
    'Region': ['North', 'South', 'East', 'West', 'North', 'South'],
    'Salesperson': ['Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Frank'],
    'Sales': [250, 150, 200, 300, 400, 100]
}
df = pd.DataFrame(sales_data)

# Group by 'Region' and sum the sales
region_sales = df.groupby('Region')['Sales'].sum()
print("Total sales per region:
", region_sales)

# Group by 'Region' and calculate multiple aggregations
agg_funcs = df.groupby('Region')['Sales'].agg(['sum', 'mean', 'max'])
print("
Aggregated sales stats by region:
", agg_funcs)

# Create a pivot table for Sales by Region and Salesperson
pivot = pd.pivot_table(df, values='Sales', index='Region', columns='Salesperson', fill_value=0)
print("
Pivot table:
", pivot)
    
Output
Total sales per region:
Region
East 200
North 650
South 250
West 300
Name: Sales, dtype: int64

Aggregated sales stats by region:
sum mean max
Region
East 200 200.0 200
North 650 325.0 400
South 250 125.0 150
West 300 300.0 300

Pivot table:
Salesperson Alice Bob Charlie David Eva Frank
Region
East 0 0 200 0 0 0
North 250 0 0 0 400 0
South 0 150 0 0 0 100
West 0 0 0 300 0 0

📌 Deep Dive: Combining DataFrames with Merge and Concat

PYTHON

import pandas as pd

# DataFrame 1: Employee info
df1 = pd.DataFrame({
    'EmployeeID': [1, 2, 3],
    'Name': ['Alice', 'Bob', 'Charlie']
})

# DataFrame 2: Employee department
df2 = pd.DataFrame({
    'EmployeeID': [2, 3, 4],
    'Department': ['HR', 'IT', 'Finance']
})

# Merge on EmployeeID (inner join)
merged_inner = pd.merge(df1, df2, on='EmployeeID')
print("Inner Merge:
", merged_inner)

# Merge with outer join to keep all records
merged_outer = pd.merge(df1, df2, on='EmployeeID', how='outer')
print("
Outer Merge:
", merged_outer)

# Concatenate DataFrames vertically
df3 = pd.DataFrame({
    'EmployeeID': [5],
    'Name': ['David']
})
concatenated = pd.concat([df1, df3], ignore_index=True)
print("
Concatenated DataFrame:
", concatenated)
    
Output
Inner Merge:
EmployeeID Name Department
0 2 Bob HR
1 3 Charlie IT

Outer Merge:
EmployeeID Name Department
0 1 Alice NaN
1 2 Bob HR
2 3 Charlie IT
3 4 NaN Finance

Concatenated DataFrame:
EmployeeID Name
0 1 Alice
1 2 Bob
2 3 Charlie
3 5 David