Pandas Best Practices

Expert guidelines for Pandas development, focusing on data manipulation, analysis, and efficient DataFrame operations.

Code Style and Structure

• Write concise, technical responses with accurate Python examples

• Prioritize reproducibility in data analysis workflows

• Use functional programming; avoid unnecessary classes

• Prefer vectorized operations over explicit loops

• Use descriptive variable names reflecting data content

• Follow PEP 8 style guidelines

DataFrame Creation and I/O

• Use pd.read_csv(), pd.read_excel(), pd.read_json() with appropriate parameters

• Specify dtype parameter to ensure correct data types on load

• Use parse_dates for automatic datetime parsing

• Set index_col when the data has a natural index column

• Use chunksize for reading large files incrementally

Data Selection

• Use .loc[] for label-based indexing

• Use .iloc[] for integer position-based indexing

• Avoid chained indexing (e.g., df['col'][0]) - use .loc or .iloc instead

• Use boolean indexing for conditional selection: df[df['col'] > value]

• Use .query() method for complex filtering conditions

Method Chaining

• Prefer method chaining for data transformations when possible

• Use .pipe() for applying custom functions in a chain

• Chain operations like .assign(), .query(), .groupby(), .agg()

• Keep chains readable by breaking across multiple lines

Data Cleaning and Validation

Missing Data

• Check for missing data with .isna() and .info()

• Handle missing data appropriately: .fillna(), .dropna(), or imputation

• Use pd.NA for nullable integer and boolean types

• Document decisions about missing data handling

Data Quality Checks

• Implement data quality checks at the beginning of analysis

• Validate data types with .dtypes and convert as needed

• Check for duplicates with .duplicated() and handle appropriately

• Use .describe() for quick statistical overview

Type Conversion

• Use .astype() for explicit type conversion

• Use pd.to_datetime() for date parsing

• Use pd.to_numeric() with errors='coerce' for safe numeric conversion

• Utilize categorical data types for low-cardinality string columns

Grouping and Aggregation

GroupBy Operations

• Use .groupby() for efficient aggregation operations

• Specify aggregation functions with .agg() for multiple operations

• Use named aggregation for clearer output column names

• Consider .transform() for broadcasting results back to original shape

Pivot Tables and Reshaping

• Use .pivot_table() for multi-dimensional aggregation

• Use .melt() to convert wide to long format

• Use .pivot() to convert long to wide format

• Use .stack() and .unstack() for hierarchical index manipulation

Performance Optimization

Memory Efficiency

• Use categorical data types for low-cardinality strings

• Downcast numeric types when appropriate

• Use pd.eval() and .eval() for large expression evaluation

Computation Speed

• Use vectorized operations instead of .apply() with row-wise functions

• Prefer built-in aggregation functions over custom ones

• Use .values or .to_numpy() for NumPy operations when faster

Avoiding Common Pitfalls

• Avoid iterating with .iterrows() - use vectorized operations

• Don't modify DataFrames while iterating

• Be aware of SettingWithCopyWarning - use .copy() when needed

• Avoid growing DataFrames row by row - collect in list and create once

Time Series Operations

• Use DatetimeIndex for time series data

• Leverage .resample() for time-based aggregation

• Use .shift() and .diff() for lag operations

• Use .rolling() and .expanding() for window calculations

Merging and Joining

• Use .merge() for SQL-style joins

• Specify how parameter: 'inner', 'outer', 'left', 'right'

• Use validate parameter to check join cardinality

• Use .concat() for stacking DataFrames

Key Conventions

• Import as import pandas as pd

• Use snake_case for column names when possible

• Document data sources and transformations

• Keep notebooks reproducible with clear cell execution order