SKILL.md

Exploratory Data Analysis (EDA)

Name: exploratory-data-analysis
Author: aj-geddes

Overview

Exploratory Data Analysis (EDA) is the critical first step in data science projects, systematically examining datasets to understand their characteristics, identify patterns, and assess data quality before formal modeling.

Core Concepts

Data Profiling: Understanding basic statistics and data types

Distribution Analysis: Examining how variables are distributed

Relationship Discovery: Identifying patterns between variables

Anomaly Detection: Finding outliers and unusual patterns

Data Quality Assessment: Evaluating completeness and consistency

When to Use

Starting a new dataset analysis

Understanding data before modeling

Identifying data quality issues

Generating hypotheses for testing

Communicating insights to stakeholders

Implementation with Python

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

# Load and explore data

df = pd.read_csv('customer_data.csv')

# Basic profiling

print(f"Shape: {df.shape}")

print(f"Data types:\n{df.dtypes}")

print(f"Missing values:\n{df.isnull().sum()}")

print(f"Duplicates: {df.duplicated().sum()}")

# Statistical summary

print(df.describe())

print(df.describe(include='object'))

# Distribution analysis - numerical columns

fig, axes = plt.subplots(2, 2, figsize=(12, 8))

df['age'].hist(bins=30, ax=axes[0, 0])

axes[0, 0].set_title('Age Distribution')

df['income'].hist(bins=30, ax=axes[0, 1])

axes[0, 1].set_title('Income Distribution')

# Box plots for outlier detection

df.boxplot(column='age', by='region', ax=axes[1, 0])

axes[1, 0].set_title('Age by Region')

# Categorical analysis

df['category'].value_counts().plot(kind='bar', ax=axes[1, 1])

axes[1, 1].set_title('Category Distribution')

plt.tight_layout()

plt.show()

# Correlation analysis

numeric_df = df.select_dtypes(include=[np.number])

correlation_matrix = numeric_df.corr()

plt.figure(figsize=(10, 8))

sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)

plt.title('Correlation Matrix')

plt.show()

# Multivariate relationships

sns.pairplot(df[['age', 'income', 'education_years']], diag_kind='hist')

plt.show()

# Skewness and kurtosis

print("\nSkewness:")

print(numeric_df.skew())

print("\nKurtosis:")

print(numeric_df.kurtosis())

# Percentile analysis

print("\nPercentiles for Age:")

print(df['age'].quantile([0.25, 0.5, 0.75, 0.95, 0.99]))

# Missing data patterns

missing_pct = (df.isnull().sum() / len(df) * 100)

missing_pct[missing_pct > 0].sort_values(ascending=False)

# Value count analysis

print("\nCustomer Types Distribution:")

print(df['customer_type'].value_counts(normalize=True))

# Advanced EDA: Groupby analysis

print("\nGroupBy Analysis:")

print(df.groupby('region')[['age', 'income']].agg(['mean', 'median', 'std']))

# Correlation with target variable

if 'target' in df.columns:

    target_corr = df.corr()['target'].sort_values(ascending=False)

    print("\nFeature Correlation with Target:")

    print(target_corr)

# Data type breakdown

print("\nData Type Summary:")

print(df.dtypes.value_counts())

# Unique value count

print("\nUnique Value Counts:")

print(df.nunique().sort_values(ascending=False))

# Variance analysis

print("\nVariance per Feature:")

numeric_cols = df.select_dtypes(include=[np.number]).columns

for col in numeric_cols:

    variance = df[col].var()

    print(f"  {col}: {variance:.2f}")

# Distribution patterns

for col in df.select_dtypes(include=[np.number]).columns:

    skew = df[col].skew()

    kurt = df[col].kurtosis()

    print(f"{col} - Skew: {skew:.2f}, Kurtosis: {kurt:.2f}")

# Bivariate analysis

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

df.groupby('region')['income'].mean().plot(kind='bar', ax=axes[0])

axes[0].set_title('Average Income by Region')

df.groupby('category')['age'].mean().plot(kind='bar', ax=axes[1])

axes[1].set_title('Average Age by Category')

plt.tight_layout()

plt.show()

# Summary statistics profile

print("\nComprehensive Data Profile:")

profile = {

    'Variable': df.columns,

    'Type': df.dtypes,

    'Non-Null Count': df.count(),

    'Null Count': df.isnull().sum(),

    'Unique Values': df.nunique(),

}

profile_df = pd.DataFrame(profile)

print(profile_df)

Advanced EDA Techniques

# Step 15: Interaction analysis

import itertools

numeric_cols = df.select_dtypes(include=[np.number]).columns

interaction_strengths = []

for col1, col2 in itertools.combinations(numeric_cols[:5], 2):

    interaction_score = abs(df[col1].corr(df[col2]))

    interaction_strengths.append({

        'Pair': f"{col1} × {col2}",

        'Correlation': interaction_score,

    })

interaction_df = pd.DataFrame(interaction_strengths).sort_values('Correlation', ascending=False)

print("\nTop Interactions:")

print(interaction_df.head())

# Step 16: Outlier summary

for col in numeric_cols:

    Q1, Q3 = df[col].quantile([0.25, 0.75])

    IQR = Q3 - Q1

    outliers = df[(df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)]

    if len(outliers) > 0:

        print(f"\n{col}: {len(outliers)} outliers detected ({len(outliers)/len(df)*100:.1f}%)")

# Step 17: Generate automated insights

print("\n" + "="*60)

print("AUTOMATED DATA INSIGHTS")

print("="*60)

for col in numeric_cols:

    skewness = df[col].skew()

    mean_val = df[col].mean()

    median_val = df[col].median()

    if abs(skewness) > 1:

        direction = "right" if skewness > 0 else "left"

        print(f"{col}: Highly {direction}-skewed distribution")

    if abs(mean_val - median_val) > 0.1 * median_val:

        print(f"{col}: Mean and median differ significantly")

print("="*60)

Key Questions to Ask

What are the data dimensions and types?

How are key variables distributed?

What patterns exist between variables?

Are there obvious data quality issues?

What outliers or anomalies exist?

What hypotheses can we generate?

Best Practices

Start with data profiling before visualization

Check data types and missing values early

Visualize distributions before jumping to analysis

Document interesting findings and anomalies

Create summaries for stakeholder communication

Use domain knowledge to interpret patterns

Common Pitfalls

Skipping data quality checks

Over-interpreting patterns in small datasets

Ignoring domain context

Insufficient data visualization

Not documenting findings systematically

Deliverables

Data quality report with missing values and duplicates

Summary statistics and distribution charts

Correlation and relationship visualizations

List of notable patterns and anomalies

Hypotheses for further investigation

Data cleaning recommendations

exploratory-data-analysis