Data Cleaning and Variable Screening

Quick Start

# Run the complete data cleaning pipeline

python ".github/skills/datanalysis-credit-risk/scripts/example.py"

Complete Process Description

The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:

• Get Data - Load and format raw data

• Organization Sample Analysis - Statistics of sample count and bad sample rate for each organization

• Separate OOS Data - Separate out-of-sample (OOS) samples from modeling samples

• Filter Abnormal Months - Remove months with insufficient bad sample count or total sample count

• Calculate Missing Rate - Calculate overall and organization-level missing rates for each feature

• Drop High Missing Rate Features - Remove features with overall missing rate exceeding threshold

• Drop Low IV Features - Remove features with overall IV too low or IV too low in too many organizations

• Drop High PSI Features - Remove features with unstable PSI

• Null Importance Denoising - Remove noise features using label permutation method

• Drop High Correlation Features - Remove high correlation features based on original gain

• Export Report - Generate Excel report containing details and statistics of all steps

Core Functions

Function

Purpose

Module

get_dataset()

Load and format data

references.func

org_analysis()

Organization sample analysis

references.func

missing_check()

Calculate missing rate

references.func

drop_abnormal_ym()

Filter abnormal months

references.analysis

drop_highmiss_features()

Drop high missing rate features

references.analysis

drop_lowiv_features()

Drop low IV features

references.analysis

drop_highpsi_features()

Drop high PSI features

references.analysis

drop_highnoise_features()

Null Importance denoising

references.analysis

drop_highcorr_features()

Drop high correlation features

references.analysis

iv_distribution_by_org()

IV distribution statistics

references.analysis

psi_distribution_by_org()

PSI distribution statistics

references.analysis

value_ratio_distribution_by_org()

Value ratio distribution statistics

references.analysis

export_cleaning_report()

Export cleaning report

references.analysis

Parameter Description

Data Loading Parameters

• DATA_PATH: Data file path (best are parquet format)

• DATE_COL: Date column name

• Y_COL: Label column name

• ORG_COL: Organization column name

• KEY_COLS: Primary key column name list

OOS Organization Configuration

• OOS_ORGS: Out-of-sample organization list

Abnormal Month Filtering Parameters

• min_ym_bad_sample: Minimum bad sample count per month (default 10)

• min_ym_sample: Minimum total sample count per month (default 500)

Missing Rate Parameters

• missing_ratio: Overall missing rate threshold (default 0.6)

IV Parameters

• overall_iv_threshold: Overall IV threshold (default 0.1)

• org_iv_threshold: Single organization IV threshold (default 0.1)

• max_org_threshold: Maximum tolerated low IV organization count (default 2)

PSI Parameters

• psi_threshold: PSI threshold (default 0.1)

• max_months_ratio: Maximum unstable month ratio (default 1/3)

• max_orgs: Maximum unstable organization count (default 6)

Null Importance Parameters

• n_estimators: Number of trees (default 100)

• max_depth: Maximum tree depth (default 5)

• gain_threshold: Gain difference threshold (default 50)

High Correlation Parameters

• max_corr: Correlation threshold (default 0.9)

• top_n_keep: Keep top N features by original gain ranking (default 20)

Output Report

The generated Excel report contains the following sheets:

• 汇总 - Summary information of all steps, including operation results and conditions

• 机构样本统计 - Sample count and bad sample rate for each organization

• 分离OOS数据 - OOS sample and modeling sample counts

• Step4-异常月份处理 - Abnormal months that were removed

• 缺失率明细 - Overall and organization-level missing rates for each feature

• Step5-有值率分布统计 - Distribution of features in different value ratio ranges

• Step6-高缺失率处理 - High missing rate features that were removed

• Step7-IV明细 - IV values of each feature in each organization and overall

• Step7-IV处理 - Features that do not meet IV conditions and low IV organizations

• Step7-IV分布统计 - Distribution of features in different IV ranges

• Step8-PSI明细 - PSI values of each feature in each organization each month

• Step8-PSI处理 - Features that do not meet PSI conditions and unstable organizations

• Step8-PSI分布统计 - Distribution of features in different PSI ranges

• Step9-null importance处理 - Noise features that were removed

• Step10-高相关性剔除 - High correlation features that were removed

Features

• Interactive Input: Parameters can be input before each step execution, with default values supported

• Independent Execution: Each step is executed independently without deleting original data, facilitating comparative analysis

• Complete Report: Generate complete Excel report containing details, statistics, and distributions

• Multi-process Support: IV and PSI calculations support multi-process acceleration

• Organization-level Analysis: Support organization-level statistics and modeling/OOS distinction