spark-engineer

$2a

Topic

Reference

Load When

Spark SQL & DataFrames

references/spark-sql-dataframes.md

DataFrame API, Spark SQL, schemas, joins, aggregations

RDD Operations

references/rdd-operations.md

Transformations, actions, pair RDDs, custom partitioners

Partitioning & Caching

references/partitioning-caching.md

Data partitioning, persistence levels, broadcast variables

Performance Tuning

references/performance-tuning.md

Configuration, memory tuning, shuffle optimization, skew handling

Streaming Patterns

references/streaming-patterns.md

Structured Streaming, watermarks, stateful operations, sinks

Code Examples

Quick-Start Mini-Pipeline (PySpark)

from pyspark.sql import SparkSession

from pyspark.sql import functions as F

from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType

spark = SparkSession.builder \

    .appName("example-pipeline") \

    .config("spark.sql.shuffle.partitions", "400") \

    .config("spark.sql.adaptive.enabled", "true") \

    .getOrCreate()

# Always define explicit schemas in production

schema = StructType([

    StructField("user_id", StringType(), False),

    StructField("event_ts", LongType(), False),

    StructField("amount", DoubleType(), True),

])

df = spark.read.schema(schema).parquet("s3://bucket/events/")

result = df \

    .filter(F.col("amount").isNotNull()) \

    .groupBy("user_id") \

    .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))

# Verify partition count before writing

print(f"Partition count: {result.rdd.getNumPartitions()}")

result.write.mode("overwrite").parquet("s3://bucket/output/")

Broadcast Join (small dimension table < 200 MB)

from pyspark.sql.functions import broadcast

# Spark will automatically broadcast dim_table; hint makes intent explicit

enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")

Handling Data Skew with Salting

import pyspark.sql.functions as F

SALT_BUCKETS = 50

# Add salt to the skewed key on both sides

skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \

    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \

    .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))

result = skewed_df.join(other_df, on="salted_key", how="inner") \

    .drop("salt", "salted_key")

Correct Caching Pattern

# Cache ONLY when the DataFrame is reused multiple times

df_cleaned = df.filter(...).withColumn(...).cache()

df_cleaned.count()  # Materialize immediately; check Spark UI for spill

report_a = df_cleaned.groupBy("region").agg(...)

report_b = df_cleaned.groupBy("product").agg(...)

df_cleaned.unpersist()  # Release when done

Constraints

MUST DO

• Use DataFrame API over RDD for structured data processing

• Define explicit schemas for production pipelines

• Partition data appropriately (200-1000 partitions per executor core)

• Cache intermediate results only when reused multiple times

• Use broadcast joins for small dimension tables (<200MB)

• Handle data skew with salting or custom partitioning

• Monitor Spark UI for shuffle, spill, and GC metrics

• Test with production-scale data volumes

MUST NOT DO

• Use collect() on large datasets (causes OOM)

• Skip schema definition and rely on inference in production

• Cache every DataFrame without measuring benefit

• Ignore shuffle partition tuning (default 200 often wrong)

• Use UDFs when built-in functions available (10-100x slower)

• Process small files without coalescing (small file problem)

• Run transformations without understanding lazy evaluation

• Ignore data skew warnings in Spark UI

Output Templates

When implementing Spark solutions, provide:

• Complete Spark code (PySpark or Scala) with type hints/types

• Configuration recommendations (executors, memory, shuffle partitions)

• Partitioning strategy explanation

• Performance analysis (expected shuffle size, memory usage)

• Monitoring recommendations (key Spark UI metrics to watch)

Knowledge Reference

Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization

Documentation