GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

• Single developer projects

• Small to medium codebases

• Simple setup without external dependencies

• Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

• Vector embeddings

• File metadata

• Chunk information

Everything is stored in a single local file.

Advantages

Benefit

Description

🚀 Simple

No external services needed

⚡ Fast setup

Works immediately

📁 Portable

Single file, easy to backup

💰 Free

No infrastructure costs

🔒 Private

Data stays local

Limitations

Limitation

Description

📏 Scalability

Not ideal for very large codebases

👤 Single user

No concurrent access

🔄 No sharing

Can't share index across machines

💾 Memory

Loads into RAM for searches

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yaml

store:

  backend: gob

Explicit Configuration

store:

  backend: gob

  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/

├── config.yaml    # Configuration

├── index.gob      # Vector embeddings

└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Codebase

Files

Chunks

Index Size

Small

100

500

~5 MB

Medium

1,000

5,000

~50 MB

Large

10,000

50,000

~500 MB

Operations

Creating the Index

# Initialize project

grepai init

# Start indexing (creates index.gob)

grepai watch

Checking Index Status

grepai status

# Output:

# Index: .grepai/index.gob

# Files: 245

# Chunks: 1,234

# Size: 12.5 MB

# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copy

cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-index

rm .grepai/index.gob

grepai watch

Moving to a New Machine

# Copy entire .grepai directory

cp -r .grepai /path/to/new/location/

# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Index Size

RAM Usage

10 MB

~20 MB

50 MB

~100 MB

500 MB

~1 GB

Search Speed

GOB provides fast searches for typical codebases:

Codebase Size

Search Time

Small (100 files)

<50ms

Medium (1K files)

<200ms

Large (10K files)

<1s

When to Upgrade

Consider PostgreSQL or Qdrant when:

• Index exceeds 1 GB

• Need concurrent access

• Want to share index across team

• Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index)

.grepai/

Why: The index is machine-specific because:

• Contains binary embeddings

• Tied to the embedding model used

• Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

• Must use identical embedding model

• File paths are absolute

• Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

• Update config:

store:

  backend: postgres

  postgres:

    dsn: postgres://user:pass@localhost:5432/grepai

• Re-index:

rm .grepai/index.gob

grepai watch

To Qdrant

• Update config:

store:

  backend: qdrant

  qdrant:

    endpoint: localhost

    port: 6334

• Re-index:

rm .grepai/index.gob

grepai watch

Common Issues

❌ Problem: Index file too large

✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase

✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index

✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob

grepai watch

❌ Problem: "Index not found" error

✅ Solution: Run grepai watch to create the index

Best Practices

• Use for small/medium projects: Up to ~10K files

• Add to .gitignore: Don't commit the index

• Backup before major changes: Copy index.gob before experiments

• Re-index after model changes: If you change embedding models

• Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

   Backend: GOB (local file)

   Index: .grepai/index.gob

   Size: 12.5 MB

   Contents:

   - Files: 245

   - Chunks: 1,234

   - Vectors: 1,234 × 768 dimensions

   Performance:

   - Search latency: <100ms

   - Memory usage: ~25 MB