Extract Skill

Extract clean content from specific URLs. Ideal when you know which pages you want content from.

Authentication

The script uses OAuth via the Tavily MCP server. No manual setup required - on first run, it will:

• Check for existing tokens in ~/.mcp-auth/

• If none found, automatically open your browser for OAuth authentication

Note: You must have an existing Tavily account. The OAuth flow only supports login — account creation is not available through this flow. Sign up at tavily.com first if you don't have an account.

Alternative: API Key

If you prefer using an API key, get one at https://tavily.com and add to ~/.claude/settings.json:

{

  "env": {

    "TAVILY_API_KEY": "tvly-your-api-key-here"

  }

}

Quick Start

Using the Script

./scripts/extract.sh '<json>'

Examples:

# Single URL

./scripts/extract.sh '{"urls": ["https://example.com/article"]}'

# Multiple URLs

./scripts/extract.sh '{"urls": ["https://example.com/page1", "https://example.com/page2"]}'

# With query focus and chunks

./scripts/extract.sh '{"urls": ["https://example.com/docs"], "query": "authentication API", "chunks_per_source": 3}'

# Advanced extraction for JS pages

./scripts/extract.sh '{"urls": ["https://app.example.com"], "extract_depth": "advanced", "timeout": 60}'

Basic Extraction

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": ["https://example.com/article"]

  }'

Multiple URLs with Query Focus

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": [

      "https://example.com/ml-healthcare",

      "https://example.com/ai-diagnostics"

    ],

    "query": "AI diagnostic tools accuracy",

    "chunks_per_source": 3

  }'

API Reference

Endpoint

POST https://api.tavily.com/extract

Headers

Header

Value

Authorization

Bearer <TAVILY_API_KEY>

Content-Type

application/json

Request Body

Field

Type

Default

Description

urls

array

Required

URLs to extract (max 20)

query

string

null

Reranks chunks by relevance

chunks_per_source

integer

3

Chunks per URL (1-5, requires query)

extract_depth

string

"basic"

basic or advanced (for JS pages)

format

string

"markdown"

markdown or text

include_images

boolean

false

Include image URLs

timeout

float

varies

Max wait (1-60 seconds)

Response Format

{

  "results": [

    {

      "url": "https://example.com/article",

      "raw_content": "# Article Title\n\nContent..."

    }

  ],

  "failed_results": [],

  "response_time": 2.3

}

Extract Depth

Depth

When to Use

basic

Simple text extraction, faster

advanced

Dynamic/JS-rendered pages, tables, structured data

Examples

Single URL Extraction

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": ["https://docs.python.org/3/tutorial/classes.html"],

    "extract_depth": "basic"

  }'

Targeted Extraction with Query

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": [

      "https://example.com/react-hooks",

      "https://example.com/react-state"

    ],

    "query": "useState and useEffect patterns",

    "chunks_per_source": 2

  }'

JavaScript-Heavy Pages

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": ["https://app.example.com/dashboard"],

    "extract_depth": "advanced",

    "timeout": 60

  }'

Batch Extraction

curl --request POST \

  --url https://api.tavily.com/extract \

  --header "Authorization: Bearer $TAVILY_API_KEY" \

  --header 'Content-Type: application/json' \

  --data '{

    "urls": [

      "https://example.com/page1",

      "https://example.com/page2",

      "https://example.com/page3",

      "https://example.com/page4",

      "https://example.com/page5"

    ],

    "extract_depth": "basic"

  }'

Tips

• Max 20 URLs per request - batch larger lists

• **Use query + chunks_per_source** to get only relevant content

• **Try basic first**, fall back to advanced if content is missing

• **Set longer timeout** for slow pages (up to 60s)

• **Check failed_results** for URLs that couldn't be extracted