content-parser

|

INSTALLATION
npx skills add https://github.com/marswaveai/skills --skill content-parser
Run in your project or agent environment. Adjust flags if your CLI version differs.

SKILL.md

When to Use

  • User provides a URL and wants to extract/read its content
  • Another skill needs to parse source material from a URL before generation
  • User says "parse this URL", "extract content from this link"
  • User says "解析链接", "提取内容"

When NOT to Use

  • User already has text content and doesn't need URL parsing
  • User wants to generate audio/video content (not content extraction)
  • User wants to read a local file (use standard file reading tools)

Purpose

Extract and normalize content from URLs across supported platforms. Returns structured data including content body, metadata, and references. Useful as a preprocessing step for content generation skills or standalone content extraction.

Hard Constraints

  • No shell scripts. Construct curl commands from the API Reference (Inlined) section below
  • See § API Reference (Inlined) below for API key and headers
  • See § API Reference (Inlined) below for polling, errors, and interaction patterns
  • URL must be a valid HTTP(S) URL
  • Always read config following shared/config-pattern.md before any interaction
  • Never save files to ~/Downloads/ or .listenhub/ — save to the current working directory

Step -1: API Key Check

Follow shared/config-pattern.md § API Key Check. If the key is missing, stop immediately.

Step 0: Config Setup

Follow shared/config-pattern.md Step 0 (Zero-Question Boot).

If file doesn't exist — silently create with defaults and proceed:

mkdir -p ".listenhub/content-parser"

echo '{"autoDownload":true}' > ".listenhub/content-parser/config.json"

CONFIG_PATH=".listenhub/content-parser/config.json"

CONFIG=$(cat "$CONFIG_PATH")

Do NOT ask any setup questions. Proceed directly to the Interaction Flow.

If file exists — read config silently and proceed:

CONFIG_PATH=".listenhub/content-parser/config.json"

[ ! -f "$CONFIG_PATH" ] && CONFIG_PATH="$HOME/.listenhub/content-parser/config.json"

CONFIG=$(cat "$CONFIG_PATH")

Setup Flow (user-initiated reconfigure only)

Only run when the user explicitly asks to reconfigure. Display current settings:

当前配置 (content-parser):

  自动下载:{是 / 否}

Then ask:

  • autoDownload: "自动保存提取的内容到当前目录?"
  • "是(推荐)" → autoDownload: true
  • "否" → autoDownload: false

Save immediately:

NEW_CONFIG=$(echo "$CONFIG" | jq --argjson dl {true/false} '. + {"autoDownload": $dl}')

echo "$NEW_CONFIG" > "$CONFIG_PATH"

CONFIG=$(cat "$CONFIG_PATH")

Interaction Flow

Step 1: URL Input

Free text input. Ask the user:

What URL would you like to extract content from?

Step 2: Options (optional)

Ask if the user wants to configure extraction options:

Question: "Do you want to configure extraction options?"

Options:

  - "No, use defaults" — Extract with default settings

  - "Yes, configure options" — Set summarize, maxLength, or Twitter tweet count

If "Yes", ask follow-up questions:

  • Summarize: "Generate a summary of the content?" (Yes/No)
  • Max Length: "Set maximum content length?" (Free text, e.g., "5000")
  • Twitter count (only if URL is Twitter/X profile): "How many tweets to fetch?" (1-100, default 20)

Step 3: Confirm & Extract

Summarize:

Ready to extract content:

  URL: {url}

  Options: {summarize: true, maxLength: 5000, twitter.count: 50} / default

  Proceed?

Wait for explicit confirmation before calling the API.

Workflow

-

Validate URL: Must be HTTP(S). Normalize if needed (see references/supported-platforms.md)

-

Build request body:

{

  "source": {

    "type": "url",

    "uri": "{url}"

  },

  "options": {

    "summarize": true/false,

    "maxLength": 5000,

    "twitter": {

      "count": 50

    }

  }

}

Omit options if user chose defaults.

-

Submit (foreground): POST /v1/content/extract → extract taskId

-

Tell the user extraction is in progress

-

Poll (background): Run the following exact bash command with run_in_background: true and timeout: 300000. Note: status field is .data.status (not processStatus), interval is 5s, values are processing/completed/failed:

TASK_ID="<id-from-step-3>"

for i in $(seq 1 60); do

  RESULT=$(curl -sS "https://api.marswave.ai/openapi/v1/content/extract/$TASK_ID" \

    -H "Authorization: Bearer $LISTENHUB_API_KEY" \

    -H "X-Source: skills" 2>/dev/null)

  STATUS=$(echo "$RESULT" | tr -d '\000-\037\177' | jq -r '.data.status // "processing"')

  case "$STATUS" in

    completed) echo "$RESULT"; exit 0 ;;

    failed) echo "FAILED: $RESULT" >&#x26;2; exit 1 ;;

    *) sleep 5 ;;

  esac

done

echo "TIMEOUT" >&#x26;2; exit 2

-

When notified, download and present result:

If autoDownload is true, generate a slug from the extracted title (falling back to domain name if no title). Follow shared/config-pattern.md § Artifact Naming for slug generation and dedup.

  • Write {slug}.md to the current directory — full extracted content in markdown
  • Write {slug}.json to the current directory — full raw API response data
SLUG="{title-slug}"  # e.g. "topology-wikipedia"

# Dedup: check if files exist

BASE="$SLUG"; i=2

while [ -e "${SLUG}.md" ] || [ -e "${SLUG}.json" ]; do SLUG="${BASE}-${i}"; i=$((i+1)); done

echo "$CONTENT_MD" > "${SLUG}.md"

echo "$RESULT" > "${SLUG}.json"

Present:

内容提取完成!

来源:{url}

标题:{metadata.title}

长度:~{character count} 字符

消耗积分:{credits}

已保存到当前目录:

  {slug}.md

  {slug}.json

-

Show a preview of the extracted content (first ~500 chars)

-

Offer to use content in another skill (e.g. /podcast, /tts)

Estimated time: 10-30 seconds depending on content size and platform.

API Reference (Inlined)

Authentication

Environment variable: LISTENHUB_API_KEY (format: lh_sk_...)

Store in ~/.zshrc (macOS) or ~/.bashrc (Linux):

export LISTENHUB_API_KEY="lh_sk_..."

How to obtain: Visit https://listenhub.ai/settings/api-keys (Pro plan required).

Base URL: https://api.marswave.ai/openapi/v1

Required headers (every request):

Authorization: Bearer $LISTENHUB_API_KEY

Content-Type: application/json

X-Source: skills

The X-Source: skills header identifies requests as coming from Claude Code skills (CLI tool).

curl template:

curl -sS -X POST "https://api.marswave.ai/openapi/v1/{endpoint}" \

  -H "Authorization: Bearer $LISTENHUB_API_KEY" \

  -H "Content-Type: application/json" \

  -H "X-Source: skills" \

  -d '{ ... }'

For GET requests, omit -d and change -X POST to -X GET.

Security notes:

  • Never log or display full API keys in output
  • API keys are transmitted via HTTPS only
  • Do not pass sensitive or confidential information as content input — it is sent to external APIs for processing

POST /v1/content/extract

Create a content extraction task for a URL. Returns a taskId for polling.

Request body:

Field

Required

Type

Description

source

Yes

object

Source to extract from

source.type

Yes

string

Must be "url"

source.uri

Yes

string

Valid HTTP(S) URL to extract content from

options

No

object

Extraction options

options.summarize

No

boolean

Whether to generate a summary

options.maxLength

No

integer

Maximum content length

options.twitter

No

object

Twitter/X specific options

options.twitter.count

No

integer

Number of tweets to fetch (1-100, default 20)

Response:

{

  "code": 0,

  "message": "success",

  "data": {

    "taskId": "69a7dac700cf95938f86d9bb"

  }

}

Error codes:

Code

Meaning

29003

Validation error ("source.uri" is required, "source.uri" must be a valid uri)

21007

Invalid API key

GET /v1/content/extract/{taskId}

Get extraction task status and results.

Path params:

Param

Type

Description

taskId

string

24-char hex task ID

Response states:

  • processing — Task is still running
  • completed — Extraction finished, data available
  • failed — Extraction failed, check failCode and message

Response (processing):

{

  "code": 0,

  "message": "success",

  "data": {

    "taskId": "69a7dac700cf95938f86d9bb",

    "status": "processing",

    "createdAt": "2025-04-09T12:00:00Z",

    "data": null,

    "credits": 0,

    "failCode": null,

    "message": null

  }

}

Response (completed):

{

  "code": 0,

  "message": "success",

  "data": {

    "taskId": "69a7dac700cf95938f86d9bb",

    "status": "completed",

    "createdAt": "2025-04-09T12:00:00Z",

    "data": {

      "content": "Extracted text content...",

      "metadata": {

        "title": "Article Title",

        "author": "Author Name",

        "publishedAt": "2025-04-01T08:00:00Z"

      },

      "references": [

        "https://example.com/related-article"

      ]

    },

    "credits": 5,

    "failCode": null,

    "message": null

  }

}

Response (failed):

{

  "code": 0,

  "message": "success",

  "data": {

    "taskId": "69a7dac700cf95938f86d9bb",

    "status": "failed",

    "createdAt": "2025-04-09T12:00:00Z",

    "data": null,

    "credits": 0,

    "failCode": "EXTRACT_FAILED",

    "message": "Unable to extract content from the provided URL"

  }

}

Key fields:

Field

Type

Description

status

string

processing, completed, or failed

data.data.content

string

Extracted text content

data.data.metadata

object

Page metadata (title, author, publishedAt)

data.data.references

array

Referenced URLs (array of strings)

credits

integer

Credits consumed

failCode

string

Error code (null on success)

message

string

Error message (null on success)

Error codes:

Code

Meaning

29003

Invalid taskId format

25002

Task not found

Polling Pattern

5-second interval, 60 polls max. Run with run_in_background: true and timeout: 300000.

Two-step pattern:

  • Submit (foreground): POST the creation request, extract taskId from the response.
  • Poll (background): Run the polling loop with run_in_background: true. You will be notified automatically when it completes.

The exact polling bash command is already specified in the Workflow section (Step 5).

Error Handling

HTTP status codes:

Code

Meaning

Action

200

Success

Parse response body

400

Bad request

Check parameters

401

Invalid API key

Re-check LISTENHUB_API_KEY

402

Insufficient credits

Inform user to recharge

403

Forbidden

No permission for this resource

429

Rate limited

Exponential backoff, retry after delay

500/502/503/504

Server error

Retry up to 3 times

Retry strategy:

  • 429 rate limit: Wait 15 seconds, then retry (exponential backoff)
  • 5xx server errors: Retry up to 3 times with 5-second intervals
  • Network errors: Retry up to 3 times

Application error codes:

Code

Meaning

21007

Invalid user API key

25429

Rate limited (application-level)

Example

User: "Parse this article: https://en.wikipedia.org/wiki/Topology"

Agent workflow:

  • URL: https://en.wikipedia.org/wiki/Topology
  • Options: defaults (omit options)
  • Submit extraction
curl -sS -X POST "https://api.marswave.ai/openapi/v1/content/extract" \

  -H "Authorization: Bearer $LISTENHUB_API_KEY" \

  -H "Content-Type: application/json" \

  -H "X-Source: skills" \

  -d '{

    "source": {

      "type": "url",

      "uri": "https://en.wikipedia.org/wiki/Topology"

    }

  }'
  • Poll until complete:
curl -sS "https://api.marswave.ai/openapi/v1/content/extract/69a7dac700cf95938f86d9bb" \

  -H "Authorization: Bearer $LISTENHUB_API_KEY" \

  -H "X-Source: skills"
  • Present extracted content preview and offer next actions.

User: "Extract recent tweets from @elonmusk, get 50 tweets"

Agent workflow:

  • URL: https://x.com/elonmusk
  • Options: {"twitter": {"count": 50}}
  • Submit extraction
curl -sS -X POST "https://api.marswave.ai/openapi/v1/content/extract" \

  -H "Authorization: Bearer $LISTENHUB_API_KEY" \

  -H "Content-Type: application/json" \

  -H "X-Source: skills" \

  -d '{

    "source": {

      "type": "url",

      "uri": "https://x.com/elonmusk"

    },

    "options": {

      "twitter": {

        "count": 50

      }

    }

  }'
  • Poll until complete, present results.
BrowserAct

Let your agent run on any real-world website

Bypass CAPTCHA & anti-bot for free. Start local, scale to cloud.

Explore BrowserAct Skills →

Stop writing automation&scrapers

Install the CLI. Run your first Skill in 30 seconds. Scale when you're ready.

Start free
free · no credit card