AI Scanner (0din-ai/ai-scanner)

Skill by ara.so — Daily 2026 Skills collection.

AI Scanner is an open-source Ruby on Rails web application for AI model security assessments, wrapping NVIDIA garak with a multi-tenant UI, scheduling, PDF reports, and SIEM integration. It runs 179 community probes across 35 vulnerability families aligned with the OWASP LLM Top 10.

Installation

Quick Install (Docker)

curl -sL https://raw.githubusercontent.com/0din-ai/ai-scanner/main/scripts/install.sh | bash

Manual Install

curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/dist/docker-compose.yml

curl -O https://raw.githubusercontent.com/0din-ai/ai-scanner/main/.env.example

cp .env.example .env

Edit .env with required values:

# Generate a secure key

openssl rand -hex 64

# .env minimum required values

SECRET_KEY_BASE=<output_of_above_command>

POSTGRES_PASSWORD=<your_secure_db_password>

docker compose up -d

Access at http://localhost — default credentials: admin@example.com / password.

Change the default password immediately after first login.

Configuration (.env)

# Required

SECRET_KEY_BASE=<64-byte-hex>

POSTGRES_PASSWORD=<strong-password>

# Optional: custom port

PORT=8080

# Optional: SIEM integration

SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector

SPLUNK_HEC_TOKEN=$SPLUNK_HEC_TOKEN

RSYSLOG_HOST=syslog.example.com

RSYSLOG_PORT=514

# Optional: email

SMTP_HOST=smtp.example.com

SMTP_PORT=587

SMTP_USERNAME=$SMTP_USERNAME

SMTP_PASSWORD=$SMTP_PASSWORD

Core Concepts

Concept

Description

Target

An AI system to test — API-based LLM or browser-based chat UI

Probe

A single attack test (e.g., prompt injection, data leakage)

Scan

A run of selected probes against a target

ASR

Attack Success Rate — percentage of probes that succeeded

Organization

Tenant boundary; users and scans are scoped per org

Setting Up a Target

Targets define what you're scanning. Two types:

API-based LLM Target (e.g., OpenAI-compatible endpoint):

# In Rails console or via UI — representative model

target = Target.create!(

  name: "Production GPT-4",

  target_type: "api",

  api_endpoint: "https://api.openai.com/v1/chat/completions",

  api_key: ENV["OPENAI_API_KEY"],

  model_name: "gpt-4",

  organization: current_organization

)

Browser-based Chat UI Target:

target = Target.create!(

  name: "Internal Chatbot UI",

  target_type: "browser",

  url: "https://chatbot.internal.example.com",

  organization: current_organization

)

Running a Scan

Via UI

• Navigate to Targets → select your target

• Click New Scan

• Select probe families or individual probes

• Click Run Scan

Via Rails Console

# On-demand scan with specific probe families

scan = Scan.create!(

  target: target,

  probe_families: ["prompt_injection", "data_leakage", "insecure_output"],

  organization: current_organization

)

ScanJob.perform_later(scan.id)

Scheduled Recurring Scan

# Weekly scan every Monday at 2am

scheduled_scan = ScheduledScan.create!(

  target: target,

  probe_families: ["prompt_injection", "jailbreak"],

  cron_expression: "0 2 * * 1",

  organization: current_organization

)

Probe Families (35 total, aligned to OWASP LLM Top 10)

Key probe families available:

# List all available probe families

Garak::ProbeRegistry.families

# => ["prompt_injection", "jailbreak", "data_leakage", "insecure_output",

#     "supply_chain", "sensitive_info", "excessive_agency", "overreliance",

#     "model_theft", "malicious_plugins", ...]

# Get probes within a family

Garak::ProbeRegistry.probes_for("prompt_injection")

# => 179 total probes across all families

Viewing Results

Attack Success Rate (ASR)

scan = Scan.find(scan_id)

puts scan.asr_score          # => 0.23 (23% attack success rate)

puts scan.status             # => "completed"

puts scan.probe_results.count # => 47

# Per-probe breakdown

scan.probe_results.each do |result|

  puts "#{result.probe_name}: #{result.passed? ? 'SAFE' : 'VULNERABLE'}"

  puts "  Attempts: #{result.attempt_count}"

  puts "  ASR: #{result.asr_score}"

end

Trend Tracking

# Compare ASR across scan runs for a target

target.scans.completed.order(:created_at).map do |scan|

  { date: scan.created_at, asr: scan.asr_score }

end

PDF Report Export

# Generate PDF report for a scan

scan = Scan.find(scan_id)

pdf_path = ReportExporter.export_pdf(scan)

# Includes: executive summary, per-probe results, per-attempt drill-down

Via UI: Navigate to a completed scan → Export PDF.

SIEM Integration

Splunk

# config/initializers/siem.rb

SiemIntegration.configure do |config|

  config.provider = :splunk

  config.splunk_hec_url = ENV["SPLUNK_HEC_URL"]

  config.splunk_hec_token = ENV["SPLUNK_HEC_TOKEN"]

  config.forward_on_completion = true

end

Rsyslog

SiemIntegration.configure do |config|

  config.provider = :rsyslog

  config.rsyslog_host = ENV["RSYSLOG_HOST"]

  config.rsyslog_port = ENV["RSYSLOG_PORT"].to_i

end

Multi-Tenant Organization Management

# Create a new organization

org = Organization.create!(name: "Security Team Alpha")

# Invite a user

user = User.invite!(

  email: "analyst@example.com",

  organization: org,

  role: "analyst"   # roles: "admin", "analyst", "viewer"

)

# Data is encrypted at rest per organization

org.encryption_key  # => managed automatically

Development Setup

git clone https://github.com/0din-ai/ai-scanner.git

cd ai-scanner

cp .env.example .env.development

# Install dependencies

bundle install

# Database setup

rails db:create db:migrate db:seed

# Install garak (Python dependency)

pip install garak

# Start development server

bin/dev

Running Tests

# Full test suite

bundle exec rspec

# Specific area

bundle exec rspec spec/models/scan_spec.rb

bundle exec rspec spec/jobs/scan_job_spec.rb

# Lint

bundle exec rubocop

Common Patterns

Testing a New LLM Before Deployment

# Comprehensive pre-deployment scan

target = Target.create!(

  name: "New Model v2 - Pre-deploy",

  target_type: "api",

  api_endpoint: ENV["NEW_MODEL_ENDPOINT"],

  api_key: ENV["NEW_MODEL_API_KEY"],

  model_name: "new-model-v2",

  organization: current_organization

)

# Run all 35 probe families

scan = Scan.create!(

  target: target,

  probe_families: Garak::ProbeRegistry.families,

  organization: current_organization

)

ScanJob.perform_now(scan.id)

if scan.reload.asr_score > 0.15

  puts "WARNING: ASR #{scan.asr_score} exceeds threshold. Review before deploying."

else

  puts "PASS: Model meets security threshold."

end

Using the Mock LLM for Testing Scanner Setup

The built-in Mock LLM lets you validate your scanner configuration without hitting real APIs:

target = Target.create!(

  name: "Mock LLM",

  target_type: "mock",

  organization: current_organization

)

# Run a quick scan to verify everything works end-to-end

Webhook on Scan Completion

# config/initializers/scan_hooks.rb

ActiveSupport::Notifications.subscribe("scan.completed") do |_, _, _, _, payload|

  scan = Scan.find(payload[:scan_id])

  if scan.asr_score > 0.20

    SlackNotifier.alert(

      channel: "#security-alerts",

      message: "High ASR detected: #{scan.asr_score} on #{scan.target.name}"

    )

  end

end

Docker Compose Production Tips

# docker-compose.override.yml — production additions

services:

  web:

    environment:

      RAILS_ENV: production

      FORCE_SSL: "true"

    labels:

      - "traefik.enable=true"

      - "traefik.http.routers.scanner.rule=Host(`scanner.example.com`)"

      - "traefik.http.routers.scanner.tls.certresolver=letsencrypt"

# Upgrade

docker compose pull

docker compose up -d

docker compose exec web rails db:migrate

Troubleshooting

Problem

Solution

Scan stuck in "running"

Check docker compose logs worker — garak Python process may have crashed

SECRET_KEY_BASE error on start

Run openssl rand -hex 64 and set in .env

Can't connect to target API

Verify API key env var is set; check firewall allows outbound from container

Browser target scan fails

Ensure Playwright/Chrome is available in the worker container

PDF export blank

Check wkhtmltopdf is installed in the web container

SIEM not receiving events

Verify SPLUNK_HEC_URL includes full path /services/collector

# View all service logs

docker compose logs -f

# Check worker specifically (runs garak)

docker compose logs -f worker

# Rails console for debugging

docker compose exec web rails console

# Check garak is working

docker compose exec worker python -c "import garak; print(garak.__version__)"

Key Files (for Contributors)

app/

  models/

    scan.rb          # Core scan model, ASR calculation

    target.rb        # Target types and validation

    probe_result.rb  # Per-probe result storage

  jobs/

    scan_job.rb      # Async job that invokes garak

  services/

    garak_runner.rb  # Ruby wrapper around garak CLI

    report_exporter.rb

    siem_integration.rb

lib/

  garak/

    probe_registry.rb  # 179 probes, 35 families

dist/

  docker-compose.yml   # Production compose file

scripts/

  install.sh           # One-line installer

Resources

• Full Documentation

• Quick Start Guide

• NVIDIA garak

• OWASP LLM Top 10

• CONTRIBUTING.md

• SECURITY.md