AI Video Analysis

Automatically analyze video content with advanced AI-powered visual recognition. Extract descriptive tags, generate summaries, and detect locations from your videos using state-of-the-art machine learning models.

Overview

VideoCascade provides intelligent video analysis powered by advanced AI vision models. When enabled, the system extracts key frames from your video and analyzes them to provide:

Tags: Array of 5 descriptive keywords that best represent the video content
Summary: Concise 2-3 sentence description of what's shown in the video
Location: Estimated geographical location based on visual cues

This feature is perfect for automatic content tagging, searchability, content moderation, and metadata enrichment.

How It Works

The vision analysis process works in three steps:

Frame Extraction: Extracts 3 frames at 10%, 50%, and 90% of video duration
- Skips first and last 10% to avoid fade in/out effects
- Frames are resized to max 1024px width for optimal processing
- Each frame is converted to base64 for API transmission
AI Analysis: Processes frames using advanced vision AI
- Analyzes all frames together for comprehensive understanding
- Identifies key objects, scenes, activities, and contexts
- Estimates location from architecture, landmarks, signs, and cultural elements
Results: Returns structured analysis data
- 5 most relevant tags (1-2 words each)
- 2-3 sentence summary
- Estimated location (or "unknown" if uncertain)
- AI cost based on token usage

Frame Selection: The system intelligently extracts frames from 10%, 50%, and 90% positions to capture the full video narrative while avoiding opening/closing transitions.

Enabling Vision Analysis

Set enableAiAnalysis: true in your video processing request:

curl -X POST https://api.videocascade.com/v1/videos \
  -H "Authorization: Bearer vca_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "fileUrl": "https://example.com/video.mp4",
    "enableAiAnalysis": true
  }'

const response = await fetch('https://api.videocascade.com/v1/videos', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer vca_your_api_key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    fileUrl: 'https://example.com/video.mp4',
    enableAiAnalysis: true,
  }),
});

const data = await response.json();
console.log(`Video ID: ${data.videoId}`);

import requests

response = requests.post(
    'https://api.videocascade.com/v1/videos',
    headers={
        'Authorization': 'Bearer vca_your_api_key',
        'Content-Type': 'application/json',
    },
    json={
        'fileUrl': 'https://example.com/video.mp4',
        'enableAiAnalysis': True,
    }
)

data = response.json()
print(f"Video ID: {data['videoId']}")

interface VideoRequest {
  fileUrl: string;
  enableAiAnalysis?: boolean;
}

const request: VideoRequest = {
fileUrl: 'https://example.com/video.mp4',
enableAiAnalysis: true,
};

const response = await fetch('https://api.videocascade.com/v1/videos', {
method: 'POST',
headers: {
'Authorization': 'Bearer vca_your_api_key',
'Content-Type': 'application/json',
},
body: JSON.stringify(request),
});

const data = await response.json();
console.log(`Video ID: ${data.videoId}`);

Response Structure

The analysis results are included in the video object under the analysis field:

interface AnalysisData {
  tags: string[];          // Array of 5 descriptive keywords
  summary: string;         // 2-3 sentence description
  estimatedLocation: string;  // Location or "unknown"
}

interface VideoAnalysis {
  status: 'not_requested' | 'pending' | 'processing' | 'completed' | 'failed';
  progress?: number;       // 0-100 progress percentage
  error?: string;          // Error message if failed
  data?: AnalysisData;     // Analysis results when completed
}

interface VideoResponse {
  videoId: string;
  status: 'queued' | 'running' | 'succeeded' | 'failed';
  progressPercent?: number;
  analysis?: VideoAnalysis;
  // ... other fields
}

{
  "videoId": "v_abc12345",
  "status": "succeeded",
  "progressPercent": 100,
  "finalVideoUrl": "https://storage.example.com/videos/final.mp4",
  "analysis": {
    "status": "completed",
    "progress": 100,
    "data": {
      "tags": [
        "temple",
        "night",
        "asia",
        "architecture",
        "outdoor"
      ],
      "summary": "The video shows a beautifully lit temple complex at night with traditional Asian architecture. The golden illuminated structures feature intricate details and ornate rooflines. The scene captures the peaceful nighttime atmosphere of this cultural landmark.",
      "estimatedLocation": "Bangkok, Thailand"
    }
  },
  "enableAiAnalysis": true,
  "createdAt": "2025-11-23T10:30:00Z",
  "lastUpdatedAt": "2025-11-23T10:31:45Z"
}

Analysis Status States

Status	Description
`not_requested`	Analysis was not enabled for this video
`pending`	Analysis queued but not started yet
`processing`	Currently analyzing frames
`completed`	Analysis finished successfully, `data` field populated
`failed`	Analysis failed, `error` field contains details

Checking Analysis Results

Poll the video status endpoint to check analysis progress:

// Poll for analysis completion
async function waitForAnalysis(videoId) {
  while (true) {
    const response = await fetch(
      `https://api.videocascade.com/v1/videos/${videoId}`,
      {
        headers: {
          'Authorization': 'Bearer vca_your_api_key',
        },
      }
    );

    const video = await response.json();

    if (video.analysis?.status === 'completed') {
      console.log('Analysis complete!');
      console.log('Tags:', video.analysis.data.tags);
      console.log('Summary:', video.analysis.data.summary);
      console.log('Location:', video.analysis.data.estimatedLocation);
      break;
    } else if (video.analysis?.status === 'failed') {
      console.error('Analysis failed:', video.analysis.error);
      break;
    }

    console.log(`Analysis ${video.analysis?.progress || 0}% complete...`);
    await new Promise(resolve => setTimeout(resolve, 5000)); // Wait 5s

}
}

await waitForAnalysis('v_abc12345');

import time

def wait_for_analysis(video_id):
    """Poll for analysis completion"""
    while True:
        response = requests.get(
            f'https://api.videocascade.com/v1/videos/{video_id}',
            headers={'Authorization': 'Bearer vca_your_api_key'}
        )

        video = response.json()

        if video.get('analysis', {}).get('status') == 'completed':
            print('Analysis complete!')
            data = video['analysis']['data']
            print(f"Tags: {data['tags']}")
            print(f"Summary: {data['summary']}")
            print(f"Location: {data['estimatedLocation']}")
            break
        elif video.get('analysis', {}).get('status') == 'failed':
            print(f"Analysis failed: {video['analysis'].get('error')}")
            break

        progress = video.get('analysis', {}).get('progress', 0)
        print(f"Analysis {progress}% complete...")
        time.sleep(5)  # Wait 5 seconds

wait_for_analysis('v_abc12345')

Use Cases

Automatic Content Tagging

Generate searchable tags for video libraries:

// Process uploaded video with automatic tagging
const response = await fetch('https://api.videocascade.com/v1/videos', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer vca_your_api_key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    fileUrl: 'https://example.com/user-upload.mp4',
    enableAiAnalysis: true,
    webhookUrl: 'https://yourapp.com/webhooks/video-analyzed',
  }),
});

// Webhook receives:
// {
//   "event": "video.completed",
//   "videoId": "v_abc12345",
//   "analysis": {
//     "tags": ["travel", "beach", "sunset", "ocean", "vacation"]
//   }
// }

// Store tags in database for search
await db.videos.update({
  id: videoId,
  tags: analysis.data.tags,
  searchableText: analysis.data.summary,
});

Content Moderation

Pre-screen uploaded videos for inappropriate content:

// Analyze video before publishing
const response = await fetch('https://api.videocascade.com/v1/videos', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer vca_your_api_key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    fileUrl: userUploadedUrl,
    enableAiAnalysis: true,
  }),
});

// Check tags for inappropriate content
const { analysis } = await getVideoStatus(videoId);
const inappropriateTags = ['violence', 'explicit', 'nsfw'];
const hasInappropriate = analysis.data.tags.some(tag =>
  inappropriateTags.some(bad => tag.includes(bad))
);

if (hasInappropriate) {
  await flagForReview(videoId);
}

Location-Based Organization

Organize travel videos by location:

// Process travel video
const response = await fetch('https://api.videocascade.com/v1/videos', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer vca_your_api_key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    fileUrl: 'https://example.com/travel-vlog.mp4',
    enableAiAnalysis: true,
  }),
});

// Get location from analysis
const { analysis } = await getVideoStatus(videoId);
const location = analysis.data.estimatedLocation;

// Group by location
await db.videos.update({
  id: videoId,
  location: location !== 'unknown' ? location : null,
  tags: analysis.data.tags,
});

// Query videos by location
const bangkokVideos = await db.videos.findMany({
  where: { location: { contains: 'Bangkok' } },
});

Enhanced Search & Discovery

Make videos searchable with AI-generated metadata:

// Build search index from analysis
async function indexVideo(videoId) {
  const { analysis } = await getVideoStatus(videoId);

  await searchEngine.index({
    id: videoId,
    tags: analysis.data.tags,
    summary: analysis.data.summary,
    location: analysis.data.estimatedLocation,
    // Combine for full-text search
    searchText: [
      ...analysis.data.tags,
      analysis.data.summary,
      analysis.data.estimatedLocation,
    ].join(' '),
  });
}

// Users can now search:
// - "temple bangkok" -> finds videos tagged with temple in Bangkok
// - "beach sunset" -> finds beach videos at sunset
// - "architecture" -> finds videos with architectural content

Combining with Other Features

Vision analysis works alongside other video processing features:

// Process video with AI analysis and transcription
const response = await fetch('https://api.videocascade.com/v1/videos', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer vca_your_api_key',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    fileUrl: 'https://example.com/video.mp4',
    enableAiAnalysis: true,      // Visual analysis
    enableTranscription: true,   // Speech-to-text
    enableThumbnail: true,       // Generate thumbnail
    aspectRatio: '16:9',         // Format for YouTube
    normalizeAudio: true,        // Clean audio
    compressionQuality: 95,      // High quality
    webhookUrl: 'https://yourapp.com/webhooks/complete'
  }),
});

response = requests.post(
    'https://api.videocascade.com/v1/videos',
    headers={
        'Authorization': 'Bearer vca_your_api_key',
        'Content-Type': 'application/json',
    },
    json={
        'fileUrl': 'https://example.com/video.mp4',
        'enableAiAnalysis': True,      # Visual analysis
        'enableTranscription': True,   # Speech-to-text
        'enableThumbnail': True,       # Generate thumbnail
        'aspectRatio': '16:9',         # Format for YouTube
        'normalizeAudio': True,        # Clean audio
        'compressionQuality': 95,      # High quality
        'webhookUrl': 'https://yourapp.com/webhooks/complete'
    }
)

Best Practices

1. Use Webhooks for Async Processing

Don't poll - use webhooks for efficient notification:

// ✅ Good: Use webhook
{
  fileUrl: 'https://example.com/video.mp4',
  enableAiAnalysis: true,
  webhookUrl: 'https://yourapp.com/webhooks/analysis-complete'
}

// ❌ Less efficient: Poll every few seconds
while (status !== 'completed') {
  await sleep(5000);
  status = await checkStatus(videoId);
}

2. Batch Process During Off-Peak Hours

For large video libraries, analyze during off-peak times:

// Process videos in batches
const videos = await db.videos.findMany({
  where: { analyzed: false },
  limit: 100,
});

for (const video of videos) {
  await fetch('https://api.videocascade.com/v1/videos', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer vca_your_api_key',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      fileUrl: video.url,
      enableAiAnalysis: true,
    }),
  });

  // Rate limit: wait between requests
  await sleep(1000);
}

3. Cache Analysis Results

Store analysis data to avoid re-processing:

// Check cache before analyzing
const cached = await redis.get(`analysis:${videoId}`);
if (cached) {
  return JSON.parse(cached);
}

// Analyze and cache
const response = await analyzeVideo(videoUrl);
await redis.set(
  `analysis:${videoId}`,
  JSON.stringify(response.analysis.data),
  'EX',
  86400 * 30 // Cache 30 days
);

4. Validate Video Quality First

Ensure video is suitable for analysis:

// Check video metadata before analysis
const metadata = await getVideoMetadata(videoUrl);

if (metadata.duration < 1) {
  throw new Error('Video too short for meaningful analysis');
}

if (metadata.width < 320 || metadata.height < 240) {
  throw new Error('Video resolution too low for analysis');
}

// Proceed with analysis
await analyzeVideo(videoUrl);

5. Handle Analysis Failures Gracefully

Always implement fallbacks:

const { analysis } = await getVideoStatus(videoId);

if (analysis.status === 'failed') {
  console.error('Analysis failed:', analysis.error);

  // Fallback: use filename or user-provided tags
  const fallbackTags = extractTagsFromFilename(video.filename);
  await db.videos.update({
    id: videoId,
    tags: fallbackTags,
    analyzed: false,
  });
}

Limitations

Video Duration

Analysis works on videos of any length
Extracts 3 frames regardless of duration
Longer videos not more expensive (same 3 frames)

Frame Quality

Frames resized to max 1024px width
Higher resolution videos provide better analysis
Very low-resolution videos (< 320px) may have limited accuracy

Content Types

Best suited for:

Outdoor scenes with landmarks
Videos with clear objects and subjects
Content with visible text/signs
Videos with distinct architectural features

Limited accuracy for:

Abstract or artistic content
Very dark or poorly lit scenes
Heavily edited or filtered videos
Pure animation or CGI without real-world elements

Location Detection

Location is estimated, not guaranteed
Requires visual cues (landmarks, architecture, signs, cultural elements)
Generic indoor scenes may return "unknown"
Best for outdoor content with identifiable features

Error Handling

Handle common error scenarios:

async function analyzeVideoSafely(videoUrl) {
  try {
    const response = await fetch('https://api.videocascade.com/v1/videos', {
      method: 'POST',
      headers: {
        'Authorization': 'Bearer vca_your_api_key',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        fileUrl: videoUrl,
        enableAiAnalysis: true,
      }),
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status}`);
    }

    const data = await response.json();
    const videoId = data.videoId;

    // Wait for analysis
    const result = await waitForAnalysis(videoId);

    if (result.analysis.status === 'failed') {
      console.error('Analysis failed:', result.analysis.error);
      return {
        success: false,
        error: result.analysis.error,
        tags: [],  // Empty fallback
        summary: '',
        location: 'unknown'
      };
    }

    return {
      success: true,
      tags: result.analysis.data.tags,
      summary: result.analysis.data.summary,
      location: result.analysis.data.estimatedLocation
    };

} catch (error) {
console.error('Error analyzing video:', error);
return {
success: false,
error: error.message,
tags: [],
summary: '',
location: 'unknown'
};
}
}

def analyze_video_safely(video_url):
    """Analyze video with error handling"""
    try:
        response = requests.post(
            'https://api.videocascade.com/v1/videos',
            headers={
                'Authorization': 'Bearer vca_your_api_key',
                'Content-Type': 'application/json',
            },
            json={
                'fileUrl': video_url,
                'enableAiAnalysis': True,
            }
        )

        response.raise_for_status()
        data = response.json()
        video_id = data['videoId']

        # Wait for analysis
        result = wait_for_analysis(video_id)

        if result.get('analysis', {}).get('status') == 'failed':
            return {
                'success': False,
                'error': result['analysis'].get('error'),
                'tags': [],
                'summary': '',
                'location': 'unknown'
            }

        analysis_data = result['analysis']['data']
        return {
            'success': True,
            'tags': analysis_data['tags'],
            'summary': analysis_data['summary'],
            'location': analysis_data['estimatedLocation']
        }

    except Exception as error:
        print(f"Error analyzing video: {error}")
        return {
            'success': False,
            'error': str(error),
            'tags': [],
            'summary': '',
            'location': 'unknown'
        }