Azure Blob Storage Integration

Overview

The Azure Blob Storage integration allows you to collect logs stored in Azure Blob Storage containers and forward them to Qorrelate for analysis and monitoring. This is useful for analyzing application logs, access logs, and other data stored in blob containers.

Prerequisites

• An Azure subscription with Blob Storage access
• Storage account connection string or SAS token
• Container name where logs are stored
• Your Qorrelate API endpoint and organization ID

1. Create an Azure Function for Log Forwarding

Create an Azure Function that triggers on new blobs and forwards logs to Qorrelate:

import logging
import azure.functions as func
import requests
import json
from datetime import datetime

def main(myblob: func.InputStream):
    logging.info(f"Processing blob: {myblob.name}")
    
    content = myblob.read().decode('utf-8')
    logs = []
    for line in content.split('\n'):
        if line.strip():
            logs.append({
                "timestamp": datetime.utcnow().isoformat() + "Z",
                "body": line,
                "attributes": {
                    "source": "azure-blob-storage",
                    "blob_name": myblob.name
                }
            })
    
    headers = {
        "Content-Type": "application/json",
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORG_ID"
    }
    
    response = requests.post(
        "https://qorrelate.io/v1/logs",
        headers=headers,
        json={"logs": logs}
    )
    
    logging.info(f"Forwarded {len(logs)} logs, status: {response.status_code}")

2. Configure Blob Trigger

Update your function.json to trigger on new blobs:

{
  "bindings": [
    {
      "name": "myblob",
      "type": "blobTrigger",
      "direction": "in",
      "path": "logs/{name}",
      "connection": "AzureWebJobsStorage"
    }
  ]
}

3. Set Environment Variables

# In Azure Portal or local.settings.json
QORRELATE_API_KEY=your_api_key_here
QORRELATE_ORG_ID=your_organization_id
QORRELATE_ENDPOINT=https://qorrelate.io

Alternative: Using Azure Event Grid

For more reliable log forwarding, you can use Azure Event Grid to trigger on blob events:

1. Create an Event Grid subscription for your storage account
2. Filter for Microsoft.Storage.BlobCreated events
3. Point the webhook endpoint to your forwarding function
4. Configure retry policies for reliability

Verifying the Integration

1. Upload a test log file to your blob container
2. Check your Azure Function logs for successful execution
3. Navigate to the Logs page in Qorrelate
4. Filter by source:azure-blob-storage to see your logs