GCP Cloud Logging Integration

Overview

Google Cloud Logging (formerly Stackdriver Logging) collects logs from GCP services and applications. This integration uses log sinks to forward logs to Qorrelate, enabling unified observability across your multi-cloud infrastructure.

Prerequisites

• GCP project with Cloud Logging enabled
• Permissions to create log sinks and Pub/Sub topics
• Service account credentials for Pub/Sub subscription
• Your Qorrelate API endpoint and organization ID

1. Create a Pub/Sub Topic

Create a Pub/Sub topic to receive log entries:

# Create topic
gcloud pubsub topics create qorrelate-logs

# Create subscription
gcloud pubsub subscriptions create qorrelate-logs-sub \
  --topic=qorrelate-logs \
  --ack-deadline=60

2. Create a Log Sink

Create a sink to route logs to Pub/Sub:

# Create sink for all logs
gcloud logging sinks create qorrelate-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT/topics/qorrelate-logs \
  --log-filter='severity>=INFO'

# Or create sink for specific logs
gcloud logging sinks create qorrelate-app-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT/topics/qorrelate-logs \
  --log-filter='resource.type="gce_instance" OR resource.type="gke_container"'

# Grant publish permissions to the sink service account
gcloud pubsub topics add-iam-policy-binding qorrelate-logs \
  --member="serviceAccount:SERVICE_ACCOUNT_FROM_SINK" \
  --role="roles/pubsub.publisher"

3. Deploy Cloud Function Forwarder

Create a Cloud Function to forward logs to Qorrelate:

import base64
import json
import os
import requests
from datetime import datetime

def forward_to_qorrelate(event, context):
    """Cloud Function triggered by Pub/Sub."""
    
    # Decode the Pub/Sub message
    pubsub_message = base64.b64decode(event['data']).decode('utf-8')
    log_entry = json.loads(pubsub_message)
    
    # Map GCP severity to standard levels
    severity_map = {
        'DEFAULT': 'INFO',
        'DEBUG': 'DEBUG',
        'INFO': 'INFO',
        'NOTICE': 'INFO',
        'WARNING': 'WARN',
        'ERROR': 'ERROR',
        'CRITICAL': 'FATAL',
        'ALERT': 'FATAL',
        'EMERGENCY': 'FATAL'
    }
    
    # Extract relevant fields
    qorrelate_log = {
        "timestamp": log_entry.get('timestamp', datetime.utcnow().isoformat() + 'Z'),
        "body": log_entry.get('textPayload') or json.dumps(log_entry.get('jsonPayload', {})),
        "severity_text": severity_map.get(log_entry.get('severity', 'INFO'), 'INFO'),
        "attributes": {
            "source": "gcp-cloud-logging",
            "log_name": log_entry.get('logName'),
            "resource_type": log_entry.get('resource', {}).get('type'),
            "resource_labels": json.dumps(log_entry.get('resource', {}).get('labels', {})),
            "project_id": log_entry.get('resource', {}).get('labels', {}).get('project_id'),
            "insert_id": log_entry.get('insertId'),
            "trace": log_entry.get('trace'),
            "span_id": log_entry.get('spanId')
        }
    }
    
    # Forward to Qorrelate
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ['QORRELATE_API_KEY']}",
        "X-Organization-Id": os.environ["QORRELATE_ORG_ID"]
    }
    
    response = requests.post(
        f"{os.environ['QORRELATE_ENDPOINT']}/v1/logs",
        headers=headers,
        json={"logs": [qorrelate_log]}
    )
    
    print(f"Forwarded log, status: {response.status_code}")

Deploy the Function

# Create requirements.txt
echo "requests==2.31.0" > requirements.txt

# Deploy function
gcloud functions deploy forward-to-qorrelate \
  --runtime python311 \
  --trigger-topic qorrelate-logs \
  --entry-point forward_to_qorrelate \
  --set-env-vars QORRELATE_API_KEY=your_key,QORRELATE_ORG_ID=your_org,QORRELATE_ENDPOINT=https://qorrelate.io

Log Filter Examples

Customize which logs to forward using filters:

# Only error logs
severity>=ERROR

# GKE container logs
resource.type="k8s_container"

# Specific service
resource.labels.service_name="my-service"

# Cloud Run logs
resource.type="cloud_run_revision"

# Cloud Functions logs  
resource.type="cloud_function"

# Audit logs
logName:"cloudaudit.googleapis.com"

# Combine filters
resource.type="gce_instance" AND severity>=WARNING

Verifying the Integration

1. Create the Pub/Sub topic and subscription
2. Create the log sink with appropriate filter
3. Grant publish permissions to the sink service account
4. Deploy the Cloud Function
5. Generate some logs (e.g., make API calls, deploy an app)
6. View logs in Qorrelate filtered by source:gcp-cloud-logging