FluentD Integration

Overview

FluentD can forward logs to Qorrelate using the HTTP output plugin. This allows you to aggregate logs from various sources and send them to Qorrelate for analysis.

Install HTTP Plugin

gem install fluent-plugin-out-http

FluentD Configuration

# fluent.conf
<source>
  @type forward
  port 24224
</source>

<source>
  @type tail
  path /var/log/app/*.log
  pos_file /var/log/fluentd/app.log.pos
  tag app.logs
  <parse>
    @type json
  </parse>
</source>

<match **>
  @type http
  endpoint https://api.qorrelate.io/v1/logs
  http_method post
  content_type application/json
  headers {"X-API-Key": "YOUR_API_KEY"}
  <format>
    @type json
  </format>
  <buffer>
    flush_interval 5s
    chunk_limit_size 1m
    retry_max_times 5
  </buffer>
</match>

Docker Compose Example

version: '3.8'
services:
  fluentd:
    image: fluent/fluentd:v1.16
    volumes:
      - ./fluent.conf:/fluentd/etc/fluent.conf
      - /var/log:/var/log:ro
    environment:
      - QORRELATE_API_KEY=your-api-key
    ports:
      - "24224:24224"

Kubernetes DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.16
        env:
        - name: QORRELATE_API_KEY
          valueFrom:
            secretKeyRef:
              name: qorrelate-credentials
              key: api-key
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: config
          mountPath: /fluentd/etc
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: fluentd-config