# Qorrelate Platform - LLM Instrumentation Guide
# Last Updated: 2025-12-17
# Purpose: Guide AI assistants in instrumenting applications with Qorrelate observability

---

## PLATFORM OVERVIEW

Qorrelate is a modern observability platform built on ClickHouse and OpenTelemetry.
- Production Website: https://qorrelate.io
- Staging: https://wartime.live
- Features: Distributed tracing, log management, metrics monitoring, session replay
- Key Advantage: Zero-code auto-instrumentation for most languages
- Technology Stack: ClickHouse (storage), OpenTelemetry (collection), FastAPI (backend)

---

## QUICK START - RECOMMENDED APPROACH

The fastest way to instrument ANY application is using the Qorrelate CLI:

```bash
# 1. Install CLI
curl -sL https://install.qorrelate.io | sh

# 2. Initialize project (user provides API key from qorrelate.io)
cd /path/to/project
qorrelate init --token <API_KEY>

# 3. Run application with auto-instrumentation
qorrelate run python app.py        # Python
qorrelate run node server.js       # Node.js
qorrelate run java -jar app.jar    # Java
qorrelate run go run main.go       # Go
```

This approach requires ZERO code changes and works for:
- Python (Flask, Django, FastAPI, Celery)
- Node.js (Express, NestJS, Next.js, Koa)
- Java (Spring Boot, Quarkus, Micronaut)
- Go (Gin, Echo, Fiber, Chi)
- .NET (ASP.NET Core)
- Ruby (Rails, Sinatra, Grape)

---

## INSTRUMENTATION BY LANGUAGE

### PYTHON

#### Method 1: CLI (Recommended - Zero Code)
```bash
qorrelate run python app.py
```

#### Method 2: Manual OpenTelemetry
```python
# Install dependencies
# pip install opentelemetry-distro opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.flask import FlaskInstrumentor  # or django, fastapi

# Setup
resource = Resource.create({"service.name": "my-service"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(
    OTLPSpanExporter(
        endpoint="https://qorrelate.io/v1/traces",
        headers={"X-API-Key": "<API_KEY>"}
    )
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Auto-instrument Flask (similar for Django, FastAPI)
from flask import Flask
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
```

#### Logging Integration
```python
import logging
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter

# Setup log exporter
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint="https://qorrelate.io/v1/logs",
            headers={"X-API-Key": "<API_KEY>"}
        )
    )
)

# Attach to Python logging
handler = LoggingHandler(logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)
```

### NODE.JS

#### Method 1: CLI (Recommended - Zero Code)
```bash
qorrelate run node server.js
```

#### Method 2: Manual OpenTelemetry
```javascript
// Install: npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http

// instrumentation.js (load BEFORE your app code)
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'my-service',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'https://qorrelate.io/v1/traces',
    headers: { 'X-API-Key': '<API_KEY>' },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Run with: node -r ./instrumentation.js server.js
```

### JAVA

#### Method 1: CLI with Java Agent (Recommended)
```bash
qorrelate run java -jar app.jar
```

#### Method 2: Manual Java Agent
```bash
# Download OpenTelemetry Java agent
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

# Run with agent
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=my-service \
  -Dotel.exporter.otlp.endpoint=https://qorrelate.io \
  -Dotel.exporter.otlp.headers=X-API-Key=<API_KEY> \
  -jar app.jar
```

### KUBERNETES

#### Method 1: Operator (Recommended for K8s)
```bash
# Install Qorrelate using CLI
qorrelate init k8s --namespace my-app --token <API_KEY>
```

#### Method 2: Manual Operator Setup
```yaml
# Install OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

# Create OpenTelemetry Collector
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: qorrelate-collector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          http:
          grpc:
    exporters:
      otlphttp:
        endpoint: https://qorrelate.io
        headers:
          X-API-Key: "<API_KEY>"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlphttp]
        metrics:
          receivers: [otlp]
          exporters: [otlphttp]
        logs:
          receivers: [otlp]
          exporters: [otlphttp]

# Auto-instrument pods with annotation
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "true"  # or java, nodejs, dotnet
    spec:
      containers:
      - name: app
        image: my-app:latest
```

### DOCKER COMPOSE

```yaml
# Generated by: qorrelate init --docker
version: '3.8'

services:
  my-app:
    build: .
    environment:
      OTEL_SERVICE_NAME: my-service
      OTEL_EXPORTER_OTLP_ENDPOINT: https://qorrelate.io
      OTEL_EXPORTER_OTLP_HEADERS: "X-API-Key=<API_KEY>"
      OTEL_TRACES_EXPORTER: otlp
      OTEL_METRICS_EXPORTER: otlp
      OTEL_LOGS_EXPORTER: otlp
      # Python-specific
      OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: "true"
      # Node.js-specific - add to Dockerfile:
      # NODE_OPTIONS: "--require @opentelemetry/auto-instrumentations-node/register"
```

---

## ENVIRONMENT VARIABLES (Universal)

When using OpenTelemetry directly (without CLI), set these:

```bash
# Required
export OTEL_SERVICE_NAME="my-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://qorrelate.io"
export OTEL_EXPORTER_OTLP_HEADERS="X-API-Key=<API_KEY>"

# Recommended
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

# Optional - Resource attributes
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=production"
```

---

## SIGNAL TYPES

### 1. DISTRIBUTED TRACES

Automatically captured by OpenTelemetry auto-instrumentation:
- HTTP requests (incoming/outgoing)
- Database queries (PostgreSQL, MySQL, MongoDB, Redis)
- Message queues (RabbitMQ, Kafka, SQS)
- gRPC calls
- GraphQL operations

Custom spans:
```python
# Python
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("custom-operation"):
    # Your code here
    span = trace.get_current_span()
    span.set_attribute("custom.attribute", "value")
```

```javascript
// Node.js
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('my-app');

const span = tracer.startSpan('custom-operation');
span.setAttribute('custom.attribute', 'value');
// Your code here
span.end();
```

### 2. LOGS

#### Automatic Log Collection (Python)
```python
import logging
logging.info("User logged in", extra={"user_id": 123})
logging.error("Payment failed", extra={"transaction_id": "abc-123"})
```

#### Structured Logging (Node.js)
```javascript
// Use with winston or pino
const logger = require('pino')();
logger.info({ userId: 123, action: 'login' }, 'User logged in');
```

#### Correlation with Traces
OpenTelemetry automatically adds trace_id and span_id to logs when available.

### 3. METRICS

#### Custom Metrics (Python)
```python
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

# Setup
reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(
        endpoint="https://qorrelate.io/v1/metrics",
        headers={"X-API-Key": "<API_KEY>"}
    )
)
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)

# Create metrics
meter = metrics.get_meter(__name__)
counter = meter.create_counter("requests.count")
histogram = meter.create_histogram("request.duration")

# Use metrics
counter.add(1, {"endpoint": "/api/users"})
histogram.record(0.125, {"endpoint": "/api/users"})
```

#### Custom Metrics (Node.js)
```javascript
const { metrics } = require('@opentelemetry/api');
const meter = metrics.getMeter('my-app');

const counter = meter.createCounter('requests.count');
const histogram = meter.createHistogram('request.duration');

counter.add(1, { endpoint: '/api/users' });
histogram.record(125, { endpoint: '/api/users' });
```

### 4. SESSION REPLAY (Frontend)

```html
<!-- Add to HTML pages -->
<script src="https://qorrelate.io/replay.js"></script>
<script>
  QorrelateReplay.init({
    apiKey: '<API_KEY>',
    serviceName: 'frontend-app',
    sampleRate: 1.0, // 100% of sessions
    maskAllText: false, // Privacy control
    blockSelector: '.sensitive-data', // Exclude elements
  });
</script>
```

Features:
- DOM recording
- User interactions (clicks, inputs, scrolls)
- Console logs
- Network requests
- Performance metrics

---

## API ENDPOINTS

### Ingestion
- **Traces:** `POST https://qorrelate.io/v1/traces`
- **Metrics:** `POST https://qorrelate.io/v1/metrics`
- **Logs:** `POST https://qorrelate.io/v1/logs`

### Authentication
All requests require `X-API-Key` header:
```
X-API-Key: <YOUR_API_KEY>
```

Get API key from: https://qorrelate.io after creating account

### Data Format
- Protocol: OTLP over HTTP
- Encoding: Protobuf (recommended) or JSON
- Content-Type: `application/x-protobuf` or `application/json`

---

## COMMON PATTERNS

### Pattern 1: Microservices Architecture
```yaml
# Each service sends telemetry directly to Qorrelate
Service A → Qorrelate
Service B → Qorrelate
Service C → Qorrelate

# OR use a collector (better for scale)
Service A ↘
Service B → OpenTelemetry Collector → Qorrelate
Service C ↗
```

### Pattern 2: Monolith with Background Jobs
```python
# Main app (Flask/Django)
qorrelate run python app.py

# Background workers (Celery)
qorrelate run celery -A tasks worker
```

### Pattern 3: Kubernetes Cluster
```bash
# Install operator
qorrelate init k8s --namespace default --token <API_KEY>

# Annotate deployments for auto-instrumentation
kubectl patch deployment my-app -p '
  spec:
    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-python: "true"
'
```

---

## TROUBLESHOOTING GUIDE

### Issue: No data appearing in Qorrelate

**Check 1: API Key**
```bash
# Verify API key is set
echo $OTEL_EXPORTER_OTLP_HEADERS
# Should show: X-API-Key=<your-key>
```

**Check 2: Endpoint Connectivity**
```bash
# Test endpoint
curl -X POST https://qorrelate.io/v1/traces \
  -H "X-API-Key: <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'
# Should return 200 OK
```

**Check 3: Instrumentation Loaded**
```bash
# Python - check for OpenTelemetry logs
OTEL_LOG_LEVEL=debug qorrelate run python app.py

# Node.js - check for instrumentation
NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register" \
OTEL_LOG_LEVEL=debug \
node server.js
```

### Issue: High cardinality / Too much data

**Solution: Sampling**
```bash
# Sample 10% of traces
export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
```

### Issue: Missing database queries in traces

**Solution: Ensure DB instrumentation enabled**
```python
# Python - explicit instrumentation
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor

Psycopg2Instrumentor().instrument()
RedisInstrumentor().instrument()
```

---

## BEST PRACTICES

### 1. Service Naming
Use consistent naming:
- Format: `{app}-{component}-{environment}`
- Example: `ecommerce-api-production`
- Avoid: dynamic names, timestamps, IDs

### 2. Resource Attributes
Always include:
```bash
export OTEL_RESOURCE_ATTRIBUTES="\
service.name=my-service,\
service.version=1.2.3,\
deployment.environment=production,\
service.namespace=ecommerce,\
host.name=$(hostname)"
```

### 3. Span Attributes
Add business context:
```python
span.set_attribute("user.id", user_id)
span.set_attribute("transaction.id", transaction_id)
span.set_attribute("cart.value", cart_total)
```

### 4. Error Handling
Always record exceptions:
```python
try:
    risky_operation()
except Exception as e:
    span = trace.get_current_span()
    span.record_exception(e)
    span.set_status(Status(StatusCode.ERROR))
    raise
```

### 5. Sensitive Data
Never log:
- Passwords
- API keys
- Credit card numbers
- PII (unless required)

Mask sensitive fields:
```python
logger.info("User login", extra={
    "email": mask_email(email),  # user@example.com → u***@example.com
    "ip": request.remote_addr
})
```

---

## ADVANCED FEATURES

### Custom Dashboards
Create in UI at https://qorrelate.io/dashboards using Apache ECharts

### Alerting
Configure alerts in UI:
1. Go to https://qorrelate.io/alerts
2. Create alert rule (PromQL-like syntax)
3. Configure notifications (Slack, Email, Webhook)

Example alert:
```
Alert: High Error Rate
Condition: error_rate > 5%
Duration: 5 minutes
Notification: #alerts-channel
```

### Multi-Tenancy
Each API key is scoped to an organization. Data is automatically isolated.

### Cost Management
- Powered by ClickHouse for efficiency
- Usage-based pricing
- Monitor costs at https://qorrelate.io/usage

---

## MIGRATION FROM OTHER TOOLS

### From Datadog
```python
# Replace Datadog tracer
# from ddtrace import tracer
# with OpenTelemetry
from opentelemetry import trace
tracer = trace.get_tracer(__name__)

# Replace Datadog logger
# import ddtrace.logger
# with standard Python logging (auto-instrumented)
import logging
logger = logging.getLogger(__name__)
```

### From New Relic
```python
# Replace New Relic agent
# import newrelic.agent
# newrelic.agent.initialize()
# with Qorrelate CLI
# qorrelate run python app.py
```

### From Prometheus
Qorrelate accepts OpenTelemetry metrics, which are compatible with Prometheus exporters.

```yaml
# Add to prometheus.yml
remote_write:
  - url: https://qorrelate.io/v1/metrics
    headers:
      X-API-Key: <API_KEY>
```

---

## CODE EXAMPLES REPOSITORY

Full working examples available at:
- https://github.com/qorrelate/examples (hypothetical)
- https://qorrelate.io/docs/

Examples include:
- Python Flask microservice
- Node.js Express API
- Java Spring Boot application
- Go Gin web server
- Kubernetes deployment with Operator
- Docker Compose multi-service setup

---

## WHEN TO USE QORRELATE CLI vs MANUAL INSTRUMENTATION

### Use CLI (Recommended):
✅ Quick setup (under 60 seconds)
✅ Zero code changes required
✅ Language-specific optimizations handled
✅ Automatic updates to instrumentation
✅ Supports Python, Node.js, Java, Go, .NET, Ruby

### Use Manual Instrumentation:
✅ Custom span attributes needed
✅ Fine-grained control over sampling
✅ Integration with existing observability setup
✅ Language not supported by CLI
✅ Advanced OpenTelemetry features required

**Best Practice:** Start with CLI, add manual instrumentation as needed.

---

## SECURITY CONSIDERATIONS

### API Key Management
- Store in environment variables (NEVER in code)
- Use secrets management (Vault, AWS Secrets Manager)
- Rotate keys regularly
- Use different keys per environment

### Data Privacy
- Session replay can be disabled per environment
- Use `maskAllText: true` for sensitive pages
- Block elements with `blockSelector`
- Log scrubbing available for PII

### Network Security
- All data encrypted in transit (HTTPS/TLS)
- Data encrypted at rest in ClickHouse
- VPC peering available (enterprise)

---

## SUPPORT AND RESOURCES

- Documentation: https://qorrelate.io/documentation.html
- API Docs: https://qorrelate.io/api-documentation.html
- OpenTelemetry Docs: https://opentelemetry.io/docs/
- ClickHouse Docs: https://clickhouse.com/docs/
- Staging Environment: https://wartime.live (for testing)

---

## VERSION COMPATIBILITY

- OpenTelemetry: v1.x (stable)
- ClickHouse: v23.x+
- Python: 3.8+
- Node.js: 14+
- Java: 8+
- Go: 1.18+
- .NET: 6.0+
- Ruby: 2.7+

---

## SUMMARY FOR LLM ASSISTANTS

When helping users instrument code with Qorrelate:

1. **Recommend CLI first**: `qorrelate run <command>` requires zero code changes
2. **Get API key**: User must create account at https://qorrelate.io (production) or https://wartime.live (staging)
3. **Set environment variables**: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS
4. **Use production URLs**: Always use qorrelate.io for production, wartime.live is staging only
4. **Auto-instrumentation covers**: HTTP, databases, queues, most common libraries
5. **Manual instrumentation**: Only needed for custom business logic spans
6. **Signals**: Traces (automatic), Logs (automatic with Python logging), Metrics (manual), Session Replay (frontend script)
7. **Troubleshooting**: Check API key, endpoint connectivity, enable debug logging
8. **Best practices**: Consistent service naming, resource attributes, error recording, no PII in logs

Remember: Qorrelate is 100% OpenTelemetry-native. Any OpenTelemetry SDK or collector can send data to it.

---

END OF LLM INSTRUMENTATION GUIDE