Observability PoC — Architecture flows

Homelab — current, with clearly marked gaps

The current log canary is real; fleet/application logs and traces are not yet in scope.

Current state + future path

Linux Docker VM · current backend

📄Synthetic canaryLocal file input
test record only

file tail

⚙️Grafana AlloyNative config
no OTLP receiver today

push · private Docker net

▤LokiDedicated volume
24-hour retention

query results

📊Grafana viewerAuthenticated
SSH tunnel to loopback

👤OperatorControl client
Mac runs no project workload

Existing Prometheus path · unchanged

📈Existing metrics GrafanaPromQL dashboards

query

🗄️Existing PrometheusPulls current target metrics

scrape

🖥Linux / K8s exportersExisting scrape targets

Arrows show the query/scrape requests; metric responses return over the same connections.

OpenTelemetry traces · future module

🔭App OTel SDKSource/runtime approval first

OTLP · not live

🧵Alloy → TempoReceiver and Tempo deferred

ℹ️

Coverage boundary: only the synthetic Alloy → Loki → Grafana flow is demonstrated. Existing metrics remain a separate Prometheus path. No Mac collector/probe, real application logs, remote Tailscale ingest, or traces are claimed.

Current foundations

Backend hostExisting Linux VM

RetentionNew Loki data: 24 hours

NetworkLoopback UI + private service network

Future source probesProxmox / UGREEN only; Plex stays on Mac mini

AWS — one Linux EC2 instance, private telemetry

Stage 1 is AWS-local synthetic only. Keep the existing homelab backend primary; a remote Linux sender is a separately approved future stage.

Single AZ · non-HA · unprovisioned

AWS account · VPC · one subnet · egress-only public IP

EC2 · Docker Compose · private data plane

🧪Local in-host synthetic generatorIsolated verification
AWS only · stage 1

loopback / local bridge

🔒Loki Push IngressPOST /loki/api/v1/push only
body and rate limits

private bridge

▤LokiNo public/LAN API
24-hour retention

internal query

📊Grafana Web UILoopback bind
login required

EC2 host boundary and sizing

Canary size2 vCPU · 4 GiB · T3 Standard credits

Storage20 GiB OS + separate 30 GiB gp3; measure before real source

Host bindingsLoopback only; Loki has no published host port

OperationsSSM / tailnet; no public SSH; IMDSv2 required

Operator access · tailnet only

🧑‍💻Operator browserMac is a viewer/control client only
no project workload

Serve HTTPS :443
operator ACL

📊Grafana Web UITailnet-authenticated
loopback backend

Future source path · gated · not connected in stage 1

⚙️Approved Linux Alloy sourceSeparate pipeline and owner approval
only after isolation evidence

Future Serve :8443
sender ACL absent in stage 1

🔒Same Loki Push IngressExact POST route
no Grafana, Loki, SSH, or Docker access

Build / change path · Gitea planned

🐈Private Gitea repoVersioned docs + IaC later

checks / review

🧑‍💻Human operatorTerraform apply + Ansible configure

Gitea Actions: static checks only for the PoC. No cloud credentials or apply-on-push.

AWS services and external dependency

EC2Linux host

EBS gp3Separate data volume

VPC + SGIsolated network policy

IAM + SSMLeast-privilege operations

S3Encrypted, locked Terraform state

CloudWatchBasic host metrics; selected alarms; filesystem needs an agent/custom metric

BudgetsSpend alert; no automatic teardown

CloudTrailAccount event history; new trail optional

Tailscale · externalTailnet transport and Serve; outbound-first

⚠️

Feasibility limits: one AZ and one VM are a single point of failure. Tempo is deferred. The local proxy enforces POST plus the exact push path; Serve only forwards ports. Stage 1 configures operator HTTPS :443 to Grafana; the future sender HTTPS :8443 listener and sender ACL are absent until approved. Reboot tests must confirm both ports, ACLs, Grafana root URL, and no 0.0.0.0 listeners. Enabling Serve HTTPS publishes the chosen machine/tailnet name in Certificate Transparency logs. Protect the data volume and measure 24-hour growth before any real source.

Illustrative AWS monthly reserve · US-East pricing basis plus ~20% regional cushion

2 vCPU / 4 GiB · 50 GiB$1.45 / $10.17 / $43.601 day / 1 week / 30 days

2 vCPU / 8 GiB · 50 GiB$2.65 / $18.57 / $79.601 day / 1 week / 30 days

4 vCPU / 16 GiB · 70 GiB$5.11 / $35.75 / $153.20Later measured trace tier; not required for PoC

Not a quoteReprice before launchExcludes egress, snapshots, extras, taxes, and CPU-credit overages

Azure — equivalent pattern, customer-region details open

Keep the workload portable; map identity, state, network, pricing, and operations to each tenant.

Reference architecture · non-authoritative · unprovisioned

Azure subscription · VNet / subnet · NSG denies unsolicited inbound

Single Linux VM · Docker Compose · illustrative only

🧪Local in-host synthetic generatorIsolated verification
Azure only · stage 1

loopback / local bridge

🔒Loki Push IngressPOST /loki/api/v1/push only
body and rate limits

private bridge

▤LokiLocal managed disk
short retention

internal query

📊Grafana Web UILoopback bind
login required

VM sizeChoose per tenant after measurement; not selected

DisksSeparate OS and telemetry data disks

IngressLoopback services; NSG denies inbound

OperationsManaged identity and approved private path

Future source and viewer paths · owner and tenant gated

⚙️Approved Linux Alloy sourceSelected logs only
separate pipeline required

Tailscale / private route
tenant review required

☁Azure VM push ingressExact private endpoint
not connected or selected

Platform and state services

Azure VM + managed disksSeparate OS and telemetry data disks

VNet + NSGInbound deny; private endpoint/egress choice TBD

Managed identityPrefer workload identity over stored credentials

Blob StorageVersioned Terraform state; locking approach review

Azure MonitorPlatform metrics and selected alerts

Cost ManagementBudget and alert by tenant

Key VaultOptional secret store if required

Gitea + human approvalChecks first; apply route determined per client

ℹ️

Reference only: choose tenant, region, data residency, identity model, private connectivity, retention, support boundary, and VM SKU before turning this mapping into an authoritative design. No Azure resources, source routes, SKU, price, or deployment duration are selected.