Why We Run k3s and Traefik on Bare Metal Instead of Managed EKS/GKE

For fast-growing engineering teams, adopting managed Kubernetes—such as Amazon EKS or Google GKE—is standard operating procedure. It delivers push-button provisioning, offloaded control planes, and turnkey integrations with cloud primitives. However, as workloads scale, an operational inflection point emerges where the benefits of managed cloud begin to fall behind its compounding costs, virtualization tax, and architectural compromises.

At Amppire, moving away from hyper-converged managed clouds toward k3s and Traefik on dedicated bare metal became our path to reclaiming resource efficiency, deterministic performance, and complete infrastructure control.

1. The Cloud Inflection Point: When Managed Becomes Bloat

Managed Kubernetes services are marketed as operational time-savers, but real-world engineering at scale exposes two compounding friction points: The Virtualization & Overlay Tax and The Compounding Cost Multiplier.
Code snippet

quadrantChart
    title Cloud vs. Bare Metal Operational Trade-Off Space
    x-axis Low Traffic / Simple Workloads --> High Throughput / Stateful Workloads
    y-axis Low Compute Overhead --> Severe Latency & Cost Penalties
    quadrant-1 Bare Metal Inflection Zone
    quadrant-2 Cloud Managed Sweet Spot
    quadrant-3 Over-engineered Bare Metal
    quadrant-4 Unsustainable Cloud Tax
    "Early MVP (Managed EKS/GKE)": [0.25, 0.30]
    "Internal Prototype": [0.15, 0.20]
    "Scale-up Ingress & DBs (EKS/GKE)": [0.85, 0.88]
    "Amppire Bare-Metal Fleet (k3s)": [0.80, 0.25]

The Cloud Resource & Latency Tax

In multi-tenant cloud virtual machines (EC2/GCE), performance variability is baked into the hypervisor layer:

• Noisy Neighbor Penalties: Shared host buses, vCPU scheduling contention, and noisy neighbors create uncontrollable tail-latency spikes ($p95$, $p99$, and $p99.9$).
• Network & Storage Intermediaries: Block storage traverses software-defined network fabrics (such as AWS EBS or GCP Persistent Disks), capping baseline IOPS and injecting millisecond-range latency bottlenecks compared to direct PCIe NVMe devices.
• Control-Plane Overhead: Standard enterprise distributions bundle cloud-provider-specific tree drivers, bloated storage controllers, and legacy components that consume node memory and compute before a single application pod boots.

The Financial Multiplier

On managed cloud platforms, costs compound across subtle, non-linear vectors:

• Cluster management fees run continuously per control plane.
• Cross-Availability Zone (AZ) egress fees silently accumulate across intra-cluster pod communications and data replication.
• Managed Network/Application Load Balancers charge continuous hourly base rates alongside Load Balancer Capacity Unit (LCU) metrics that surge under traffic.
  

2. Architecture: Bare-Metal k3s + Traefik Topology

flowchart TD
    Internet[Internet / Border Gateway] --> IngressGW[Anycast DNS / Cloudflare / BGP]
    
    subgraph EdgeLayer["Bare-Metal Ingress Layer (HostNetwork / BGP VIPs)"]
        Traefik[Traefik Ingress Controller]
        Traefik --- ACME[Automated ACME / TLS]
        Traefik --- CRDs[IngressRoute CRDs]
        Traefik --- Middlewares[Middleware Rate Limiting]
    end
    
    IngressGW -->|Public VIPs| Traefik
    
    subgraph ControlPlane["k3s HA Control Plane"]
        Master1[k3s Server 1]
        Master2[k3s Server 2]
        Master3[k3s Server 3]
        Master1 <-->|Embedded Raft / etcd| Master2
        Master2 <-->|Embedded Raft / etcd| Master3
        Master3 <-->|Embedded Raft / etcd| Master1
    end

    subgraph ClusterMesh["Amppire Production Workload Mesh (Flannel WireGuard-Native)"]
        direction TB
        subgraph Worker1["Worker Node 1"]
            PodA[Pod A]
            PodB[Pod B]
            NVMe1[(Local PCIe NVMe)]
            PodA --> NVMe1
        end

        subgraph Worker2["Worker Node 2"]
            PodC[Pod C]
            PodD[Pod D]
            NVMe2[(Local PCIe NVMe)]
            PodC --> NVMe2
        end
    end

    Traefik -->|WireGuard Tunnel| ClusterMesh
    ControlPlane -.->|API & Scheduling| ClusterMesh

Why k3s on Bare Metal?

Rancher’s k3s is a certified lightweight Kubernetes distribution engineered for edge, embedded, and bare-metal environments:

• Single Binary Footprint: Combines the API Server, Controller Manager, Scheduler, kubelet, and containerd into a single binary that initializes in seconds and consumes under 512MB RAM.
• Stripped Cloud Bloat: Eliminates legacy, proprietary in-tree storage volume plugins and external cloud controllers.
• Embedded Production Datastore: Uses embedded Raft (etcd) out of the box, removing the operational burden of orchestrating external etcd fleets.

Why Traefik Ingress?

Rather than relying on layered cloud load balancers paired with heavy ingress controllers, Traefik natively integrates with Kubernetes Custom Resource Definitions (CRDs):

• Dynamic Endpoint Auto-Discovery: Traefik monitors the k3s API server directly, constructing routing tables, internal weightings, and health checks as pods scale.
• IngressRoute Custom Resources: Replaces fragile Kubernetes ingress annotations with structured, version-controlled YAML for rate-limiting, circuit breakers, and header manipulations.
• Native ACME Automation: Built-in Let’s Encrypt challenge solvers eliminate external certificate managers and recurring webhook validation failures.

3. Bare-Metal Infrastructure Building Blocks

Eliminating managed cloud providers means taking deliberate ownership of three foundational primitives:

flowchart LR
    subgraph EdgeNetworking["1. Edge Ingress Layer"]
        BGP[Top-of-Rack BGP Router] -->|Layer 2 / BGP VIPs| MetalLB[MetalLB Controller]
        MetalLB -->|Binds VIP| HostPort[Traefik :80 / :443 hostNetwork]
    end

    subgraph InternalNetworking["2. Cluster Pod Mesh"]
        HostPort -->|Direct Kernel Path| WireGuard[Flannel WireGuard Tunnel]
        WireGuard --> PodEndpoints[Workload Pods]
    end

    subgraph StateStorage["3. Direct NVMe Storage"]
        PodEndpoints -->|Raw Block / File I/O| LocalCSI[Local Path CSI Provisioner]
        LocalCSI -->|Direct PCIe Access| NVMeDrives[(Enterprise NVMe Array)]
    end

1. Ingress Routing & Layer 2/3 Load Balancing

Bare-metal clusters do not have an AWS ALB or GCP Load Balancer to synthesize public endpoints:

• The Layer 2 / BGP Bridge: Using MetalLB or Cilium BGP Control Plane, the cluster advertises service Virtual IPs (VIPs) directly to top-of-rack (ToR) switches or upstream gateway routers.
• HostNetwork Direct Binding: Edge proxy instances bind Traefik directly to physical ports 80 and 443 with hostNetwork: true, eliminating network translation hops and ingress packet mangling.

2. High-Performance Persistent Storage (CSI)

Without managed cloud volumes, persistent state is mapped directly to physical silicon:

• Direct Local Storage: Leveraging Local Path Provisioner (bundled by default with k3s), databases allocate dedicated directories directly on host PCIe NVMe arrays, capturing millions of IOPS at microsecond latencies.
• Distributed Volume Replication: For workloads requiring synchronous cross-host replication, lightweight distributed block systems such as Longhorn or Rook-Ceph provide fault tolerance across physical nodes.

3. High Availability Control Plane via Embedded etcd

For production environments, k3s initializes as an HA 3-node cluster utilizing embedded Raft:

# /etc/rancher/k3s/config.yaml (Bootstrap Master)
cluster-init: true
token: "amppire-production-bootstrap-token"
tls-san:
  - "k8s-api.internal.amppire.io"
  - "10.0.0.10"
disable:
  - servicelb       # Replaced by MetalLB / Direct Traefik HostNetwork
  - local-storage   # Handled via custom NVMe storage classes
flannel-backend: "wireguard-native"
write-kubeconfig-mode: "0640"

Joining nodes register instantly against the internal control plane VIP:

curl -sfL https://get.k3s.io | K3S_URL=https://10.0.0.10:6443 \
  K3S_TOKEN="amppire-production-bootstrap-token" sh -

4. Production Traefik Configuration & IngressRoutes

sequenceDiagram
    autonumber
    actor Client as Client / HTTPS
    participant NIC as Bare-Metal Host NIC
    participant Traefik as Traefik Ingress (hostNetwork)
    participant MW1 as Middleware: RateLimit
    participant MW2 as Middleware: SecureHeaders
    participant Pod as Backend Microservice Pod
    
    Client->>NIC: TLS Handshake (Port 443)
    NIC->>Traefik: Direct Socket Delivery (Zero Hypervisor/NAT Hop)
    Traefik->>Traefik: Auto ACME Cert Validation (Let's Encrypt)
    Traefik->>MW1: Evaluate Request Frequency & Token Bucket
    MW1-->>Traefik: Bucket Within Threshold
    Traefik->>MW2: Inject HSTS, XSS, & Security Headers
    MW2-->>Traefik: Headers Attached
    Traefik->>Pod: Forward over Flannel WireGuard Mesh
    Pod-->>Traefik: HTTP 200 Response Payload
    Traefik-->>Client: Streaming Response (Sub-millisecond latency)

Traefik Helm Configuration (values.yaml)

deployment:
  enabled: true
  replicas: 3

hostNetwork: true # High-throughput direct socket binding

ports:
  web:
    port: 80
    expose: true
    redirectTo:
      port: websecure
  websecure:
    port: 443
    expose: true
    tls:
      enabled: true

providers:
  kubernetesCRD:
    enabled: true
    allowCrossNamespace: true
  kubernetesIngress:
    enabled: false

service:
  type: ClusterIP

additionalArguments:
  - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
  - "[email protected]"
  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"

Amppire IngressRoute & Security Middlewares

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: core-api-route
  namespace: production
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`api.amppire.io`) && PathPrefix(`/v1`)
      kind: Rule
      services:
        - name: core-api-service
          port: 8080
          strategy: RoundRobin
      middlewares:
        - name: api-ratelimit
        - name: secure-headers
  tls:
    certResolver: letsencrypt
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: api-ratelimit
  namespace: production
spec:
  rateLimit:
    average: 1500
    burst: 2000
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: secure-headers
  namespace: production
spec:
  headers:
    sslRedirect: true
    stsSeconds: 315360000
    browserXssFilter: true
    contentTypeNosniff: true

5. Storage Tier Architecture & Production Workload Placement

Stateful infrastructure is where managed clouds impose both performance penalties and escalating billing tiers. Amppire’s bare-metal architecture separates persistent layers by operational profile:

graph TD
    subgraph TrafficEntry["External Traffic Ingress"]
        TraefikProxy[Traefik Ingress Router]
    end

    subgraph WorkloadBoundary["Amppire Workload Placement Layer"]
        APIApps[Next.js / NestJS / FastAPI Apps]
        BaaS[Self-Hosted Supabase Engine]
    end

    subgraph LocalPersistentLayer["Bare-Metal NVMe Storage Partitions"]
        direction TB
        Dragonfly[DragonflyDB / High-Speed Cache]
        Postgres[CloudNativePG Cluster]
        Analytics[ScyllaDB / ClickHouse Ingest]
    end

    subgraph ObservabilityLayer["Cluster Management & Telemetry"]
        Rancher[Rancher Container Ops]
        GitOpsPipeline[GitOps Deploy Pipeline]
    end

    TraefikProxy --> APIApps
    TraefikProxy --> BaaS
    APIApps -->|In-Memory Bus| Dragonfly
    APIApps -->|PCIe Local Path IOPS| Postgres
    APIApps -->|Direct Block Append| Analytics
    BaaS --> Postgres

    ObservabilityLayer -.-> WorkloadBoundary
    ObservabilityLayer -.-> LocalPersistentLayer

• CloudNativePG (PostgreSQL): Powers primary relational workloads. Running CloudNativePG inside bare-metal k3s provides automated HA failover, streaming replication, and WAL archival directly to off-site object stores without the overhead of cloud RDS instances.
• DragonflyDB / High-Performance Redis: Serves sub-millisecond cache hits and active pub/sub event streams, maximizing memory bus bandwidth without hypervisor CPU quota throttling.
• ScyllaDB / ClickHouse: Manages high-velocity telemetry, heavy ingest workloads, and analytical aggregations, writing directly to physical NVMe blocks without network hop contention.
• Self-Hosted Supabase: Delivers integrated auth, real-time subscriptions, and database interfaces directly within our Kubernetes boundary.
• Dual-Tier Management: Operational deployments run via GitOps pipelines, while Rancher is maintained alongside kubectl for visual container inspections, real-time log monitoring, and container health checks without relying on heavy external SaaS observability agents.
  

6. Performance Benchmarks: Managed Cloud vs. Bare-Metal k3s

To quantify the architectural difference, synthetic benchmarks were run comparing a Managed Cloud Cluster (AWS EKS c6i.8xlarge / GCP c2-standard-30 + Provisioned IOPS EBS + ALB) against our Bare-Metal k3s Cluster (AMD EPYC / Ryzen 9 + PCIe Gen4 NVMe + HostNetwork Traefik).

gantt
    title Latency Breakdown: Network Packet Traversal (ms)
    dateFormat X
    axisFormat %s ms

    section Managed EKS (ALB -> Pod)
    DNS & Edge Hop              :0, 15
    Managed ALB Queuing         :15, 35
    vSwitch / Hypervisor Bridge :35, 60
    Pod Overlay & DNAT Hop      :60, 84

    section Bare-Metal k3s (Traefik)
    DNS & Border BGP            :0, 4
    Direct NIC Traefik Engine   :4, 8
    WireGuard Pod Transit       :8, 11

1. HTTP Ingress & API Tail Latency

Benchmarked using wrk2 generating 40,000 requests/sec of mixed HTTPS microservice traffic over 15 minutes:

Managed cloud ingress routes traffic through edge distributions, virtual switches, hypervisors, and container bridges before hitting the pod. Bare-metal k3s with HostNetwork Traefik terminates TLS on the physical NIC directly with minimal context switching.

2. Disk I/O & Database Throughput (Fio & pgbench)

Evaluated via random 4KB read/write workloads ($70/30$ ratio) and transactional PostgreSQL pgbench testing:

3. Node Resource Tax & Idle Overhead

Resource consumption by Kubernetes system components prior to launching application containers:

7. Cost Comparison: Total Cost of Ownership (TCO)

To examine the economic inflection point, we evaluated the monthly and annualized run-rate of supporting an equivalent production compute footprint: 128 Dedicated Cores / 512GB RAM / 8TB NVMe-tier Storage / 20TB Outbound Transit.

pie title Monthly Fleet Cost: Managed Cloud ($4,658/mo) vs Bare Metal ($1,085/mo)
    "Cloud Compute Instances (EKS)": 2180
    "Cloud Egress Penalties": 1420
    "Cloud EBS Provisioned Storage": 840
    "Cloud ALB & Control Plane": 218
    "Bare Metal Dedicated Servers": 780
    "Bare Metal Bandwidth (10Gbps)": 140
    "Bare Metal Ops & S3 Backups": 165

Itemized Monthly Cost Breakdown

Key Cost Drivers Decoded

1. The Egress Penalty: Cloud hyperscalers charge upwards of $0.05 to $0.09 per GB for public egress, plus $0.01 to $0.02 per GB just for traffic crossing Availability Zones within your own VPC. In bare-metal colocation or dedicated hosting, multi-gigabit unmetered bandwidth or high allowances are included by default.
2. Storage Provisioning Surcharges: Cloud providers bill separately for volume size, baseline IOPS, and burst throughput. High-end PCIe Gen4 NVMe drives deliver hundreds of thousands of IOPS as a physical characteristic of the hardware, incurring no recurring usage surcharge.
3. Hypervisor Tax: Because managed nodes consume 2.5GB to 4GB of RAM and background CPU cycles on cloud monitoring agents, teams must purchase roughly 15% to 20% more compute instances simply to match bare-metal application capacity.

8. Architectural Comparison: Measured Outcomes

9. The Operational Trade-Offs: When Bare Metal Is Not for You

Operating your own bare-metal Kubernetes fleet requires specific operational capabilities that managed providers abstract:

flowchart TD
    Decision{Should You Adopt Bare-Metal k3s?}
    
    Decision -->|Unpredictable, 10x intraday spikes| StickWithCloud[Stick with Managed Cloud EKS/GKE]
    Decision -->|Zero Linux systems engineers on staff| StickWithCloud
    Decision -->|Single-region, minimal bandwidth bill| StickWithCloud
    
    Decision -->|Sustained high bandwidth / heavy egress| EvaluateBareMetal[Adopt Bare-Metal k3s]
    Decision -->|Write-intensive NVMe databases| EvaluateBareMetal
    Decision -->|Need predictable sub-millisecond p99 latency| EvaluateBareMetal
    
    EvaluateBareMetal --> PrepOps[Operational Prerequisites: Remote IPMI/iKVM, MetalLB BGP Routing, Off-site Backups]

1. Hardware Lifecycle Management: In managed cloud, a failed node is resolved by spinning up another virtual instance. On bare metal, your engineering team manages remote IPMI/iKVM interventions, coordinates hardware component replacements, and configures automated node-drain automation for failing hardware.
2. Elastic "Burst" Capacity: If your platform experiences unpredictable 10x traffic spikes within minutes, public cloud Auto Scaling Groups provision virtual capacity faster than physical servers can be racked or re-imaged.
3. Systems Engineering Discipline: Operating bare metal demands foundational Linux systems skills—including kernel tuning, systemd process supervision, local network route advertisements, and isolated disaster recovery workflows.

The Verdict

For Amppire, the decision came down to workload mechanics: our traffic is predictable, throughput-intensive, and sensitive to latency outliers. Paying hyperscalers a recurring premium for virtualized abstractions no longer aligned with our technical or economic objectives.
By deploying k3s and Traefik on dedicated bare metal, paired with local NVMe storage engines (CloudNativePG, DragonflyDB, ScyllaDB) and clean operational tooling, we reclaimed our infrastructure stack: shaving tail latencies under 1ms, unlocking direct hardware IOPS, and cutting infrastructure expenditures by more than 75%.