Kovenlab Systems engineering AI infrastructure Doc KVL-000 Sheet 1 of 1 Rev. C [email protected]

Kovenlab

Complex systems, and the compute they run on.

Kovenlab designs and builds complex software systems end to end, and provides the layer underneath them: AI models, inference and hosting, GPU compute, servers, networking and storage.

Practice
Systems engineering and AI infrastructure
Builds
Distributed systems, data and generation pipelines, real-time platforms, AI systems
Provides
Models, inference and hosting, GPU compute, servers, networking, storage
Terms
By contract, by service class
Reference stack, from compute nodes to interfaces Six layers drawn bottom to top: GPU nodes and servers; network and storage; orchestration and scheduling; model serving with a router in front of model replicas; systems such as streams, pipelines, state and AI components; and interfaces such as REST, gRPC, WebSocket, server-sent events, container images and telemetry. Two rails run through every layer: observability, and isolation and security. A request path runs from the interfaces down to the hardware. KVL-000 / REFERENCE STACK / REV. C OBSERVABILITY ISOLATION + SECURITY OPERATED AS ONE STACK REQUEST PATH INTERFACES L5 SYSTEMS L4 MODEL SERVING L3 ORCHESTRATION + SCHEDULING L2 NETWORK + STORAGE L1 GPU NODES + SERVERS L0 REST gRPC WS SSE OCI OTEL STREAM PIPELINE STATE AI A/1 A/2 B/1 B/2 C/1 C/2
Fig. 1 Reference stack, from compute nodes to interfaces. Two rails run through every layer. Schematic.
  1. L0 GPU nodes and servers
  2. L1 Network and storage
  3. L2 Orchestration and scheduling
  4. L3 Model serving: a router in front of model replicas
  5. L4 Systems: streams, pipelines, state, AI components
  6. L5 Interfaces: REST, gRPC, WebSocket, SSE, OCI, telemetry

1.0 Two practices, one stack

Kovenlab works in two connected practices. The first designs and builds complex software systems end to end: distributed systems, data and generation pipelines, real-time platforms and AI systems. The second provides what those systems run on: AI models, inference and hosting, GPU compute, servers, networking and storage.

Because we operate the layers underneath, the systems we build are designed against real hardware, real failure modes and real capacity. The infrastructure we provide is shaped by the systems that run on it.

Practice A
Systems engineering: design, build, commission, operate
Practice B
Infrastructure: models, inference, compute, servers, network, storage
Boundary
One operator from hardware to interface
Engagement
Fixed-scope build, continuing operation, or capacity by service class

2.0 Lines of work

Seven lines, each with a part number used across contracts, capacity plans and this document. They can be taken alone or combined into one deployment.

Table 2.1 Lines of work
LineScopeDelivered as
KVL-100 Systems engineeringDistributed systems, data and generation pipelines, real-time platforms and AI systems, designed and built end to endFixed-scope build or continuing engagement
KVL-200 ModelsModel provision, adaptation to a domain, evaluation against task-specific test setsWeights under licence terms, or served endpoints
KVL-300 Inference and hostingManaged endpoints, dedicated serving, batch inference, streaming outputEndpoints by service class
KVL-400 GPU computeManaged clusters with reserved, on-demand and preemptible capacity, batch schedulingCapacity by service class
KVL-500 Servers and edgeDedicated servers, edge nodes, private deployments on a site the tenant controlsHardware we operate, or hand over under contract
KVL-600 Network and storagePrivate networking, encrypted links, object and block storageAttached to any line above
KVL-700 Content and generation APIsStructured, validated content packages: interactive items, narration audio, assets, manifestVersioned packages over an API

3.0 Systems we design and build

We take on systems where the difficulty is the point: many moving parts, strict latency, large volumes, or components that are probabilistic by nature. We carry them from first design through build, commissioning and operation, and we stay responsible for how they behave in production.

Every design states its failure modes before it states its features: what happens when a node, a link, a region or a model misbehaves, and how the system degrades instead of stopping.

Two regions and an edge tier under one control plane One control plane sends configuration to two regions and an edge tier. Nodes inside each region form a mesh. Two data links join the regions; one of them is down, and its traffic is rerouted through the edge tier. REGION A REGION B EDGE TIER CONTROL PLANE LINK DOWN
Fig. 2 Two regions and an edge tier under one control plane. Solid lines carry data, dashed lines carry control. One link is down and its traffic takes the edge path. Schematic.
Table 3.1 System classes
System classWhat it involvesHard part
SC-1 Distributed systemsServices spread across nodes and regions, with replication, consensus and failoverConsistency and recovery when parts of the system disagree or disappear
SC-2 Data and generation pipelinesIngest, transform, generate and validate at volume, with lineage for every outputKeeping output correct and traceable as volume and sources change
SC-3 Real-time platformsEvent streams, live state and low-latency fan-out to many connected usersHolding latency at the tail while load moves
SC-4 AI systemsModels in production: retrieval, tool use, evaluation, guardrails, feedback loopsMaking probabilistic components behave predictably inside a system

4.0 Models, compute and infrastructure

We provide the layer that complex systems run on, as a managed service or as hardware we operate for a single tenant. Models and inference come as managed endpoints or dedicated serving. GPU compute comes as managed clusters with reserved, on-demand and preemptible capacity. Servers are available dedicated, at the edge, or deployed privately on a site the tenant controls, with networking and storage attached.

Capacity, objectives and support are stated per service class and fixed per contract. Nothing on this sheet is a published figure.

Capacity and scheduling across six GPU nodes Six GPU nodes over time. Reserved work holds fixed slots, on-demand work fills gaps, and preemptible batch work uses what is left. On node three a batch job is preempted when reserved work starts; its remainder resumes later on node six. Free capacity after the current moment is headroom. Schematic. NOW TIME N01 N02 N03 N04 N05 N06 PREEMPTED, RESUMED HEADROOM RESERVED ON-DEMAND PREEMPTIBLE PREEMPTION
Fig. 4 Capacity and scheduling across six GPU nodes. Reserved work holds its slots; preemptible work yields and resumes elsewhere. Schematic.
Table 4.1 Service classes
ClassTenancySchedulingCapacityObjectives
S0 SharedMulti-tenantBest effort, preemptiblePooledNone stated
S1 StandardMulti-tenant, isolated runtimeAdmission-controlled queueQuota per accountPer class schedule
S2 ReservedMulti-tenant hardware, reserved slicesGuaranteed slotsReserved per contractPer contract
S3 DedicatedSingle tenantExclusive nodesFixed allocationPer contract
S4 PrivateSingle tenant, private site or zoneOperator definedSized to the sitePer contract, site specific
Table 4.2 Isolation levels
LevelBoundaryStill shared
I1 ProcessSeparate processes, namespaces and resource limitsKernel, node, network
I2 Virtual machineHardware-virtualised guest per workloadHost hardware
I3 Dedicated nodeWhole machines for one tenantNetwork fabric, facility
I4 Dedicated segmentOwn nodes, own network segment, own storageFacility
I5 Private siteHardware at a location the tenant controlsNothing with other tenants
Table 4.3 Deployment modes
ModeHardwareOperated byTypical use
D-M ManagedOurs, sharedKovenlabEndpoints and APIs without running infrastructure
D-D DedicatedOurs, single tenantKovenlabSteady workloads that need fixed capacity
D-P PrivateTenant site or private zoneKovenlab, or handed overData and models that must stay in one place
D-E EdgeSmall nodes near the data sourceKovenlabLow latency or intermittent connectivity
D-H HybridAny mix of the aboveShared, per contractBursting from private to managed capacity

5.0 A request, end to end

One inference request passes nine stages between the caller and the first byte of the answer. Each stage holds part of a latency budget, and the parts add up to the objectives of the request's service class. When a stage cannot meet its share, the request is refused early with a clear signal instead of being served late.

Lifecycle of one inference request, with objectives A request passes nine stages: client, edge with TLS, gateway with authentication and quota, router choosing model and version, queue with admission control, continuous batcher, GPU worker, validation against a schema, and the output stream. Three objectives span the path: admission, first token and completion, each set by service class. A request that cannot be admitted within budget is shed with status 429 and a retry-after header. A budget bar below splits the latency budget across network, authentication, queueing, prefill, decode and validation. Proportions are illustrative. COMPLETE WITHIN SLO.C, BY SERVICE CLASS FIRST TOKEN WITHIN SLO.B ADMIT WITHIN SLO.A CLIENT request EDGE TLS GATEWAY auth, quota ROUTER model, version QUEUE admission BATCHER continuous WORKER GPU VALIDATE schema STREAM tokens out OVER BUDGET: SHED, 429 + RETRY-AFTER NET AUTH QUEUE PREFILL DECODE CHECK LATENCY BUDGET, SPLIT BY STAGE. PROPORTIONS ILLUSTRATIVE
Fig. 3 Lifecycle of one inference request, with objectives SLO.A to SLO.C set by service class. Proportions are illustrative.

6.0 Interfaces and protocols

Everything we provide is reached through a small set of standard interfaces, the same in managed, dedicated and private deployments.

Table 6.1 Interfaces and protocols
InterfaceProtocolUsed for
IF-1 Request and responseHTTPS, JSON, HTTP/2Synchronous calls and control operations
IF-2 StreamingServer-sent events, WebSocketToken and event streams
IF-3 Service to servicegRPC over HTTP/2 with mutual TLSInternal and private integrations
IF-4 Asynchronous jobsJob queue with signed webhooksBatch inference and package generation
IF-5 WorkloadsOCI container imagesBringing code to dedicated and private nodes
IF-6 TelemetryOpenTelemetry traces, OpenMetricsObservability exported to the tenant
IF-7 Private networkingWireGuard tunnels, private linksJoining sites and segments

Content and generation APIs KVL-700

One product line among several. A request names a topic, a level and a content specification. The response is a finished, versioned package: interactive items, narration audio, assets and a manifest, validated before it leaves the system.

Input
Topic, level, content specification
Output
Versioned package: items, narration audio, assets, manifest
Interface
IF-1 or IF-4. One request returns one finished package
Hard part
Generated material that is correct, consistent and checkable before it ships
An exploded package A request with a topic, a level and a specification enters on the left. The package is drawn as four separated layers: manifest, items, audio and assets. One package leaves as the response. REQUEST topic level spec MANIFEST ITEMS AUDIO ASSETS RESPONSE: ONE PACKAGE
Fig. 5 An exploded package: request in, layered contents, one response out. Schematic.

7.0 Engineering principles

  1. Measure before claiming

    Every objective has a measurement behind it. Figures are stated per contract, taken from the systems that produce them.

  2. Design for failure

    Every component is expected to fail. A design states what happens when it does, and drills prove it.

  3. The tail is the latency

    Systems are judged at the slow end of the distribution, where people actually feel them.

  4. Isolation is explicit

    Each workload runs at a stated isolation level. Nothing is shared by accident.

  5. Backpressure over collapse

    Under overload a system admits what it can serve and refuses the rest clearly, instead of slowing down for everyone.

  6. Correct before plentiful

    Generated output is checked against its specification before it leaves the system.

  7. One owner per layer

    Every layer, from hardware to interface, has a named operator, so a fault never falls between two parties.

  8. Plain surfaces, deep internals

    Interfaces stay simple to use. The complexity lives behind them, where it can be tested.

8.0 Operating model

Work moves through six stages, each closed by a stated exit condition. Once a system is in operation, responsibility for every layer is written down per deployment mode.

Table 8.1 Stages of work and their exit conditions
StageWorkExit condition
OP-1 ScopeDefine the workload, its constraints, its service class and what success is measured bySigned scope and acceptance tests
OP-2 DesignArchitecture, capacity plan, failure modes, isolation levelDesign review passed, risks written down
OP-3 BuildImplement and integrate against the acceptance testsTests pass in a production-like environment
OP-4 CommissionLoad, failure and recovery drills on the real deploymentObjectives met under drill conditions
OP-5 OperateRun, monitor, respond and report against the agreed objectivesNone. This stage continues
OP-6 EvolveRevise capacity, models and design as the workload changesA new revision of the design
Table 8.2 Responsibility by layer and deployment mode. K Kovenlab, T tenant, S shared.
LayerManagedDedicatedPrivate
Facility and powerKKT
HardwareKKS
Firmware and OSKKK
OrchestrationKKK
Model servingKKK
ApplicationSTS
DataTTT
Change control
Every change to a running system is reviewed, staged and reversible
Incidents
Detected, handled and written up against the service class
Reporting
Objectives reported from live measurements, per contract
Exit
Designs, runbooks and data handed over on request

9.0 Glossary

Admission control
Deciding at the entrance whether a request can be served within its objective, and refusing it early if not.
Continuous batching
Grouping inference requests on a GPU as they arrive, so new work joins without waiting for the current batch to finish.
Control plane
The part of a system that decides placement, configuration and policy. It carries no user data.
Data plane
The part that carries requests and data. It keeps working while the control plane is briefly unavailable.
Edge node
A small server placed near the source of data or the people using it.
Error budget
The amount of failure an objective allows over a period. Spending it pauses risky changes.
Isolation level
The boundary between workloads, from separate processes to a separate site. See Table 4.2.
Package
A versioned unit of structured content: items, narration audio, assets and a manifest, validated before delivery.
Placement
Choosing which node runs a workload, given capacity, isolation and locality.
Preemption
Stopping lower-priority work to free capacity for higher-priority work, then resuming it elsewhere.
Private deployment
Hardware and software at a location the tenant controls, operated under contract.
Service class
A named bundle of tenancy, scheduling and objectives that a workload runs under. See Table 4.1.
SLO
Service level objective: the target a service is operated against, measured from live traffic.
Tail latency
Response time at a high percentile: the slowest requests rather than the average.
TTFT
Time to first token: how long a streaming model response takes to begin.

10.0 Company

Kovenlab is an independent engineering company. We design, build and operate complex systems and the infrastructure underneath them.

Correspondence
[email protected]
Engagement
By contract. Capacity, objectives and terms are fixed per service class.
Document
KVL-000, Rev. C, sheet 1 of 1
Colophon
Set in Archivo and Fragment Mono, both under the SIL Open Font License. This page makes no external requests, sets no cookies and runs no scripts.