AI That Operates at the Edge of the World

Edge AI infrastructure is the software layer that makes AI inference happen on the device you actually have, rather than on a server somewhere else. It is not a cloud product with an offline mode bolted on. It is built from the ground up for the constraint that there is no cloud available.

It matters because the environments where AI decisions are most consequential, defense operations, clinical care, autonomous systems, industrial control, are exactly the environments where cloud connectivity is unavailable, prohibited, or a security liability. A soldier in a denied communications environment cannot use ChatGPT. A surgeon in a hospital with HIPAA obligations cannot route patient queries through a third-party API. A drone operating beyond radio range has no datalink to reach.

Edge AI infrastructure closes this gap. It brings the reasoning, memory, and language capability of modern AI inside the trusted boundary of the device and the organization that operates it.

We prototype and validate on NVIDIA Jetson Nano (4GB), NVIDIA Jetson Orin Nano (8GB), Raspberry Pi 4 (8GB), and Raspberry Pi 5. These are our primary development targets and reflect real hardware constraints, not theoretical ones.

On the Jetson Nano (472 GFLOPS, 4GB RAM), we run 1B to 3B parameter models at 4-bit quantization using llama.cpp, producing roughly 10 to 15 tokens per second. On Raspberry Pi 4 with 8GB RAM, running CPU-only inference, we achieve 4 to 6 tokens per second with 1B parameter models. On the Jetson Orin Nano (40 TOPS, 8GB RAM), 7B models at Q4_K_M quantization deliver 20 to 30 tokens per second. These are real prototype numbers.

We are at an early stage and say so plainly. What we are building is the software and runtime layer that makes these numbers useful in real operational contexts: persistent memory, session continuity across power cycles, and hardware-aware resource allocation. The hardware runs today. The production hardening is on the roadmap.

We use open-weight models in GGUF format, distributed through llama.cpp and Ollama. On Raspberry Pi 4 and 5, we target models at or below 1B parameters: Llama 3.2 1B and Qwen 2.5 0.5B are our current primary targets. These run entirely in RAM with no GPU required.

On the Jetson Nano (4GB), we run Llama 3.2 3B and Phi-3 Mini at Q4 quantization. On the Jetson Orin Nano (8GB), we run Llama 3.1 8B and Mistral 7B comfortably at Q4_K_M quantization. The same model file and runtime that operates on Jetson hardware operates identically on an x86 server, meaning prototypes written on edge hardware deploy to command infrastructure without code changes.

All inference is 100% local. No tokens leave the device. No API is called. No network connection is required or used at any point during inference.

At the edge, there is no reach-back to a cloud memory store. Either memory lives on the device or it does not exist. Most local AI deployments today are stateless by design because persistent memory was considered a cloud problem. We disagree.

Consider the operational difference. A stateless edge AI deployed on a forward operating base starts fresh every session. The analyst who built 200 hours of target understanding loses everything when the session ends. A device with our tiered memory architecture retains hot context for the immediate session, warm summaries for recent operational history, and a permanent vault of critical intelligence that survives power cycles and reboots.

The vault layer uses vector embeddings generated entirely on-device. On Raspberry Pi, lightweight sentence transformer models run on CPU. On Jetson hardware, GPU-accelerated embedding delivers lower latency. Nothing is sent to an external vector database. The accumulated knowledge belongs to the device and the organization that operates it, permanently.

Once deployed, the system has zero external dependencies of any kind. There are no licensing servers to check in with. There is no telemetry. There are no model download requirements at runtime. There is no cloud fallback path. Everything the system needs to operate lives on the local device.

This is not a degraded offline mode. It is the only mode. The system was designed from first principles to treat network connectivity as permanently unavailable. This is why Raspberry Pi and Jetson Nano are our development targets: if the software works correctly under those constraints, it works correctly everywhere.

For defense and sensitive applications, this means data never crosses a network boundary. For robotics and field deployments, it means the system keeps working when communications are jammed, unavailable, or actively hostile.

Raspberry Pi and Jetson support is currently in Alpha. The inference engine runs. The memory layer runs. The SDK integrations work. What we are building toward for production is: deployment tooling that configures the system for a specific hardware target without manual tuning, production-grade error handling and recovery for field conditions, and comprehensive documentation for each supported hardware class.

We do not publish a specific date because hardware certification for defense and regulated industry use involves external validation steps we cannot fully control. What we can say is that our SaaS applications are live, our SDK is published, and our edge runtime is the next focus of hardening effort.

If you are working on a defense program, robotics platform, or clinical deployment that requires edge AI today, reach out. We work directly with early deployment partners and the feedback from real operational environments is what drives the production roadmap.

At startup, our inference engine reads the available VRAM and assigns GPU layers accordingly. 4GB VRAM gets 12 GPU layers. 8GB gets 25. 12GB gets 32. 16GB and above gets 40 to 50. This maximizes inference speed for the hardware present without exceeding memory bounds. CPU fallback is automatic when no GPU is available, with thread count optimized for the available core count.

CPU thread allocation follows a similar pattern. 1 to 2 cores gets 1 thread. 3 to 8 cores gets 60% of cores. 9 to 16 cores gets 50%. Above 16 cores, we cap at 16 threads to avoid contention. Context and batch sizes are inferred from the model file name and available RAM, with safety limits to prevent memory exhaustion on constrained hardware.

The result is that the same deployment package works on a 4-core Raspberry Pi and a 64-core server without any configuration changes between them. The system adapts to what it finds.

In connected environments, yes. Edge nodes can sync their memory layer to a local hub device or private server without any data leaving the organization's network. A team of field devices operating within radio range of a hub can share accumulated operational context. The hub device maintains the consolidated vault and pushes relevant context back to individual nodes.

In fully disconnected environments, each device maintains its own independent memory. When connectivity is restored, even intermittently, devices can sync their vault layers. The architecture is designed to treat synchronization as opportunistic, not required.

Swarm coordination for autonomous systems and distributed memory sync across disconnected networks are active areas of development on the roadmap. If this is a requirement for your deployment, we want to hear the specific operational scenario.

Ollama is a model runner. It handles downloading, managing, and serving GGUF models through an OpenAI-compatible API. It is a good tool for developers who want to run models locally on a workstation. It is not built for operational deployment in constrained environments.

Offline Intelligence adds the layers that Ollama does not have: persistent tiered memory that survives power cycles, hardware-aware resource allocation for embedded targets, production-grade API gateway with rate limiting and health checks, Prometheus metrics for operational monitoring, multi-language SDKs for integration into existing software ecosystems, and architecture specifically designed for classified and regulated environments where even the model runner itself cannot make external calls.

If you are building a demo or doing personal research, Ollama is excellent. If you are deploying AI to a forward operating base, a hospital, or an autonomous platform that will operate for months in the field, you need the infrastructure layer we are building.