Needle: The Open-Source Foundation Model With 4,400+ GitHub Stars That Runs AI on a 14MB Binary

What Is Needle and Why Should Developers Care?

The AI industry has been in a "bigger is better" arms race for years. Models with hundreds of billions of parameters dominate headlines. But what if the most impactful AI isn't the largest — it's the smallest?

Needle 2, created by Cactus Compute, challenges every assumption about what a foundation model needs to be. It's a 45-million-parameter model compressed into a single 14MB binary that performs tool calling, structured data extraction, and device control — all running locally on tiny hardware with just 28MB of RAM.

Today, Needle is trending on GitHub with over 4,400 stars and 315 new stars in a single day. It's gaining traction because it solves a real problem: how do you bring intelligent, tool-calling AI to devices that can't afford a cloud connection, a GPU, or even a gigabyte of memory?

The answer is elegant — and it fits in your pocket.

How Does a 14MB Model Actually Work?

Needle 2 is built on what Cactus Compute calls a Simple Attention Network (SAN) — a novel architecture designed from the ground up for efficiency at extreme scale constraints. Here's what makes it tick:

Hadamard MLP Instead of Traditional FFN

Instead of the standard feed-forward network layers found in most transformers, Needle uses a Walsh-Hadamard transform — a fixed mathematical matrix applied in O(n log n) time with zero weights to read from memory. This alone saves enormous amounts of parameter space.

CQ2-bit Quantization

The model uses Cactus Quants (CQ2-bit), an aggressive quantization scheme that compresses weights to just 2 bits. Despite this extreme compression, Needle trades wins with models 5x to 70x larger, including FunctionGemma (270M parameters), LFM2.5 (230M), and Apple Foundation Models — all running at full f16 precision.

Bounded Memory Architecture

Needle uses a 256-token sliding window with tool definitions pinned as KV sinks. This means total memory stays near 28MB regardless of conversation length. The model doesn't grow unbounded — it's designed for devices where every megabyte counts.

Engram Key-Value Memory

Rather than traditional KV caches that balloon with context, Needle uses hashed n-gram tables for memory retrieval. This "engram" system provides persistent context without the memory explosion typical of transformer inference.

# Install Needle in seconds
pip install cactus-needle

# The inference engine auto-downloads from HuggingFace and caches locally
# No separate model files, no GPU required, no network during inference

Tool Calling on the Edge: A Developer's Dream

Needle's killer feature isn't just its size — it's what it can do within that size. The model treats every problem as a function call, making it a purpose-built agent for IoT, smart home, and embedded AI applications.

Simple API, Three Levels of Control

Needle offers three tiers of tool definition, from zero-config to fully constrained:

Level 1: Decorate a function

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("what's it like in Lagos right now?")
print(result["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

Level 2: Describe arguments with docstrings

from typing import Literal

@needle.tool
def set_thermostat(temperature: int, mode: Literal["heat", "cool", "auto"] = "auto"):
    """Set the thermostat.

    Args:
        temperature: target temperature in Celsius
        mode: heating strategy to use
    """
    return {"temperature": temperature, "mode": mode}

agent = needle.Needle(tools=[set_thermostat])
agent.run("make it 21 and cool the room")

Level 3: Full schema constraints with needle.Field

from typing import Annotated

@needle.tool
def send_money(
    amount: Annotated[float, needle.Field(gt=0, le=10000, description="USD, up to 10,000")],
    to: Annotated[str, needle.Field(pattern=r"^@[a-z0-9_]+$", description="recipient handle")],
    memo: Annotated[str, needle.Field(max_length=80)] = "",
):
    """Send money to a handle."""
    return {"sent": amount, "to": to}

The constraints aren't just validated after the fact — they're compiled into a byte-level grammar that constrains every token the model can emit. The model literally cannot produce invalid output.

Confidence-Gated Responses

Every response from Needle carries a calibrated confidence score. This is critical for production edge deployments:

• Above threshold: Execute the action automatically
• Below threshold: Escalate to a larger model or ask the user

The confidence is the minimum of two signals: a calibrated post-hoc scoring head and the raw decoding probability. This dual-signal approach means the failure mode is escalation, not wrong execution.

{
  "type": "call",
  "success": true,
  "function_calls": [{
    "name": "set_lights",
    "arguments": { "room": "living room", "on": true, "brightness": 30 }
  }],
  "reasoning": "'living room' → room; 'dim' → on true, brightness 30",
  "confidence": 0.94,
  "prefill_tps": 4300.0,
  "decode_tps": 850.0
}

Real-World Example: Building a Smart Home Controller

Let's build a practical smart home controller that runs entirely on a Raspberry Pi (or even smaller devices) using Needle:

import needle
from typing import Literal, Annotated

# Define your home automation tools
@needle.tool
def set_lights(
    room: str,
    on: bool,
    brightness: Annotated[int, needle.Field(ge=0, le=100)] = 100
):
    """Control room lighting.

    Args:
        room: which room to control (kitchen, bedroom, living room)
        on: whether lights should be on or off
        brightness: light intensity from 0 to 100
    """
    # Your IoT bridge code here
    return {"room": room, "on": on, "brightness": brightness, "status": "ok"}

@needle.tool
def set_alarm(
    time: str,
    label: str = ""
):
    """Set an alarm for a specific time.

    Args:
        time: alarm time in HH:MM format (24h)
        label: optional label for the alarm
    """
    return {"time": time, "label": label, "status": "scheduled"}

@needle.tool
def play_music(
    room: str,
    genre: Literal["jazz", "classical", "rock", "pop", "ambient"] = "ambient",
    volume: Annotated[int, needle.Field(ge=1, le=100)] = 40
):
    """Play music in a specific room.

    Args:
        room: target room
        genre: music genre to play
        volume: playback volume 1-100
    """
    return {"room": room, "genre": genre, "volume": volume, "status": "playing"}

# Add environment context
agent = needle.Needle(
    tools=[set_lights, set_alarm, play_music],
    system="date: 2026-08-13 Thu 21:30; device: raspberry-pi; locale: en-US"
)

# Natural language commands
agent.run("dim the bedroom lights and play some jazz")
# → set_lights(room="bedroom", on=True, brightness=30) + play_music(room="bedroom", genre="jazz")

agent.run("set an alarm for 7am tomorrow called morning run")
# → set_alarm(time="07:00", label="morning run")

agent.run("turn off all the lights, I'm going to bed")
# → set_lights for each room with on=False

This entire system runs on a $35 Raspberry Pi with no cloud connection, no API costs, and sub-second latency. Your home AI works even when the internet goes down.

Structured Data Extraction Without the Cloud

Needle treats extraction as the same operation as tool calling — just with a single "tool" representing your data schema. This unified approach means the same confidence gating, grammar constraints, and fine-tuning pipeline work for both use cases.

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

# Extract structured data from unstructured text
invoice = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-09-01",
    Invoice
)
print(invoice.vendor)   # → Acme Corp
print(invoice.total)    # → 1200.0
print(invoice.due_date) # → 2026-09-01

This is powerful for mobile apps that need to process receipts, forms, or documents offline. Scan a receipt on a phone, extract the data, sync when you're back online.

Fine-Tuning: Make Needle an Expert in Your Domain

Needle supports LoRA fine-tuning on the frozen base model, then merges the adapter at export. The result is still a single .cact file that runs on the same engine — no special deployment needed.

# Step 1: Generate synthetic training data (optional, uses OpenRouter)
export OPENROUTER_API_KEY=sk-or-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# Step 2: Fine-tune with LoRA
needle finetune data.jsonl --epochs 3 --lora-rank 16 --lora-alpha 32

# Step 3: Build a tuned .cact binary
needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

# Step 4: Run your custom model
# The engine is weights-agnostic — same runtime, different brain

The training data format is simple JSONL:

{
  "query": "dim the kitchen to 10",
  "tools": [{"name": "set_lights", "parameters": {...}}],
  "answers": [{"name": "set_lights", "arguments": {"room": "kitchen", "brightness": 10}}],
  "reasoning": "'kitchen' → room; 'dim to 10' → brightness 10"
}

Benchmark Performance: Punching Above Its Weight

Despite being 5x to 70x smaller than competitors, Needle trades wins on standard benchmarks:

At 2 bits against their 16-bit, Needle achieves comparable tool-calling accuracy while using a fraction of the compute, memory, and storage. For edge deployments where every byte matters, this is transformative.

Key Benefits

• Truly on-device: No cloud dependency. Works offline, in air-gapped environments, and on battery power.
• 14MB single binary: One file to deploy. No model files, no config hell, no dependency chains.
• 28MB RAM ceiling: Runs on microcontrollers, wearables, and IoT devices that can't spare a gigabyte.
• Grammar-constrained output: The model literally cannot produce malformed JSON or invalid arguments.
• Confidence gating: Built-in uncertainty quantification means you can trust or escalate automatically.
• LoRA fine-tuning: Customize for your domain without retraining from scratch.
• Tool retrieval: Declare hundreds of tools — only the top 5 most relevant enter context per turn.
• Open source: MIT-licensed weights on HuggingFace, Python package on PyPI.

Getting Started in Under 5 Minutes

Needle ships with a built-in playground for interactive testing:

# Install
pip install cactus-needle

# Launch the browser-based playground
needle playground
# Opens at http://127.0.0.1:7860

# Or use your fine-tuned weights
needle playground --weights my_needle.cact

The playground lets you define tools, test prompts, and even trigger fine-tuning from the UI — all in your browser.

Who Should Use Needle?

• IoT developers building smart home and industrial automation systems
• Mobile app developers who need offline AI for forms, receipts, and data extraction
• Robotics engineers integrating voice-controlled tool use on embedded hardware
• Privacy-first applications where data must never leave the device
• Edge computing teams deploying AI on constrained infrastructure
• Startup founders who want AI features without per-request API costs

Frequently Asked Questions