Using LLM for Autonomous Vehicles: A Practical Guide

We are building a lightweight LLM-powered decision agent that consumes structured scene descriptions from an autonomous vehicle perception stack and returns a JSON driving plan with reasoning, controls, and risk flags. This is the same kind of prototype I ship to test new behavior policies before committing them to the C++ planning stack. Because Oxlo.ai charges a flat rate per request instead of per token, you can feed the model long sensor logs and detailed prompt context without the cost ballooning.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai

Step 1: Define the scene schema and Oxlo.ai client

I start with standard library dataclasses to keep the scene structured, then initialize the Oxlo.ai client. I use the OpenAI SDK because Oxlo.ai is fully compatible and requires zero client-side changes.

import json
from dataclasses import dataclass, asdict
from typing import List, Dict, Any

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

@dataclass
class DetectedObject:
    label: str
    distance_m: float
    velocity_ms: float
    lane: str

@dataclass
class Scene:
    ego_speed_ms: float
    weather: str
    traffic_light: str
    objects: List[DetectedObject]

def scene_to_prompt(scene: Scene) -> str:
    return json.dumps(asdict(scene), indent=2)

Step 2: Craft the system prompt for the driving agent

The system prompt is the contract. It tells the model exactly what JSON keys to emit and encodes hard safety biases, like always stopping for pedestrians within 10 meters.

SYSTEM_PROMPT = """
You are an autonomous vehicle planning agent.
Your input is a structured scene description in JSON.
Your output must be a single JSON object with exactly these keys:
  reasoning: string explaining the situation and trade-offs,
  action: one of [MAINTAIN_SPEED, DECELERATE, STOP, CHANGE_LEFT, CHANGE_RIGHT, PULL_OVER],
  target_speed_ms: float,
  risk_level: one of [LOW, MEDIUM, HIGH, CRITICAL],
  notes: string with any caveats or uncertainties.
Be concise and safety-first. If any pedestrian or cyclist is within 10 meters of the ego vehicle in the ego lane, your action must be STOP or DECELERATE.
"""

Step 3: Build the decision engine with structured output

Here is the core inference loop. I send the serialized scene to an Oxlo.ai reasoning model and parse the JSON response. I use kimi-k2.6 because its advanced reasoning and long context window handle complex multi-agent scenes well, but you can drop in llama-3.3-70b or qwen-3-32b without changing any other code.

def plan_driving_action(scene: Scene) -> Dict[str, Any]:
    user_message = scene_to_prompt(scene)

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )

    content = response.choices[0].message.content
    return json.loads(content)

Step 4: Add safety guardrails and fallback logic

No LLM output should reach actuators unverified. I add a lightweight rules engine that overrides dangerous commands, such as accelerating through a red light. This runs client-side in milliseconds.

SAFETY_RULES = {
    "max_speed_ms": 35.0,
    "forbidden_at_red": ["MAINTAIN_SPEED", "CHANGE_LEFT", "CHANGE_RIGHT"],
}

def validate_plan(plan: Dict[str, Any], scene: Scene) -> Dict[str, Any]:
    if scene.traffic_light == "RED" and plan["action"] in SAFETY_RULES["forbidden_at_red"]:
        return {
            "reasoning": "Guardrail override: traffic light is red.",
            "action": "STOP",
            "target_speed_ms": 0.0,
            "risk_level": "HIGH",
            "notes": "LLM plan rejected by safety layer.",
        }

    if plan.get("target_speed_ms", 0) > SAFETY_RULES["max_speed_ms"]:
        plan["target_speed_ms"] = SAFETY_RULES["max_speed_ms"]
        plan["notes"] = plan.get("notes", "") + " Speed capped by guardrail."

    return plan

Run it

This main block simulates a tricky urban scene: red light, a pedestrian crossing 4.5 meters ahead, and a stationary vehicle in the left lane. The agent should call STOP.

if __name__ == "__main__":
    scene = Scene(
        ego_speed_ms=13.4,
        weather="light_rain",
        traffic_light="RED",
        objects=[
            DetectedObject("pedestrian", 4.5, 0.0, "ego"),
            DetectedObject("vehicle", 12.0, 0.0, "left"),
        ],
    )

    raw_plan = plan_driving_action(scene)
    safe_plan = validate_plan(raw_plan, scene)

    print(json.dumps(safe_plan, indent=2))

Example output:

{
  "reasoning": "Traffic light is red and a pedestrian is 4.5 meters ahead in the ego lane. Continuing is unsafe.",
  "action": "STOP",
  "target_speed_ms": 0.0,
  "risk_level": "CRITICAL",
  "notes": "Pedestrian proximity and red light require full stop."
}

Next steps

Wire this module into a ROS2 node or a CARLA simulator bridge to close the loop with simulated camera and LiDAR feeds. You can also benchmark planning latency across Oxlo.ai models by swapping in deepseek-v3.2 or qwen-3-32b to see which fits your operational design domain.