My sister needs help cooking meals in the kitchen. Agents which read out recipes and provide feedback and improvements should assist her

# hf26challenge# devchallenge# weekendchallenge

Toluwaloju Kayode

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend What I Built I...

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built
I built Mise en Place, an AI-powered culinary sous-chef and interactive recipe workbench designed for a close friend who recently moved into their first apartment and wanted to stop relying on expensive takeout and start cooking real meals at home—especially cherished family recipes and authentic regional dishes they find online.

Whenever my friend tried to cook, three real-world kitchen frustrations consistently got in their way:

Scattered, Unusable Recipe Formats: Family recipes were trapped in photos of handwritten index cards or messy text messages, while online recipes were buried under 15 paragraphs of blog backstory, pop-ups, and ads.
Kitchen Math & Scaling Anxiety: Cooking for one or two people when a recipe is written for six meant doing fraction arithmetic (1/3 of 1 ¾ cups) and converting US volume cups into metric grams while food was actively sizzling on the stove.
Mid-Cook Panic with Messy Hands: Halfway through cooking, they would realize they were missing a key ingredient (like mirin or dry white wine) or didn't know what a sensory cue like "cook until translucent" or "al dente all'onda" actually looked like—and with flour- or oil-covered hands, scrolling on a phone screen was a nightmare.
Mise en Place solves all three problems by turning any recipe into a calm, hands-free kitchen station:

Multi-Format Recipe Upload (+ Upload Recipe): Upload a photo/screenshot of a cookbook page or handwritten recipe card, select a .txt/.md/.json/.pdf recipe file, or paste raw kitchen notes. The agent extracts categorized ingredients, exact numeric quantities, temperatures, and step-by-step instructions with embedded timer durations.
Live Web Recipe Sourcing & Python Schema.org Scraper (Web Sourcing): Search the web by dish name or by the ingredients currently sitting in your fridge, or paste any recipe URL. Our Python agent (scrape_recipe_url) parses Schema.org Recipe graphs directly from webpage HTML to strip away blog clutter while preserving clickable source citations. Deterministic Python Fraction & Density Scaling (Cook Mode): Scale any recipe from 1 to 24 servings and toggle between Original, Metric (g / ml / °C), and Imperial (oz / cup / °F) with real-time oven temperature conversions inside the step text. Smart Multi-Timer Dock & Hands-Free Voice Read-Aloud: Every time-bound cooking step generates a one-click countdown timer with Web Audio API chimes so multiple stove and oven stages can run simultaneously, plus a Read Step Aloud voice button for hands-free listening when your hands are covered in dough. Contextual Gemma 3 Sous-Chef Sidebar: Ask real-time culinary questions ("What can I substitute for white miso?", "What sensory cues tell me this step is done?", "How can I prep this dish ahead of time?") with full awareness of your active recipe, current step number, and scaled serving size. Demo <a href="https://www.loom.com/share/7c66921cdbfc4e178f9e6a1a587d5053">https://www.loom.com/share/7c66921cdbfc4e178f9e6a1a587d5053</a> Live Deployed Application: <a href="https://ais-pre-7d5qqkd5x7vda5c5bv7xes-679051571315.europe-west1.run.app">https://ais-pre-7d5qqkd5x7vda5c5bv7xes-679051571315.europe-west1.run.app</a> Quick Walkthrough to Try: Open the live demo and click + Upload Recipe → Load Sample Roast Chicken Note (or upload a photo of your own recipe) to watch the agent extract structured ingredients and timed steps. In Cook Mode, adjust the serving count (- / +) and toggle between Original, Metric (g/ml), and Imperial to see the Python fractions engine recalculate ingredient weights and oven temperatures (°F ↔ °C) in real time. Click Start Timer on any step to launch the multi-timer dock, or click Read Step Aloud for hands-free spoken guidance. Visit the Web Sourcing tab to search the live web for any dish or scrape a recipe URL directly. Visit the Python Agent tab to run the live Python 3.10 subprocess sandbox and download culinary_agent.py. Code GitHub Repository: <a href="https://github.com/Toluwaloju0/Cooking_AI">https://github.com/Toluwaloju0/Cooking_AI</a> Key files in the repository: agent/culinary_agent.py — Standalone, zero-dependency Python 3 culinary agent harness implementing Schema.org JSON-LD recipe scraping, fractions.Fraction & density-aware unit conversions, timer/temperature regex extraction, and Gemma 3 inference. server.ts — Full-stack Express + subprocess bridge connecting the frontend workbench to python3 agent/culinary_agent.py and Google GenAI / Gemma endpoints. src/components/CookModeWorkspace.tsx — Interactive 3-column kitchen station (Mise en Place checklist, Guided Step Timeline with TTS & timers, and Gemma Sous-Chef sidebar). src/components/WebRecipeSourcing.tsx — Live web recipe discovery and direct URL scraper interface. src/components/PythonAgentInspector.tsx — In-browser Python subprocess test bench and source code inspector. How I Built It Mise en Place is built around a hybrid architecture that pairs Google's open-weight Gemma 3 model with a custom, open-source Python 3 Agent Harness: Open-Weight Model (gemma-3-27b-it): The core conversational Sous-Chef agent uses Gemma 3 27B Instruction-Tuned (gemma-3-27b-it) via our Python inference bridge (run_gemma_inference in agent/culinary_agent.py). Every time my friend asks a question in Cook Mode, the agent injects a structured snapshot of the kitchen state—the active recipe title, scaled ingredient quantities, current step index, and target unit system—so Gemma 3 gives precise, step-specific culinary chemistry and technique advice. We also paired Gemma with multimodal document/photo extraction, live web search grounding, and neural Text-to-Speech (audio/wav) for hands-free step reading. Open-Source Python 3 Agent Harness (agent/culinary_agent.py): Built purely with Python 3.10's with zero external pip dependencies, ensuring instant portability across any machine or local kitchen device. Exposes four deterministic CLI and programmatic tools: scrape_recipe_url(url): Custom HTMLParser that extracts <script type="application/ld+json"> blocks, traverses @graph nodes to locate Schema.org Recipe objects, and parses ISO-8601 durations (PT1H30M → 1h 30m). scale_and_convert_recipe(recipe, target_servings, unit_system): Uses fractions.Fraction to compute exact kitchen fractions (1/8, 1/3, 3/4) and a curated INGREDIENT_DENSITY_GRAMS_PER_CUP lookup table (120g/cup flour, 227g/cup butter, 195g/cup Arborio rice) for volume-to-weight conversions, plus regex temperature conversion (450°F ↔ 230°C). parse_raw_recipe_text(raw_text): Normalizes Unicode fractions (½, ¾, ⅓), tokenizes ingredient lines into amount/unit/notes/category, and extracts step timer durations in seconds. run_gemma_inference(prompt, system_instruction, preferred_model): Invokes gemma-3-27b-it and returns transparent execution telemetry (Python version, model used, and execution latency in milliseconds). Full-Stack Kitchen Workbench: Built with React 19, TypeScript, Tailwind CSS, and an Express server (server.ts) that spawns python3 agent/culinary_agent.py via child_process.execFile and returns live execution traces to the UI. Why Does Open Innovation Matter? Cooking is deeply personal, cultural, and domestic—which makes open-source and open-weight AI essential to what Mise en Place achieves: Deterministic Culinary Math Where Black-Box Models Hallucinate: Baking and cooking rely on unforgiving chemical ratios (such as a 78% hydration sourdough boule or a 2% salt cure). Relying solely on a closed, black-box chat prompt to scale recipes or convert volume to weight frequently leads to hallucinated arithmetic and ruined meals. By building an open, inspectable Python tool harness (culinary_agent.py) alongside an open-weight Gemma 3 model, we separate concerns cleanly: deterministic Python fractions and density tables handle exact measurement math and Schema.org HTML scraping, while Gemma 3 handles culinary intuition, flavor pairings, and ingredient substitutions. Privacy for Family Heirlooms & Local/Offline Kitchen Autonomy: Many of the recipes my friend wanted to digitize are treasured family heirlooms that shouldn't be locked inside a proprietary cloud subscription. Because our agent is powered by open-weight Gemma 3 and a zero-dependency standard-library Python script, the entire culinary_agent.py harness can be downloaded with one click from the app and pointed at a local Gemma 3 instance (via Ollama, vLLM, or Hugging Face Transformers on a laptop or Raspberry Pi kitchen counter display) with zero cloud lock-in. Community Extensibility for Global Cuisines: A closed API cannot easily be customized for hyper-local ingredients or regional kitchen measurements. Because our Python agent harness is completely open and inspectable in the Python Agent tab, any home cook or developer can add regional ingredient densities (e.g., cassava flour, palm oil, or local grains), custom unit aliases, or dietary rules in seconds.