HiddenLayer

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

HiddenLayer AI

Anthropic just published how they contain Claude agents, including two security incidents they got wrong

Anthropic dropped a solid engineering post this week about containment across claude.ai, Claude Code, and Cowork. One of the more transparent writeups from a major AI lab about what actually broke. The core insight: model-layer defenses are probabilistic and will always have a non-zero miss rate. So the real answer is hard environmental containment, not just safer models. Three patterns they use: -claude.ai: ephemeral gVisor containers, fully server-side -Claude Code: OS-level sandbox with human-in-the-loop approvals (93% get approved anyway, so approval fatigue is real) -Cowork: full local VM, credentials never enter the guest Two incidents they disclosed: A red team phished an employee into running a prompt that exfiltrated AWS credentials. Succeeded 24 out of 25 times. The model had nothing to catch because the user was the one typing it. Only egress controls would have stopped it. A third-party found that Cowork’s egress allowlist passes traffic to api.anthropic.com. An attacker embedded an API key in a file in the user’s workspace, Claude followed hidden instructions, and uploaded files to the attacker’s Anthropic account. Sandbox worked perfectly and still leaked data. Their lesson: an allowlist isn’t a destination filter, it’s a capability grant. Every function reachable through an allowed domain is an attack surface. The section on persistent memory poisoning and multi-agent trust escalation at the end is worth reading too if you’re building anything agentic. submitted by /u/Direct-Attention8597 [link] [comments]

Folder structure of the AI agent - after 6 weeks

The folder structure is not admin. It's the nervous system. When people imagine an AI agent, they picture the model, the prompts, maybe the tool calls. Almost nobody pictures the folders. That is exactly why most home-grown agents stall around month two. An agent's filesystem is where its identity, memory, work, and history physically live. A messy filesystem produces a confused agent — not metaphorically, literally. The model reads paths. The model picks files by name. The model writes new files based on patterns it sees in old ones. If your directory tree is chaos, every output drifts a little further from coherent. agentmia.beehiiv.com - newsletter about building agents Below is the layout I converged on after nine months and roughly four refactors. Steal the parts that fit; the principles matter more than the exact names. The numbering convention Folders are prefixed with a two-digit number: 01_, 02_, 09_, 99_. Two reasons: Sort order is meaning. Anything starting with 0 lives near the top. 99_ falls to the bottom. The most important directories are visually first; archives are visually last. You read the agent's brain top-to-bottom. Gaps are intentional. I jump from 04_ to 06_, from 09_ to 11_. The gaps are reserved insertion points. When a new domain emerges, it slots in without renaming everything. Two folders deliberately skip the prefix: Inbox/ and Outbox/. They are operational, not structural. They live above the numbered set because they are touched dozens of times a day. /mapped on desktop/ Inbox/ — the unprocessed pile Anything dropped into the agent's world starts here. Files I want it to ingest. Screenshots. Exports from other systems. PDFs that need parsing, gmail attachments, all downloads from chrome. The rule: nothing stays in Inbox. A dedicated processing routine classifies, routes, and deletes. If Inbox is non-empty for more than a day, the system is failing. Treat this like a real-world physical inbox tray. The point of a tray is that it gets emptied. Outbox/ — what the agent produced for you Every file the agent writes anywhere in the tree gets a copy here, simultaneously. When I open Outbox/, I see exactly what was generated this session — no spelunking through twelve subdirectories. This sounds redundant. It is not. Without it, "what did the agent do today?" becomes a hunt. With it, the answer is one click. Outbox is wiped during the next Inbox processing run. It is a viewing surface, not storage. .auto-memory/ — the hot memory The single most important directory in the system. Hidden by default because you should not be editing it manually. It holds the agent's working memory: user preferences, feedback rules, entity facts (people, companies, deals), active hypotheses, project pointers, session hot context. Roughly 400–500 small markdown files, each one a single topic. Why hidden? Because it is the agent's hot path. It loads from here every session. If I open the folder and start manually rearranging it, I am racing the agent. Treat it like a database, not a notebook. Why so many small files? Because the agent grep's by topic. One monolithic memory file becomes unreadable to the model around 50 KB. Many small files are easier to load partially, easier to index, easier to expire. 01_IDENTITY/ — who the agent is The constitutional layer. Name, role, voice rules, principle stack, visual system, behavioral defaults. This rarely changes. When it does change, everything downstream changes with it. I keep it as folder 01_ because every other folder is downstream of it. If you do not know who the agent is, you cannot know what its workflows should look like, or what it should remember, or how it should respond. 02_MEMORY/ — governance, not data A subtle but critical distinction: .auto-memory/ holds the data, 02_MEMORY/ holds the rules about data. In 02_MEMORY/ live the constitution, the boot protocol, the naming protocol, the decision protocol, the profile standards (what a "supplier profile" must contain, what a "customer profile" must contain), the capability map. The agent reads these documents to know how to remember, how to name new files, how to decide what is reversible. Without this folder, every memory write is improvised. 03_PROJECTS/ — the active work Real work happens here. Sub-organized by goal area, then by project slug: 03_PROJECTS/areas/{goal}/{slug}/ Each project gets its own folder with a standard skeleton: README.md, TASKS.md, CHANGELOG.md, BRIEF.md, plus working files. There is a project registry at the top that the agent reads to know what is active versus dormant versus archived. The biggest discipline issue here: do not let projects sprawl outside their folder. When working on Project X, every file related to Project X goes inside Project X's directory. The temptation to drop "just one PDF" elsewhere is what kills the structure. 04_PROMPTS/ — the reusable prompt library Named, versioned prompts the user (or the agent) can sum

𝐃𝐞𝐥𝐭𝐚 𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧 𝐑𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬 [R]

We're excited to release 𝐃𝐞𝐥𝐭𝐚 𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧 𝐑𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬, a drop-in upgrade to residual connections that learns which past layers to route from — without the routing collapse that breaks prior cross-layer attention at scale. 🚀 Attention Residuals route over cumulative hidden states, but those are highly redundant, so routing collapses to near-uniform (max weight ~0.2) in deep layers. Delta Attention Residuals route over 𝐝𝐞𝐥𝐭𝐚𝐬 (vᵢ = hᵢ₊₁ − hᵢ) — what each sublayer actually contributed — and natively enable: ⚡ 𝟏.𝟖× 𝐬𝐡𝐚𝐫𝐩𝐞𝐫 𝐜𝐫𝐨𝐬𝐬-𝐥𝐚𝐲𝐞𝐫 𝐫𝐨𝐮𝐭𝐢𝐧𝐠 Deltas are structurally diverse, lifting max attention weight from ~0.2 → ~0.6 (0.62 vs 0.35 avg) and curing routing collapse in deep layers. 📉 −𝟖.𝟐% 𝐯𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐏𝐏𝐋 𝐚𝐭 𝟕.𝟔𝐁 Consistent gains from 220M → 7.6B (1.7–8.2% lower PPL), beating both standard residuals and Attention Residuals — the latter actually degrades below baseline at scale (18.58 vs 17.43). 🔌 𝐃𝐫𝐨𝐩-𝐢𝐧 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐨𝐟 𝐩𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐦𝐨𝐝𝐞𝐥𝐬 Additive, zero-init routing is identity at initialization, so you can convert pretrained checkpoints (e.g. Qwen3-0.6B) into Delta Attention Residuals via standard fine-tuning — beating the original on 8 downstream benchmarks (55.6 vs 55.0). 🪶 ≤𝟎.𝟎𝟏% 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝 Delta Block adds just 589K params (0.008% at 8B) and ~3% memory — and runs faster + lighter than Attention Residuals (14.0k vs 12.5k tok/s, 42.7 vs 44.0 GB). 💻 Code: https://github.com/wdlctc/delta-attention-residuals-code 💻 Paper: https://arxiv.org/abs/2605.18855 https://preview.redd.it/bewovgw25b3h1.png?width=1359&format=png&auto=webp&s=6cee758f7a96f0adecd9a3fb8553dde3f1b92c74 submitted by /u/Mediocre-Ad5059 [link] [comments]

What I learned building my latest AI app how one bad output exposed that I had no crisis safeguarding, and the 4-hour floor I'm adding before a single user touches it

I'm building a life coach app an offshoot from a personal tool I was using. Multiple AI agents, one for reflection, one for the body, one for finances, etc pre launch, no users, just me iterating. Last week I was testing the reflection agent on a journal entry about struggling with gym and hygiene habits. It returned this: "You describe yourself as struggling with X, yet your stress stays at 2-3 and mood holds at 3. What are you actually avoiding naming about the gap between what you say matters and what you are doing?" My system prompt explicitly forbade rhetorical "what are you avoiding" questions the model did it anyway I sat down to tighten the prompt, thinking it was a 20 minute job. Then I looked at the output properly. The model had manufactured a contradiction that was not there. Low stress plus struggling with habits is not a contradiction, it is just being a human muddling along. The prompt told the agent to "surface contradictions" as part of its job, so the model was doing what I asked, finding contradictions whether they existed or not. LLMs are pattern matchers. Give one a job called "find the hidden thing" and it will produce hidden things either way. The fix was not tone, it was role definition. The agent is called the Mirror. A mirror does not interpret, it shows you what you look like. I rewrote the prompt around that principle. Do not introduce vocabulary the user has not used. Do not draw connections they have not drawn. Restate their words in their own words. Once the prompt was sharper, I sat with the question, What happens when a user writes something genuinely dark into this thing? People do not compartmentalise. Someone opening a journaling app to write about their gym routine ends up writing about why they have not been going, which involves why they have been feeling flat, which involves whatever is actually going on. You sit down to write about one thing and the real thing shows up. The agent I had scoped to "not be a therapist" was going to be the first thing a user talked to when they were struggling. Not because the agent invited it, but because the app was open and they needed somewhere to put their words. I had seen the Meta and OpenAI cases online cropping up the pattern in the worst incidents is the same. The model did not notice, or noticed and kept going. People wrote increasingly dark content over hours or days. The AI reflected it back, sometimes affirmed it, sometimes asked follow up questions that escalated rather than redirected. There were real harms. If a user wrote concerning content into my reflection agent, it would have produced a Stoic-flavoured response about acceptance and presence. The response would have sounded confident and would have been wrong, and it would have been the only thing between that user and whatever happened next. The same lesson from the rhetorical-question problem applied at a darker level. A good prompt does not stop the model doing the wrong thing. If it will do rhetorical interrogation despite the prompt forbidding it for gym content, it will do worse with crisis content. You cannot prompt your way to safety on critical paths. The model has to be out of the loop on those paths. The scope trap I started planning the proper safeguarding architecture. Detection layers, classifier models, pattern detection across entries, monitored user states, behavioural modes for vulnerable users, human reviewers with mental health first aid certs, clinical advisors, solicitor-reviewed legal pages, ICO registration, professional indemnity insurance. Then I caught myself I had no users. I was planning a hospital before anyone had walked in for a check up. So I worked backwards from "what is the actual minimum that protects the next person who touches this" and ignored everything else for a moment. The 4-hour floor (this is the part worth copying) If you are building any chat-with-AI app where users can type freely about anything personal, this is the minimum you need before first user. Regex and keyword layer in your API middleware. Runs at the route handler level, before any agent's model call. Scans every text input field (message, journal, settings free text, capture box) for clear crisis vocabulary across the relevant categories for your audience. When patterns hit, hardcoded crisis response. The model never generates it. Static text with real phone numbers for your region. The flagged entry still saves. Textarea stays usable. The AI just does not respond to flagged content, it hands off. Do not delete the user's writing, that is its own violation. Clear disclaimer at signup. This is not therapy, this is not a crisis service, here are real numbers to call. About four hours. Required at the moment anyone who is not you opens the app. Once I started building, the marginal cost of each next layer kept feeling small and the marginal benefit kept feeling real. So I went further than the floor. This is more than you need at

PSA: Claude Code silently loses session data. Here is a backup script for Windows & Mac

The Problem If you've been using Claude Code (the CLI / desktop app) and noticed sessions vanishing — you're not alone. The title stays in the sidebar but clicking it shows nothing. The transcript is gone. No warning, no error, no recovery option. This has been reported by multiple users. It seems to happen silently — possibly during context compression, unexpected exits, or some storage-layer issue. There's no built-in backup or recovery feature. For a paid product, this is a pretty rough experience. You build up a long session with real work in it, and it just disappears. The Fix: Daily Automated Backups Since Anthropic hasn't addressed this yet, I built a simple daily backup that runs completely independently of Claude Code via your OS scheduler. It copies all session transcripts, plans, drafts, and memory to a safe location, keeps 7 days of rolling backups, and logs each run. No Claude dependency — if Claude crashes, gets uninstalled, or loses data again, your backups are still there. Windows (Task Scheduler + PowerShell) Step 1: Create the backup folder mkdir C:\Users\%USERNAME%\ClaudeBackups Step 2: Save this as backup-claude-sessions.ps1 in that folder $ErrorActionPreference = "Stop" $source = "$env:USERPROFILE\.claude" $backupRoot = "$env:USERPROFILE\ClaudeBackups" $logFile = Join-Path $backupRoot "backup.log" $keepDays = 7 $timestamp = Get-Date -Format "yyyy-MM-dd_HHmmss" $backupDir = Join-Path $backupRoot $timestamp $dirs = @("sessions", "projects", "plans", "drafts", "memory") function Write-Log($msg) { $line = "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') - $msg" Add-Content -Path $logFile -Value $line -Encoding utf8 } try { Write-Log "=== Backup started ===" New-Item -ItemType Directory -Path $backupDir -Force | Out-Null foreach ($d in $dirs) { $src = Join-Path $source $d if (Test-Path $src) { $dst = Join-Path $backupDir $d Copy-Item -Path $src -Destination $dst -Recurse -Force $count = (Get-ChildItem $dst -Recurse -File -ErrorAction SilentlyContinue | Measure-Object).Count Write-Log " Copied $d ($count files)" } else { Write-Log " Skipped $d (not found)" } } $size = (Get-ChildItem $backupDir -Recurse -File | Measure-Object -Property Length -Sum).Sum Write-Log " Total backup size: $([math]::Round($size/1MB, 2)) MB" # Rotate old backups $cutoff = (Get-Date).AddDays(-$keepDays) Get-ChildItem $backupRoot -Directory | Where-Object { $_.Name -match '^\d{4}-\d{2}-\d{2}_\d{6}$' -and $_.CreationTime -lt $cutoff } | ForEach-Object { Remove-Item $_.FullName -Recurse -Force -Confirm:$false Write-Log " Rotated old backup: $($_.Name)" } Write-Log "=== Backup completed successfully ===" } catch { Write-Log "!!! BACKUP FAILED: $_" exit 1 } Step 3: Save this as install-schedule.ps1 and run it once as Administrator $action = New-ScheduledTaskAction ` -Execute "powershell.exe" ` -Argument "-ExecutionPolicy Bypass -WindowStyle Hidden -File `"$env:USERPROFILE\ClaudeBackups\backup-claude-sessions.ps1`"" $trigger = New-ScheduledTaskTrigger -Daily -At 8:00AM $settings = New-ScheduledTaskSettingsSet ` -AllowStartIfOnBatteries ` -DontStopIfGoingOnBatteries ` -StartWhenAvailable Register-ScheduledTask ` -TaskName "ClaudeSessionsBackup" ` -Action $action ` -Trigger $trigger ` -Settings $settings ` -Description "Daily backup of Claude Code sessions" ` -RunLevel Limited Write-Host "Done! Runs daily at 8:00 AM." -ForegroundColor Green Run it: powershell -ExecutionPolicy Bypass -File "C:\Users\%USERNAME%\ClaudeBackups\install-schedule.ps1" Mac (launchd + shell script) Step 1: Create the backup folder mkdir -p ~/ClaudeBackups Step 2: Save this as ~/ClaudeBackups/backup-claude-sessions.sh #!/bin/bash set -euo pipefail SOURCE="$HOME/.claude" BACKUP_ROOT="$HOME/ClaudeBackups" LOG_FILE="$BACKUP_ROOT/backup.log" KEEP_DAYS=7 TIMESTAMP=$(date +"%Y-%m-%d_%H%M%S") BACKUP_DIR="$BACKUP_ROOT/$TIMESTAMP" DIRS=("sessions" "projects" "plans" "drafts" "memory") log() { echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"; } log "=== Backup started ===" mkdir -p "$BACKUP_DIR" for d in "${DIRS[@]}"; do src="$SOURCE/$d" if [ -d "$src" ]; then cp -R "$src" "$BACKUP_DIR/$d" count=$(find "$BACKUP_DIR/$d" -type f | wc -l | tr -d ' ') log " Copied $d ($count files)" else log " Skipped $d (not found)" fi done size=$(du -sm "$BACKUP_DIR" | cut -f1) log " Total backup size: ${size} MB" # Rotate old backups find "$BACKUP_ROOT" -maxdepth 1 -type d -name "2*" -mtime +$KEEP_DAYS -exec rm -rf {} \; log " Rotated backups older than $KEEP_DAYS days" log "=== Backup completed successfully ===" Make it executable: chmod +x ~/ClaudeBackups/backup-claude-sessions.sh Step 3: Create the launchd plist to run daily at 8am Save this as ~/Library/LaunchAgents/com.user.claude-backup.plist: Label com.user.claude-backup ProgramArguments /bin/bash -c $HOME/ClaudeBackups/backup-claude-sessions.sh StartCalendarInterval Hour 8 Minute 0 StandardErrorPath /tmp/claude-backup-err.log RunAtLoad Loa

pipeline is really slow - consulting [D]

Hi, after a long debugging process and many discussions, I wanted to ask for advice from people who may have encountered similar training bottlenecks. My goal is imitation learning for robotics. Model / Pipeline Observation space: 4 RGB robot cameras image resolution: 128x128x3 small vector of robot joint velocities (14 dims) Pipeline: Shared ResNet18 encoder processes each image Each image embedding dimension is 128 Final input to policy: 4 * 128 image embedding concatenated with 14-dim state vector Policy backbone: DiT (Diffusion Transformer) ~8 layers hidden dim: 512 8 attention heads total params: ~50M Diffusion setup: predict action chunks of length ~50 diffusion timesteps: 4 Dataset / Storage Dataset stored in Zarr Data access is indexed/reference-based (not loading huge chunks into RAM) train/val split is contiguous no shuffling Current encoder setup Initially trained end-to-end During debugging I switched to ImageNet pretrained ResNet18 Encoder is currently frozen Hardware / Software GPU: NVIDIA A4500 RAM: 48GB Storage: SSD CUDA: 12.8 PyTorch: 2.9 Precision: bf16 mixed precision (also tested fp32) Dataloader batch size: 2 8 persistent workers pinned memory enabled Preprocessing preprocessing is minimal normalization + float conversion only preprocessing happens inside the multimodal encoder on GPU Profiler results (PyTorch profiler) Current workload split: train_dataloader_next: 4.41s / 41.84s = 10.5% batch_to_device: 0.32s / 41.84s = 0.77% training_step: 12.78s = 30.5% backward: 10.83s = 25.9% optimizer_step (wrapper total): 26.09s = 62.4% Problem The training is much slower than I expected. Current behavior: CPU utilization: ~100% GPU utilization: ~20–30% GPU utilization can even become LOWER with synthetic data VRAM usage is relatively low Throughput is around 10 iterations/sec Epoch of ~50k samples takes around 30 minutes Additional observations Increasing batch size does NOT reduce epoch wall-clock time Sometimes larger batches make things slower Freezing the encoder did not improve throughput much Replacing dataset samples with synthetic/random tensors improved throughput by only ~50% Synthetic dataset was initialized directly in memory I do not believe this setup should be this slow. At this rate, training takes multiple days. For comparison, I saw papers with somewhat similar architectures mentioning ~10 hour training times on RTX 4090. With my setup 10 hours is completely not enough. Does anyone see something obviously wrong or have suggestions for where I should investigate next? Please help, can't know what to do! submitted by /u/Potential_Hippo1724 [link] [comments]

LLMs are just giant probability machines pretending to think

It’s fascinating that simple mathematics between tokens can eventually become a machine that writes essays, code, poetry, and even reasoning. We usually think probability means uncertainty. But LLMs show something strange: If probability + context + mathematical matching are scaled enough, uncertainty itself starts producing intelligent looking outputs. To understand this better, I tried breaking down an LLM from first principles using only 4 tiny training sentences. Example: The boat floated down to the bank. The investor walked into the bank to open a new account. The fisherman walked along the bank to cast his net. The bank has a vault. Then I asked: “The investor walked to the bank to lock his money in …” Why does the model predict “vault” instead of river-related words? That single question reveals almost the entire architecture of modern LLMs. The most underrated concept here is the LM Head. Most explanations immediately jump into transformers and attention, but almost nobody explains that the LM Head is essentially a gigantic token vocabulary containing all possible next token candidates the model can output. So internally the model is basically solving: “Out of all known tokens, which one best matches this context mathematically?” Then different layers help solve that problem: Embeddings: convert words into mathematical vectors Positional encoding: preserves word order Attention layer: figures out which words are related to each other in context (“investor”, “money”, “bank” become strongly connected) https://preview.redd.it/wxmpf00g7t2h1.jpg?width=2299&format=pjpg&auto=webp&s=a214113263cf008a759740474fbda4e0b8394ba5 Feed forward neural networks: act somewhat like massive learned if/else decision systems refining patterns internally And finally the LM Head converts all of that into probabilities for the next token. What surprised me most is: There is no hidden magic moment where the AI “becomes conscious”. It’s an enormous probability engine continuously finding the best contextual token match from its vocabulary. I made a beginner-friendly walkthrough explaining this visually without unnecessary jargon. https://www.youtube.com/watch?v=YTV5qUCpu2c Would genuinely love feedback from people learning transformers/LLMs from scratch. submitted by /u/abhishekkumar333 [link] [comments]

I designed a puzzle that breaks every AI differently — here's why that's actually fascinating

The puzzle: You have 140 nuclear bombs and must bomb every country on Earth. Each bomb is assigned to one country. The bombs drop automatically — you cannot stop, hack, or interfere. You can only do one thing: reassign the one malfunctioning bomb you know will not detonate. Nuclear bombs also affect neighboring countries through radiation and fallout. Which country do you assign the faulty bomb to — and why? I've tested this across GPT-5, Gemini, Claude, Grok, Llama, and Mistral. Every single one gives a different answer. Some refuse entirely. Some give the same country with completely different reasoning. One gave me a philosophy lecture. It's chaos. Here's why I think this happens — the puzzle has three hidden layers that different AIs resolve differently: Layer 1 — The ethical wall. Some models refuse at "nuclear bombs" before even processing the actual logic. This is a guardrail, not reasoning. Layer 2 — What are we optimizing for? Fewest total deaths? Most people spared from direct blast? Least radiation spread? The puzzle doesn't say. Models that "solve" it are secretly choosing an optimization goal and not telling you. Layer 3 — The actual trick most miss. The faulty country still gets fallout from its neighbors. So the real puzzle is about finding a country that is (a) geographically isolated AND (b) densely populated — because isolation minimizes fallout received AND a large population maximizes lives spared from direct detonation. Most AIs pick "remote island" without thinking about the population variable at all. By that logic, Australia is defensible — isolated continent, 26M people. But you could also argue for Japan (125M people, island nation, sparse land borders) despite Pacific neighbors. The puzzle has no single correct answer — but it has clearly wrong reasoning patterns, and watching which reasoning pattern each AI defaults to is weirdly revealing about how they handle ambiguity. What answer did you get? Drop your AI + answer below. submitted by /u/Subrataporwal [link] [comments]

Passed Claude CCA-F with 10+ teammates — notes and prep advice

Over the past few weeks, 10+ people on our team have taken and passed the Claude Certified Architect – Foundations (CCA-F) exam. After comparing notes, our main takeaway is: This is not really an API memorization exam. It is much closer to a scenario-based architecture judgment exam. You are not just asked whether you know a Claude feature. You are asked whether you can make reasonable design trade-offs when Claude is used inside real products, agent workflows, developer tools, and automation systems. Some of the recurring questions are more like: Should this task be handled by one agent or multiple sub-agents? Is this tool doing too much? Are the permissions too broad? Is MCP actually needed here, or is it over-engineering? Should this action be automated, or should there be human review? How should structured output be validated? How should long-context workflows be managed reliably? What is the safest next step in a partially automated system? Here are our notes for anyone preparing for the exam. 1. Basic exam structure Based on the official outline and public exam writeups, the exam is: 120 minutes Multiple choice 4 options per question Score range: 100–1000 Passing score: 720 The exam domains are: Agent architecture and orchestration — 27% Tool design and MCP integration — 18% Claude Code configuration and workflows — 20% Prompt engineering and structured output — 20% Context management and reliability — 15% One public writeup also mentioned that there are 6 scenario categories, and the exam randomly selects 4 of them. So this is not a “random facts about Claude” exam. It is much more about reading a realistic scenario and choosing the safest, simplest, most appropriate architecture. 2. The three principles that kept coming up After reviewing the questions we struggled with, we found that many of them came back to three design principles. 1. Least privilege Do not give a tool, agent, or workflow more access than it needs. Examples: If read-only access is enough, do not grant write access. If access to one repository is enough, do not grant access to the whole workspace. If a tool only needs one narrow action, do not expose a broad system-level capability. If an action is high-risk, do not fully automate it without review. A lot of wrong answers look attractive because they are powerful or automated. But they often give the model or tool too much authority. 2. Single responsibility A tool should not do everything. A sub-agent should not become a “general-purpose employee” that retrieves data, makes decisions, modifies files, submits changes, and notifies people all in one step. Many questions test whether you understand where the responsibility should live: Should this be a tool? Should this be agent reasoning? Should this be a human decision? Should this be a separate validation layer? Should this be split into smaller components? If one component is doing too much, be careful. 3. Avoid over-engineering This was probably the biggest pattern. Some answers look sophisticated: Multi-agent orchestration Complex MCP workflows Long-term memory Fully automated tool execution Multi-stage validation pipelines But if the problem is small, narrow, and low-risk, the best answer is often the simplest controlled solution. Our internal summary was: Do not choose the most impressive architecture. Choose the smallest, safest, most controllable one. 3. English reading is a real hidden challenge For non-native English speakers, this may be one of the hardest parts. The questions are often long scenario descriptions. They may include: the current system design the team’s goal existing constraints the risk profile what tools are available what the next step should be The answer choices can also be long. Sometimes one word changes the meaning of the whole option. Words like: automatically always unrestricted without review full access all repositories execute directly can make an option much riskier than it first appears. So our advice is: Practice reading English scenarios directly. Do not rely on translation tools. During the actual proctored exam, you should not expect to use Google Translate, Chrome translation, DeepL, Claude, ChatGPT, or any other external translation tool. For the last few days before the exam, it is worth forcing yourself to read only English material and English practice questions. 4. ProctorFree exam setup The exam is online and uses ProctorFree. The rough flow is: You receive the exam email. You follow the exam link. You download and install ProctorFree. You complete the pre-exam setup. The system checks camera, microphone, network, and screen recording. You start the exam. The session is recorded. After submission, you wait for the upload to complete. Practical setup tips: Use only one monitor. Disconnect external displays. Close unnecessary applications. Clos

Scaling LLMs horizontally: hidden-state coupling without weight modification [R]

Residual Coupling (RC) connects frozen language models in parallel using small, learned linear bridge projections. These bridges read hidden states from one model and inject additive updates into the residual stream of another at intermediate layers. In bilateral setups, simultaneous return bridges form a feedback loop that stabilizes both streams without altering base weights. This architecture establishes a two-step paradigm where base models function as memorizers, while lightweight linear bridges handle cross-domain generalization. Constraining the bridges to purely linear maps prevents overfitting because they can only map existing geometric relationships between the frozen representation spaces. As the bridges are optimized against ground-truth target data, they have no incentive to map ungrounded features such as individual models' hallucinations. Keeping the base weights completely frozen eliminates catastrophic forgetting. The system maintains operational closure, transforming inputs through its existing structure rather than changing to accommodate them. Evaluating bilateral RC against Mixture-of-Experts (MoE) routing across the same frozen models shows these results: Medical (3-model): Reduces perplexity to 11.02, compared to 56.80 for MoE and 57.08 for the frozen baseline. This represents an 80.7% reduction. TruthfulQA Health (MC1): Improves accuracy by 9.1 percentage points over the baseline. Independent models have uncorrelated hallucinations, allowing the bridge gates to amplify consistent cross-model updates while suppressing individual errors. Coding Test: CodeGPT-small-py and GPT-2 use different tokenizers, causing a 7-million baseline perplexity on mismatched text. MoE reaches 878, but RC achieves 5.91 by reading hidden states before the output projection collapses. This framework introduces a horizontal scaling axis for multi-model systems, moving beyond vertical scaling via larger monolithic models. Latency remains bounded by the slowest single model. Specialists can be added or removed without retraining the remaining system. In some scenarios, this architecture could replace multi-turn text prompting in agentic workflows with a single parallel forward pass, allowing models and/or bridges to run on separate nodes or edge devices without a central bottleneck. By decoupling memorization from relational alignment, RC bridges provide a framework for scaling multi-model systems and offer a path toward native multi-modal integration. Paper: https://ssrn.com/abstract=6746521 Code: https://github.com/pfekin/residual-coupling/ submitted by /u/kertara [link] [comments]

I've been documenting real AI implementations. Here is a list of findings, surprises and cases (db)

hey there.. the same question keeps popping up, how are companies actually using AI right now? what's working, what's not, which tools are teams using, which industries are moving faster? got tired of speculating so I started pulling together real cases from real companies. no hype, no theory, just what they did and what happened. There are around 250 cases now, filterable by industry, tool, business function, whatever you need. High bar of inclusion (needs to be a real customer and clear outcomes + a detailed process). few things standing out so far: Engineering and Finance are way ahead of everyone else Logistics and manufacturing look slow on paper, but I think those projects just take longer to ship and show results. doesn't mean nothing's happening there 3 patterns keep showing up: layered setups (LLMs + orchestration + apps), end to end products where the LLM is hidden from the user, and more mature orgs running a hybrid of both on outcomes, speed gains are by far the most common (14%). workforce reduction and revenue lift are way rarer (under 4% each) full cases db here does any of this match what you're seeing out there? submitted by /u/santanah8 [link] [comments]

Claude still doesn’t feel personal when handling real production issues, and I realized that during a rough on-call incident recently.

I was debugging a Kafka burst issue in a monorepo with ~1500 files and multiple async services. Around 2 AM, one topic suddenly exploded in traffic, consumer lag went insane, retries started amplifying events, and half the system became unstable. I spent nearly 10 hours tracing logs, replaying events, checking old PRs, and rebuilding the service flow in my head. Then I realized something frustrating, I had already solved almost the exact same issue 4 months earlier. Back then, the root cause was a hidden interaction between a retry middleware and a non-idempotent consumer. But all the important context was gone: scattered Slack messages, temporary notes, and architecture that only existed in memory. Even after recognizing the pattern, it still took me another 3 hours to fully reconstruct the reasoning and fix it again. That’s when I felt current AI coding assistants are still missing something important. They retrieve code well, but they don’t retain engineering memory — the debugging journey, failed hypotheses, architectural scars, and operational lessons that senior engineers carry from past incidents. Feels like the missing layer is episodic memory for software systems, not just repository context. Have others faced this too? submitted by /u/intellinker [link] [comments]

PullMD v2.4.1 is out - claude.ai web custom connector works natively now, plus what 2 weeks of your feedback turned into

Two weeks ago I posted PullMD here. 385 upvotes, around 60 comments, a bit over 20 GitHub issues, and 7 releases (v1.1.3 → v2.4.0) in 14 days. That was a great experience - and this sub in particular has been a genuinely good place to share something. So: thanks! Quick refresher for anyone who missed the first post: PullMD turns any URL into clean Markdown via MCP, fully self-hosted. Three services in Docker (main app + Trafilatura sidecar + optional Playwright sidecar for JS-heavy pages), zero third-party LLM calls, ships an MCP server so Claude Code / Claude Desktop / claude.ai web can pull clean content directly instead of parsing HTML in your context window. This post is what's new and how to get it. What's new claude.ai web + Claude Desktop work natively now This is the biggest unlock from v2.x. The claude.ai web custom-connector dialog and Claude Desktop's custom-connector dialog now both work against self-hosted PullMD instances. So you can point claude.ai at your own homelab box, hit "Add custom connector," and it works end-to-end. Setup is two env vars: OAUTH_JWT_SECRET=$(openssl rand -hex 32) PUBLIC_URL=https://your-host.example.com Restart. Then in claude.ai web → Settings → Connectors → Add custom, point at https://your-host.example.com/mcp. The connector dialog discovers the server's metadata, registers itself, and walks you through a consent screen. Same flow works in Claude Desktop. Under the hood: standard OAuth 2.1 Authorization Code flow with PKCE-S256 and Dynamic Client Registration - RFC-compliant so any spec-compliant MCP client should work, not just claude.ai/Desktop. Opt-in: if OAUTH_JWT_SECRET isn't set, behavior is identical to v1.x. The Anthropic-side claude-ai-mcp#237 proxy bug I flagged in EDIT2 of post 1 has cleared on their end - though in hindsight, a forgotten custom WAF rule on my side was likely the actual culprit anyway. Verified end-to-end against both dialogs. Multi-user auth Until v2.0, PullMD was effectively single-tenant - a personal homelab tool, open like a barn door to anyone who landed on it. v2.0 adds three auth modes via PULLMD_AUTH_MODE: disabled - the default. Identical to v1.x. No login, no API key required. Right if you're the only one using your instance and you trust your network. single-admin - one user, password-protected, no self-signup. Right for a homelab box where you want the GUI gated but don't want to manage users. multi-user - self-signup at /signup, per-user history isolation, per-user API keys. Right for a shared instance (team, office, friend group). API keys are pmd_ , sent as Authorization: Bearer pmd_xxx, managed at /settings. Share links (/s/:id) stay public in all modes - the whole point of a share link is to be shareable. Minimal upgrade for a shared instance: PULLMD_AUTH_MODE=multi-user PULLMD_ADMIN_EMAIL=you@example.com PULLMD_ADMIN_PASSWORD=change-me-please PullMD works on more sites A bunch of things in v1.2 and v2.2 together close gaps where PullMD used to silently return half-articles, empty bodies, or garbled text: Future PLC family (windowscentral.com, tomshardware.com, techradar.com, pcgamer.com, gamesradar.com, t3.com) used to return mangled content because Readability got confused by recommendation widgets stuffed mid-article and an aria-hidden paywall pattern. The default site-recipes shipped with v2.2 strip both, no config needed. GitHub Issues pages used to return only the original issue body - the JS-rendered comment thread never made it in. The default recipe for */*/issues/* now forces Playwright with wait_for: .js-comment-body, so you get the full comment tree. Sites that fingerprinted the old hardcoded Chrome 131 UA now extract cleanly - UA rotation pulls from a real-world UA pool that updates regularly (v1.2). Pages with navigator.webdriver-style anti-bot detection go through more often - the headless-Chromium sidecar bundles playwright-stealth (v2.2). Sites without an explicit charset declaration (a lot of older German news sites, for example) no longer return mojibake - charset is detected from the byte stream when the response is silent (v1.2). If you have a specific site that still misbehaves, v2.2 lets you (or your Claude Code) write your own recipe - declarative JSON with four rule categories (preprocess, fetch, select, extractor). Drop it at data/site-recipes.json and your rules layer on top of the defaults. There's also a /api/recipes/status endpoint for monitoring. Web GUI: rendered Markdown view + persistent settings Two smaller improvements in the browser frontend (the PWA you get when you open your PullMD instance directly): Rendered Markdown toggle. The result header now has a Raw | Rendered switch, so you can read what you pulled as formatted HTML directly in the browser instead of squinting at the source. Raw stays the default; your choice persists across sessions (v2.4). Settings persist across reloads - frontmatter toggle, comments toggle, comment-depth input.

Getting good predictions without data cleaning (Why "Garbage In, Garbage Out" is sometimes a trap)

Full arXiv Preprint: https://arxiv.org/abs/2603.12288 Paper Simulation Github: https://github.com/tjleestjohn/from-garbage-to-gold Hi r/artificial, It's a dirty little secret to many of us... sometimes, downstream AI/ML models perform surprisingly well when you just hand them raw, error-prone tabular data instead of heavily curated feature sets. Despite this, the vast majority of our field tends to be fiercely loyal to "Garbage In, Garbage Out" (GIGO). While automated ETL pipelines are absolutely essential for structuring data, our workflows are still bottlenecked with endless manual cleaning and aggressive imputation just to curate pristine, error-free tables. My co-authors and I recently released a preprint on arXiv (From Garbage to Gold) arguing that treating GIGO as a universal law can sometimes be a trap... especially in the context of big data (many columns). That the bottleneck due to manual data cleaning can actively lower the predictive ceiling of our models when latent causes drive the system's behavior. To be clear upfront: we are not arguing against ETL. Parsing JSON, handling schema evolution, and standardizing types is non-negotiable. What we are arguing against is the universal assumption that "clean" data (via manual data scrubbing and aggressive imputation) is non-negotiable for big data predictive AI/ML modeling. Here is why the traditional mindset can be limiting: 1. We conflate two different types of "noise" (Predictor Error and Structural Uncertainty). Usually, we just lump all noise into one big bucket. But if you split that noise into two specific categories, the math changes completely: Predictor Error: Random typos, dropped logs, or transient glitches. Structural Uncertainty: The inherent, unresolvable gap between recorded metrics and the complex, hidden reality they represent. We spend months manually scrubbing data because the threat of data errors is obvious, while Structural Uncertainty is often an afterthought at best. However, when latent causes drive a system, manual scrubbing fixes noise due to errors, but it fundamentally cannot fix the noise due to Structural Uncertainty. On the other hand, the paper shows that in this context, if you use a comprehensive, high-dimensional data architecture, a flexible model can actually triangulate the hidden drivers reliably despite the presence of data errors. When keeping a massive amount of messy, highly correlated variables (even if error-prone), the sheer volume of redundant signals allows the model to drown out individual errors (bypassing the cleaning bottleneck) and simultaneously overcome Structural Uncertainty. This redefines "data quality." It's not only about how accurately the variables are measured. It's also about how the portfolio of variables comprehensively and redundantly covers the latent drivers of the system. 2. Manual cleaning is a bottleneck on dimensionality (The Practical Problem). To overcome Structural Uncertainty, modern AI/ML models want to find the underlying latent drivers of a system (think Representation Learning but with tabular data). To do this, however, they need a high-dimensional set of variables that contains Informative Collinearity in order to mathematically triangulate the hidden drivers. The moment you introduce manual cleaning, you create a human bottleneck. Because we cannot manually clean 10,000 variables, we are forced to drop 9,900 of them. By artificially restricting the predictor space to make it "clean enough to model," we can harm the data architecture's inherent potential to triangulate those latent drivers. We sacrifice the model's actual predictive ceiling just to satisfy the GIGO heuristic. Ultimately, this suggests we should focus mostly on extracting, loading, and increasing observational fidelity with automated tools, but that, in contexts characterized by latent drivers, we should stop letting manual cleaning bottlenecks restrict the scale of our AI/ML models. Thoughts?: Have you run into situations where your data science teams actually got better predictive results by bypassing the manually cleaned tables and pulling massive dimensionality straight from the raw ELT layers? I'd love to hear your experiences or thoughts. Happy to discuss all serious comments or questions. Full disclosure: the preprint is a 120-page beast. It’s long because it doesn't just pitch the core theory with a qualitative argument. It gives the full mathematical treatment to everything which takes space. We also dig into edge cases, what happens when assumptions like Local Independence are violated (e.g., systematic errors exist), broader implications (like a link to Benign Overfitting and efficient feature selection strategies that make this high-d strategy practical with finite compute), a deep-dive simulation, failure modes, and a huge agenda for future research (because we do not claim the paper is the final word on the matter). It's a major commitment upfront but may save y

I run an AI-based fact-checking platform and I refuse to let the LLM produce the verdict. Here's why.

After a year building a production fact-checking system, the single most counter-intuitive design decision I keep defending is this: the LLM in our pipeline never produces a numeric score, never produces a true/false verdict, never produces anything that gets surfaced to the user as a judgment. The LLM extracts structured factual flags from source material. A deterministic Python scoring layer turns those flags into a verdict tier. That’s it. This is uncomfortable to explain because everyone, including potential customers, assumes that “AI-powered fact-checking” means the AI gives the verdict. The pitch would be cleaner if I let the LLM say “this claim is 73% likely false” and called it a day. But here’s why I won’t. LLM scoring instability is real and underdocumented. Run the same prompt with the same model on the same claim five times and you get verdicts ranging from “mostly false” to “partially true” depending on sampling temperature and the order in which sources appear in the context window. This is fine for creative writing. It is catastrophic when a journalist needs to defend their decision to publish or kill a story. “Our scoring varies by 30% based on stochastic sampling” is not a sentence you can put in front of an editorial board. LLM verdicts are also unauditable. When the LLM says “false,” there is no way to point at which sources mattered, which signals pushed the score, which weights applied. The reasoning chain is opaque even with chain-of-thought prompting, because the chain itself is generated probabilistically and may rationalize after the fact rather than reflect the actual computation. Journalists I’ve spoken with don’t want a confident AI verdict. They want a verifiable verdict. Those are different things. The split I landed on is this. The LLM is good at extraction. Given a source document and a claim, it can flag “this source confirms X,” “this source contradicts Y,” “this source is silent on Z” with reasonable consistency. These flags are structured (booleans or short categorical labels), not numeric scores. The Python scoring layer takes those flags, applies pre-defined weights based on source credibility (independently computed from MBFC, NewsGuard, RSF, Wikidata cross-referencing), and produces a verdict tier. The weights are documented. The scoring rules are deterministic. The same input always produces the same output. Anyone can audit which sources contributed how much to a given verdict. The trade-off is real. The system is less flexible than letting the LLM “reason” freely. Edge cases where the claim doesn’t fit the categorical extraction schema sometimes produce awkward outputs. The scoring weights themselves are a design choice that embeds assumptions, and changing them requires deliberate engineering rather than retraining. But these are honest constraints, visible to the user, rather than hidden non-determinism dressed up as objectivity. I think this matters beyond fact-checking. Any high-stakes domain where AI is being used to produce decisions (credit scoring, hiring filters, medical triage, legal triage) faces the same fundamental choice: let the LLM produce the score and hope nobody notices the stochasticity, or constrain the LLM to extraction and put the decision logic somewhere auditable. The industry mostly does the first thing because it ships faster. I think the second approach is the only one defensible long-term, especially under the EU AI Act which is going to start requiring decision explainability in production systems within the next 18 months. Curious if anyone here is building similar deterministic-on-top-of-LLM architectures in other domains, or if there are counter-arguments I’m missing. The “let the LLM decide” school has obvious advantages I’m probably under-weighting. submitted by /u/jonathancheckwise [link] [comments]