WLCG Open Technical Forum #12 · 2026‑08‑25

AI and Facilities

Towards trustworthy adoption of agentic management and use

Rob Gardner, Giordon Stark, Ilija Vukotic, Fengping Hu · University of Chicago

Special acknowledgement: Giordon Stark, original author of most slides.

Blueprint diagram: a systems administrator (MCP broker + Kubernetes API access) and a physics analyst (agentic analysis pipelines) each connect via USB-C-style links through an MCP broker to a WLCG AF or site's compute, GPUs, storage, and Kubernetes control plane

Slides AI-assisted by Claude (Anthropic) · title illustration AI-generated.

The argument

Facility context and trust challenges

The model layer

Moves fast. Not ours to own.

The facility layer

Context, identity, trust, packaging, equity.

Two trust problems: AI acting on the facility, and AI acting through it.

This is the spine of the whole talk; everything later is evidence or consequence. Say the claim in full (it came off the slide for the 40-word rule): an LLM becomes useful on YOUR cluster the moment it knows your storage, your scheduler, your data, and your identity model — and stays safe only when those are real. That layer is a computing-infrastructure problem and it ports across experiments; the model layer (which model is best this month, the runtime, the viral demo) is interchangeable. Spell out the two problems: ON = privileged actions a sysadmin would do; THROUGH = a physicist's analysis driven from their laptop. Deliver it as a thesis, then promise the structure: (1) a short existence proof from one WLCG site, (2) the sysadmin's view — monitoring, privileged operations, and the trust principles that let an agent earn them, (3) the physicist's view — laptop to facility through a credential-brokered gateway, and (4) the WLCG question: what here, if anything, is worth coordinating. Map to the charge out loud once: (2) is site operations, troubleshooting, and security, (3) is data-management and interfaces. Keep this slide tight; it's the frame.

Grounding

Assumed vocabulary

Agent

An LLM loop: decide, act, observe, repeat.

MCP server

A service exposed as tools + live context.

Skill

A reusable recipe, loaded on demand.

Runtime

The agent’s hands.

skills + MCPs + agents → a plugin; plugins → a marketplace.

Full definitions, spoken (trimmed off the slide for the 40-word rule): an agent uses an LLM to decide a step, call a TOOL, read the result, repeat. An MCP server (Model Context Protocol) is a standard adapter exposing a service (Rucio, a notebook, a DB) as tools plus live context any agent can call. A skill is STATIC context loaded on demand, a reusable recipe ("how we do a fit here"). A runtime is the app that gives the agent hands: runs commands, wires MCPs, enforces permissions (Claude Code, Codex, ...). A discoverable collection of plugins is a marketplace. Keep FAST — the OTF room is technical but not uniformly agent-literate. 60–90 s max. "Runtime" is the deliberate word choice; say once that the AI engineering community also says "harness" and move on. Timing: main-line but compressible.

Existence proof

In production today:
systems agents & a user gateway

Agents operating the infrastructure · physicists reaching it through an MCP broker.

Existence proof

AI at every layer of the facility

Analyst-facing

Find data, generate, submit, fit.

Operations

Reports, triage, human-approved emails.

Sysadmin-facing

Privileged copilots; trust earned, gated.

Daily HTCondor cluster report posted to Slack by the AI monitor agent

448 of 14,671 held

Drafted held-job email posted to Slack, approved by a human, then sent by the agent

draft → “approve” → sent

Same context, different blast radius.

Say the full card content aloud (trimmed for the 40-word rule): analyst-facing = find data, generate code, submit, recover, fit — low autonomy, high convenience; operations = cluster health reports, hold-reason triage, drafted user emails, always human-approved; sysadmin-facing = config/runbook copilots in the privileged lane. And the callout's tail: the facility decides where the human gate sits. Frames "agents on your behalf" as a LEVELS-OF-AUTONOMY spectrum (the brief asked for "AI/LLM at varying levels, user-facing to sysadmin"). Both screenshots approved by Giordon explicitly (the "AI monitor" poster is a bot user, not a person). Real detail: a scheduled agent scans HTCondor every morning and posts a Slack report — state summary, hold-reason breakdown, top users, jobs held >7 days, recommended actions. When YOUR jobs are stuck it drafts an email with a specific fix (real OOM case: raise request_memory based on actual observed usage vs the limit, write big outputs to /data) — a human replies "approve" in Slack, then it sends. Never autonomous. No HTCondor MCP for this one, just skills around the condor CLI.

Deep-dive · one triage run

Root causes, not symptom lists

HTCondor Queue Triage — 2026-04-01
Total held: 825 / 2395 (34%) — above threshold

 #  User      Held  Cause                    Action
 1  user-01    284  PERIODIC_HOLD >7d wall   Restructure, resubmit
 2  user-02    165  errno 13: /scratch/      Fix node perms, release
                    perms c111,c113          (node bug — blameless)
 3  user-03    155  PERIODIC_HOLD >7d wall   Restructure, resubmit
 4  user-04    121  Short queue >4h          Split <4h chunks
 5  user-05     65  errno 2: .tgz missing    Investigate, resubmit

Wall-time holds: 560/825 (68%) — three users, policy issue
Priority: fix /scratch perms c111+c113, then release user-02
(the agent does NOT do this autonomously)

Reads HTCondor via MCP; no ssh.
Root causes: a blameless node bug ≠ user error.
Never releases jobs autonomously.

Our expert caught 2 errors in 10 minutes: AI proposes, experts decide.

The user gateway, live

“One gateway. Your entire facility.”

mcp-portal.af.uchicago.edu · live

AI Assistant

MCP Gateway

Auth Policy Audit Broker

Rucio

HTCondor

Jupyter

Storage

Credentials stay behind the broker

Raw tokens never reach the assistant.

Add tools, not complexity

Authorize every call

Group-based access; audit logging.

How it works: part 03.

mcp-portal.af.uchicago.edu · live diagram reproduced from the portal’s landing page (af-mcp-platform).

The second thing the 01 divider promised: the user gateway, shown as the live product, 30-45 seconds only. The diagram is the portal landing page's own animation, reproduced faithfully in the slide (ported from af-mcp-platform portal/src/lib/gatewayPulse.ts): three scripted cycles on loop. Let it run one cycle while talking; when the third cycle plays, point at it: "Delete my /data datasets" gets a green Auth check, then Policy fails, the gateway buzzes red, and the assistant comes back with "I am not authorized". That's the whole governance story in eight seconds. The three messages on the right are the portal's own: credentials stay behind the broker, add tools not complexity, authorize every call. Do NOT explain the architecture here; say "hold that thought, part 03 opens it up" (the broker diagram and the one-command connect slide live there). If asked now: single endpoint, ~800 AF users behind it. Say the "not a demo" list aloud (trimmed off the slide): Rucio, AMI, HTCondor, JupyterLab, and filesystem access exposed to AI-powered workflows today; short-lived credentials minted per call. Fallbacks: with no network or reduced motion, the static diagram stands on its own (that's how the portal itself degrades too).

AI acting on the facility

The sysadmin’s view:
earning trust for privileged actions

Monitoring · privileged operations · trust principles — earned like a new operator: incrementally, observed, revocable.

Earned autonomy

A trust ladder, with promotion gates

Observe

read-only diagnostics

Suggest

a human approves

Submit job

scoped, reversible

Restart service

restarts stuck workers

Operate production

remediates, deploys

Promotion by evaluation: fault scenarios, scored, regression-tested. Autonomy measured, not assumed.

Everything below this slide in part 02 is a down-arrow deep-dive: git governance, levels, memory, the Lustre/Ceph honesty beat, enforcement, open-web policy — pull up what the room wants. Say the callout's cut tail aloud: fault scenarios curated in a staging environment, re-scored when the model changes — the same way society learned to trust autonomous vehicles: evidence first, hands off the wheel later, never everywhere at once. The Waymo model, named out loud: driver present, then hands off the wheel on mapped streets, then no driver — each step gated by accumulated evidence, bounded by geography (for us: bounded by blast radius). Concrete ops verbs on the cards (Observe -> Suggest -> Submit job -> Restart service -> Operate production). The eval-as-promotion-gate idea comes from the IRIS-HEP/OSG-LHC concepts doc (trust ladder + facility-ops eval harness: injected faults, scored outcomes, regression on model change). This is the governance answer to "what may an agent do?" — quantified reliability a site operations team can defend to its own management.

Deep-dive

Governance lives in a git repository

Principles

Auditability, minimal privilege, human-in-the-loop.

Policies

“Read-only by default;” “no outbound web.”

Runbooks: the atomic unit of trust

PR-gated: autonomous vs human-required.

governance/
├── principles/
│   ├── auditability.md
│   └── minimal-privilege.md
├── policies/
│   ├── condor/query-rate.yaml
│   └── network/egress.yaml
└── runbooks/
    ├── htcondor/queue-health.md
    └── kubernetes/cluster-health.md

AI drafts → expert reviews → merged. The PR is the audit record.

Card detail, spoken: principles are immutable commitments (human-in-the-loop UNTIL TRUST IS EARNED); policies are scoped YAML derived from them ("wait for human ack after restart"); a runbook is a human-reviewed, PR-gated spec per capability — what it does, what it may do autonomously, what requires a human. The full loop: AI drafts from cluster ground truth, the domain expert reviews and corrects, merged = blessed, runbook + agent config = a deployed capability. The tools sysadmins already trust — git, code review, PRs — become the trust machinery for agents. Nothing exotic: a principle is a value, a policy is that value made specific to one service, a runbook is the contract for one capability. The runbook is deliberately the atomic unit of trust: an action type is either listed as autonomous or it requires a human, and changing that list is itself a PR. Say out loud: "if you can review a colleague's change, you can review an agent's charter." This and the next two slides are the concepts behind our UChicago implementation; the framework names are deliberately left off — the concepts are what port.

Deep-dive · blast radius

Three governance levels

Level 1 · Assisted

Humans execute. Blast radius: a document.

Level 2 · User-scope

User credentials only. A quota.

Level 3 · Privileged

Scoped accounts. The account.

Dev sandbox

Staging cluster

Production

Nothing skips a step.

Card detail, spoken: L1 = AI drafts runbooks and diagnoses, no autonomous action; L2 = agents act for a physicist with that user's credentials only, no admin access ever; L3 = scheduled agents, scoped service accounts, read-only defaults, explicit write allowlists, PR-gated changes. The callout's cut tail: governance level = what it may do, environment = where; the framing is blast radius, not compliance. Environment stair: dev proves the pattern, staging proves it under load. Level 1 is where every site can start tomorrow: AI-assisted runbook writing, humans at the keyboard. Level 2 is the physicist story from Part 2 — the agent holds the user's own credentials and nothing more, so the worst case is what that user could already do. Level 3 is the interesting one for this room: scheduled, privileged automation, made tolerable by scoped service accounts, read-only defaults, explicit write allowlists, and PR-gated promotion. The environment progression is mandatory at our site: a dev sandbox cluster, then a staging cluster, then the AF — patterns are proven twice before they touch production. Say: "the question we ask is never 'is it smart enough' but 'what is the blast radius if it is wrong.'"

Deep-dive

What the agent knows

Normative source

What it may do.

Runtime state record

What it has learned.

DR archive

Session memory

Runbook

Agent learns

Weekly review

Autonomy changes are PRs, a trust escalation log: when you stopped watching.

Card detail, spoken: normative source = runbooks, policies, agent configs; runtime state record = accumulated knowledge + daily session logs, auto-pushed to git, human-reviewed; DR archive = daily snapshots of agent state and conversations, restart = restore + wait for human ack; session memory = live in-process state, lost on crash, and that's fine. This is the answer to "how would I even know what it's doing?" — the question every experienced admin asks first. Two distinctions to say out loud: runbooks are read-only and procedural (what it CAN do); memory is read-write and experiential (what it HAS learned) — a window into the running agent's mind, in git, diffable, reviewable at the weekly ops meeting like any other change. The promotion loop is the payoff: the agent's operational experience flows back into the human-owned runbook only through review. And the trust escalation log is the single most quotable governance idea we have: autonomy changes are themselves version-controlled — you can point to the commit where you stopped watching, who approved it, and on what evidence. Down-arrow: the "no context — stay quiet" honesty beat (Lustre-vs-Ceph) if the room wants it.

Deep-dive · the flip side

No context? It should stay quiet

What happened

Our agents confidently recommended Lustre tuning, for a Ceph filesystem. Plausible, fluent, wrong.

The rule: ungrounded → silent. Context is what makes an agent safe to trust.

Deep-dive · enforcement

Enforcement is dual

Policy awareness inside the agent; technical enforcement outside.

Podsk8s default

Sandboxbubblewrap, sydbox

Policynode-level

Privilegeearned

The honest gap: runtimes advise; they can’t hard-deny. Blocking needs a node policy engine or sandbox.

Cut from the slide for the 40-word rule, say aloud: two mechanisms, both required, never one without the other — the agent reasons about the rule before acting (managed CLAUDE.md, allow-lists), and RBAC, scoped service accounts, network isolation, allowlists enforce outside it. Lincoln's line ("an agent just lets users 100x the bad behaviour we've had for decades") moved fully off-slide — deliver it with the meme. Say each step out loud: pods are the k8s default (resource caps, network policy, no host access); off k8s, sandboxes like bubblewrap or sydbox fence files/network/hardware on bare login nodes; policy is the fine-grained node-level layer (policy engines — keep it high-level, details are not my lane); privilege is earned, per the trust ladder earlier in this section. A system-wide /etc/claude-code/CLAUDE.md + permission allow/deny is stronger than "notes" but is NOT kernel-level enforcement; cf. claude-code issue #70184 (can't reliably deny commands at user level). Lincoln's line: "Claude lets users 100x the same bad behaviour we've had for 30 years" — good honest laugh beat, attribute lightly if asked. The meme in the corner is the punchline for that line — let it land for a beat before moving on, don't over-explain it. NIKHEF ANGLE: a Tier-1 with mixed bare-metal can't rely on k8s isolation everywhere. HONEST NEAR-MISS (tell live, don't put on the slide): our held-job email agent hit a race condition once — two agents grabbed the same approval and it nearly double-sent — so we stood email-sending down until it was fixed. Human-in-the-loop gates caught it; "honest over flawless."

Deep-dive · a security reality

Agents don’t get the open web

Facilities block agent web access: prompt injection.

Stale-prone

use /path/2024-egamma-recs.config

Durable

“ask chATLAS for the current path.”

Minimal durable guidance + a trusted live source: a facility decision.

Cut for the 40-word rule, say aloud: NERSC, UChicago, SLAC, BNL, AmSC, NRP are all blocking agent web access against prompt injection from untrusted pages. A skill can't just say "go read the docs": it encodes JUST ENOUGH guidance and delegates "what's the latest?" to a curated MCP (the MCP owns "what's current"), not the open internet. What an agent may reach is a facility decision, not a user convenience. From the Giovanni Guerrieri / ATLAS-AMG email thread + Giordon's reply (Jun 2026). Core point: the "just point AI at the docs + basic guardrails" instinct fails because AFs block agent web access (prompt-injection hardening); hence skills hard-code "just enough" and use an MCP (e.g. chATLAS) for the latest paths/recommendations. Curated webfetch via an MCP (cf. Giovanni's cern-mkdocs-mcp docs_sources.json) is the controlled alternative to open web access. research.md §skills-governance. NEW, opinionated, very current slide — plays well with the technical/security crowd.

AI acting through the facility

The physicist’s view:
laptop → facility

The physicist hardly leaves their laptop; the facility’s job is a safe port to plug an agent into — the broker · self-service access · pipelines · identity.

The USB-C port of the facility

A credential-brokered MCP gateway

One socket. The client never holds raw credentials: the broker authenticates, mints short-lived credentials, audits every call.

Claude, Gemini or any MCP client to the mcp.af.uchicago.edu front door (oauth2-proxy, AF Keycloak OIDC), through a FastMCP aggregator and AF credential broker (FastAPI /v1) doing identity, authZ, credential brokering and audit, fanning out to rucio-mcp, ami-mcp, openmagic, panda-mcp, condor-mcp, gitlab-mcp, jupyter-control and an Nth backend with no code change

Live at mcp.af.uchicago.edu: Giordon Stark’s af-mcp-platform.

af-mcp-platform · rucio-mcp · ami-mcp · atlasopenmagic · panda-mcp · condor-mcp (Bockelman, CHEP 2026).

Cut for the 40-word rule, say aloud: the STRATEGIC BOUNDARY — identity, authZ, credential brokering, and audit in one place, the same boundary a site needs regardless of AI. THE highlighted named artifact of the talk: Giordon's af-mcp-platform (github.com/maniaclab/af-mcp-platform), the new way to interface with the AF. Say the USB-C line out loud: one standard socket, any device on the other end; the facility standardizes the port, not the laptop. Real numbers: ~800 users, single endpoint mcp.af.uchicago.edu, brokers to Rucio/PanDA/AMI/GitLab/Jupyter/HTCondor. Next slide: how a physicist actually connects — the self-service portal. Diagram is images/NEW-af-mcp-arch.svg: client (bearer token only) -> mcp.af.uchicago.edu (oauth2-proxy -> AF Keycloak OIDC) -> FastMCP aggregator + AF credential broker (FastAPI /v1; Identity / AuthZ / Credential brokering / Audit) -> rucio-mcp, ami-mcp, openmagic (theory/MC gen), panda-mcp, condor-mcp, gitlab-mcp, jupyter-control, Nth backend (no code change). Say out loud: the broker is the ONLY thing that ever touches raw x509/IAM; the client just carries a short-lived bearer token. Deployed via Flux CD/Helm; the portal at mcp-portal.af.uchicago.edu issues bearer tokens. ~800 users behind the one endpoint.

Self-service, today

Getting connected takes one command

Any MCP client, one endpoint:

claude mcp add --transport http atlas-af \
  https://mcp.af.uchicago.edu/mcp

401 → OAuth discovery → browser login. No manual tokens.
No OAuth flow? Mint a token at the portal.

mcp-portal.af.uchicago.edu

Link identities, mint x509/VOMS proxies, manage tokens, see your backends.

mcp-portal.af.uchicago.edu · github.com/maniaclab/af-mcp-platform.

Deep-dive · the bottleneck

Who is this client, and may it act as you?

Who is it?

Do we trust it?

How much authority?

Today: DCR. Worth watching: CIMD (identity as a URL). Atop WLCG tokens + AARC.

Cut for the 40-word rule, say aloud: every MCP client — Claude, an internal tool, something a student wrote — wants to connect; once AGENTS are the ones connecting this stops being one-time setup and becomes a continuous, scalable decision. The three questions: a durable identity (not just a name); trust by policy, not vibes; authority scoped and auditable. Identity is becoming the bottleneck, not an afterthought, with WLCG's x509→OIDC transition as the backdrop. DCR is the adopted, stateful mechanism; CIMD is a newer DRAFT (stateless, client identity as a URL the server fetches) — worth watching, not yet the default. All of it sits ON TOP OF WLCG token profiles, the AARC blueprint, IAM — not instead of them. SIMPLIFIED main line (moved DCR/CIMD jargon to the two backups) — for a mixed colloquium room, the memorable takeaway is "identity becomes the bottleneck," not "DCR vs CIMD." Say the three questions out loud as the research problem. IMPORTANT ACCURACY NOTE: don't imply CIMD is "the solution" — DCR is what's actually adopted today (an approved RFC, what OIDC-based MCP clients follow in practice); CIMD is still an IETF draft, not yet a real-world default, even though the MCP spec itself prefers it (SHOULD vs MAY). Frame CIMD as "a newer proposal worth watching," not "the standard." AARC/WLCG line matters for Groep/Campana in the room. This is a place to shape the standard, not just consume it — a facility/Tier-1 voice in how research infrastructure adopts client-identity models. Full DCR-vs-CIMD mechanics + the real trace are the two backups (down-arrow) if pressed.

Deep-dive · CIMD vs DCR

Client ID = a URL the server fetches

GET /authorize?client_id=https://app.example/oauth.json&…
        │  authn server fetches the JSON, validates, applies policy
        ▼
{ "client_id":"https://app.example/oauth.json",
  "client_name":"…","redirect_uris":[…] }   ← hosted by the client

Wins: no /register abuse; stable identity; domain allow/deny.
Costs: outbound fetches (SSRF); localhost ambiguity; domain ≠ trust.

Auth today reuses grid creds (x509) or Keycloak OIDC — OAuth bridge docs.

BACKUP. Cut from the slide: SSRF mitigations (block private IPs, HTTPS-only, size/timeout caps), and the concrete FastMCP knob — CIMDTrustPolicy(trusted_domains=["claude.ai", "cursor.com"], auto_approve_trusted=True): the facility decides which domains skip consent. The client_id IS the HTTPS URL to a metadata doc the client hosts; the authn server fetches it on demand, validates client_id == source URL and the redirect_uris, then applies its own policy. Read-only, stateless, fetch-on-demand. Spec status: adopted into MCP 2025-11-25 via SEP-991, from an IETF draft promoted to WG status Oct 2025 (origins in IndieAuth; in production at Bluesky); MCP Python SDK added minimal client-side support in v1.23.0, server-side still maturing. Stance: CIMD = SHOULD, DCR = MAY, pre-registration still allowed. Software statements (RFC 7591 §2.3) / OS attestation are the next layer if asked. The second backup (down-arrow) has a real trace, the actual Claude client_id URLs, and the full spec list — pull that up only if someone like Groep or Campana pushes on mechanics.

One pipeline, four personas

Agent personalities

data_agent“get the dataset”

transform_agent“make the histograms”

batch_agent“run the cluster”

inference_agent“fit the limit”

Rucio · dCache

coffea · ServiceX

HTCondor · Dask

pyhf · RooFit

Personas speak intent, never backend names: portable across facilities.

Say aloud: four personas the physicist's assistant adopts; the facility wires each to its backends. Reframed (Rob, night before): the four intent tools presented as agent PERSONALITIES / personas the physicist's assistant adopts — a data steward, a transformer, a batch runner, a statistician — each mapped facility-side to local backends. Say the full backend lists aloud: Rucio/dCache/Open Data; coffea/uproot/ServiceX; HTCondor/Slurm/Dask/REANA; pyhf/cabinetry/RooFit/SBI. The persona never names "coffea" or "condor"; that indirection is what lets the same laptop plug into a second facility by adding a second broker endpoint — the multi-facility punchline, say it out loud. HS3 + pyhs3 are the pyhf/RooFit bridge if the stats-interop question comes: inference_agent privileges no single fitter.

Beyond one facility

The WLCG question:
coordinate what, exactly?

Options

What could be common?

Standardize the port

a conformant interface profile

Authorize the clients

client conventions on WLCG tokens

Share the knowledge

runbooks and skills, single-source

Operate together

shared eval scenarios

Which, if any, do we do together?

AARC · EOSC · ESCAPE · IRIS-HEP/OSG-LHC concepts doc.

Cut for the 40-word rule, say aloud: the governance model and interface patterns PORT; policies, playbooks, and trust surfaces stay local. Each candidate could become a WLCG activity, an HSF-style community project, or stay experiment/site-specific. What likely does NOT need coordination: model choice, runtime choice, per-site policy content. Card detail: the port = the way WLCG already standardizes site interfaces; clients = identity, token exchange, audit; knowledge = reviewed, not a dozen forks; operate = eval scenarios for ops agents across sites. Deliberately neutral: four candidates, laid out evenhandedly, with an explicit not-worth-coordinating list so the menu is credible. Sources: the IRIS-HEP/OSG-LHC concepts doc (conformant facility-MCP interface + conformance suite; RFC 8693 token exchange from WLCG/SciTokens to AI backends; the eval harness), plus the skills-commons debate in ATLAS. If pressed for a personal view, the honest answer: the interface profile and the client-identity conventions are the two with the clearest WLCG shape — both look like things WLCG has standardized before; the knowledge commons wants a community home more than a committee; and evaluation sharing is the youngest idea. Down-arrows: the skills-commons debate, and the ESCAPE bridge as evidence this crosses sciences, not just experiments.

Deep-dive · the commons

One commons, not a dozen forks

Who hosts, who maintains, who stops the forks?
ATLAS today: a single-source marketplace; reference, don’t copy.
An eval/lint harness keeps skills orthogonal.

Open: the cross-experiment contribution. ESCAPE is one model.

US AI proto-collaboration update: DPF 2026.

Cut for the 40-word rule, say aloud where useful: skills are the durable context layer. The live ATLAS debate: single-source marketplace (reference skills by URL, fix upstream) vs duplicated copies; docs next to the source code; official e-group-governed repos over personal ones; still open how to LINK, not copy (usatlas/marketplace#60). Semantic confusability: overlapping descriptions degrade routing — hence skill-design guides plus the eval/lint harness. Third option worth watching: package skills like software — prefix.dev's skill-forge ships skills as versioned conda/pixi packages, lockfile discipline for context. US ATLAS already operates the marketplace others converge toward (agentskills.io practice); the US AI proto-collaboration is an umbrella over IRIS-HEP and other funded projects + research groups, open to international collaborators. From the same AMG thread (Giordon, Zach Marshall, Gordon Watts, Tomas Dado, Giovanni). Points to draw on: single-source via marketplace.json/plugin.json — reference, don't duplicate (usatlas/marketplace#60); docs-next-to-source; official egroup-backed repos (Zach + Watts: CERN/AMG-hosted eventually, but US ATLAS is ahead today and shouldn't be slowed); semantic confusability + eval/lint to keep skills orthogonal (Giovanni's Lumi skill-design guide). agentskills.io = the practice Giordon follows. research.md §skills-governance. Ties directly to #discuss "who maintains context". NIKHEF ANGLE: positions Giordon as already running the commons, and frames a concrete European/ESCAPE role. LIVE FORUM (the half-on-slide answer): we do NOT yet have the European contribution to the commons; pose it as open. ESCAPE is the model for how such a collaboration could form and persist. On the US side (open to international collaborators) there's an AI proto-collaboration update at the upcoming DPF meeting (indico.fnal.gov/event/72820, 2026-07-23): CLARIPHY runs a dedicated Slack workspace for the AI proto-collaboration; the framing is that the AI collaboration is an UMBRELLA bringing together multiple funded projects (IRIS-HEP is just one) plus individual research-group contributions. Name CLARIPHY/DPF aloud if asked; keep the slide to "US-side AI proto-collaboration, DPF July 2026".

Deep-dive · ESCAPE

The agentic layer, meet ESCAPE

Data Lake (Rucio), VRE (k8s, Jupyter, REANA), OSSR: multi-science.
The hosted Rucio MCP already serves an escape site.

# escape.cfg — a real, shipped site config
[client]
rucio_host = https://vre-rucio.cern.ch
auth_host = https://vre-rucio-auth.cern.ch
oidc_issuer = escape
oidc_audience = rucio

Gateway + playbooks on the VRE: built once, serves LHC and astroparticle science.

rucio-mcp/data/escape.cfg.

Equitable inference

Who pays for inference?

Credit-card-per-student entrenches inequality, between groups and countries.
Host open-weight models on facility GPUs (NRP-style).
Bridge identity: WLCG token → scoped inference token (RFC 8693).

A facility responsibility: GPUs, model hosting, fair-share policy.

Agents already perform full HEP analyses end-to-end — Moreno et al., arXiv:2603.20179 — and that run kept stalling on usage rate-limits: access is already the binding constraint.

Cut for the 40-word rule, say aloud: if agentic analysis becomes normal, access to inference is a fairness issue. Hosting foundational models the experiments depend on keeps reconstruction/inference services from being hostage to a single vendor or budget line. The token exchange (RFC 8693) is MCP-aware — it scopes which models AND which tools. The cost is already real: tens to hundreds of dollars per run at frontier rates, numbers in the down-arrow. The brief's "equitable access to LLMs through hosting open-models + hosting foundational models used by experiments." Tie to the model-swap slide (§2): self-hosting open weights was already framed as cost control; here it becomes EQUITY. NRP-hosted LLMs (nrp.ai/llms) as the precedent. NIKHEF ANGLE: Nikhef has GPUs + a Tier-1 + a DEI commitment in the posting — equitable inference is a natural institutional value to name out loud. Also connects to "foundational models for experiments" = e.g. hosting large reconstruction/sim-surrogate models centrally. The token-exchange bullet is OSG-LHC doc Concept 4 (RFC 8693: WLCG/SciTokens → scoped token for nrp.ai-style backends, MCP-aware). COST: down-arrow backup has the per-run numbers. Be clear the 1M-in + 1M-out figure is MY illustrative floor (code-gen is ~1:1), NOT a number from the Moreno paper — the paper reports only a ~$200/mo Claude Max subscription and ~10h wall-clock with rate-limit idle. The rate-limit detail IS the equity hook.

Deep-dive · cost

Per analysis: tens to hundreds of dollars

Illustrative: ~1M in + 1M out tokens per analysis.

Frontier (hosted)

$100–200+

Mid-tier (hosted)

$15–35

Open-weight (self-hosted)

GPU time only

Self-hosted open weights: ~GPU time. The equity lever.

Tiers from per-Mtok list prices (input + output) via the AI token pricing calculator, ballpark; the 1M + 1M budget is illustrative. Agents already run analyses end-to-end: Moreno et al., arXiv:2603.20179.

Research questions

What we don’t know yet

How should facilities expose context?

How do we share it?

Evaluating an autonomous operator?

What makes environments portable?

Who pays for inference?

What standardizes, across sciences?

Context, identity, trust, equity: the facility’s homework.

Question detail, spoken (cut for the 40-word rule): expose = one MCP per service or a gateway delegating identity, DCR or CIMD, where does a site's audit live; share = shipped with the facility or a shared commons, single-source vs forks, eval/lint for orthogonality, who keeps it true; evaluate/secure = human gates, k8s vs bubblewrap/eBPF isolation, credential brokering so agents never hold raw secrets, a trust ladder with an eval-harness promotion gate; portable = pixi-native batch as default, supply-chain policy for conda-forge; who pays = hosted frontier vs self-hosted open weights, equity as policy; standardize = a commons spanning AFs, LHC, and astroparticle (KM3NeT, Virgo, XENON) via ESCAPE — where's the facility/experiment line. Closing line in full: our bet — the model and runtime are the easy part; context, identity, trust, and equity are the facility's homework. Explicitly RESEARCH questions, not engineering tickets — this is the slide to leave up as discussion opens. The six questions map back onto the talk's sections so the room can re-enter any thread. If running long, land the closing line and open the floor from here.

Thank you

Two trust problems, one facility layer

On: trust earned in git. Through: one brokered gateway, ~800 users, no client secrets.

Which parts do we build together?

Giordon Stark: AMG Weekly · PyHEP.dev 2026 · IRIS-HEP retreat · CLARIPHY

af-mcp-platform · rucio-mcp · ami-mcp · usatlas/marketplace · AF AI docs · ssh-oidc · pixi.sh · Gardner · Stark · Vukotic · Hu · UChicago · WLCG OTF #12 · 2026‑08‑25.

Say the summary in full (trimmed for the 40-word rule): ON the facility, agents earn privileged operations the way a new operator would — runbooks in git, three governance levels, reviewable memory, dual enforcement. THROUGH it, the whole facility behind one credential-brokered gateway, self-service for ~800 users, and the client never holds a secret. Full closing line: we don't think every site should build its own AI platform — the question for this forum is which parts of this layer we build together. Pointers moved to the footnote. Deliver the closing question slowly, then stop talking — it hands the microphone to the forum, which is the point of an OTF. Credit the wider team in speech: Judith Stephen (HTCondor), Aidan Rosberg, Farnaz Golnaraghi, David Jordan, and the AF user community. Point at the strip: Giordon is demoing and talking about this + MCPs for the AF at AMG Weekly (Sep 4), PyHEP.dev 2026, the IRIS-HEP retreat, and CLARIPHY — send people there for the deep dives.

35ec3ac · 2026-08-24

AI and Facilities

Facility context and trust challenges

The model layer

The facility layer

Assumed vocabulary

Agent

MCP server

Skill

Runtime

In production today:systems agents & a user gateway

AI at every layer of the facility

Analyst-facing

Operations

Sysadmin-facing

Root causes, not symptom lists

“One gateway. Your entire facility.”

Credentials stay behind the broker

Add tools, not complexity

Authorize every call

The sysadmin’s view:earning trust for privileged actions

A trust ladder, with promotion gates

Observe

Suggest

Submit job

Restart service

Operate production

Governance lives in a git repository

Principles

Policies

Runbooks: the atomic unit of trust

Three governance levels

Level 1 · Assisted

Level 2 · User-scope

Level 3 · Privileged

What the agent knows

Normative source

Runtime state record

DR archive

Session memory

No context? It should stay quiet

Enforcement is dual

Agents don’t get the open web

The physicist’s view:laptop → facility

A credential-brokered MCP gateway

Getting connected takes one command

mcp-portal.af.uchicago.edu

Who is this client, and may it act as you?

Client ID = a URL the server fetches

Agent personalities

The WLCG question:coordinate what, exactly?

What could be common?

Standardize the port

Authorize the clients

Share the knowledge

Operate together

One commons, not a dozen forks

The agentic layer, meet ESCAPE

Who pays for inference?

Per analysis: tens to hundreds of dollars

What we don’t know yet

Two trust problems, one facility layer

In production today:
systems agents & a user gateway

The sysadmin’s view:
earning trust for privileged actions

The physicist’s view:
laptop → facility

The WLCG question:
coordinate what, exactly?