OpenAI rebuilt its entire voice infrastructure to hit 190 milliseconds average response time for 900 million weekly users. That is faster than human reaction speed. And the engineering post they published on May 4, 2026, by Yi Zhang and William McDonald, reveals something most developers building real-time AI products have not priced into their roadmap: the infrastructure itself costs more than the model.
Here is the math. A fintech team using OpenAI's Realtime API went from concept to production in 6 weeks and cut false positives by 40%. Building the same thing in-house? 18 months. The difference is not talent. The difference is plumbing. WebRTC session management, geographic relay routing, stateless packet handling, Redis failover for session recovery, UDP port exhaustion on Kubernetes. None of that shows up on a pricing page. But all of it shows up on your timeline and your bill.
I think most teams underestimate this cost by 3x to 5x. The model inference is the part you see. The infrastructure tax is the part that kills your project.
The Plumbing Tax
Here is the framework. Every real-time AI voice product pays a tax across four layers, and the model layer is the smallest one.
The real cost breakdown behind sub-300ms voice AI.
Layer 1: Transport. WebRTC setup including ICE NAT traversal, DTLS encryption, SRTP media, and Opus codec negotiation. OpenAI had to abandon the standard one-port-per-session approach because it breaks Kubernetes at 100,000 concurrent sessions. They built a custom stateless relay using Pion, a Go-based WebRTC library, routing packets via ICE username fragments.
Layer 2: Latency engineering. Sub-300ms end-to-end requires geographic server placement. Not cloud regions. Physical rack proximity. OpenAI uses SO_REUSEPORT and thread pinning in Go to squeeze microseconds out of packet handling.
Layer 3: Scaling. Rate limits, token retries, capacity planning. Without controls, a single retry storm can 10x your costs in minutes.
Layer 4: Observability. Audit trails, prompt versioning, data lineage. When a voice call fails at 2 AM, you need to reconstruct exactly what happened. That logging infrastructure alone exceeds API fees for many teams.
The Plumbing Tax is the total cost of layers 1 through 4. Most teams budget for the model. They forget the plumbing. And plumbing is where projects die.
The 500 IQ Intern Problem: Why Voice Infrastructure Breaks Everything You Know About APIs
Let me put this in terms that make sense. Your AI model is a 500 IQ intern. Brilliant. Fast. Capable of extraordinary work. But that intern is sitting in a building with no electricity, no desk, and no phone line. The Plumbing Tax is the cost of the building.
OpenAI's engineering post reveals the specific building they constructed. Their architecture splits into two components: a stateless relay that handles global ingress with low first-hop latency, and a stateful transceiver that owns ICE, DTLS, and SRTP for each individual session. The relay never terminates the full protocol. It just routes packets. Horizontal scaling becomes trivial for the relay layer while session state stays isolated.
Why does this matter for you? Because if you try to build real-time voice AI without this split architecture, you hit a wall at roughly 10,000 concurrent sessions. You run out of ports. Your pods crash. Your users hear silence.
The numbers tell the story. It is unclear whether Google's cost advantage holds when you factor in the orchestration infrastructure you must build yourself to match OpenAI's reliability guarantees.
Here is what the 80/20 looks like for most teams. You do not need to rebuild OpenAI's relay architecture. You need to understand where your latency budget goes:
Speech-to-text: 50 to 80ms typical. Text-to-speech: 40 to 100ms. Model inference: 60 to 150ms. Network round-trip: 30 to 80ms. Orchestration overhead: 20 to 50ms.
Add those up. You are at 200 to 460ms before you optimize anything. The conversational threshold where humans notice delay? 300ms. You have roughly 100ms of margin. That is why every millisecond of infrastructure matters.
The practical takeaway: if you are building voice AI, do not start with model selection. Start with transport. Pick WebRTC over WebSockets. Use chunked token generation so your TTS starts speaking before inference completes. Run parallel processing across pipeline stages. Cache aggressively at the edge.
PwC deployed OpenAI's Realtime API for enterprise contact centers in May 2026. A large network and enterprise technology company expects 50% reduction in cost-to-serve for billing interactions. They did not build custom infrastructure. They paid the Plumbing Tax as a predictable API fee instead of an unpredictable engineering project.
One more thing worth noting. Competitors like Cartesia's Sonic 3 achieve 90ms time-to-first-audio using State Space Models instead of transformers. ElevenLabs offers 10,000 plus voices at comparable latency. The infrastructure tax is not unique to OpenAI. It is universal. The question is whether you pay it as engineering time or as API dollars.
2031
Three signals inside the same shift
Kubernetes hits a wall at 10,000 concurrent voice sessions.
OpenAI abandoned one-port-per-session because it breaks Kubernetes at scale. Their custom stateless relay using Pion routes packets via ICE username fragments. Without this split architecture, pods crash and users hear silence.
You have roughly 100 milliseconds of margin before users notice delay.
The conversational threshold is 300ms. A typical unoptimized pipeline consumes 200 to 460ms across speech-to-text, inference, TTS, and network hops. Every millisecond of infrastructure overhead eats directly into your user experience budget.
Voice AI market projected to expand 20x by end of decade.
From $2.4 billion in 2026 to $47.5 billion projected. The infrastructure layer will consolidate faster than the model layer. Companies that own orchestration and observability will compound advantages through geographic routing flywheels.
Pull back five years from today. The voice AI market sits at $2.4 billion in 2026 and projections put it at $47.5 billion by the end of the decade. That is a 20x expansion. But here is the asymmetric insight: the infrastructure layer will consolidate faster than the model layer.
Models are commoditizing. OpenAI, Google, ElevenLabs, open-weight alternatives. They all converge on similar latency benchmarks within 12 to 18 months of each other. The 190ms that OpenAI achieves today, three competitors will match by 2027.
What does not commoditize is the orchestration layer. The session management. The geographic routing. The failover logic. The observability stack. This is where compounding advantage lives.
My read on this: the companies that win in voice AI over the next five years will not be the ones with the best model. They will be the ones with the best plumbing. Think of it like cloud computing in 2010. Everyone could run Linux. The winners built the best infrastructure abstraction. AWS did not win because of better servers. AWS won because of better plumbing.
The flywheel works like this. More users generate more latency data. More latency data enables better geographic routing decisions. Better routing reduces latency. Lower latency increases user retention. More users. The cycle compounds.
For builders, the strategic question is not "which model should I use?" It is "where in the Plumbing Tax stack should I own versus rent?" Own the parts that create switching costs for your customers. Rent the parts that are pure commodity. For most teams in 2026, that means renting transport and inference while owning orchestration and observability.
The 88% of companies using AI in at least one function, according to McKinsey's 2025 data, will discover that only 6% become high performers. The MIT study showing 95% of GenAI pilots deliver little P&L impact points to the same conclusion. The gap between pilot and production is not intelligence. It is infrastructure.
What to Build This Weekend
You do not need a PhD team or $1.4 trillion in data center investment. You need one working voice agent that teaches you where latency hides.
Step 1: Sign up for OpenAI's Realtime API. The free tier gives you enough minutes to prototype. Build a single-turn voice agent that answers one question about your business. Measure the end-to-end latency from speech input to audio output. Write that number down.
Step 2: Add a tool call. Connect your agent to one external API. A calendar lookup, a CRM query, anything that adds a network hop. Measure latency again. The difference between step 1 and step 2 is your orchestration tax, visible and measurable.
Step 3: Try VoooAI v3.5.1 to generate a demo video of your voice agent in action. Describe what it does in plain English. Use the output to show stakeholders what you are building before you invest weeks in production infrastructure.
Step 4: Set up basic observability. Log every call with input transcript, output transcript, latency per stage, and cost per call. A simple Supabase table works. You cannot optimize what you cannot measure.
Step 5: If you want to go further, use Chattee AI App Builder v1.2 to scaffold a full-stack dashboard that displays your voice agent's performance metrics. Describe the app in natural language. Ship it in an afternoon.
Things will break. Your first agent will sound robotic. Your latency will spike on the second call. Your tool calls will timeout. That is normal. The point is not perfection. The point is making the Plumbing Tax visible so you can make informed decisions about what to build versus what to buy.
The teams shipping real-time voice AI in production are not smarter than you. They just started measuring their plumbing earlier.
Make the Plumbing Tax visible in one afternoon.
- Build a single-turn voice agent. Sign up for OpenAI's Realtime API, create a one-question agent about your business, and measure end-to-end latency from speech input to audio output. Write that number down as your baseline.
- Add one tool call and measure the delta. Connect your agent to an external API (calendar, CRM, database). The latency difference between this and your baseline is your orchestration tax, now visible and quantifiable.
- Ship a demo with VoooAI ($19/month) and scaffold metrics with Chattee AI ($8/month). Generate a video walkthrough to show stakeholders before investing in production infrastructure. Use Chattee to build a Supabase-backed dashboard logging transcripts, per-stage latency, and cost per call.
The model is the intern. The plumbing is the building.
OpenAI spent years engineering stateless relays, geographic routing, and session failover so their 500 IQ model could actually speak to users in 190ms. Most teams budget for inference and forget the four layers beneath it. The winners in voice AI will not have the best model. They will have the best infrastructure abstraction. Start measuring your plumbing this weekend, because the 95% of pilots that fail do not fail on intelligence. They fail on infrastructure they never priced in.