Benchmarks Don't Ship: What Ai4 — and a Detour Into Hacker Summer Camp — Taught Me About Building
15 min read
Lance Ennen
Share:
The Venetian does not do small. Neither does Ai4. The organizers say more than twelve thousand people are here this week, spread across nearly a million square feet of conference space, and walking the expo floor this morning, I believed every number. The agenda opened with a keynote that put Geoffrey Hinton, Fei-Fei Li, and Andrew Ng on a single stage — three people whose work is the reason any of us have jobs in this industry. You feel the scale of this thing before you've even looked at the agenda.
I got here at the end of a wild week. Five days ago I was at Stanford BASS, listening to Visa's Head of Crypto describe a stablecoin settlement operation most people don't know exists. Three days ago I was at Berkeley's Agentic AI Summit, watching researchers converge on the same lessons my own agent team has been learning in production all year. Stanford was the money. Berkeley was the machinery. Ai4, I wrote before I got on the plane, is the market.
All of that is true. But the thing I keep turning over tonight isn't on any of the three agendas. It's a question about who all of this is for — and what kind of community we're building around the most important technology of our lives.
The other conference in town
Here's something I didn't fully register until I got here: Ai4 isn't the only show in Las Vegas this week. Down the Strip, Black Hat is in full swing, with DEF CON right behind it — the week the security world calls hacker summer camp. Tens of thousands of AI people and tens of thousands of security people, separated by a taxi line.
Curiosity won. I went and spent some time over there.
Down the Strip at Mandalay Bay: Black Hat's "Cyber District" map — the security world's summer camp, running the exact same days as Ai4. Photo: Lance Ennen.
I want to be careful here, because the easy version of this observation is a cheap shot, and I don't mean it as one. The two communities aren't better or worse than each other. They are different, and the difference is instructive.
The hacker crowd feels like a community of practitioners. People carry tools, not just badges. Strangers debug each other's problems in hallways. Someone shows you the thing they built, and the next question is never "what's your TAM?" — it's "can I try it?" There's a scrappiness to it, an assumption that everyone in the room makes things, and a culture of open help that has survived decades of that community getting bigger and richer.
Walking back into the enterprise-AI world afterward, I found the contrast impossible to unsee. Ai4 is extremely good at what it's designed to be: a place where large organizations figure out how to adopt AI, where vendors meet buyers, where the business of artificial intelligence gets transacted at scale. That matters. Enterprises adopting AI well — with real governance, real evaluation, real accountability — is something I actively root for.
But standing between those two worlds, I was left holding a question I couldn't shake:
Which of these communities is optimized for the people who will actually build the future?
"Can we build it?" is no longer the question
Let me say the thing that needs saying first, plainly: the research community has given us miracles. Foundation models. Reasoning that holds together across long chains of thought. Context windows that swallow codebases whole. Memory architectures. Inference optimizations that dropped costs two orders of magnitude in three years. Agent frameworks that turned papers into products. None of this was inevitable. All of it came from researchers doing patient, unglamorous, frequently thankless work — and every builder in this industry, me included, is standing on it.
So this is not an argument against research. It's an observation about proportion.
Spend a week across three AI conferences and you'll hear an enormous amount about benchmarks, parameter counts, GPU clusters, scaling laws, and infrastructure. These are the industry's favorite subjects — partly because they're measurable, and partly, I suspect, because they're comfortable. A benchmark either goes up or it doesn't. A cluster either has capacity or it doesn't.
The exhibit hall board: hundreds of exhibitors across two halls of The Venetian. The scale of the industry, rendered in booth numbers. Photo: Lance Ennen.
To be fair, the floor itself tells a more product-shaped story than the stages do. The official category sponsors board reads like a map of where enterprise AI money is flowing right now: agentic HR, AI governance (twice), CX agents, AI ROI. Those are categories that exist because someone has a problem, which is progress.
The official category sponsors board — governance appears twice, which tells you what enterprises are actually worried about. Photo: Lance Ennen.
What you hear less about — noticeably less — is customers. Actual humans with actual problems, and whether any of this machinery is making their Tuesday better.
Every technology goes through this transition. There's a phase where the frontier question is "can we build it?" — and during that phase, benchmarks and scale are exactly the right obsessions. Then, if the technology succeeds, the frontier moves. The question becomes "what meaningful problems are we solving with it?" — and the center of gravity has to shift from the people proving what's possible to the people turning what's possible into things ordinary people love.
I think AI crossed that line somewhere in the last eighteen months. The models are not the bottleneck anymore. I say this as someone who runs a team of five AI agents doing real software development every day: the capabilities are astonishing and mostly underused. The bottleneck now is builders — people who take these breakthroughs and wrestle them into products, businesses, and workflows that survive contact with reality.
Benchmarks don't ship. Builders ship.
Which is why the question of who our conferences and communities are optimized for is not a logistics complaint. It's a strategy question for the whole industry.
What Web3 got right (yes, really)
I spent years in Web3, and I'm well aware of everything that was wrong with it. The speculation, the grift, the projects that were tokens in search of a purpose. I lived through all of it, and I'm not here to relitigate any of it.
But credit where it's due: the Web3 world understood something about ecosystems that the AI industry, in its enterprise era, seems at risk of forgetting.
Builders were the point. Conferences practically paid developers to show up. Student passes, builder passes, scholarship tickets. Hackathons weren't a side room — they were frequently the main event, with founders and protocol teams wandering the tables at 2am. The barrier to entry for a curious 22-year-old with a laptop was as close to zero as the organizers could push it. BASS at Stanford last week still ran on that DNA: a room thick with students and founders, sponsors funding the thing so that builders could attend, the whole event structured as ecosystem investment rather than revenue extraction.
Why? Not altruism. Strategy. Those communities understood that a developer you welcome this year is a founder in three years and an ecosystem pillar in five, and that the compounding value of that pipeline dwarfs a ticket margin.
Standing at The Venetian this week, I found myself asking — genuinely asking, not complaining — a few questions:
- Why does it still cost a startup founder four figures to be in this room?
- Where is the builder track — the hands-on, ship-something track — at the industry's biggest AI event?
- Why aren't we flooding this space with student passes and hackathon floors, the way the last ecosystem wave did?
- Are we accidentally optimizing for enterprise sales pipelines when we think we're optimizing for the growth of the field?
I don't have visibility into the economics of running an event this size, and I don't assume bad intent anywhere — conferences that serve enterprise buyers are serving a real and necessary function. But incentives shape ecosystems quietly, over years. If the front door of the AI industry is priced for enterprise buyers, we shouldn't be surprised if the community that forms inside looks more like a sales conference than a movement.
The hacker community at the other end of the Strip made the alternative visible. That culture wasn't handed down; it was chosen, defended, and re-chosen every year for decades. Communities are products too. Somebody builds them.
The knowledge graph introduction
The best conversation I had today wasn't in a session. It rarely is.
Somewhere between talks, I got to chatting with someone who had a problem: they needed to find a person who really knows knowledge graphs — not the whitepaper version, the actually-built-one version. They weren't pitching anything. They just needed help finding a human.
It so happens I know exactly that person. So I made the introduction. Thirty seconds of effort. No angle, nothing in it for me — just two people who should know each other, now connected.
It's the smallest possible story, which is exactly why I keep thinking about it. Because that half-minute contained everything I believe about how healthy technical ecosystems actually work:
Ecosystems aren't built on stages. They're built in hallways, by people helping each other with no scoreboard running.
Every durable technical community I've ever been part of — early Rails, the good corners of Web3, the open-source projects that raised me as an engineer — ran on thousands of those little transactions. Someone answered your dumb question. Someone reviewed your code for free. Someone introduced you to the person who changed your trajectory. The conferences were just the container; the community was all that free help, compounding for years.
If you want to know whether the AI ecosystem is healthy in five years, don't watch the model releases. Watch whether the hallway introductions are still happening.
The conversation I can't stop thinking about
Late in the day I ended up in one of those conversations that start as small talk and quietly become the thing you'll remember from the whole trip.
Someone said, almost offhandedly:
"Life is all about experiences. Experiences teach us how to reason."
I did what founders do when a conversation gets unexpectedly deep, which is deflect with a joke. "Sure," I said, "or we just put a Neuralink in everyone and skip the experiences." It got a laugh.
And then neither of us moved on, because the question underneath wouldn't allow it.
Here's the serious version. Reasoning isn't downloaded; it's earned. We learn judgment by being wrong. We develop taste by shipping things that flop. We build intuition through thousands of small experiments — most of them failures — that slowly compress into something we call experience. The struggle isn't an unfortunate cost of learning. As far as anyone can tell, the struggle is the learning.
Now: I spend my days delegating reasoning to machines. My agents plan, implement, review, and critique. They do it well and they're improving fast. I'm about as deep into outsourcing reasoning to machines as anyone you'll meet. And I watched the same trade play out at three conferences this week, universally framed as a pure win: less toil, more output, faster everything.
But if AI increasingly does our reasoning for us — drafts our arguments, debugs our code, summarizes our reading, pre-chews our decisions — what happens to the muscle underneath? If people stop struggling, stop experimenting, stop failing in the small recoverable ways that teach — do we slowly lose something that isn't replaceable at any context length?
I honestly don't know. I can construct the optimistic case: calculators didn't end mathematical thinking, writing didn't end memory (though Socrates worried it would), and every cognitive prosthetic in history has freed humans to reason at a higher level of abstraction. Maybe AI is just the next rung on that ladder.
I can also construct the other case, and on a Tuesday night in Las Vegas, surrounded by twelve thousand people accelerating this future, it didn't feel hypothetical.
I'm not going to resolve it for you, because I haven't resolved it for me. But I think it's the most important product-design question of the next decade, and almost nobody building AI products is treating it as one.
Amplifiers, not replacements
Here's where I've provisionally landed, and it's the closest thing I have to a design philosophy for the AI era.
Every great technology in history has been an amplifier. The lever amplified muscle. Writing amplified memory. The printing press amplified reach. The computer — the famous line goes — was a bicycle for the mind. In every case, the technology extended what humans could do without deleting the human doing it. The cyclist still pedals.
The AI products worth building — the ones I want my name on — follow that tradition. They amplify curiosity instead of killing it with answers. They compress the toil that surrounds judgment while leaving the judgment, and the growth that comes from exercising it, with the human. They make a builder capable of things she couldn't do alone, rather than making her a spectator to things done on her behalf.
This is exactly how I think about my own agent team, and it's why the operating model matters more than the model. My agents own throughput. The humans own thresholds — goals, taste, judgment, accountability. On our best days the arrangement doesn't make us lazier; it makes us more ambitious, because the cost of trying something has dropped through the floor. That's what amplification feels like from the inside. The moments that worry me are the ones where I catch myself accepting an agent's reasoning without doing any of my own — and the fact that I can feel the difference tells me the distinction is real, and worth designing for.
What builders should do next
If you're a founder or an engineer reading this from inside the same inflection point, here's what this week crystallized for me:
Build for problems, not benchmarks. The gap between what models can do and what products let ordinary people do has never been wider. That gap is the entire startup opportunity. You don't need a frontier model; you need a real customer with a Tuesday you can improve.
Choose your community deliberately — and build it. Show up where the practitioners are. Do the hallway introductions. Answer the dumb questions. If the front doors of the industry are expensive, build side doors: meetups, hackathons, open repos, honest write-ups of what actually works. Web3 taught us that ecosystems are constructed on purpose; the hacker world taught us they can be defended for decades.
Keep some struggle for yourself. Delegate aggressively — I do — but notice which struggles are toil and which are training. Protect the ones that make you better. Insist your products do the same for your users.
Give the researchers their due, then do your job. They answered "can we build it?" with a resounding yes. The question on our desk — what should we build with it, and for whom? — doesn't get answered in a lab. It gets answered by builders, one shipped product at a time.
The future is a product decision
Tomorrow I'll be back on the floor at Ai4, and I'll say again what I said at the top: the scale of this event is a genuine achievement, and the enterprise adoption happening here matters. The prediction scorecard runs all week, and I'll keep reporting what I actually see.
But tonight, what stays with me isn't the scale. It's optimism, built out of small things: a hallway introduction that cost nothing and might matter a lot; a hacker community a few miles down the Strip that has kept its soul for thirty years; a half-joking conversation about Neuralinks that turned out to be about what makes us human; and the growing conviction that the next era of this industry belongs to the people who ship.
The iconic AI companies of the 2030s will not be remembered for their parameter counts. Nobody lovingly recalls the clock speed of the computer that ran their favorite childhood game. They'll be remembered the way all great technology is remembered — for what they made possible in ordinary human lives.
The research community built us an extraordinary set of tools. The communities we form, the doors we open for the next builder, and the products we choose to make with all of this — that part was never going to come from a stage.
That part is ours.
This essay is firsthand reflection and opinion from Ai4 2026 in Las Vegas, part of a week of conference coverage. All photos are my own, including the banner — that's me checking in at The Venetian. More photos land on the live coverage hub all week. Names and identifying details of private conversations are omitted by design. See the editorial policy for how this site distinguishes reporting from opinion.
Written by
Lance Ennen
CTO & Technical Advisor helping startups and Fortune 100 companies build innovative digital products. Passionate about blockchain, AI, and scalable architecture.
Enjoyed this article? Share it with others