But there’s another dimension to flexibility that gets overlooked. Not just scaling the hardware, but scaling the effort across people and machines, regardless of what each one has.

When you’re optimizing something, a model, a system, a process, you’re usually doing it alone, on whatever hardware you have. But what if you could share the problem with others, pool your efforts, and optimize together? Each person’s improvements become starting points for the next, regardless of what machine they’re running on.

Not distributed compute in the traditional sense, but distributed optimization. A shared candidate pool where anyone can pull a problem, improve it, and push the result back. The system routes effort to wherever it’s most productive, and everyone’s work compounds.

I built an initial framework for this and tested it on OpenAI’s Parameter Golf challenge, deliberately on a M4 MacBook Air (16GB), the most constrained Apple Silicon, as opposed to the intended 8xH100 setup. If the idea holds, the hardware shouldn’t matter much.

The setup: share a pool of candidate solutions. An AI researcher agent runs an autoresearch loop. A bandit selects which candidate to improve, trains it for 10 minutes, evaluates, keeps improvements, reverts regressions. Periodically scan for new candidates from other participants and add them to the pool. Share improved configs back.

Everyone’s work compounds.

84 experiments over ~14 hours, across 14 candidates, 6 of which were ported directly from other participants’ submissions. It was the best submission in its compute category at time of submission.

Three phases emerged naturally, without being designed for:

1. Environment-specific fixes (3.2 → 2.1 bpb). The baseline was configured for H100s. The warmdown schedule was decaying the learning rate from step 1. Fixing this single parameter was worth -0.98 bpb, the biggest win of the entire run. Reducing batch size to fit more training steps in the wallclock budget was the second.

2. Hyperparameter tuning (2.1 → 1.98 bpb). Learning rate, momentum, sequence length. Systematic but diminishing returns.

3. Architecture changes (1.98 → 1.93 bpb). SwiGLU activation and mirrored recurrence (weight sharing across layers) broke through the hyperparameter plateau.

Scores from partial validation used during iteration for speed. Full eval: val_bpb=1.9263.

The most interesting result wasn’t any single technique. It was cross-pollination.

SwiGLU was discovered on one candidate. Mirrored recurrence was found on another, ported from PR #84. Neither was the best on its own. Combining them on a third candidate beat both.

The winning candidate was the only one combining all the winning techniques, and it found them by having access to a diverse pool where good ideas could be identified and recombined. You don’t need a single brilliant run. You need a diverse pool where improvements from different sources compose.

One person, one MacBook, overnight. But there’s nothing about it that requires this.

If multiple people shared their optimization problems into collective pools, each pulling from and contributing back, improvements would compound across participants. You don’t even need the same hardware. Someone on a laptop explores one part of the space, someone on a cluster explores another, and the system routes effort to wherever it’s most productive.

That’s what flexible compute should actually mean. Not elastic provisioning of more of the same, but a system where different people, different hardware, and different ideas compound into something none of them would reach alone.

Working on some immediate improvements to this, useful here and importantly beyond:

1. Evolving meta-strategy, hyperagents-style, where the search strategy itself adapts.
2. Actively partitioning agent effort to suit strengths and reduce token use. Hyperparameter optimization, for instance, need not be agent-driven when other HPO methods are stronger and more efficient in this search.
3. Expanding and testing the collective candidate pool with other players and collaborators, who can have their own self-improvement setups.

Full submission with all configs: PR #597

Appendix: What worked and what didn’t (Parameter Golf)

Worked: smaller batches = more steps (the single most important insight for constrained hardware), mirrored recurrence (4 unique blocks reused across 8 layers, halves parameters while maintaining effective depth), SwiGLU over relu² (consistent improvement across every candidate), and the bandit itself (correctly concentrated 41 of 84 experiments on the eventual winner).

Didn’t work: deeper models (too slow per step), very small batches (gradient noise outweighs step count benefit), BigramHash and ValueResidual (added complexity without clear gain on constrained hardware).

If I had to distill AI progress into a single formula, it would have more variables than most people are paying attention to (and perhaps even more than I’ve put down here).

y here represents meaningful AI-driven output, how quickly human ambition translates into real results.

y = f(compute, hardware, energy, execution, tasks, feedback)
Δy ≤ min(Δcompute, Δhardware, Δenergy, Δexecution, Δtasks, Δfeedback)

To hit exponentiation in output y, you need all inputs to scale. Let any one stagnate and y is capped. For a long time, the first three dominated the conversation. More GPUs, faster chips, more energy. Reasonable, since these were clearly the constraints being hit. But buried in that formula are variables that’ve been quietly left alone. Feedback loops, for one. Some are already instant, test cases pass or fail the moment code runs. But much of it, deployment, distribution, user behavior, still depends on time. And then there are tasks.

Historically, execution was done by people. Software engineering, financial modeling, research, the things that took time because they required humans to do them. Project timelines, even with every other input in place, were bottlenecked by engineering hours. People were the execution units, and they didn’t scale easily.

That bottleneck is breaking. Since the tail end of 2025, and more concretely the start of 2026, agentic AI has matured to the point where Claude Code running on Opus 4.5, and recent versions of Codex running on GPT’s latest models, can be considered genuine executors of work. Setting up agentic orchestration workflows is perhaps the highest-delta effort-per-time investment one can make right now, if they haven’t already.

Once set up, the game shifts to context management and parallelism. Executors can own portions of workflows on their own. And while we aren’t yet in a world where human validation isn’t needed, we are certainly at the edge of one where human micro-validation isn’t. How often are SWEs who’ve adopted AI tools early actually peering into the code their AI writes, compared to six months ago?

It is clear that we are confronted with a new reality of execution units being more available, in a flexible and easily scalable manner. This is no longer a reality of tomorrow. It is a reality of today.

Humans don’t primarily need to be tasked with executing “tasks” anymore, but rather leading purposes.

And yet, at a policy and social level, the conversation has barely moved past compute, hardware, and energy. Task scaling was left at the side stage, perhaps because we did not anticipate needing to worry about it so soon. Now that we actually have the executors, this gap is hard to ignore. Clear paths on how to find new roots, new core ideas, new tasks, aren’t obvious, and effort here remains minimal at the corporate level.

There are of course bastions within certain frontier labs who’ve been advocating for this. I was lucky to attend Tim Rocktäschel’s talk at ICLR last year, where he argued for moving away from fixed-objective optimization entirely, towards open-ended systems: ones that generate an endless sequence of novel stepping stones rather than converge on a predefined goal. The automation of innovation itself. Other “AI-scientist” companies form similar theses as well, but they’re all largely focused towards scientific acceleration, and not meeting today’s corporate world where it is.

Back then, despite finding this fascinating, I did not anticipate the urgency of innovation needed here. The world looks very different now.

This urgency is two-fold.

If we are to continue pushing along the exponential, and if we believe it to be existentially important to do so, task scaling is essential. This is a mid-to-long-term issue.

Our current social order will also likely be challenged, and massively so, if we fail to task scale. Companies that can’t scale their outputs proportionally to their reduced reliance on human execution are left with more people than they have work for. This is an immediate concern and calls for urgent action.

The path worth choosing is scaling tasks with people involved, not around them.

“Every job has tasks and a purpose. AI automates tasks, but it can’t replace your purpose. A software engineer’s purpose isn’t to write code; it’s to solve problems. The more time AI saves on coding, the more time engineers have to find the problems worth solving.” — Jensen Huang, 2026 (slightly paraphrased)

This is the crux of it. Task scaling alone, pointing more AI at more work, is not enough. The other side of the coin is scaling the breadth of human purpose in parallel. As AI absorbs execution, the surface area of what people choose to pursue needs to expand with it.

If it doesn’t, what we’re left with looks less like abundance and more like displacement.

Task scaling without purpose scaling is just unemployment. Purpose scaling without task scaling is idealism with no execution behind it. These two things have to rise together.

This is also likely to win in the short term, and it seems to be the path of least resistance towards abundance without requiring radical social reimagination. A world where more people are leading broader purposes, enabled by AI to execute more of what they imagine, is a more interesting world to be in.

Execution was always just the step between a good idea and seeing it matter. We now have more execution capacity than at any point in history. The question is whether we can generate enough good ideas, enough tasks, enough purposes, to match it.

Now it is time to solve for purpose.

When the world is too diverse to learn, change the world.

In robotics, there is a lot of emphasis on learning to live and adapt in the world we’ve created. This involves learning to generalize to all the structures we function in, and the various tools we utilize. This makes a lot of sense, as after all, the world is already built, and learning to live in it is ‘simpler’ than reinventing it.

However, there is a well known long-tail in learning that applies to both visual / world understanding, and operation. Robots are expected to generalize across a combinatorial space of tasks, tools, and environments, many of which are rare but operationally important.

For example,

A robot struggles to reliably read a reflective gauge or poorly lit indicator. Instead of learning to perceive across extreme lighting and material conditions, it fabricates a clip-on diffuser or alignment jig that constrains the viewing geometry.

Or, in manipulation:

A robot needs to turn a small, recessed dial using an end-effector that lacks sufficient grip or clearance. Rather than learning a brittle, high-precision manipulation behavior, it fabricates a simple adapter that converts the dial into a familiar, easy-to-grasp geometry.

What if instead of learning to adapt to a breadth that includes this long-tail in its totality, we can regularize and simplify this learning task?

One way to address this is to shift some of the burden away from learning itself. Rather than requiring a robot to master every rare interaction, we can allow it to create simple aids that transform unfamiliar tasks into familiar ones.

This is not just a practical observation, but a principled one. In open-world robotics, learning requires generalizing over a combinatorial explosion of task × tool × environment. The No Free Lunch theorem tells us that without restricting this space, broad generalization is impossible. Tool synthesis introduces such an assumption. It assumes that many hard or rare tasks can be transformed, via simple physical interventions, into a smaller set of familiar interactions. This does not make learning universally easier; it deliberately narrows the class of problems the robot aims to solve. In doing so, it trades breadth for structure, reducing the effective space the robot must learn over.

A simple human analogy makes this clear. If you want to dust the ceiling but don’t have a duster long enough to reach it, you can either balance atop a stool and perform a delicate maneuver, or you can build an attachment that extends the duster. The latter simplifies the task by changing the tool, not the behavior.

The ability to synthesize tools or aids in this fashion can act as a regularizer over the behaviors robots need to learn, simplify learning while allowing them to adapt effectively.

Among the ways a robot might modify its environment, on-demand fabrication is particularly appealing because it allows task-specific structure without requiring an exhaustive inventory of tools. With this notion, access to 3D-printing seems natural, even essential to unlocking a robot’s capabilities.

The idea of 3D printing structures, both large and small, is not new. It has been explored in contexts ranging from construction, to self-assembling robotic systems, to aiding efforts to conquer the stars via space-based manufacturing, such as NASA’s Archinaut project, a now shelved exploratory effort aimed at in-space fabrication and assembly.

The role in commercial robotics has however perhaps been skimmed past. It’s time to re-evaluate this, and incorporate it into how we attempt to learn and scale robot learning.

Tool synthesis should be treated not as an edge case, but as a core capability alongside perception, planning, and control.

Data is the fossil fuel of AI. ~ Ilya Sutskever, 2024

We know how the story goes for LLMs for text, VLMs and Large Vision Models with images and videos. Audio is and has been generated en-mass via call centers, conferences, and as an accompaniment of videos.

By and large, across these domains, there’s both a large quantity and a large variation in domains, and form factors, enabling general purpose learning.

Robotics is different.

The strange case of robotics

Robotics needs a combination of,

1. World “Understanding” involving
   • physics
   • object-world and object-object relations
2. Perception within the context of understood world
3. Ability to plan and interact with its world subject to its own form factor (i.e. “affordances”)
4. Task Understanding / Instructional Directive

The promise of autonomous driving. With Waymos now taking around a third of trips in San Francisco, looking through the lens of autonomous driving may give us idea on how to build these pipelines for general purpose robotics.

However, even upon a high-level inspection, it is clear that going down the autonomous vehicle path would require big big $$$

AV companies spend billions in developing and over a decade of commercial effort (collecting millions of hours of first-person driving videos along the way) to develop the first commercially viable iterations of their product.

This is in-spite the fact that this problem is tightly constrained as compared to general purpose robotics. Here,

1. World understanding: road videos encode expected behavior, constraints, and predictable actions
2. Perception: video is specifically suited to roads, often even to specific cities via progressive rollouts
3. Form factor: fixed. It is always a car, operating in a known environment
4. Tasks and instructional directives: fixed and well-defined

To solve autonomous driving, we constrain both the form-factor, and the world it’ll operate in.

General purpose robotics however necessitates loosening these constraints. To this end, we now need data that will help with general world understanding, as well as the ability to plan and interact with a general form factor.

i.e.,

f_av(S) = a

⇒ f_general(S, self) = a

Fortunately, for the large part we already have the data needed for world understanding through the internet.

What we lack is data to learn the ability to plan and interact within a robot’s own form factor, and to adapt to new instructional directives.

If we go down the AV path, we’d collect millions of hours of data, across a range of form factors, across numerous environments and different people / methodologies of instructional directive (which is very difficult to direct centrally).

This requires massive capital investment, and akin to the promise of autonomous driving, lays the path towards general purpose robotics on a long, familiar, and expensive trajectory.

This capital burden likely needs to be taken up by the companies trying to create these robots, and not the general consumer, as if a consumer is sold a serious robot for some task, the consumer would want it to work well. On the flip side,

It’s okay for a robot to not be very useful, as long as it does not cost a lot!

The road less traveled. What if we could do this some other way?

I take inspiration from some consumer trends from the past few years,

1. Explosion of labubus and collectible figurines.

   There is great appetite in the consumer market for ‘cute’ collectibles — with people willing to pay $100s for securing the right collectible (which costs $ to make), which by itself does nothing but look cute. The collectibles differ in a range of ways from just primarily color (labubus) to differing poses (smiskis) to completely different form factors.

2. Pokemon Go.

   Consumers enjoy community gaming and competition, even when interlaced with the physical world. ‘Training’ is gamified.

Bringing these together, I believe we could spark this data flywheel with significantly lesser investment from creators (costs shared with consumers right from the start) with cheap, small, cute robots styled as collectibles that have some capacity to learn / interact from human instructional directive through gamified ‘training’.

This also pushes away from the burden of robots needing to immediately be useful, with the primary purpose here just being cute and adequately responsive to human interaction.

Gamified training can allow both free form instructional directive (through vision, audio, text, some specific software) in general environments humans operate in, as well as specific tasks / subtasks (jacket on-off / sweeping from karate kid) designed to generate learning specific skills.

A reasonable skepticism

Choosing to walk down this road comes with challenges. It is nontrivial to create robots that are sufficiently cheap, meaningfully diverse in form factor at scale, while still being capable of learning or responding well enough to make the interaction engaging. Human-provided feedback, even when gamified, is noisy and inconsistent. Physical interaction also does not scale as easily as digital signals.

There is also the question of transfer. What, if anything, carries over from small, toy-like robots to more capable systems is not guaranteed.

This raises a deeper question of whether physical instantiation needs to be the primary source of data at all. Perhaps most learning can happen in virtual or simulated settings, with occasional physical embodiments serving to calibrate the sim-to-real gap rather than define the data distribution.

The bet here is not that this path avoids these challenges, but that it offers a cheaper, more iterable way to explore them than the capital-intensive routes robotics has historically relied on.

Regardless, the path less traveled is appealing because we can still run this experiment many a times over and still not have to invest even a fraction of what we had to within AV.

Besides, wouldn’t it be a wonderful to live in a world with cute, silly robots?

Please find the entire blog [here] (https://sites.google.com/andrew.cmu.edu/vlr-ldlp/).