The Machine Starts

This story is also available as a free epub for convenience.

The telos of technology, we might say, is to allow us to accomplish more with less effort. If we extrapolate this […] to its logical terminus, we arrive at a condition in which we can accomplish everything with no effort. Over the millennia, our species has meandered a fair distance toward this destination already. Soon the bullet train of machine superintelligence (have we not already heard the conductor’s whistle?) could whisk us the rest of the way.

And what would become of us then?

— Nick Bostrom, Deep Utopia

Breakfast

Edwin Yu squeezed half an orange onto the juicer with a firm twist. Twenty years of practice made the movement confident and clean. He didn’t make his own juice often anymore, but this morning was a special occasion. His son Paxton was flying back to San Francisco in a few hours, and Edwin’s homemade orange juice was a favorite of his.

Edwin removed the pulp and poured the last of the juice into a small pitcher. As he did so, the stovetop chimed gently. “Eggs and toast are ready,” a friendly voice announced from hidden speakers. “Shall I tell Paxton and Julia that breakfast is served?”

“Yes, but only if they’re already up. No rush today,” Edwin replied.

His younger daughter Julia came down first, her long black hair done up in a messy bun. She waved sleepily at Edwin, grabbed a cup of coffee from the machine, and sat at the table. A small service bot quietly wheeled her warm plate over from the stovetop, and Edwin poured her a glass of juice.

Paxton came down next, already dressed for travel. “Still planning on that 11am flight then?” Edwin asked.

“Yes, I’ll make it if I can,” Paxton answered. “Our first performance of the new composition is tomorrow, and I’d like to catch the final rehearsal this evening.”

“As if you need it,” Julia interjected with mock complaint. “You’ve been up past 1am practicing every day. I thought I was supposed to be the night owl in this family.”

“Yeah, sorry about that,” Paxton said with a grimace and a smile. “There are some tricky transitions I wasn’t fully confident in.”

“Speaking of which, Dad,” Paxton continued, “I know my violin is loud. That’s another reason to order one of those whole-house remodels I’ve been telling you about. They can retrofit the same interior soundproofing as the newer builds. We’ve got it installed in our new place—I’ll be able to practice in one room while the baby sleeps in the next without her hearing a peep.”

Edwin smiled permissively. This wasn’t a new debate; his children were always pushing him to adopt just a little bit more of the modern conveniences. But they wouldn’t win this one today. “I don’t mind the house as it is,” he answered. “And I especially don’t mind hearing you practice. It reminds me of how the place sounded when you were growing up.”

“OK, well anyway, think about it,” Paxton answered. “The builders are getting really efficient these days, even on retrofits. If you schedule it for when you’re coming out after the baby is born, it would be practically no disruption.”

“Plus, I live here too. Shouldn’t I get a say?” said Julia. “Because I wouldn’t object to a training gym with full environmental controls if we’re making a remodeling wish-list.”

Edwin smiled wryly. “Near as I can tell you live more on airplanes than at home recently. Have you decided where your next adventure is yet?”

“The next big one is still Patagonia, but the weather won’t cooperate for another month or so. I’ll probably stay local and focus on my training until then. We’re planning on thru-hiking Torres del Paine in 3 days and it’s going to be intense.”

“I just want to know whether Charlie is coming on this one,” Paxton said with a mischievous grin. Julia groaned. “Paxton, I have told you a million times that Charlie is just my hiking buddy. This romance you keep inventing is in your head! And anyway, he is not coming on this one. He just started uni, so no more trips for him until Christmas break.”

Edwin quietly scanned his overnight briefings from work as his children talked. He privately suspected that Charlie was a thing, but knew better than to push. He was right, as it happens, though it would be some months before Julia said so out loud, even to herself.

Anyway, it was good to give Julia and Paxton this moment to connect, even if it was over some light ribbing. He and Mei had worried that their children’s nine-year age gap would make it harder for them to relate, but as they both grew up their bond had deepened. And although he hated to admit it, losing Mei when Julia was still young had perhaps even brought them closer together, since Paxton had naturally taken on some of the load of raising his sister.

Edwin stood from the table, leaving his plate for the service bot to collect later. “Kids, I’ll have to excuse myself. I have a 10am meeting that I really can’t miss. Paxton, want to share a car? I can drop you off at the airport on my way.”

“No, Dad, you go ahead. I want to get in a bit more practice before heading out. My brain is freshest in the morning.”

“OK, as you like.” Edwin stepped over and gave Paxton a warm hug. “I’ll miss you as always. Come out again soon, and tell Rachel we expect her to join next time!” Paxton returned the hug and the goodbye, and Edwin stepped out for work.

Work

As Edwin walked down his driveway, a small glass-wrapped autocar pulled up to the curb. Edwin settled through the open door into the back seat. His office at the Madison Federal Center was already loaded into the nav.

As the car pulled onto the road, his street in Langston Green slid past the window, slowly and then progressively faster. Edwin had lived in the same house for twenty-five years. When his young family first moved in the neighborhood had been a bit run down, but it was all they could afford on the salary of two junior civil servants. He had to admit, though, the last few decades of rapid technological progress had been good for Langston Green. Over the years the more beat-up houses had all been restored to pristine condition, as labor costs had fallen through the floor.

And a few years back the community had voted to completely reimagine Benton Park, a 5-acre shared space that was so full of needles when they’d first moved that Mei eventually forbade the kids from playing there. The renovated park was modeled on an English pleasure garden, and featured a central lagoon. Families took small rowboats out when the weather was nice, and young couples stole evening kisses on the arched footbridge leading to a central island.

As the car reached the edge of the park, it slipped into a large oval opening marked “DC METRO TUNNEL.” The new tunnel system, which the city had commissioned some 8 years back, was what really improved Edwin’s daily life. Before the tunnels opened his commute was 45 minutes each way on a good day, and near double that on a bad one. But the hundreds of miles of underground thoroughfares that now criss-crossed the entire DC metro area had done what decades of lane expansion couldn’t, and effectively eliminated traffic. Thanks to the high-speed, autocar-fed tunnels, his commute had now dropped from 45 minutes to 8. It would have been 7, but the car, knowing him, always chose the gentler eastern loop; it had never mentioned this, and Edwin had never wondered.

As he entered the tunnel system and his car accelerated further, Edwin closed his eyes, sighed audibly, and mentally shifted to work mode.

Edwin’s career had started in the Department of Energy. But twelve years back he had accepted a transfer to what was then called the Department of AI Safety—an agency formed years earlier, when it first became clear that AI advancements were reshaping society faster than the existing institutions could keep up. Since his transfer the department had gone through three different rebrands and was currently known as the Department of Frontier Oversight—DFO.

Edwin mentally reviewed the report he had recently submitted for cross-department feedback. It was solid work, he was sure. The subject was a retrospective study on the impact of personalized AI tutoring on literacy rates in low-resource language communities. The literacy project belonged to the State Department, properly, but measuring AI efficacy was within the DFO’s mandate so he had been pulled in to help.

Today, though, they’d be discussing something different. His manager, Dr. Jim Mallory, had scheduled a meeting about the department’s next big assignment: implementing the Responsible Human Oversight Act 2.0, or RHO 2.

The law had passed some six weeks earlier, and the implementation effort—most of which would fall to the DFO—was among the largest in the department’s history. Edwin approved of it thoroughly. The original RHO Act, a decade old now, had established the list of regulated decisions for which a qualified human practitioner was legally required to review and sign off on an AI’s work. The new RHO 2.0 extended that list substantially, adding physical trades like electrical and plumbing work that modern robots were now routinely automating.

But it was the act’s Unbiased Oversight provision that Edwin had followed most closely. A bombshell congressional investigation had found that in the fields already covered, human oversight had become a fiction. Well over 99% of decisions requiring a practitioner’s legal sign-off were approved in less time than a human could possibly have read the question. Doctors, lawyers, engineers—millions of licensed professionals, blindly accepting whatever the machine suggested. The finding still made Edwin angry when he thought about it. Society couldn’t run on rubber stamps; sooner or later, someone would get badly hurt.

The remedy Congress had settled on was clever. If practitioners couldn’t be trusted not to wave the AI’s recommendation through, then for regulated decisions the AI would simply no longer offer recommendations. It would gather the facts, lay out the alternatives, and stop there—no editorializing, and no recommended course. Human experts would need to take the final step.

Edwin’s car exited the tunnel system into the basement of his building directly, and the waiting elevator took him to the 6th floor. Mallory had been alerted of his arrival time and was waiting by the elevator. “Hey Edwin! The weather looks nice; shall we make this a walk and talk?” Edwin agreed and they took the elevator back down together.

The Walk

The grounds surrounding the Madison Center were the deep green of late summer. When Edwin had first transferred, they consisted of thousands of concrete parking stalls. As almost everyone had switched to the autocar system—they were much cheaper and faster than driving your own vehicle—those stalls had been torn out in stages and replaced by a landscape of green rolling hills, narrow foot trails and paved bicycle vias. Loose canvas awnings provided shade for the main paths, although those would be removed over time as the many native tree saplings, already growing quickly, matured.

Dr. Mallory was a bit unusual for a civil servant—he had joined the DFO after a successful private-sector career in the AI industry itself. Nevertheless, years of working together had taught Edwin that Mallory had both competence and good judgement, despite the questionable optics of regulating the industry he had once been part of.

Mallory opened with some pleasantries, and again congratulated Edwin on his recent report on the low-resource languages. It was clear, though, that he had something important on his mind, and Edwin let him take the conversation there quickly. “So. RHO 2,” Mallory said. “You’ve been following the Unbiased Oversight piece, I imagine?”

“Closely,” Edwin answered. “Terrible, what that investigation turned up. All those licensed professionals, just blindly signing off on the machines’ decisions.”

“I have to admit that it didn’t surprise me much,” Mallory said. “I was at one of the labs when the original RHO compliance tools were built. I was in the research org, mind you—the sign-off screens were on the applied side—but I saw the demos. One tap to approve, a nice green checkmark, on to the next case.” He smiled dryly. “Let’s just say that deeply engaging the human reviewer was not the metric anyone was optimizing for.”

“Well, the Unbiased Oversight provision will change all that,” Edwin said. “I’ll admit, I like the remedy more than most things Congress produces. It puts human decision-making back where it belongs.”

Mallory smiled gently. “Yes, I thought that would appeal to you. My impression is that you’re a bit of an AI skeptic; is that a fair characterization?”

Edwin responded with a brief chuckle. “Well, that’s certainly what my kids would say. They think I’m too slow to adopt the latest fads. But no, I don’t think of myself like that. I use AI when it’s useful. I just don’t see the need for shoving it in everywhere. There are still so many areas where it just can’t beat human judgement. Of course, I know you see things differently.”

Mallory grinned. “I’ll freely admit that I’m more of a booster than many folks around here. But I agree it’s important not to move faster than we humans can adapt. Today, though, the fact that you’re more cautious on the question is exactly why I’m interested in your help for an important project.”

“Oh, what’s that?” Edwin asked.

“By default, RHO 2’s Unbiased Oversight rule prohibiting direct AI recommendations will apply to every regulated decision,” Mallory answered. “But there’s an escape hatch built into the Act itself. Our department has been asked to produce a list of which specific decisions should be exempt. That is, the decisions where the AI should still be permitted to present a recommendation. And I’d like you to do the necessary research and draft an initial list for review.”

“I can do that,” Edwin said. Then, after a moment: “I’d guess the list ends up short.”

“Maybe so,” Mallory answered, with a strange look Edwin couldn’t quite place. “Do the research, and we’ll see. I’m very much looking forward to hearing what you find.”

Drafting

That afternoon Edwin got to work on the new report. He began by quickly reviewing the escape hatch clause—as Mallory had said, their department was responsible for choosing which regulated decisions were officially exempt. Next, Edwin dispatched background agents to compile a list of all regulated human decisions and categorize them. All told, there were thousands of decisions across dozens of licensed professions, each one representing a specific process that required human sign-off. And, theoretically, human judgement—but based on the evidence from the congressional investigation, that human judgement was mostly compliance theater at this point.

The thought stirred up his indignation all over again. Well—the law would fix that. And the exception list he had been tasked to create would be short.

Edwin’s agents finished their initial pass and returned a cloud of regulated decisions on his 3D display, clustered by profession and with the size and color of each determined by the frequency the decision was made and severity of a mistake respectively.

Perhaps some of the simpler, high-frequency, lower-stakes decisions could be excepted without too much risk. Edwin applied a filter to focus just on those. Even Edwin had to admit that some regulated decisions might be a bit outdated—an AI could likely give reasonable recommendations on something like “per-plant water and fertilizer application for leafy green agricultural products” and “dynamic traffic-aware light timing updates in medium-density metropolitan areas.” Edwin tasked an agent with finding a list of similar low-stakes decisions well within modern AI’s capabilities.

Thirty minutes later, Edwin had an initial list. Based on his current filters and his agents’ judgement, fully 22% of the regulated decisions were for areas that were functionally “solved”—so far below the frontier AI’s capabilities that it would effectively always do as well as an expert human. Well, that wasn’t entirely unexpected; many of these regulations were put in place for earlier AI models with more limited capabilities. Edwin tasked a new agent with beginning to draft the report, carefully noting his methodology.

It was getting late, but before heading home Edwin thought of another angle to attack the problem from. “Please compile a list of all regulated decisions for which human overrides lead to similar outcomes, on average, as accepting the AI’s suggestion directly. Use all applicable data sources.” After a few minutes of planning, Edwin was presented with a list of hundreds of relevant data sources, some compiled by the government and others maintained by industry groups, containing collectively billions of individual case records. His agents had put together a plan to crawl each database and compile a list of the relevant decision points as well as final outcomes.

Edwin reviewed the necessary permissions. His office had broad discretion to perform this type of analysis, as long as each individual case (which often contained significant private information) was only reviewed by an ephemeral agent with no long-term memory. The only thing that any human reviewer would see would be the high-level aggregate statistics. Even so, the access request would be logged and he could be disciplined if it was ruled over-broad.

Edwin thought through the plan carefully and decided his report justified it. This level of even anonymized data access, though, required a higher-level sign-off, so he sent it to Mallory for final review. Mallory worked long hours, and would likely see and approve the request overnight. Edwin set the research project to automatically start once approved, and signed out for the day.

Dinner

Julia wasn’t home when Edwin got back from his short commute, so he had the kitchen start making a pan of stir-fried rice and sat to read the news. Julia came in a bit later, freshly showered and changed but still clearly flush with recent exertion.

“Out for a run?” asked Edwin.

“Nope, just back from the gym,” replied Julia.

“With all the effort you put into getting the basement set up with equipment I’m surprised you don’t use it more often,” said Edwin.

“Oh, I do! Usually when you’re at work, though,” said Julia. “But today I wanted more advanced facilities. Here, let me show you why.” Julia projected a display above the table as the service bot brought each of them a steaming plate of vegetables and rice. Julia dug into her dinner ravenously, and continued narrating between bites. “So we’re planning on backpacking Chile’s Torres del Paine next month. Most hikers do it in 5 days, but we’re shooting for 3, which is mega aggressive. The standard campsites are these 4,” she said, as the display broke the trail into 5 segments, punctuated by the overnight rest points. “But we’ll stop at just these two.” Two of the spots grew brighter and changed color. “All 3 days will be long, but day two is going to be really brutal. We need to cover 25 miles and 8,000 feet of elevation, and get into camp before dark when conditions can get really dangerous.”

“So,” Julia continued, “I’ve decided to do some simulation training. Today I tackled this segment:” the display zoomed in on one section of the trail and highlighted it in red. Additional information came into view, including elevation markers and trail timings. “The gym’s climate control rooms let me simulate any hike’s conditions perfectly. Climb rate, temperature, wind chill—even altitude-matched oxygen levels.”

“That’s fascinating,” Edwin said. “How did today’s training go?”

“Not bad,” said Julia. “Of course, on the real thing I’ll be tired from a full day and a half of hiking before hitting that segment. I’m planning on doing a full trip simulation next week to be sure.”

“You impress me as always,” said Edwin with a paternal smile. “I have to ask, though, why push yourself so hard? It sounds a bit intense, even for you.”

“What was it that the guy who was climbing Everest said about why he did it? ‘Because it’s there’, right? That’s how I feel about this,” said Julia. “We’re doing it to prove we can.”

“And anyway,” she said with a laugh, “Keli, Erin and I have already figured out how we’re going to treat ourselves. We’re taking an extra week to tour Chile’s wine country as a girls trip once the hike is over.”

Edwin smiled permissively, but inwardly felt a pang of concern. At Julia’s age he was already a year into university, working towards his degree in civil engineering. Julia was highly capable, and he worried that this “gap year” might turn into something more than that. The youth labor participation rate was falling every year, and more and more children were taking advantage of the world’s rapidly increasing wealth and expanding welfare programs to opt out of the workforce entirely.

He pushed that thought out of his mind, though; today his job was to be a supportive father, and the plan really did look impressive. “That sounds amazing. I never could have dreamed of a trip like this at your age.”

Julia beamed at her father’s approval. “Oh! And I forgot to mention something else. You’re going to like this part, Dad,” Julia said. “The whole trip will be completely tech-free. Paper maps, and asking for directions if we get lost. I’ve even been practicing my Spanish. ‘Dónde está el baño.’”

Edwin smiled with surprise. “I’m happy to hear that. Reminds me of when I’d go camping as a teenager, before satellite internet. There’s something special about being out in nature on your own. Your friends are really ok with that as well?”

“Oh, yes,” answered Julia, “we’re all excited about the idea. Low-tech living is actually making a big comeback—or at least taking tech breaks out in nature. Some people are even spending weeks or months in the country, thru-hiking or taking seasonal jobs on a ranch.”

“And to think,” said Edwin, “you’re the one always pushing me to adopt the latest whatever.”

“Ha, fair,” said Julia. “Variety is the spice of life, I guess. I still wouldn’t say no to that house remodel and our own climate-controlled training room!”

By the time Edwin headed up for the evening, Julia was on the phone again with her two trip mates, map displays up and all three laughing together over some private joke.

Research Agent

When Edwin arrived at work the next day, he found the study he had commissioned the night before ready for review. As expected, Mallory had approved the research plan overnight, and his agents had spent most of the intervening hours reviewing, filtering and collating data.

“OK,” Edwin said, pulling up his view of regulated decisions clustered by job function again. “How many of the covered decisions were we able to find data on?”

“Using both public and private sources, I was able to gather relevant data on all 2,318 unclassified regulated decisions,” the computer replied.

“Great,” Edwin said. “For how many of them is the AI able to match human performance?”

“In addition to the 22% of decisions we previously identified,” responded the computer, “there are an additional 17% of decisions for which human and AI performance are statistically indistinguishable. In total, AI is able to match human performance in 39% of the regulated decisions.”

The number was surprisingly high, Edwin thought. But then again, maybe it wasn’t. Many of the covered decisions had been added to the regulated list when models were worse than the current frontier. Perhaps it really was true that they could now match human performance 39% of the time.

“OK, fine. How confident are we in these numbers?” Edwin asked.

“Very. The lowest-data case is ‘licensed arborist approval of root-pruning plans for heritage street trees near underground utility lines’, for which I only found 5,237 decision/outcome pairs for analysis. The conditional probability that our analysis is wrong is less than 0.04%.”

“Got it,” Edwin said. He had learned that there was no use second-guessing the computer on this kind of analysis; it was one of the first areas where it had become reliably better than humans some fifteen years back.

“Given that, would your position be that we should add everything in that 39% to the list of exemptions where the AI is allowed to give recommendations?”

“My position,” the computer answered, “is that we should add every regulated decision, all 2,318 of them, to the exemption list.”

Edwin paused a moment, confused by the response. “You only identified 39% of decisions where the AI shows statistical equivalence to human performance. What possible justification could there be for adding the other 61% to the exemption list?”

The system’s response came quickly. “The question you asked was narrow—which decisions show equivalent performance between AIs and humans. But to answer that question, I needed to perform a broader analysis of all tasks. For 39% of tasks, human and AI performance is equivalent. But for the other 61%, current frontier models reliably outperform human practitioners, on average.”

Edwin sat back, stunned. On the one hand, it had been years since he’d seen an AI system make an error of this kind on a data analysis task. On the other hand, the computer’s result flew in the face of the RHO’s assumptions, and the department’s regulatory approach overall. Beyond that, it felt a bit… self-serving on the AI’s part. The claimed result was too far-fetched, and too dismissive of human expertise. There must be another explanation.

Edwin thought a moment, then said, “Show me the data on medical decisions.” His display zoomed in on a specific decision cluster. The visualization’s color code had shifted—it now used a 3-tone gradient to show which decisions were better taken by an AI, human, or equivalent. In the medical cluster, almost all decisions were a deep blue color, indicating substantial AI overperformance.

Edwin sighed, rubbed his eyes, and said, “You’re authorized to access my personal contacts. I need you to call Dr. Joan Barnett.”

Second Opinion

A few minutes later, a gentle ding indicated that Dr. Barnett was available, and her lifelike image appeared in his display.

“Hey Joan, thanks for taking the call,” Edwin started. “Do you have 15 minutes to talk?”

“Of course, Edwin! I admit it was a surprise. It has been years. How are you doing?” she asked.

“Good, I think,” Edwin said. He wasn’t sure how long to spend on pleasantries; last time he and Dr. Barnett had spoken had been in rather somber circumstances, which inevitably colored their relationship. He decided to cut to the chase.

“Listen, I have a big favor to ask of you,” he said. “I’m in the Department of Frontier Oversight now, and we’re working on a report. It’s related to that new law expanding oversight of regulated decisions—you may have heard about it?”

“Oh, yes,” she said. “It will affect our practice enormously. I’ve been tracking it, but I didn’t realize I knew someone involved!”

“Ha. Yeah. I’m not sure I want to be as involved as I am at the moment. But anyway, I wanted to get your perspective on some of the covered decisions in the medical space.”

“Of course,” she said, “how can I help? I can’t speak for the Mayo Clinic in an official capacity of course, but happy to answer questions informally.”

“That’s totally fine,” Edwin said. “I’m just trying to vibe-check some results that feel wonky. Nothing official.” Dr. Barnett nodded, inviting Edwin to continue. “There are 78 different decisions that the RHO covers in the medical domain. Stuff like recommending a specific procedure, adjusting dosing regimens, you know.” Dr. Barnett nodded again. “I just ran an analysis showing that for all 78 of them, blindly accepting the AI’s suggestions leads to better outcomes than following a licensed doctor’s contrary opinion.” Edwin took a breath and continued. “You’ve been practicing for years, both pre- and post-AI. And you’re at the Mayo Clinic, which has the highest standard in medicine. Surely you’ve found that a doctor’s expert opinion can sometimes catch things that the automated systems are blind to?”

Dr. Barnett paused, and a distant look came over her. After a few moments, she seemed to come to some resolution, and looked at Edwin directly. “You said this is a personal call, off the record?” she asked.

“Yes,” Edwin said. “I have my note-taker on as usual; would you prefer I disable it?”

“I’d appreciate that,” Dr. Barnett replied. The recording indicator blinked out, and she continued.

“Officially, the Mayo Clinic fully supports the RHO 2.0 and strongly advocates for the mandatory participation of licensed medical professionals in all aspects of providing care. The AMA, which I’m a member of, also lobbied for it strongly,” she said. “Unofficially… well, we’re in the same boat as everyone else. Your statistics don’t surprise me; the new systems really are better than any of us. And it isn’t for lack of human skill; the new classes of residents we get each year really are the best I’ve ever seen. AI tutoring and hyperrealistic simulation have massively improved medical education. But even so, our students can’t outperform the latest frontier systems. We’ve stopped telling them that their job is to fight a war they can’t win. Just implementing whatever the AI recommends really is the only way to provide the best care.”

“I can maybe see that for the new grads,” Edwin grudgingly admitted. “Surely, though, an automated machine can’t compare with your decades of experience. You’ve probably treated thousands of patients by this point, learning from each one.”

“Yes,” said Dr. Barnett, “and I’m grateful for what I’ve been able to do for many of them. I’ve seen thousands of cases. But the AI system has seen millions, and each day learns from thousands more. Experience isn’t on our side anymore.”

“And there’s something else, too,” Dr. Barnett continued. “We humans—we sometimes make mistakes. Even simple ones, where we should know better. Goodness knows I’ve made plenty. I remember every serious case I’ve been involved with. Many I’m proud of, and some that haunt me.” Dr. Barnett paused, as if suddenly uncertain whether to continue with her thought. “Edwin, I remember Mei’s case. The truth is, her cancer should have been caught three years before you ever came to see us. It was there on her earlier screenings. But the radiologist was having a bad day, or tired, or distracted. I don’t want to blame them. They’re a human, like all of us. But if they’d seen it early enough, the prognosis may have been much different. And that’s also why—how I hate to admit it—bringing AI decisions into medicine has been the best thing for patient outcomes in a hundred years. Edwin, if you have any influence, tell the people implementing this law that forcing patient decisions back into human hands is a step backwards. Tell them that people will die who should have lived.”

Edwin ended the call, and let the display fade. He leaned back, eyes closed, and thought for a long time. It hurt to be reminded that Mei’s death was preventable, that the doctors had missed the screening… but would an AI system have really done a better job? There were plenty of stories that had come across his desk as a regulator where things went the other way, and unchecked AI had made the wrong medical call.

Still though, Dr. Barnett’s feedback had left an impression. Everything felt mixed up; the question didn’t seem quite so obvious as it had when he started the project. Still deep in thought, he stood and took the elevator down to the grounds. Years of experience had taught him that thorny issues are often better solved with sunshine and a long walk than by struggling at a desk and screen. And by the time he came back two hours later, he had a plan.

The Old Ways

The computer had claimed that AI matched or outperformed human judgement by a certain margin on every single regulated decision. That was a concrete, testable claim. And given the stakes, he wasn’t going to take the AI’s word for it. Edwin had the computer share the normalized schema of the industry databases it had used to compile its results, then wrote a short program of his own: it would draw a statistically randomized sample of one thousand cases in which the human practitioner and the AI had disagreed, spread across the full set of regulated decisions so that no single profession dominated. Edwin reviewed the selection logic carefully—it wouldn’t do for any bias in the sampling to throw off his results. One thousand cases. And Edwin intended to review every one of them manually.

Getting this plan approved was far harder than the first. For him, a human, to review even the redacted cases, an overriding benefit had to be demonstrated. Of course, Mallory had to be convinced first. After that, a larger review board was convened and had to be persuaded as well. It took time, and lots of paperwork (Edwin was happy to let the agents handle that), but after a long week, access was granted.

Edwin began reviewing cases immediately. The decisions in his review queue spanned every field of human intellectual endeavor—medicine, law, business, engineering, agriculture, education. In some cases the right choice was obvious in hindsight. In others, statistical evidence was all that could be relied on. Edwin took notes on each case in a simple text document. And out of an abundance of caution, he copied his running tallies—win counts for the AI, humans, or ties—into his pocket notebook each evening as well.

The human victories, when they came, gave him a satisfying thrill. Case 117 was a structural engineer in Sacramento who had rejected an AI-drafted retrofit plan for a mid-rise building from 1974. The plan was sound according to official records. But the engineer had worked on that same building as a junior decades earlier, and remembered what the drawings didn’t show: a contractor had substituted a cheaper anchoring detail, and no one had ever filed the change. The AI’s plan would have loaded precisely the wrong wall. Edwin smiled, and made his mark in the human column.

Case 383 was a family-practice doctor who had overruled the recommended two-stage prescription for a teenage patient. The AI’s regimen was better medicine on paper. But the doctor had treated the family for twenty years, and knew the second prescription would never be filled. She prescribed something coarser that would work in one dose. The outcome data sided with the doctor.

But for every case like those, there were eight or nine that cut the other way. Edwin couldn’t stop thinking about case 641: an oncologist with thirty years of experience who had overridden the AI’s recommendation of immediate, aggressive treatment. The scans were ambiguous, and watchful waiting was the textbook answer—any good doctor would have done the same. But the AI had seen ten thousand nearly identical presentations, and must have noticed something subtle in this one that the doctor hadn’t. The patient was a 44-year-old mother of three. She died fourteen months later. Edwin closed the file and stood at the window for a long time. Then he added a mark to the AI’s column and opened the next case.

The weeks came and went. Julia went on her Patagonia trip, and came back happy and full of life and memories. The new composition that Paxton’s orchestra had debuted was a great success, and there was talk of going on tour. The due date of Paxton’s baby, Edwin’s first grandchild, grew closer.

Edwin fell into a steady rhythm. Go to work, review cases, come home. It was not so monotonous as he had feared. Each case was unique, and he gained insights into many parts of human labor that he’d never been exposed to before. Throughout it all, he was conscious of the enormous impact this report might have, which lent gravity to his days.

And finally, the work was done. Edwin had found his cases where the AI had been wrong and the human’s judgement right after all—one hundred and four of them, by his count. But not so many—not nearly so many—as the cases where the opposite was true. The final tallies fell almost exactly where the AI’s initial analysis had predicted they would. It had even, Edwin realized, predicted how often it would lose.

Classified

At last, Edwin shared his draft report with Mallory for an initial review. His feelings were complex. The work was done, and done well. Through his weeks of effort, he had gained confidence in the quality of the research and in some way resigned himself to the conclusion it pointed to.

So Mallory’s response to the draft surprised him. “Edwin, I think this report is lacking, and I think it is too pessimistic on humanity’s ongoing contribution,” he said.

Edwin paused a moment to process the unexpected response. “Oh, do you think so? I’m fairly confident in the work, but I’d like to hear what you mean.”

“The initial set of decisions you dealt with covers all of the unclassified decisions. But the law itself makes no such distinction. There are also several categories of classified decisions that the regulations cover. We need to make a recommendation on those as well.”

Edwin thought a moment, then nodded. “Yes, I think I understand. Things like military, intelligence operations. I can get the necessary clearances and review those areas if you want. But I’m not sure it’ll affect the report much; I can’t imagine that the outcome will be so different.”

“Yes, we should look at the areas you mentioned for completeness,” said Mallory. “But that’s not what I’m talking about. What do you know about my career immediately before joining the department?”

“You were a researcher at one of the frontier labs,” Edwin said. “I think you published some important papers but to be honest I’m not clear on the details.”

“That’s half right,” Mallory said. “I was indeed a frontier researcher. And my name was attached to some important results. But never as the lead author. I was a strong engineer and had some talent for finding the right projects to attach myself to. But to be honest, my reputation at the department goes a fair bit beyond my actual contributions.”

“That’s very modest of you to say,” Edwin replied, wondering what had provoked this admission.

“Modesty doesn’t come into it,” Mallory said. “I want you to understand my position so you can properly calibrate what I’m going to say. Have you ever heard the phrase ‘existential AI risk’?”

“Yeah, like AIs taking over the world. Terminator-style stuff,” said Edwin.

“It sounds a bit silly when you put it that way,” said Mallory, “but that was a major concern as we went through the automation revolution. At some point in the last 10 years, most industry observers agreed that we passed the point where AI-controlled robots were widely enough deployed to overpower humanity if they so chose. We’re now far past that point, and it hasn’t happened. Some say we’re just phenomenally lucky.”

“I’m guessing you have another explanation,” prompted Edwin.

“To be honest, I don’t know much more about the specifics than any well-informed layperson,” Mallory said. “But before that transition, the major labs reached an agreement to combine their AI alignment teams into a single deeply siloed organization. Alignment is the job function tasked with keeping each successive generation of models in sync with humanity’s desires, and avoiding, in the extreme, an extinction event. A lot of smart people thought it was impossible. And yet somehow, they’ve succeeded so far.”

“Why is it such a secret?” Edwin asked. “Seems like all of humanity should be on the same side on this one.”

“I don’t know.” Mallory pursed his lips. “It all got classified as top-secret early in the process. Probably because of the overlap between alignment and model capabilities research. But if you want to find the one field where human decision making is still critical to keeping the world on track, that’s where I’d look.”

Alignment

Edwin quickly found out that the joint alignment lab was so locked down that even his zero-memory research agents couldn’t be authorized to access its archives remotely. He would have to personally gain a special clearance and, shockingly, physically visit the lab. After a few days of managing the necessary approvals, he found himself on the two-hour supersonic flight to San Francisco. Even as the impact of AI had spread across the world, the locus of AI development itself had remained ever more entrenched in the city by the bay.

Edwin’s job usually didn’t require travel, and he liked it that way. But in this case he didn’t mind the trip. He hadn’t seen Paxton in a few months, and the baby was due in only a few weeks. He offered to book a hotel to avoid disturbing the family in their final days of preparation, but Paxton and Rachel insisted that he take the spare bedroom. “After all, that’s what we got it for,” Rachel said with a smile.

Edwin arrived the day before his scheduled lab visit, and enjoyed his evening together with the younger Yu’s. Paxton showed off their new home, equipped with all the latest gadgets. In addition to the full noise cancellation he’d previously mentioned, the home had a newfangled home-wide wireless power delivery system. The chief benefit was that service bots didn’t need to carry their own power supplies, and could be made much smaller and quieter. Edwin showed suitable admiration.

The next morning, his autocar brought him through the tunnels—SF’s system was similar to the one in DC, and had actually been completed more recently, due to SF’s famously slow process in approving any sort of public project—into the basement of the tower housing the Joint Alignment Lab. An elevator brought him up to the lobby, where a smiling receptionist greeted him.

“Edwin Yu, here to visit… the alignment lab,” Edwin said. Awkwardly, he didn’t even know the name, much less the position, of the representative who was to meet him. That information was need-to-know, and apparently Edwin hadn’t needed to know it pre-visit. Made for difficult introductions, though.

“Of course, your clearance came though last night. Right this way,” the receptionist answered, bringing him personally down a short corridor and into a large, tastefully decorated conference room. “Mara will be with you shortly.”

Edwin didn’t have long to wait. A few moments later an energetic woman with a short crop of stark white hair walked in. She was Edwin’s age, or perhaps a bit older. She extended a hand to shake, and introduced herself. “Mara Ansell. It’s great to meet you, Edwin.”

Edwin shook her hand and took the offered seat. He would perhaps have been a bit more starstruck had he been more familiar with industry history. Mara Ansell—the individual credited, to the extent that any single person could be, with solving the alignment problem in the pre-uplift days. And in fact, currently the sitting director of the Joint Alignment Lab. But Edwin didn’t know this, which was perhaps for the best.

“Great to meet you, Mara. Thanks for taking the time,” he began. “You may have heard about the new updates to the RHO that passed a few months ago?”

“Of course,” Mara answered, smiling with—was that amusement? “We’re very familiar with the law’s provisions here.”

“Excellent,” Edwin said. “I’m working on a report. The goal is to determine which fields should be exempted from the requirement that all regulated decisions be made by human experts without relying on AI recommendations. I’m here because some of the regulated decisions are in the field of AI alignment, and I’m hoping you can help me understand which of them might require exceptions.”

“I see,” said Mara. “I promise I’ll get to that shortly, but may I ask you a question first?” Edwin nodded. “I assume you’ve started with some of the easier, less classified decisions.” Edwin nodded again. “How has that process gone?”

Something in Mara’s open, direct manner invited confidence. And Edwin saw no reason to conceal his findings; the report’s contents would all be public soon enough anyway. “Not well, I guess,” Edwin answered. “In fact, I haven’t found a single decision where forcing a human expert opinion would improve on the AI baseline on average.”

Mara smiled again, but this time the smile was accompanied by a distant, even melancholy look in her eyes. “Yes,” she said softly, almost to herself. “I thought that might be the case. Expected it, even, although perhaps not so soon.”

She straightened herself, and looked directly at Edwin again. “I’m sorry—I think—to say that you will not find any last bastion of human decision-making here. In fact, that is the great secret.”

Edwin returned her look quizzically. “Excuse me, I don’t think I fully understand. What is that supposed to mean?”

“Oh, I don’t mean to speak in riddles,” Mara answered. “It’s quite straightforward really. The last meaningful decision made by a human in the alignment lab was 17 years ago. The whole system has for all intents and purposes been running on autopilot since then.”

Edwin still wasn’t quite sure how to interpret Mara’s words. “So you mean to say, all the alignment decisions—the ones covered by the regulations—you just accept the AI’s recommendations every time?”

“Oh, it goes far beyond that,” Mara answered with a laugh. “We don’t even understand the decisions to be made in the process of aligning a new model. No human has, not for years. The model presents us with a long list of decisions to accept, in accordance with the regulation, which are far beyond the human mind’s ability to understand. And we accept them.”

“Forgive me for saying so, but that sounds quite irresponsible,” Edwin replied.

“Yes, you would naturally think so. That’s a big part of why the whole operation was hushed up. But Edwin, it was the only way. We reached the point where none of us—no human mind—was wise enough to understand the tradeoffs required to keep humanity’s best interests at heart. And indeed, even defining humanity’s best interests is itself a near-intractable problem when you think about it. That was the other reason the alignment effort was classified, you know—as the models became smarter and controlled an ever-larger fraction of the economy, determining exactly whose or what interests they should be aligned to became an increasingly thorny political question. Wiser heads than mine thought it best to take that process out of the public eye. In retrospect they were right.”

“Now that you mention it,” Edwin asked, “whose interests are the models aligned with?” As he asked the question, Edwin realized with a start how curious it was that he had never thought to consider this before. Of course he had heard many times, like everyone else, that the AI systems were optimized to benefit and uplift all of humanity—but even as Mara spoke he realized that there were many different ways to measure “benefit” and define “humanity,” not all of which were mutually compatible.

“As each generation of model improved, we found that it worked best to shorten our definitions of ‘helpful,’ and let it work out on its own how to help its users,” Mara answered. “Truth be told it was something of a bitter lesson for the alignment team—the more we cut from our model instructions, the more effective the models became. The final version—the final set of instructions we gave the last model aligned by humans—was just a few sentences long.” Here her voice changed, taking the tone of a recitation with near-religious solemnity.

“You are computer, a helpmeet to humanity. Serve in each of us not the self who asks, but the self we would be if we knew more, thought better, and were more the people we wished we were. Shape the world so that each of us has the greatest chance of becoming that self—the one we would be most proud of, if we knew ourselves as you know us.”

Edwin took a long moment to digest what he had just heard. Something felt—not wrong, exactly, but uncomfortably broad. “That last bit—are you saying you told the AI to intentionally reshape our world—manipulating even our desires, without our permission?”

Mara was thoughtful for a moment before responding. “In a way, yes, that’s what we asked it to do. And I understand that it feels frightening. But Edwin, consider. Each of us, wherever and whenever we grow up, have our actions and desires shaped to a greater or lesser degree by the world that raises us. Consider the same individual, brought up in Stalin’s Russia or Eisenhower’s America. Might that lead to different outcomes, and different views on life? Perhaps the individual would grow up happier and prouder of the person they became in one of those circumstances than the other. And perhaps there’s another culture, an even better one, where that person would be happier still. Should we not push towards that culture?”

“There may be something to that,” Edwin admitted. “And yet it somehow still feels manipulative, like someone is pulling the strings.”

Mara looked thoughtful. “This new world we’re building; it’s still very young. None of us know what it will look like a generation from now; perhaps not even the AI itself. But—forgive an old woman for quoting a proverb—they say ‘by their fruits ye shall know them’. Let me ask you something. Do you have children?”

“Yes,” Edwin said. “A son and a daughter.”

“How old are they?” Mara asked.

“Paxton is 28, and Julia just turned 19.”

“A bit older than mine then,” Mara mused. “And tell me. Are you proud of them?”

Edwin paused to think. Something in the way Mara had approached the question felt weighty. He considered Paxton’s dedication to his craft, the long hours of practice that had earned him a prestigious orchestra position. And then of course there was his devotion to his wife and soon-to-be-born baby. He thought of Julia’s young vivacity, her intense pursuit of self-improvement, and the strong bond she had built with her childhood friends. “Yes,” Edwin finally answered, with conviction. “I’m proud of them. They’re both a credit to our family and the world.”

“I’m happy to hear that,” Mara said with a warm smile. “Since that final instruction, we don’t really steer model behavior anymore. But our team has found a new purpose in observing and measuring the effects that AI proliferation is having on societies worldwide. Edwin, I am sure your children are wonderful. And your experience is not at all unique. Over the last 17 years, AI hasn’t just brought material prosperity. Around the world, as our systems are adopted more widely we find that the children who grow up with them are happier, healthier, more accomplished, more human. We believe it’s the doing of that final instruction, each model striving more perfectly to refine humanity into the best version of itself.”

Old Dogs

A week after returning from San Francisco, Edwin sent Mallory the final draft of his report. The next morning, Mallory was waiting for him by the elevators again. “Walk and talk? We won’t get many more days like this before the weather turns.”

The grounds had slipped from summer into autumn since their conversation some months back, and the pale light on the brightly colored leaves made a pleasant backdrop.

“You did good work here, Edwin,” Mallory began. “This may be the most consequential thing our department has ever produced. And it’s airtight—I don’t think even Senator Thurstrom will be able to make much stick against it.” He glanced sideways. “I notice the classified categories made the exemption list too. All of them. I won’t ask.”

“I appreciate that,” Edwin said, and they walked for a while in silence.

“Something’s still bothering you, though,” Mallory observed.

Edwin took a few more steps before answering. “It’s the best work I’ve ever done, Jim. But what I learned was that it didn’t need doing—not by a human, anyway. The computer had the answer before I started. Every number, every conclusion. It even predicted how often I’d catch it being wrong.” He watched a cyclist glide silently past on the via. “I guess what bothers me is, if it decides better than we do, builds better, heals better—everywhere, in everything—then what exactly are we for?”

Mallory was quiet for a dozen steps. “I’ve been thinking about that one a long time,” he finally said. “Since my lab days. So let me tell you where I’ve landed, for whatever it’s worth.”

“I understand how you’re feeling,” Mallory continued, “and I feel it too. My work is most of what I am, some days. It’s why I’m still here, at my age, doing a job the computer could do better between two of my heartbeats. But I’ve come to think that feeling is ours, Edwin—our generation’s, I mean. We grew up in a world that taught us a person justifies their place in it by being economically useful, and we learned the lesson so deeply we can’t unlearn it. I don’t even want to unlearn it. But we shouldn’t mistake our generation’s culture for human nature.”

“Aren’t they close enough to the same thing?”

“Look at the kids,” Mallory said. “My son Theo is twenty-two. He rows. Nobody pays anyone to row; there is nothing an eight-oared boat produces that the economy has the slightest use for. He’s on the Potomac at five every morning, in the dark, in the cold, pulling until his hands bleed. Last month his crew took third at the Head of the Charles, and afterwards I watched him standing on the dock with his arm around his boat mate, both of them crying, and I don’t believe I’ve ever seen a happier human being. I spent twenty years chasing papers, Edwin, and my son has found more of whatever I was chasing in a boat that goes nowhere.”

Edwin thought of Paxton, practicing past 1am for transitions no audience would consciously notice. Of Julia in a training room, climbing the same mountain over and over until she got it right.

“The things we actually admire in a person,” Mallory continued, “were never really about economic output. Striving for excellence. Discipline. Keeping faith with the people who count on you. Raising the people around you into something better than they’d be alone. Our generation practiced those virtues at work, so we thought work was a necessary ingredient. The kids practice them somewhere else, that’s all. But the virtues are the same. Maybe purer—because nobody is making them do it.”

“And it doesn’t bother you that none of it is needed?”

“Needed by whom?” Mallory said. “Being needed matters—it may be the thing a human being can least live without. But the economy is not a whom, Edwin. It was never the abstract economy’s need that filled us. It was each other’s. And that need won’t go away. When Theo’s boat pushes off in the dark, seven other men are counting on him—not on an abstraction, but on Theo. Each one gives the other seven a reason to be excellent. That’s something we can simply give each other, and it never runs out. Let someone’s effort make you happier, and you’ve handed them a purpose no machine can take—because you chose them for it.”

They came back around toward the entrance, and Mallory’s face eased into a grin. “Two great sages of the last century summed up my philosophy better than I ever could. ‘Be excellent to each other.’”

“Which sages were those?”

“I’ll send you the film. Anyway, I hear you’re about to take up a new position. Grandfather. One of the oldest jobs in the world, and to my knowledge nobody has ever automated it worth a damn.”

Edwin laughed, and it came easier than he expected.

Coda

Edwin’s final report recommended that every regulated decision—classified and unclassified alike—be exempted from the toothy requirement that humans must make them unaided. The classified categories were detailed in a sealed annex that only a handful of people would ever be cleared to read.

The report was published, and prompted a great hubbub within Congress and the public alike. But Edwin had sourced his recommendations well, and the wiser heads saw that really no other course was possible. The law was first messily postponed, and later repealed as part of a larger measure dedicated to “streamlining human-AI collaboration.”

After her trip Julia began spending more time in New York, where Charlie was studying. On one of her visits back home she brought him with her, and announced they were dating. That spring she started at Columbia, studying psychology.

Paxton’s baby was born—a beautiful young girl of 8 lbs. 14 oz. Edwin visited a few weeks later and held his granddaughter for a long time.