Leslie Lamport may not be a household name, but he’s behind a few of them for computer scientists: the typesetting program LaTeX and the work that made cloud infrastructure at Google and Amazon possible. He’s also brought more attention to a handful of problems, giving them distinctive names like the bakery algorithm and the Byzantine Generals Problem. This is no accident. The 81-year-old computer scientist is unusually thoughtful about how people use and think about software.

In 2013, he won the A.M. Turing Award, considered the Nobel Prize of computing, for his work on distributed systems, where multiple components on different networks coordinate to achieve a common objective. Internet searches, cloud computing and artificial intelligence all involve orchestrating legions of powerful computing machines to work together. Of course, this kind of coordination opens you up to more problems.

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable,” Lamport once said.

Among the biggest sources of problems are “concurrent systems,” where multiple computing operations happen during overlapping slices of time, leading to ambiguity: Which computer’s clock is the right one? In a seminal 1978 paper, Lamport introduced the notion of “causality” to solve this issue, using an insight from special relativity. Two observers may disagree on the order of events, but if one event causes another, that eliminates the ambiguity. And sending or receiving a message can establish causality among multiple processes. Logical clocks — now also called Lamport clocks — provided a standard way to reason about concurrent systems.

With this tool in hand, computer scientists next wondered how they could systematically make these connected computers even bigger, without adding bugs. Lamport came up with an elegant solution: Paxos, a “consensus algorithm” that allows multiple computers to execute complex tasks. Without Paxos and its family of algorithms, modern computing could not exist.

In the early 1980s, as he developed the field, Lamport also created LaTeX, a document preparation system that provides sophisticated ways to typeset complex formulas and format scientific documents. LaTeX has become the standard for formatting papers not only in math and computer science but also in most scientific domains.

Lamport’s work since the 1990s has focused on “formal verification,” the use of mathematical proofs to verify the correctness of software and hardware systems. Notably, he created a “specification language” called TLA+ (for Temporal Logic of Actions). A software specification is like a blueprint or a recipe for a program; it describes how software should behave on a high level. It’s not always necessary, since coding a simple program is akin to just boiling an egg. But a more complicated task with higher stakes — the coding equivalent of a nine-course banquet — requires more precision. You need to prepare each component of each dish, combine them in a precise way, then serve them to every guest in the correct order. This requires exact recipes and instructions, written in unambiguous and succinct language, but descriptions written in English prose could leave room for misinterpretation. TLA+ employs the precise language of mathematics to prevent bugs and avoid design flaws.

Using your recipe, or specification, as an input, a program called a model checker will check whether the recipe makes sense and works as intended, producing a dish the way the chef wants it. Lamport laments how programmers often cobble together a system before writing a proper specification, whereas chefs would never cater a banquet without first knowing that their recipes will work.

*Quanta* spoke with Lamport about his work on distributed systems, what’s wrong with computer science education, and how using TLA+ can help programmers build better systems. The interview has been condensed and edited for clarity.

People were building a system with some code, and I had the hunch that what their code was trying to accomplish was impossible. So I decided to try to prove it, and instead came up with an algorithm that the people should have been using for their system.

Well, they didn’t have an algorithm, just a bunch of code. Very few programmers think in terms of algorithms. When trying to write a concurrent system, if you just code it without having algorithms, there’s no way that your program is not going to be full of bugs.

**The paper**** that introduced Paxos wasn’t very widely read at first. Why was that? **

What made it impossible for people to read the paper was that I like explaining things with stories, and I made up names for characters in sort of pseudo-Greek letters. For example, in the paper there was a cheese inspector named Γωυδα. Having grown up as a mathematician, where Greek letters were used all over the place, I was just unaware that nonmathematicians get completely freaked out by those letters. Apparently, the readers couldn’t deal with it, and it caused that paper not to be read as it should have been.

So that didn’t work as well at first. Although in the long run it did, because people call this family of consensus algorithms Paxos instead of “viewstamped replication,” which was another name for the same algorithm from [the computer scientist] Barbara Liskov.

In the 1970s, when people were reasoning about programs, they were proving properties of the program itself stated in terms of programming languages. Then people realized that they should really be stating what the program is supposed to accomplish first — the program’s behaviors.

In the early 1980s, I realized that one practical method of writing these higher-level specifications for concurrent systems was writing them as abstract algorithms. With TLA+, I was able to express them mathematically in a completely rigorous fashion. And everything clicked. What that involves is basically not trying to write algorithms in a programming language: If you really want to do things right, you need to write your algorithm in the terms of mathematics.

Model checking is a method for exhaustively testing all executions of a small model of the system. It just shows the correctness of the model, not of the algorithm. While model checking tests for correctness, coding just produces code. It doesn’t test anything. Before there was model checking, the only way to be sure that your algorithm worked was to write a proof.

In practice, model checking checks all executions of a small instance of the algorithm. And if you’re lucky, you can check large enough instances that it gives you enough confidence in the algorithm. But the proof can prove its correctness for a system of any size and for any use of the algorithm.

Coq was designed to do real mathematics and to be able to capture the reasoning that mathematicians do. It’s what Georges Gonthier used to prove the four-color theorem, for example. A machine-checked proof of a mathematical statement shows that the statement is almost certainly true.

TLA+ is designed not for mathematicians but for engineers who want to prove the properties of their systems. In the 1990s, after having spent about 15 years writing proofs of concurrent algorithms, I learned what you needed to do in order to prove the correctness of a concurrent algorithm. TLA was the logic that allowed it to be all completely formal. And TLA+ is the complete language based on that.

Well, I’m doing what I can. But basically, programmers and many (if not most) computer scientists are terrified by math. So that’s a tough sell.

Secondly, every project has to be done in a rush. There’s an old saying, “There’s never time to do it right. There’s always time to do it over.” Because TLA+ involves upfront effort, you’re adding a new step in the development process, and that’s also a hard sell.

True, most of the code written by programmers across the world doesn’t require very precise statements about what it’s supposed to do. But there are things that are important and need to be correct.

When people build a chip, they want that chip to work right. When people build a cloud infrastructure, they don’t want bugs that will lose people’s data. For the kind of application where precision is important, you need to be very rigorous. And you need something like TLA+, especially if there’s concurrency involved, which there usually is in these systems.

Yes, the importance of thinking and writing before you code needs to be taught in undergraduate computer science courses and it’s not. And the reason is that there’s no communication between the people who teach programming and the people who teach program verification.

From what I’ve seen, the fault lies on both sides of that divide. The people who teach programming don’t know the verification that they need to know. The people who are teaching verification don’t understand how it should be applied and used in practice.

Until that divide is bridged, TLA+ is not going to find a large number of users. I hope I could at least get the people who teach concurrent programming to understand that they need it. Then maybe there’s some hope.

On mathematical thinking, yeah.

I’m not an educator, so I don’t know how to teach it to them. But I know what people should have learned. They shouldn’t be afraid of math. It’s just simple math that they’ve probably taken a course in, but they don’t know how to use it. They don’t know what good it is. They learn enough to pass the exam and then they forget about it.

I don’t think in terms of aesthetics. I probably have the kinds of feelings that other people do, but I just use different words to express them. Being beautiful is not something I would say about an algorithm. But simplicity is something that I value highly.

Any way you want. I don’t advise spending very much time thinking about it.

The quantum physicists Sandu Popescu__,__ Yakir Aharonov and Daniel Rohrlich have been troubled by the same scenario for three decades.

It started when they wrote about a surprising wave phenomenon called superoscillation in 1990. “We were never able to really tell what exactly was bothering us,” said Popescu, a professor at the University of Bristol. “Since then, every year we come back and we see it from a different angle.”

Finally, in December 2020, the trio published a paper in the *Proceedings of the National Academy of Sciences* explaining what the problem is: In quantum systems, superoscillation appears to violate the law of conservation of energy. This law, which states that the energy of an isolated system never changes, is more than a bedrock physical principle. It’s now understood to be an expression of the fundamental symmetries of the universe — a “very important part of the edifice of physics,” said Chiara Marletto, a physicist at the University of Oxford.

Surprising as superoscillation is, it doesn’t contradict any laws of physics. But when Aharonov, Popescu and Rohrlich applied the concept to quantum mechanics, they encountered a situation that’s downright paradoxical.

In quantum mechanics, a particle is described by a wave function, a kind of wave whose varying amplitude conveys the probability of finding the particle in different locations. Wave functions can be expressed as sums of sine waves, just as other waves can.

A wave’s energy is proportional to its frequency. This means that, when a wave function is a combination of multiple sine waves, the particle is in a “superposition” of energies. When its energy is measured, the wave function seems to mysteriously “collapse” to one of the energies in the superposition.

Popescu, Aharonov and Rohrlich exposed the paradox using a thought experiment. Suppose you have a photon trapped inside a box, and this photon’s wave function has a superoscillatory region. Quickly put a mirror in the photon’s path right where the wave function superoscillates, keeping the mirror there for a short time. If the photon happens to be close enough to the mirror during that time, the mirror will bounce the photon out of the box.

Remember we’re dealing with the photon’s wave function here. Since the bounce doesn’t constitute a measurement, the wave function doesn’t collapse. Instead, it splits in two: Most of the wave function remains in the box, but the small, rapidly oscillating piece near where the mirror was inserted leaves the box and heads toward the detector.

Because this superoscillatory piece has been plucked from the rest of the wave function, it is now identical to a photon of much higher energy. When this piece hits the detector, the entire wave function collapses. When it does, there’s a small but real chance that the detector will register a high-energy photon. It’s like the gamma ray emerging from a box of red light. “This is shocking,” said Popescu.

The clever measurement scheme somehow imparts more energy to the photon than any of its wave function’s components would have allowed. Where did the energy come from?

The mathematician Emmy Noether proved in 1915 that conserved quantities like energy and momentum spring from symmetries of nature. Energy is conserved because of “time-translation symmetry”: the rule that the equations governing particles stay the same from moment to moment. (Energy is the stable quantity that represents this sameness.) Notably, energy isn’t conserved in situations where gravity warps the fabric of space-time, since this warping changes the physics in different places and times, nor is it conserved on cosmological scales, where the expansion of space introduces time-dependence. But for something like light in a box, physicists agree: Time-translation symmetry (and thus energy conservation) should hold.

Applying Noether’s theorem to the equations of quantum mechanics gets complicated, though.

In classical mechanics, you can always check the initial energy of a system, let it evolve, then check the final energy, and you’ll find that the energy stays constant. But measuring the energy of a quantum system necessarily disturbs it by collapsing its wave function, preventing it from evolving as it otherwise would have. So the only way to check that energy is conserved as a quantum system evolves is to do so statistically: Run an experiment many times, checking the initial energy half the time and the final energy the other half. The statistical distribution of energies before and after the system’s evolution should match.

Popescu says the thought experiment, while perplexing, is compatible with this version of conservation of energy. Because the superoscillatory region is such a small part of the photon’s wave function, the photon has a very low probability of being found there — only on rare occasions will the “shocking” photon emerge from the box. Over the course of many runs, the energy budget will stay balanced. “We do not claim that energy conservation in the … statistical version is incorrect,” he said. “But all we claim is that that is not the end of the story.”

The problem is, the thought experiment suggests that energy conservation can be violated in individual instances — something many physicists object to. David Griffiths, a professor emeritus at Reed College in Oregon and author of standard textbooks on quantum mechanics, maintains that energy must be conserved in each individual experimental run (even if this is normally hard to check).

Marletto agrees. In her opinion, if it looks as if your experiment is violating this conservation law, you’re not looking hard enough. The excess energy must come from somewhere. “There are a number of ways in which this alleged violation of the energy conservation could come about,” she said, “one of which is not fully taking into account the environment.”

Popescu and his colleagues think they have accounted for the environment; they suspected that the photon gains its extra energy from the mirror, but they calculated that the mirror’s energy does not change.

The search continues for a resolution to the apparent paradox, and with it, a better understanding of quantum theory. Such puzzles have been fruitful for physicists in the past. As John Wheeler once said, “No progress without a paradox!”

“If you ignore such questions,” Popescu said, “you’re never really going to … understand what quantum mechanics is.”

Today, more than three years after the release of the first-ever image of a black hole, scientists from the Event Horizon Telescope (EHT) shared an image of Sagittarius A* (pronounced A-star) — the supermassive specimen sitting at the center of our own Milky Way galaxy.

“It is a dream that comes true after decades of work,” said Heino Falcke, an astrophysicist at Radboud University in the Netherlands. “I always knew this day would come, but I never expected it to be so clear and impressive right away.”

The image immediately reveals new information about the Milky Way’s monster. “The major things we found out about Sag A* were: Is the black hole spinning? Yes, it is,” said Sara Issaoun, an astrophysicist and member of the EHT team. “And what is the orientation of the black hole with respect to us? Now we are fairly confident it is pointed more or less face on to us,” with one of the poles pointed in our direction.

The new images were taken in April 2017 during the same window in which the EHT was taking the now-famous image of M87’s black hole. Eight telescopes gathered views of Sagittarius A* over the course of 10 consecutive nights. The volume of information collected was enormous, said Lindy Blackburn, an EHT data scientist at the Harvard-Smithsonian Center for Astrophysics — billions of gigabytes’ worth. The resulting files were too large to go out over the internet. Instead, more than 1,000 hard drives were physically transported back to two processing facilities, one at the Haystack Observatory near Boston, and another at the Max Planck Institute for Radio Astronomy in Bonn, Germany.

The EHT uses a technique called very-long-baseline interferometry to produce its images, turning Earth into a giant virtual telescope by combining the views of multiple observatories from the South Pole to Spain. The spread-out telescopes can create sharper images, just as a larger mirror on an optical telescope affords better views. Except in this case, the observations were performed at a wavelength of 1.3 millimeters rather than the wavelength of visible light. “That wavelength is a sweet spot,” said Carl Gwinn, an astrophysicist at the University of California, Santa Barbara who was not involved in the result. It allowed astronomers to peer through the hot gas surrounding a supermassive black hole, but it also provided the necessary resolution to reveal the shadow resulting from its event horizon — the point at which no light can escape.

Imaging Sagittarius A* is the end result of decades of observations that began with our first tantalizing hints of its presence in 1918, when the astronomer Harlow Shapley first noticed stars congregating toward the center of the Milky Way. Later observations detected powerful radio emissions coming from that spot, pointing to the presence of a massive yet compact object, most likely a black hole — a phenomenon predicted by Einstein’s general theory of relativity.

In the 21st century, scientists solidified this idea by tracking the motions of stars, research that was awarded the 2020 Nobel Prize in Physics. One star in particular, named S2, has a 16-year elliptical orbit that scientists were able to follow in its entirety, gaining a clear (though indirect) view of the black hole. “It has a beautiful trajectory,” said Luciano Rezzolla, a theoretical astrophysicist at Goethe University Frankfurt in Germany and a member of the EHT’s board. “It’s a gift nature has given us.”

Now that we’ve directly seen this black hole, scientists will probe its intricacies and compare it to its much larger sibling inside M87. “Going from a sample of one to two is a big jump,” said Sarah Gallagher of Western University, an astrophysicist who was not part of the EHT collaboration.

Both observations are glorious in their own right: beautiful results that are “an affirmation of the scientific process,” said Gallagher. But seeing Sagittarius A* is just a bit more extraordinary for many. It’s an object that has fascinated us for so long, and now, right before our eyes, it dazzles us with its dance. “This is an even greater technical accomplishment,” said Blandford. “This is up close and personal. This is our home.”

* Correction: May 16, 2022
The original version of this article described “face on” incorrectly. It means that the axis of rotation is more or less pointed in our direction. In addition, the original graphic contained incorrect numbers for the radii of the two black holes. *Quanta

Last summer, three researchers took a small step toward answering one of the most important questions in theoretical computer science. To paraphrase Avi Wigderson of the Institute for Advanced Study, that question asks something simple but profound: Can we solve all the problems we hope to solve?

More precisely, computer scientists want to know whether all the problems we hope to solve can be solved efficiently, in a reasonable amount of time — before the end of the universe, say. If not, they are simply far too difficult.

Many problems seem to be this hard, but we won’t know for certain until we can mathematically prove their difficulty. And in a paper from last year, a trio of computer scientists showed that a broad category of problems are indeed too difficult to be solved efficiently, thereby providing one of the best examples yet of what the field has been seeking.

“We’re looking for stronger footholds as we’re climbing this mountain,” said Paul Beame of the University of Washington. “And this is another foothold on this route.”

The problems the researchers studied require only addition and multiplication. When these problems are restricted to being solved in specific ways, with alternating patterns of addition and multiplication, they become extremely difficult to solve.

“It just changed our knowledge dramatically,” said Amir Shpilka of Tel Aviv University in Israel. “We didn’t know how to do anything close to that.”

Surprisingly, the findings required no new frameworks or tools. Instead, the authors figured out how to bypass a mathematical roadblock that had emerged in decades-old work by Wigderson in collaboration with Noam Nisan of the Hebrew University of Jerusalem.

“We realized there was a very silly way of getting around it,” said Srikanth Srinivasan of Aarhus University in Denmark, who authored the new work along with Nutan Limaye of the IT University of Copenhagen and Sébastien Tavenas of Savoy Mont Blanc University in Chambéry, France. “And it feels like if there’s such an easy way to do something we didn’t think possible, surely there should be a way to go further.”

In the 1960s, after computers had been around for a couple decades, a troubling trend emerged. Scientists had successfully created algorithms to get their computers to solve various problems, but sometimes these algorithms took too long — longer than was practical.

They began to suspect that some problems were just fundamentally hard, regardless of the problem’s scale. For example, consider a collection of points with edges connecting them, called a graph. An important problem is to determine whether there exists a path, called a Hamiltonian path, that travels along the edges to visit each point just once. It stands to reason that if you increase the number of points (and edges), it will take longer to determine whether such a path exists. But the best algorithms took exponentially longer with increasing scale, making them impractical.

Many other problems seemed to take exponential time as well. Computer scientists would go on to show that any algorithm that could somehow solve one of these hard problems efficiently could be transformed to do the same for others of similar difficulty. They called this category of problems NP. When Wigderson referred to “all the problems we hope to solve,” he meant the problems in NP, which come up in many contexts.

Of course, there were also many problems that did not seem to be hard, and that did not take exponential time to solve. Computer scientists showed that many of these problems were also equivalent in a certain sense, and they called this category P. They suspected that NP problems were truly harder than P problems, and that problems in NP could never be solved efficiently. But without proof, there was always a chance that they were wrong.

Computer scientists began to search for ways to prove that NP problems really were harder, a question formally known among researchers as P versus NP. To answer it, it would suffice to prove that a hard problem really required exponential time, but they had no idea how to do that. So they sought out other avenues where hardness might not be so hard to pin down.

One particular set of problems seemed just right. This was the set that depended only on addition and multiplication. For example, given a set of points, it is possible to count all the possible Hamiltonian paths (if any exist) merely by adding and multiplying data about the points. (The same count can also be performed if other operations are allowed, such as comparing values, but then the problem and solution become more complicated.)

Further, this set of simpler “arithmetic” problems mirrored the larger landscape of more complicated tasks. That is, some arithmetic problems (like counting Hamiltonian paths) seemed to take exponentially more time as the scale of the problem increased. In 1979, Leslie Valiant of Harvard University showed that many arithmetic problems were equivalent in their difficulty, and that others were equivalent in their lack of difficulty. Computer scientists would later name these categories after him — VNP and VP, respectively.

As with the P versus NP question, the hardness of VNP problems could not be proved. Computer scientists had rediscovered the P versus NP problem in a new form — VP versus VNP — only now they had a key advantage. VNP problems are even more difficult than NP problems, because they build on them: Counting paths requires you to be able to determine if a path exists. And if you want to prove something’s hard, you want as much hardness as possible.

“It’s harder than NP,” said Shpilka. “And therefore proving that it’s hard should be easier.”

In the ensuing decades, computer scientists made much more progress on the VP versus VNP question than they had on P versus NP, but most of it was limited to the subfield that Valiant had created known as algebraic complexity. Until the recent work by Limaye, Srinivasan and Tavenas, they still had trouble telling whether there were any arithmetic problems that were hard in a general sense.

To understand the new work, it helps to learn how computer scientists think about addition and multiplication problems. Mathematically, these problems are completely captured by expressions called polynomials — like *x*^{2} + 5*y* + 6 — which consist of variables that are added and multiplied together.

For any particular problem, like counting Hamiltonian paths, you can build a polynomial that represents it. You can represent each point and edge with a variable, for example, so that as you add more points and edges, you also add more variables to the polynomial.

To prove an arithmetic problem like counting Hamiltonian paths is hard, you need to show that when you add more points and edges, the corresponding polynomial takes exponentially more operations to solve. For example, *x*^{2} requires one operation (multiplying *x* by *x*), whereas *x*^{2} + *y* takes two operations (multiplying *x* by *x* and then adding *y*). The number of operations is called a polynomial’s size.

But a polynomial’s size is a difficult thing to be certain about. Take the polynomial *x*^{2} + 2*x* + 1. It appears to have a size of 4 (two multiplications and two additions). But the polynomial can be rewritten as the product of two sums, (*x* + 1)(*x* + 1), which has fewer operations — still two additions, but now only one multiplication. Generally, as a problem scales and more variables are added to its polynomial, you can keep simplifying and shrinking its size.

A few years after Valiant’s work, computer scientists found a way to make the size of a problem easier to analyze. To do so, they came up with a property known as depth, which specifies the number of times the polynomial switches, or alternates, between sums and products. The polynomial *x*^{2} + 2*x* + 1 has a depth of two, for example, because it is a sum of products (like *x ^{2}* and

To simplify polynomials, computer scientists restricted them to a form in which they had a property called constant depth, where the pattern of sums and products doesn’t change as the problem grows. This makes their size more static, since the size of a polynomial tends to decrease as its depth increases. Representations of a certain constant depth (such as depth three) are known as formulas.

By studying polynomials of constant depth, as well as graphs that represent them (called arithmetic circuits), computer scientists were able to make more progress. Gradually, they uncovered a sequence of findings that eventually culminated with the new work.

The first step toward the new result came in a 1996 paper by Nisan and Wigderson. The pair focused on a frequently occurring problem that involved the multiplication of tables of numbers called matrices. They simplified this problem in two ways. First, they represented it with formulas of a constant depth — depth three. Second, they only considered formulas with a certain simple structure, where each variable has a maximum exponent of 1 — a property that makes them “multilinear.” (Actually, they only considered formulas with a slight variation of this property, known as set-multilinear formulas.) Computer scientists already knew that certain problems could be converted to this relatively simple set-multilinear structure, at the cost of a sub-exponential increase in their size — a rate of growth comparable to exponential growth.

Nisan and Wigderson then showed that the matrix multiplication problem took exponential time to solve as the matrices scaled up. In other words, they proved that an important problem was hard, a notable victory in the broader enterprise of proving hardness. However, their result was limited because it only applied to formulas with a simplistic, set-multilinear structure.

“If you were working outside of algebraic complexity, that might not have meant very much for you,” said Beame.

But over the next 20 years, computer scientists came to better understand the properties of depth and structure, building on what we’ve already seen: Increasing the depth of a polynomial tends to cause a decrease in its size, and giving it a simple structure increases it. Depth and structure therefore play a kind of tug of war with hardness.

Over time, computer scientists made the balancing act between these two properties precise. They showed that adding two levels of depth to a depth-three, set-multilinear polynomial balanced out the gain in size from its set-multilinear structure. They could then expand on this to show that if a structured formula of depth five took exponential time, so would a depth-three formula of a general, unstructured nature.

The authors of the new work then showed that depth-five set-multilinear formulas for the matrix multiplication problem do indeed grow at a rate comparable to exponential. By the earlier relation, that means that general depth-three formulas also take exponential time. They then showed that a similar balancing act held for all depths — not just three and five. With that relationship in hand, they proved for the same problem that the size of a general formula of any depth grows at exponential rates as the problem scales.

In so doing, they proved that matrix multiplication is hard whenever it is represented by a formula of a constant depth, regardless of what that depth may be. Although the depth three formulas had been intensively studied before, we still didn’t know if they were hard — and we knew nothing about the hardness (or easiness) of formulas of greater depths. “It’s a humongous breakthrough,” said Shubhangi Saraf of the University of Toronto. “It’s a coming together of a lot of beautiful results from the last 30 years.”

The result provides the first general understanding of when an arithmetic problem is hard — at least, when it is restricted to being represented by formulas of constant depth. The specific problem of matrix multiplication the researchers focused on was already known to be a VP problem. And since VP problems are known to be relatively easy when not restricted to a constant depth, the result isolates the constant-depth restriction as the source of hardness.

“The model is so restricted that even things that should be easy in an unrestricted world become hard,” said Shpilka.

The ultimate question in the field of algebraic complexity is whether VNP problems are hard compared to problems from VP. The new result does not speak to this directly, since it only shows that constant-depth formulas are hard. Nevertheless, researchers are working to build on the result in ways that might allow them to reach an answer.

“That might still be a long shot. It likely is,” said Saraf. “But it is still a big milestone on the way to [showing] VP is not equal to VNP.”

And for the greater P versus NP question, we can now be a little more optimistic about the prospects of one day finding an answer. After all, in order to solve the problems we hope to solve, we first need to know which ones are hopeless.

*Correction: **May 12, 2022*

*An earlier version of the “What Is Size?” figure had a formula with the wrong coefficients. It has been replaced.*

In 2017, Roger Guimerà and Marta Sales-Pardo discovered a cause of cell division, the process driving the growth of living beings. But they couldn’t immediately reveal how they learned the answer. The researchers hadn’t spotted the crucial pattern in their data themselves. Rather, an unpublished invention of theirs — a digital assistant they called the “machine scientist” — had handed it to them. When writing up the result, Guimerà recalls thinking, “We can’t just say we fed it to an algorithm and this is the answer. No reviewer is going to accept that.”

The duo, who are partners in life as well as research, had teamed up with the biophysicist Xavier Trepat of the Institute for Bioengineering of Catalonia, a former classmate, to identify which factors might trigger cell division. Many biologists believed that division ensues when a cell simply exceeds a certain size, but Trepat suspected there was more to the story. His group specialized in deciphering the nanoscale imprints that herds of cells leave on a soft surface as they jostle for position. Trepat’s team had amassed an exhaustive data set chronicling shapes, forces, and a dozen other cellular characteristics. But testing all the ways these attributes might influence cell division would have taken a lifetime.

Instead, they collaborated with Guimerà and Sales-Pardo to feed the data to the machine scientist. Within minutes it returned a concise equation that predicted when a cell would divide 10 times more accurately than an equation that used only a cell’s size or any other single characteristic. What matters, according to the machine scientist, is the size multiplied by how hard a cell is getting squeezed by its neighbors — a quantity that has units of energy.

“It was able to pick up something that we were not,” said Trepat, who, along with Guimerà, is a member of ICREA, the Catalan Institution for Research and Advanced Studies.

Because the researchers hadn’t yet published anything about the machine scientist, they did a second analysis to cover its tracks. They manually tested hundreds of pairs of variables, “irrespective of … their physical or biological meaning,” as they would later write. By design, this recovered the machine scientist’s answer, which they reported in 2018 in *Nature Cell Biology*.

Four years later, this awkward situation is quickly becoming an accepted method of scientific discovery. Sales-Pardo and Guimerà are among a handful of researchers developing the latest generation of tools capable of a process known as symbolic regression.

Symbolic regression algorithms are distinct from deep neural networks, the famous artificial intelligence algorithms that may take in thousands of pixels, let them percolate through a labyrinth of millions of nodes, and output the word “dog” through opaque mechanisms. Symbolic regression similarly identifies relationships in complicated data sets, but it reports the findings in a format human researchers can understand: a short equation. These algorithms resemble supercharged versions of Excel’s curve-fitting function, except they look not just for lines or parabolas to fit a set of data points, but billions of formulas of all sorts. In this way, the machine scientist could give the humans insight into why cells divide, whereas a neural network could only predict when they do.

Researchers have tinkered with such machine scientists for decades, carefully coaxing them into rediscovering textbook laws of nature from crisp data sets arranged to make the patterns pop out. But in recent years the algorithms have grown mature enough to ferret out undiscovered relationships in real data — from how turbulence affects the atmosphere to how dark matter clusters. “No doubt about it,” said Hod Lipson, a roboticist at Columbia University who jump-started the study of symbolic regression 13 years ago. “The whole field is moving forward.”

Occasionally physicists arrive at grand truths through pure reasoning, as when Albert Einstein intuited the pliability of space and time by imagining a light beam from another light beam’s perspective. More often, though, theories are born from marathon data-crunching sessions. After the 16th-century astronomer Tycho Brahe passed away, Johannes Kepler got his hands on the celestial observations in Brahe’s notebooks. It took Kepler four years to determine that Mars traces an ellipse through the sky rather than the dozens of other egglike shapes he considered. He followed up this “first law” with two more relationships uncovered through brute-force calculations. These regularities would later point Isaac Newton toward his law of universal gravitation.

The goal of symbolic regression is to speed up such Keplerian trial and error, scanning the countless ways of linking variables with basic mathematical operations to find the equation that most accurately predicts a system’s behavior.

The first program to make significant headway at this, called BACON, was developed in the late 1970s by Patrick Langley, a cognitive scientist and AI researcher then at Carnegie Mellon University. BACON would take in, say, a column of orbital periods and a column of orbital distances for different planets. It would then systematically combine the data in different ways: period divided by distance, period squared times distance, etc. It might stop if it found a constant value, for instance if period squared over distance cubed always gave the same number, which is Kepler’s third law. A constant implied that it had identified two proportional quantities — in this case, period squared and distance cubed. In other words, it stopped when it found an equation.

Despite rediscovering Kepler’s third law and other textbook classics, BACON remained something of a curiosity in an era of limited computing power. Researchers still had to analyze most data sets by hand, or eventually with Excel-like software that found the best fit for a simple data set when given a specific class of equation. The notion that an algorithm could find the correct model for describing any data set lay dormant until 2009, when Lipson and Michael Schmidt, roboticists then at Cornell University, developed an algorithm called Eureqa.

Their main goal had been to build a machine that could boil down expansive data sets with column after column of variables to an equation involving the few variables that actually matter. “The equation might end up having four variables, but you don’t know in advance which ones,” Lipson said. “You throw at it everything and the kitchen sink. Maybe the weather is important. Maybe the number of dentists per square mile is important.”

One persistent hurdle to wrangling numerous variables has been finding an efficient way to guess new equations over and over. Researchers say you also need the flexibility to try out (and recover from) potential dead ends. When the algorithm can jump from a line to a parabola, or add a sinusoidal ripple, its ability to hit as many data points as possible might get worse before it gets better. To overcome this and other challenges, in 1992 the computer scientist John Koza proposed using “genetic algorithms,” which introduce random “mutations” into equations and test the mutant equations against the data. Over many trials, initially useless features either evolve potent functionality or wither away.

Lipson and Schmidt took the technique to the next level, ratcheting up the Darwinian pressure by building head-to-head competition into Eureqa. On one side, they bred equations. On the other, they randomized which data points to test the equations on — with the “fittest” points being those which most challenged the equations. “In order to get an arms race, you have to set up two evolving things, not just one,” Lipson said.

The Eureqa algorithm could crunch data sets involving more than a dozen variables. It could successfully recover advanced equations, like those describing the motion of one pendulum hanging from another.

Meanwhile, other researchers were finding tricks for training deep neural networks. By 2011, these were becoming wildly successful at learning to tell dogs from cats and performing countless other complex tasks. But a trained neural network consists of millions of numerically valued “neurons,” which don’t say anything about which features they’ve learned to recognize. For its part, Eureqa could communicate its findings in human-speak: mathematical operations of physical variables.

When Sales-Pardo played with Eureqa for the first time, she was amazed. “I thought it was impossible,” she said. “This is magic. How could these people do it?” She and Guimerà soon began to use Eureqa to build models for their own research on networks, but they felt simultaneously impressed with its power and frustrated with its inconsistency. The algorithm would evolve predictive equations, but then it might overshoot and land on an equation that was too complicated. Or the researchers would slightly tweak their data, and Eureqa would return a completely different formula. Sales-Pardo and Guimerà set out to engineer a new machine scientist from the ground up.

The problem with genetic algorithms, as they saw it, was that they relied too much on the tastes of their creators. Developers need to instruct the algorithm to balance simplicity with accuracy. An equation can always hit more points in a data set by having additional terms. But some outlying points are simply noisy and best ignored. One might define simplicity as the length of the equation, say, and accuracy as how close the curve gets to each point in the data set, but those are just two definitions from a smorgasbord of options.

Sales-Pardo and Guimerà, along with collaborators, drew on expertise in physics and statistics to recast the evolutionary process in terms of a probability framework known as Bayesian theory. They started by downloading all the equations in Wikipedia. They then statistically analyzed those equations to see what types are most common. This allowed them to ensure that the algorithm’s initial guesses would be straightforward — making it more likely to try out a plus sign than a hyperbolic cosine, for instance. The algorithm then generated variations of the equations using a random sampling method that is mathematically proven to explore every nook and cranny in the mathematical landscape.

At each step, the algorithm evaluated candidate equations in terms of how well they could compress a data set. A random smattering of points, for example, can’t be compressed at all; you need to know the position of every dot. But if 1,000 dots fall along a straight line, they can be compressed into just two numbers (the line’s slope and height). The degree of compression, the couple found, gave a unique and unassailable way to compare candidate equations. “You can prove that the correct model is the one that compresses the data the most,” Guimerà said. “There is no arbitrariness here.”

After years of development — and covert use of their algorithm to figure out what triggers cell division — they and their colleagues described their “Bayesian machine scientist” in *Science Advances* in 2020.

Since then, the researchers have employed the Bayesian machine scientist to improve on the state-of-the-art equation for predicting a country’s energy consumption, while another group has used it to help model percolation through a network. But developers expect that these kinds of algorithms will play an outsize role in biological research like Trepat’s, where scientists are increasingly drowning in data.

Machine scientists are also helping physicists understand systems that span many scales. Physicists typically use one set of equations for atoms and a completely different set for billiard balls, but this piecemeal approach doesn’t work for researchers in a discipline like climate science, where small-scale currents around Manhattan feed into the Atlantic Ocean’s gulf stream.

One such researcher is Laure Zanna of New York University. In her work modeling oceanic turbulence, she often finds herself caught between two extremes: Supercomputers can simulate either city-size eddies or intercontinental currents, but not both scales at once. Her job is to help the computers generate a global picture that includes the effects of smaller whirlpools without simulating them directly. Initially, she turned to deep neural networks to extract the overall effect of high-resolution simulations and update coarser simulations accordingly. “They were amazing,” she said. “But I’m a climate physicist” — meaning she wants to understand how the climate works based on a handful of physical principles like pressure and temperature — “so it’s very hard to buy in and be happy with thousands of parameters.”

Then she came across a machine scientist algorithm devised by Steven Brunton, Joshua Proctor and Nathan Kutz, applied mathematicians at the University of Washington. Their algorithm takes an approach known as sparse regression, which is similar in spirit to symbolic regression. Instead of setting up a battle royale among mutating equations, it starts with a library of perhaps a thousand functions like *x*^{2}, *x*/(*x* − 1) and sin(*x*). The algorithm searches the library for a combination of terms that gives the most accurate predictions, deletes the least useful terms, and continues until it’s down to just a handful of terms. The lightning-fast procedure can handle more data than symbolic regression algorithms, at the cost of having less room to explore, since the final equation must be built from library terms.

Zanna re-created the sparse regression algorithm from scratch to get a feel for how it worked, and then applied a modified version to ocean models. When she fed in high-resolution movies and asked the algorithm to look for accurate zoomed-out sketches, it returned a succinct equation involving vorticity and how fluids stretch and shear. When she fed this into her model of large-scale fluid flow, she saw the flow change as a function of energy much more realistically than before.

“The algorithm picked up on additional terms,” Zanna said, producing a “beautiful” equation that “really represents some of the key properties of ocean currents, which are stretching, shearing and [rotating].”

Other groups are giving machine scientists a boost by melding their strengths with those of deep neural networks.

Miles Cranmer, an astrophysics graduate student at Princeton University, has developed an open-source symbolic regression algorithm similar to Eureqa called PySR. It sets up different populations of equations on digital “islands” and lets the equations that best fit the data periodically migrate and compete with the residents of other islands. Cranmer worked with computer scientists at DeepMind and NYU and astrophysicists at the Flatiron Institute to come up with a hybrid scheme where they first train a neural network to accomplish a task, then ask PySR to find an equation describing what certain parts of the neural network have learned to do.

As an early proof of concept, the group applied the procedure to a dark matter simulation and generated a formula giving the density at the center of a dark matter cloud based on the properties of neighboring clouds. The equation fit the data better than the existing human-designed equation.

The roboticist Hod Lipson helped jump-start the study of symbolic regression.

In February, they fed their system 30 years’ worth of real positions of the solar system’s planets and moons in the sky. The algorithm skipped Kepler’s laws altogether, directly inferring Newton’s law of gravitation and the masses of the planets and moons to boot. Other groups have recently used PySR to discover equations describing features of particle collisions, an approximation of the volume of a knot, and the way clouds of dark matter sculpt the galaxies at their centers.

Of the growing band of machine scientists (another notable example is “AI Feynman,” created by Max Tegmark and Silviu-Marian Udrescu, physicists at the Massachusetts Institute of Technology), human researchers say the more the merrier. “We really need all these techniques,” Kutz said. “There’s not a single one that’s a magic bullet.”

Kutz believes machine scientists are bringing the field to the cusp of what he calls “GoPro physics,” where researchers will simply point a camera at an event and get back an equation capturing the essence of what’s going on. (Current algorithms still need humans to feed them a laundry list of potentially relevant variables like positions and angles.)

That’s what Lipson has been working on lately. In a December preprint, he and his collaborators described a procedure in which they first trained a deep neural network to take in a few frames of a video and predict the next few frames. The team then reduced how many variables the neural network was allowed to use until its predictions started to fail.

The algorithm was able to figure out how many variables were needed to model both simple systems like a pendulum and complicated setups like the flickering of a campfire — tongues of flames with no obvious variables to track.

“We don’t have names for them,” Lipson said. “They’re like the flaminess of the flame.”

Machine scientists are not about to supplant deep neural networks, which shine in systems that are chaotic or extremely complicated. No one expects to find an equation for catness and dogness.

Yet when it comes to orbiting planets, sloshing fluids and dividing cells, concise equations drawing on a handful of operations are bafflingly accurate. It’s a fact that the Nobel laureate Eugene Wigner called “a wonderful gift we neither understand nor deserve” in his 1960 essay “The Unreasonable Effectiveness of Mathematics in the Natural Sciences.” As Cranmer put it, “If you look at any cheat sheet of equations for a physics exam, they are all extremely simple algebraic expressions, but they perform extremely well.”

Cranmer and colleagues speculate that elementary operations are such overachievers because they represent basic geometric actions in space, making them a natural language for describing reality. Addition moves an object down a number line. And multiplication turns a flat area into a 3D volume. For that reason, they suspect, when we’re guessing equations, betting on simplicity makes sense.

The universe’s underlying simplicity can’t guarantee success, though.

Guimerà and Sales-Pardo originally built their mathematically rigorous algorithm because Eureqa would sometimes find wildly different equations for similar inputs. To their dismay, however, they found that even their Bayesian machine scientist sometimes returned multiple equally good models for a given data set.

The reason, the pair recently showed, is baked into the data itself. Using their machine scientist, they explored various data sets and found that they fell into two categories: clean and noisy. In cleaner data, the machine scientist could always find the equation that generated the data. But above a certain noise threshold, it never could. In other words, noisy data could match any number of equations equally well (or badly). And because the researchers have proved probabilistically that their algorithm always finds the best equation, they know that where it fails, no other scientist — be it human or machine — can succeed.

“We’ve discovered that that is a fundamental limitation,” Guimerà said. “For that, we needed the machine scientist.”

*Editor’s note: The Flatiron Institute is funded by the Simons Foundation, which also funds this **editorially independent publication**.*

*Correction: **May 10, 2022*

*A previous version of this article omitted the names of two coauthors of a sparse regression algorithm developed at the University of Washington.*

*Correction: **May 11, 2022*

*A word was added to the article to clarify that John Koza proposed using genetic algorithms to generate new equations, rather than inventing genetic algorithms himself.*

For scientists studying de-extinction — the ambitious effort to resurrect extinct species — a paper that appeared in *Current Biology* in March was a sobering reality check. Thomas Gilbert, a genomics researcher and professor at the University of Copenhagen, led a team of researchers who tested the feasibility of de-extinction by sequencing the genome of the Christmas Island rat, a species that went extinct in the late 19th or early 20th century.

“Look, this is like the best-case scenario,” Gilbert said. The samples of DNA from the extinct species were relatively new and well preserved, and the extinct rat was very closely related to the standard brown Norway rat, for which there is abundant DNA reference data. This was a far cry from trying to figure out the DNA of some jungle cat from the Pleistocene, let alone a dinosaur. Reconstructing the extinct rat’s genome should have been relatively simple.

Yet despite their best efforts, the scientists were unable to recover nearly 5% of the Christmas Island rat’s genome. Many of the missing genes were related to immunity and olfaction, two highly important functions for the animal. “It’s not just the irrelevant stuff that you’re not going to get back,” Gilbert said. “And so what you’ll end up with is nothing like what went extinct.”

Though the results from Gilbert’s group are new, in many ways they underscore something that many scientists have understood for a long time. “The biggest misconception about de-extinction is that it’s possible,” said Beth Shapiro, a professor of ecology and evolutionary biology at the University of California, Santa Cruz.

Ben Novak, a lead scientist for Revive & Restore, one of the front-running non-profit organizations in the de-extinction effort, readily acknowledges this. “You can never bring something back that is extinct,” he said. But for Novak and most other de-extinction researchers, creating a proxy instead of the real thing is not a problem — it’s the goal.

Most de-extinction researchers aren’t looking to resurrect a charismatic ancient beast just for the sake of putting it into the nearest zoo for viewer pleasure. Rather, they are aiming to create proxies for educational or conservation purposes, such as to fill the void left by their extinct counterparts in ecosystems or to boost the numbers of modern-day endangered species.

The challenges facing de-extinction start with DNA, the genomic molecule that makes the hope of de-extinction possible. In the *Jurassic Park* novel and films, dinosaur DNA from more than 65 million years ago could be extracted from a mosquito preserved in amber. But in real life, DNA is too delicate to survive that long: It has a half-life of only around 521 years.

Even in well-preserved tissues left over from recently extinct species, the contained DNA is often fragmented. “And because those fragments are tiny, it’s impossible to actually reassemble them [digitally] like puzzle pieces into the entire picture that they used to be,” Novak said.

In particular, it’s not always clear what the order of the genes should be on the reconstructed chromosomes. Those details matter because studies of living species have shown that slight alterations in gene order can have significant effects on behavior and other traits. De-extinction researchers typically use the genome of a closely related living species as a guide, but that approach has limitations.

“Even if we could get 100% of its genetic code, we would still be creating an organism that has the same gene order and chromosome count as its living relative,” Novak said. And as Gilbert’s new work makes clear, getting close to all of the genetic code may often be impossible.

(Left to right, top to bottom) Proceedings of the Zoological Society of London; Iconographia Zoologica; Smithsonian Libraries; Proceedings of the Zoological Society of London 1887

Gilbert’s work speaks to the difficulties of de-extinction through genetic engineering, a popular approach favored by researchers such as George Church, a professor of genetics at Harvard University who is leading a project aimed at bringing back the woolly mammoth of prehistory. With a large recent funding boost from the startup Colossal, Church is hopeful that they’ll make headway in the next decade or so by genetically editing mammoth genes into Asian elephants, a closely related living pachyderm.

But the de-extinction field encompasses more than genetic engineering. Using an approach called selective back-breeding, some groups are restoring ancient traits from extinct species by selectively breeding individuals that still carry the genes for them. For instance, the Tauros Program aims to back-breed modern cattle to make them more like their pre-domesticated ancestors, the aurochs, and the Quagga Project in South Africa is selecting for zebras that still have genes from the quagga, a subspecies hunted to extinction in the 19th century.

Still, even if these genetic engineering and selective breeding efforts succeed, they can only create a kind of hybrid rather than a purely resurrected species.

The closest you can get to an exact genetic replica of an extinct species is a clone created from a living or preserved cell from that species. Scientists don’t have useable cells from woolly mammoths, dodos, the Tasmanian tiger or most other species that are hyped up in the realm of de-extinction, but they do from some more recently extinct species. In 2003, researchers used cloning to bring back the bucardo, a species of wild goat, using a modern goat as a surrogate parent and egg donor. The baby bucardo, the only extinct species to ever be cloned, died after only seven minutes because of a lung malformation.

But even if cloning is someday more successful, according to the International Union for Conservation of Nature (IUCN), it could also lead to proxies “that differ in unknown and unpredictable ways from the extinct form.” For example, researchers may not know everything about potential epigenomic differences affecting DNA activity or the microbiome needed to support the species’ health. They also may not be able to recreate the exact learning environment in which the original species was reared, which could cause the behavior of the de-extinct species to deviate from that of the original.

Despite these differences, Novak said, “from an evolutionary standpoint, a clone is an authentic, or ‘true’, de-extinct organism.” In fact, although cloning is officially included in the IUCN guidelines and some other researchers would disagree, Novak doesn’t think cloning should even be considered de-extinction but rather a “true recovery.”

The problems that riddle the field don’t dissuade de-extinction researchers. For them, a good proxy or functional equivalent of a lost species may be good enough. “I don’t actually know anyone who said we have to get a perfect copy of anything,” said Church. The practical goal of the woolly mammoth project he’s leading is to help endangered Asian elephants adapt to the frigid environments of the Arctic tundra.

“Make sure people don’t think they’re going to get a mammoth, because they’re not,” said Gilbert, who is not involved in that research. They will instead get a “hairy elephant” that can live in the cold.

Mammoth-elephant hybrids could be relocated to places such as Pleistocene Park, a large area of tundra in Russia where scientists are trying to restore the much more biodiverse and climate-friendly grasslands ecosystem it once was, when large grazers including mammoths populated the area. By trampling the soil and allowing cold air to seep in, the mammoth hybrids could in theory slow the melting of permafrost and the release of greenhouse gases that are warming the globe. The team also hopes that in the process, they can rescue the endangered elephant species by placing them in a large open area free from human conflict.

Similarly, Novak is working to resurrect the extinct passenger pigeon and the heath hen as genetically engineered hybrids of modern species, in the hope that they might help to restore their respective ailing ecosystems and motivate restoration efforts. The San Diego Zoo is trying to save the northern white rhino, a species that is functionally extinct because two females are the only ones left in the world. The zoo’s scientists are developing stem cells that could differentiate into northern white rhino sperm and eggs, and any resulting embryos might be carried to term by surrogate southern white rhinos.

“I’m excited about [de-extinction] and keep talking about it and keep doing interviews about it, not because I think we really are going to get a mammoth — I don’t think we will,” Shapiro said. “But because the path to getting us there is so important for conservation of living species.”

And if resurrected species are introduced into the wild, some of de-extinction’s successes may go even further in the long run. “If we get our proxies close enough,” Novak said, “evolution itself is probably going to converge them even closer to the original form than we can actually succeed in doing.” That is, if the forces that felled the original species don’t render their replacements extinct too.

After a lengthy experiment with tantalizing implications for origin-of-life studies, a research group in Japan has reported creating a test tube world of molecules that spontaneously evolved both complexity and, surprisingly, cooperation. Over hundreds of hours of replication, a single type of RNA evolved into five different molecular “species” or lineages of hosts and parasites that coexisted in harmony and cooperated to survive, like the beginning of a “molecular version of an ecosystem,” said Ryo Mizuuchi, the lead author of the study and a project assistant professor at the University of Tokyo.

Their experiment, which confirmed previous theoretical findings, showed that molecules with the means to replicate could spontaneously develop complexity through Darwinian evolution, “a critical step for the emergence of life,” the researchers wrote.

“We can provide the direct evidence; we can see what can actually happen” when a replicating molecule complexifies in a test tube, Mizuuchi said.

This was the first and probably most important step toward evolving a complex network of replicators in the lab, said Sijbren Otto, a professor of systems chemistry at the University of Groningen in the Netherlands who was not involved in the study. “With what is shown here, the path ahead becomes a lot clearer, and one becomes a lot more optimistic that this can actually work out.”

Joana Xavier, a computational biologist at University College London, hailed the work by Mizuuchi and his colleagues as a “great proof of concept” for how a minimal system can complexify. It’s “a very significant advance,” she said.

The roots of the new experiments reach back to the 1960s, when the molecular biologist Sol Spiegelman created what he called “the little monster” in his laboratory. Despite the overtones of *Frankenstein* in that label, his little monster was not green, square-browed, growling or even alive. It was a synthetic molecule that filled test tubes with copies of itself.

In the 1960s, the molecular biologist Sol Spiegelman performed the first demonstration of Darwinian evolution at the molecular level, using an evolving strand of viral RNA that he called “the little monster.”

Spiegelman’s monster was a mutating strand of RNA based on a viral genome. The biologist had discovered he could indefinitely replicate it simply by heating and mixing it in the presence of nucleotide building blocks and a polymerizing enzyme called a replicase. He soon realized, however, that his molecules were getting smaller over time: The copies that shed unnecessary genes replicated more quickly, which improved their chances of being collected in samples and transferred to new test tubes for further replication. Just like living species, his molecules had started mutating and evolving under the pressure of natural selection to better survive inside their glass world.

These studies were the world’s first experimental demonstration of Darwinian evolution at the molecular level — “evolution by natural selection, survival of the fittest,” said Eugene Koonin, a National Institutes of Health distinguished investigator at the National Center for Biotechnology Information. “In those conditions, the fittest simply meant the fastest replicating.”

Spiegelman’s work inspired decades of further study, much of which was foundational to research on life’s origins and provided fuel for the RNA world hypothesis that life sprang from self-replicating RNA molecules. But those studies left unanswered a crucial question: Could a single molecular replicator evolve into a complex network of multiple replicators?

About a decade ago, when Norikazu Ichihashi was an associate professor of bioinformatic engineering at Osaka University in Japan, he set out to learn the answer by tweaking Spiegelman’s test tube world. “We tried to develop our system to be a little more lifelike,” Ichihashi said.

Ichihashi and his team developed an RNA molecule that encoded a replicase, which can make copies of RNA. But for the molecule to translate its own code, the scientists needed to add something more: ribosomes and other gene translation machinery that they borrowed from the common gut bacteria *Escherichia coli*. They embedded the machinery inside droplets and added them to a mixture of RNAs and raw materials.

Then came years of tedious mixing and waiting.

Their long-term experiment involved incubating their replication system at 37 degrees Celsius (the temperature of a human body or a hot summer’s day), adding new droplets with fresh translation systems, and stirring the mixture to induce replication. Every few days or so they analyzed RNA concentrations in the test tubes, and every week or so they froze samples from the latest mixture. Every half year or so, they sequenced large batches of the collected samples to see if the RNA had acquired new mutations and evolved into a new lineage.

After 215 hours and 43 rounds of replication, the researchers began to see interesting results, which they reported in the *Proceedings of the National Academy of Sciences* in 2016. The original RNA had been replaced by lineages of two other RNAs. One, which the researchers described as a “host,” could use its own replicase to copy itself, like the original molecules. The other lineage, a “parasite,” needed to borrow the gene expression machinery of the hosts.

When Ichihashi and his colleagues extended the experiment to 120 rounds of replication over 600 hours, they found that the host lineage had split into two separate host lineages, and one of the hosts had evolved two distinct parasites. But it wasn’t just the number of lineages that had increased; so had the complexity of their interactions. The hosts had acquired mutations that interfered with the ability of the parasites to hijack their replicative resources — but the parasites had also developed mutations that served as a defense against those obstacles. The hosts and parasites seemed to be coevolving.

The populations of parasites and hosts greatly fluctuated as they competed for the realm in “evolutionary arms races,” the scientists reported in 2020 in *eLife*. Each RNA lineage transiently rose to dominance, then lost its place to another one. “If one lineage dominated, then another lineage decreased,” said Ichihashi, who is now a professor at the University of Tokyo.

To Ryo Mizuuchi, a University of Tokyo researcher and lead author of the new study, the work demonstrates how parasites and hosts can push each other to evolve. “Without parasites, this level of diversification is probably not possible,” he said.

But the researchers kept the experiments going, and by round 130, another host had evolved. By round 160, one of the parasites had disappeared; some rounds later, another parasite had appeared. By round 190, the researchers had hit on a new surprise: The huge dynamic swings in the population of each lineage had started giving way to smaller waves. This stabilization suggested that the lineages were no longer competing to replicate. Instead, they had started to interact as a network and cooperate in a state of quasi-stable coexistence.

Mizuuchi and Ichihashi, who did the experiments with Taro Furubayashi (who was a doctoral student in Ichihashi’s lab at the time and is now a research fellow at the University of Tokyo), were floored by the findings, which they reported in *Nature Communications* in March. They’re just “mere molecules,” Mizuuchi said. “It’s pretty unexpected.”

Koonin agrees that their findings are striking. Their “experimental setup is more elaborate, it’s more realistic, and the results are more complex and rich, but [it’s] fully compatible” with Spiegelman’s, he said. They watched a single type of molecule replicate and gather mutations under natural selection — but then went further by letting the divergent molecules evolve into a community under one another’s influence, just as communities of living cells, animals or people would. In the process, the researchers explored some of the rules governing what it takes for such complex communities to become stable and enduring.

Some of these results confirmed the predictions of earlier experimental studies of how complexity can arise in viruses, bacteria and eukaryotes, as well as some theoretical work. A study from Koonin’s lab, for instance, also suggested that parasites were inevitable in the emergence of complexity.

“Without parasites, this level of diversification is probably not possible,” Mizuuchi said. Evolutionary pressures that parasites and their hosts place on each other lead both sides to split into new lineages.

A more surprising fundamental principle that emerged was the critical role of cooperation. The five lineages belonged to different small networks of cooperation, and some were more cooperative than others. By round 228, for example, one of the three hosts had evolved into a “super cooperator” that could replicate itself and all the other lineages; the other two hosts could each replicate only themselves and one of the parasites.

Scientists have focused on studies of competition in evolution for so long that the role of cooperation “has been a bit overlooked,” Xavier said. “I think cooperation is also going to start having a major role, especially in origins, because there are so many things that have to come together in the right way.”

The cooperation among RNAs was focused entirely on replication in the system that Ichihashi, Mizuuchi and their colleagues observed. But the researchers hope that it will be possible to coerce the RNAs to evolve a completely different function too, such as a metabolic one, by adjusting the natural selection criteria inside the test tubes.

“Scientists like to entertain each other, and the best entertainment is a surprise,” said David Deamer, a research professor of biomolecular engineering at the University of California, Santa Cruz. He considers it a good paper but noted that what happened in the laboratory may not translate to what happened at the dawn of life.

Indeed, the scenario in Ichihashi’s lab could not reflect what played out at the start of life since the experiments depended on translation machinery from *E. coli*. “The quintessential problem with the origin of life is: How did protein synthesis itself begin?” said Charlie Carter, a professor of biochemistry and biophysics at the University of North Carolina School of Medicine.

But Koonin thinks that if researchers found a way to evolve complexity using truly self-replicating systems of molecules, they would see something very much resembling the networks depicted in the paper. “They, in the very least, beautifully illustrate the types of processes that likely occurred in the origin of life,” Koonin said.

To Otto, the study suggests that once you’ve solved the problem of accurate replication with molecules at this level of complexity, they will complexify further: The experiment “doesn’t show you how you got there, but once you’re there at this stage, it does chart the way ahead,” he said.

Carrying on with their work, Ichihashi and his colleagues wanted to see if they could re-create the same sustainable network in a separate experiment, so they extracted samples of the five lineages. This time, however, they found that while four of the lineages continued to replicate and survive for at least 22 more rounds, the fifth disappeared. “I don’t know why,” Ichihashi said. “It’s a very strange point.”

One possibility is that the system was even more complex than the researchers thought, and when they isolated the five lineages, they missed a sixth one critical for the survival of the lineage that disappeared. With theoretical models, Ichihashi’s group confirmed that the four remaining lineages could sustainably and interdependently replicate, and that knocking out any one of the four would lead to the extinction of at least one of the others. Their simulation also pointed to the counterintuitive discovery that knocking out one of the parasites would lead to the extinction of its host.

Meanwhile, the researchers have continued their main test tube experiments and are waiting to see whether their network will complexify further. They have also begun similar experiments that use DNA instead of RNA.

“We observed just the beginning” of how these communities of molecular replicators can evolve, Ichihashi said. “I think that they have a different destiny in the future — we cannot predict what happens.”

*Correction: May 5, 2022
*

Math has a certain logic to it. If you use it to accurately describe a situation, sometimes you can predict the inevitable — for instance, the moment an eclipse will take place — centuries in advance. To those unfamiliar with the math behind the prediction, this outcome might seem like magic. Indeed, the science fiction author Arthur C. Clarke famously wrote, “Any sufficiently advanced technology is indistinguishable from magic.”

In today’s Insights puzzle we’ll explore four examples of mathematical magic that can seem, at first glance, like mind reading. Just like stage magic, these examples can leave you wondering, “How did they know that?”

Many of us have experienced this as children. We are asked by a friend to think of a certain number without revealing it. We are then asked to do a series of simple arithmetical operations on it. Finally, our friend astonishes us by telling us our secret number. So how is this done?

Here’s a simple example that might astonish a child in your life and even inspire a fascination with math. It was used for this purpose by Lewis Carroll, the author of *Alice’s Adventures in Wonderland*. Carroll was an Oxford mathematician and avid puzzler, whose math puzzles have entertained many mathematicians, including the famous Terence Tao.

This magic trick is designed for a child who can do multiplication reliably (get an assistant to help them if not). Ask your own little Alice (or Bob) to think of a three-digit number without telling you what it is. Then tell them that you will reveal the number by producing two copies of it side by side! First ask the child to multiply their number by 7. Then ask them to multiply the answer by 11. Finally — and at this point you can add feigned concentration and appropriate magical phrases — ask them to multiply the second answer by 13.

If the child has done this right, you might find a smile start to light up their face. The question for you, or for an older child, is — why does this work?

If you’re interested in more Lewis Carroll math puzzles, you can find them here. Now let’s get into something a little harder.

There are two unknown numbers between 2 and 9. The two numbers can include 2 or 9 and they can both be the same number. S and P are two mathematicians with perfect logic. S is given only the sum of the two numbers, and P is given only the product. Both S and P know everything we’ve just specified. Here is their subsequent conversation.

*S: I cannot deduce what the two numbers are.*

*P: Neither can I.*

*S: Aha! Now I know what the two numbers are!*

*P: So do I!*

This seems like magical mind reading at first glance — where could they be getting the new information to solve the problem? Can you figure out the two numbers (there are two possible answers). Can you explain how S and P did this?

Hint: Tabulate all the information that S and P have at every point, figuring out what they can infer. Remember to consider all possible cases. This doesn’t involve any difficult calculations, but it’s easy to get confused about what each of them knows.

Now for a harder version that has many more possibilities.

Again, there are two unknown numbers that could be equal, but now they are between 2 and 70. Again S is given only the sum of the two numbers, and P is given only the product. Here’s how their conversation goes this time.

*P: I cannot deduce what the two numbers are.*

*S: I could have told you that, even though I can’t deduce the numbers either.*

*P: Aha! Now I know what the two numbers are!*

*S: So do I!*

Again, find the two numbers and explain how P and S did this.

In puzzle 2, you had 15 unique sums with 64 products. In this one you have 137 unique sums with 4,761 products. Tabulation on paper was good enough for puzzle 2, but what now? You can, of course, use a spreadsheet or write a computer algorithm as some readers will no doubt try. But if you pay careful attention to the conversation, especially S’s first remark, you can significantly limit the possibilities by applying some elementary number theory, and yes, you can still solve the problem on paper. So our second question is: Using number theory principles, what’s the smallest number of cases (sums and products) that you have to examine manually after judicious exclusion?

Our next problem shows how to read minds in a card trick. But this trick is strictly mathematical and has some interesting mathematics underlying it.

A “mathemagician” prepares a deck of 32 cards by throwing away the 2s through 6s. She arranges the remaining cards in a certain order and then places the deck, face down, on a table. Five people are randomly selected to go to the table. Each of them cuts the deck, one after another. Then the first person takes the top card and passes the deck to the second person, who takes the current top card, and so on, in order. Once each one has a card, the last person puts the deck back on the table face down, and they all return to their seats.

Now the performer asks the five people to beam her their card telepathically. A frown of concentration crosses her face. Finally she shakes her head in a resigned manner. “It’s getting harder these days,” she says. “The expansion of the universe is causing a redshift that’s interfering with the colors I am receiving. Will the people who have red cards please stand up?”

The second and fifth participants stand up. A relieved look crosses the mathemagician’s face. “Now it’s clear,” she says. “You have the 10 of hearts, and you have the king of diamonds.” They do, indeed. She proceeds to correctly guess the cards of the other three people as well.

How did she do it? Well, since I told you the trick is strictly mathematical, it’s clear that she memorized the card order (probably using some sort of code), and the card order somehow survived the five random cuts. Can you think of a way this might be done? Hint: Consider the sequence 00010111. It contains, cyclically, all eight possible triplets formed from zero and 1: 000, 001, 010, 101, 011, 111, 110 and 100. What happens if you “cut” it, as you would a card deck?

Was the performer’s stage banter mere theater, or was it essential to the trick?

Can you come up with a card-order code that’s easy to remember (for when you perform this trick)?

That’s it for today. I hope you were impressed with how math and logic can help you infer so much from so little information. Now, that’s magical!

Have fun unraveling how all these tricks were done.

**Correction: May 8, 2022**

*In puzzle 3, the upper limit should be 70, not 50 as originally stated, for it to have a unique solution. There is no solution for the originally stated range even though the numbers comprising the unique solution for the new range are both below 50! The reason for this is interesting and will be discussed in the solution column. The numbers of unique sums and products for the problem were also corrected.*

*Editor’s note: The reader who submits the most interesting, creative or insightful solution (as judged by the columnist) in the comments section will receive a* Quanta Magazine *T-shirt** or one of the two *Quanta* books, *Alice and Bob Meet the Wall of Fire* or *The Prime Number Conspiracy* (winner’s choice). And if you’d like to suggest a favorite puzzle for a future Insights column, submit it as a comment below, clearly marked “NEW PUZZLE SUGGESTION.” (It will not appear online, so solutions to the puzzle above should be submitted separately.)*

General relativity and quantum mechanics are the two most successful conceptual breakthroughs of modern physics, but Einstein’s description of gravity as a curvature in space-time doesn’t easily mesh with a universe made up of quantum wavefunctions. Recent work that tries to bring those theories together is revealing some mind-bending truths. In this episode, the physicist and author Sean Carroll talks with host Steven Strogatz about how space and time might be emergent properties of quantum reality, not fundamental parts of it..

Listen on Apple Podcasts, Spotify, Google Podcasts, Stitcher, TuneIn or your favorite podcasting app, or you can stream it from *Quanta*.

**Steven Strogatz** (00:03): I’m Steve Strogatz, and this is *The Joy of Why*, a podcast from *Quanta Magazine* that takes you into some of the biggest unanswered questions in science and math today. In this episode, we’re going to be discussing the mysteries of space and time, and gravity, too. What’s so mysterious about them?

Well, it turns out they get really weird when we look at them at their deepest levels, at a super subatomic scale, where the quantum nature of gravity starts to kick in and become crucial. Of course, none of us have any direct experience with space and time and gravity at this unbelievably small scale. Up here, at the scale of everyday life, space and time seem perfectly smooth and continuous. And gravity is very well described by Isaac Newton’s classic theory, a theory that’s been around for over 300 years now.

(00:53) But then, about 100 years ago, things started to get strange. Albert Einstein taught us that space and time could warp and bend like a piece of fabric. This warping of the space-time continuum is what we experience as gravity. But Einstein’s theory is mainly concerned with the largest scales of nature, the scale of stars, galaxies and the whole universe. It doesn’t really have much to say about space and time at the very smallest scales.

And that’s where the trouble really starts. Down there, nature is governed by quantum mechanics. This amazingly powerful theory has been shown to account for all the forces of nature, except gravity. When physicists try to apply quantum theory to gravity, they find that space and time become almost unrecognizable. They seem to start fluctuating wildly. It’s almost like space and time fall apart. Their smoothness breaks down completely, and that’s totally incompatible with the picture in Einstein’s theory.

(01:54) As physicists try to make sense of all of this, some of them are coming to the conclusion that space and time may not be as fundamental as we always imagined. They’re starting to seem more like byproducts of something even deeper, something unfamiliar and quantum mechanical. But what could that something be? Joining me now to discuss all this is Sean Carroll, a theoretical physicist who hosts his own podcast, *Mindscape*. Sean spent years as a research professor of physics at Caltech [California Institute of Technology], but he is now moving to Johns Hopkins as the Homewood Professor of Natural Philosophy. He’s also an external professor at the Santa Fe Institute. But no matter where he is, Sean studies deep questions about quantum mechanics, gravity, time and cosmology. He’s the author of several books, including his most recent, *Something Deeply Hidden: Quantum Worlds and the Emergence of Spacetime*. Sean, thank you so much for joining us today.

**Sean Carroll** (02:54): Thanks very much for having me, Steve.

**Strogatz** (02:56): It’s very exciting to me to be talking with the master of emergent space-time. Really mind-boggling stuff, I enjoyed your book very much. I hope you can help us make some sense of these really thorny and fascinating issues in, I’d say, at the frontiers of physics today.

Why are you guys, you physicists, worrying so much about space and time again? I thought Einstein took care of that for us a long time ago. What’s really missing?

**Carroll** (03:21): Yeah, you know, we think of relativity, the birth of relativity in the early 20th century, as a giant revolution in physics. But it was nothing compared to the quantum revolution that happened a few years later. Einstein helped the beginning of special relativity, which is the theory that says you can’t move faster than the speed of light, everything is measured relative to everything else in terms of velocities and positions and so forth. But still, there was no gravity in special relativity. That was 1905. And then 10 years later, after a lot of skull sweat and heavy lifting, Einstein came up with general relativity, where, he had been trying to put in gravity to special relativity, and he realized he needed a whole new approach, which was to let space-time be curved, to have a geometry, to be dynamical. It’s the fabric of space-time itself that responds to energy and mass, and that’s what we perceive as gravity.

(04:14) And as revolutionary as all that was, sort of replacing fundamental ideas that had come from Isaac Newton, both special relativity and general relativity were still fundamentally classical theories. You know, we sometimes prevaricate about the word “classical,” but usually what physicists mean is, the basic framework set down by Isaac Newton in which you have stuff, whether it’s particles or fields, or whatever. And that stuff is characterized by what it is, where it is, and then how it’s moving. So for a particle, that would be its position and its velocity, right? And then, from that, you can predict everything, and you can observe everything and it’s precise and it’s deterministic, and this gives us what we call the clockwork universe, right? You can predict everything. If you knew perfect information about the whole world, you would be what we call “Laplace’s demon,” and you’d be able to precisely predict the future and the past.

(05:08) But even general relativity, which says that space-time is curved, that still falls into that framework. It’s still a classical theory. And we all knew, once quantum mechanics came along, circa 1927, let’s say. It was bubbling up from 1900, and then sort of — it triumphed in 1927, at a famous conference, the fifth Solvay Conference, where Einstein and Bohr argued about what it all meant.

(05:32) But since then, we’ve accepted that quantum mechanics is a more fundamental version of how nature works. I know — you said this for all the right reasons, but it’s not that quantum mechanics happens at small scales. Quantum mechanics is the theory of how the world works. What happens at small scales is that classical mechanics fails. So you need quantum mechanics. Classical mechanics turns out to be a limit, an approximation, a little tiny baby version of quantum mechanics, but it’s not the fundamental one.

And since we discovered that, we have to take all of what we know about nature and fit it into this quantum mechanical framework. And we have been able to do that for literally everything we know about nature, except for gravity and curved space-time. We do not yet have a full, 100% reliable way of thinking about gravity from a quantum point of view.

**Strogatz** (06:24): I appreciate that correction. You’re right, I was being a little bit loose there in saying that quantum mechanics only applies at the smallest scales. I mean, there’s — on mathematical grounds, we can see how quantum mechanics becomes classical mechanics. It’s consistent with it, it’s — in fact, it implies classical mechanics, once the scale gets to be the more familiar one.

**Carroll** (06:45): Yeah, not only is that true, but it’s kind of crucially important, and I like to emphasize it even more than most people do, because we’re not born understanding quantum mechanics. We kind of have a much more intuitive grasp of classical mechanics. And we kind of tend to think of the world in classical terms. Classically, things have positions, and they have locations — positions and velocities. Quantum mechanically, that’s not true. And it’s really hard to wrap your brain around that. And so, we tend to speak in exactly the way that you said, like, classical mechanics works on large scales, quantum mechanics works on small scales, because we kind of don’t want to face the fact that quantum mechanics is everywhere in everything and we should learn to understand what’s going on.

**Strogatz** (07:28): But you say that gravity has been this kind of outlier, that it’s very hard — or at least it hasn’t been incorporated in a fully satisfactory way yet, into any kind of quantum mechanical framework. Is there a way to sum up what the nature of the difficulty is? Why is it so hard to come up with a theory that merges quantum theory and gravity?

**Carroll** (07:47): Yeah, there are kind of two sets of issues that come up. What you might call technical issues, and conceptual issues. We human beings start classically. When you’re a physics student as an undergraduate, and you’re learning quantum mechanics, what does that mean? That means that you’re taught the classical model for something, like a harmonic oscillator, or the hydrogen atom, or whatever. And then you’re given rules for quantizing that classical theory, okay? So there’s supposed to be, in some sense — you mathematicians out there in the audience will appreciate — a map from the space of classical theories to quantum theories, okay? The quantization procedure.

(08:26): This is all a complete fake. I mean, it sort of is a kludge that works sometimes, but this purported map from classical theories to quantum theories is not very well-defined. You can have the same classical theory that maps on to two different quantum theories. You can have two different classical theories mapping onto the same quantum theory. So, there’s no direct correspondence and after all, why should there be?

(08:46): But again, nevertheless, it has worked for electromagnetism, the nuclear forces and everything else. When you straightforwardly apply that quantization procedure to gravity — we have a classical theory, general relativity, we can quantize it. It just blows up. It just gives us infinite crazy answers.

(09:04) This has happened before in the history of trying to quantize classical theories. Richard Feynman and Julian Schwinger and Sin-Itiro Tomonaga famously won the Nobel Prize for showing how to get rid of the infinities in quantum electrodynamics. But the infinities you get in gravity are of a different character, they’re not get-rid-able, they’re not “renormalizable,” as we say. So, at a very fundamental mathy level, you know, the procedure that you were relying on all along just stalls and you don’t know what to do.

(09:35): But then there’s a whole set of more deep conceptual issues, not only do you not know what to do, you don’t know what you’re doing. Because, with everything else, every other theory other than gravity, it’s very clear what’s going on. You have stuff inside space-time. The stuff has a location, right? It has a point in space, it’s moving through time. Even if you have a field, it has a value at every point in space, etc.

But in gravity, you’re sort of combining a whole bunch of different possible geometries of space-time. And what that means is, you’re not really sure what time is, for one thing, and you’re not really sure where things are in space, because if you don’t know the geometry of space, it is impossible to identify a point in space uniquely throughout all the possible quantum combinations of the geometry of space-time. So, we really, at a fundamental level, have difficulty knowing what we’re talking about, when it comes to quantum gravity.

**Strogatz** (10:33): It certainly does sound very thorny, that the arena itself, like in traditional thinking, physics, as you say, there’s stuff and fields and particles and things happening, moving around from place to place, from moment to moment inside this arena of space-time. But now it’s the arena itself. Einstein already took us a little bit in that direction by making the arena a dynamical thing where space and time could warp and have, as you say, dynamics. But it’s now, it seems like it’s getting much worse.

**Carroll** (11:02): Well, it is, because remember, I alluded to the idea that, classically, for a particle, you have a very clear notion of where it is, its location, and how fast it’s moving. And you could measure those things. The whole spookiness of quantum mechanics is that to define what you mean by quantum mechanics, you have to use words like “observation” and “measurement.” That was never true in classical mechanics, you just measure whatever you want, it was perfectly trivial and straightforward. Quantum mechanics is a little bit different from that.

(11:03) And so, one of the lurking things here, in this whole discussion, you know, there’s many, many theoretical physicists who would say, yes, quantum gravity, very, very important, we should try to understand this. But we don’t understand quantum mechanics. Even though it’s been around for almost 100 years. We don’t agree on what quantum mechanics is saying, because of these weird words like measurement and observation. So, I tried to explain why quantum gravity is hard but I’m going to reveal my prejudices, because I can’t do that without explaining what I think quantum mechanics is. Or at least, referring to what I think quantum mechanics is.

**Strogatz** (11:32): So I think that segues very nicely into the next thing I was going to ask you. We’re hoping, by the end of this episode, to give people a feeling of what it means for space-time to be emergent. But what would it mean for you, or anybody studying space and time, for them to be emergent?

**Carroll** (12:05): So I don’t think that there is any such thing as a position or a velocity of a particle. I think those are things you observe, when you measure it, they’re possible observational outcomes, but they’re not what is — okay, they’re not what truly exists. And if you extend that to gravity, you’re saying that what we call the geometry of space-time, or things like location in space, they don’t exist. They are some approximation that you get at the classical level in the right circumstances. And that’s a very deep conceptual shift that people kind of lose their way in very quickly.

(12:58) It’s a tricky word. We have to think about it. Emergence is kind of like morality. Sometimes we agree on it when we see it. But other times, we don’t even agree on what the word is supposed to mean. So, the physicists, and mathematicians, and other natural scientists tend to — but not always — rely on what a philosopher would call weak emergence. And weak emergence is basically a convenience, in some sense. The idea is that you have a comprehensive theory, you have a theory that works at some deep level. Let’s say, the standard example is gas in a box, okay? You have a box full of some gaseous substance, and it’s made of atoms and molecules, right? And that’s the microscopic theory. And you say that, okay, I could — in principle, I could be Laplace’s demon, I could predict whatever I want, I know exactly what’s going on.

(13:47) But, we human beings, when we look at the gas in the box with our eyeballs, or our thermometers, or whatever, we don’t see each individual atom or molecule, and its position and its velocity, we see what we call coarse-grained features of the system. So we see its temperature, its density, its velocity, its pressure, things like that. And the happy news — which is not at all obvious or necessary, it’s kind of mysterious when it happens and when it doesn’t — but the happy news is that we can invent a predictive theory of what the gas is going to do just based on those coarse-grained macroscopic observables. We have fluid mechanics, right? We can model things without knowing what every atom is doing. That’s emergence, when you have a set of properties that are only approximate and coarse-grained, that you can observe at the macroscopic level, and yet you can predict with them. And weak emergence just means, there’s nothing new that happened along the way. You didn’t say that, oh, when you go to the larger scales and you zoom out, fundamentally new essences or dynamics are coming in. It’s just sort of the collective behavior of the microscopic stuff. That’s weak emergence.

(15:01) There’s also strong emergence where spooky new stuff does come in. And people talk about the necessity of that when they think about consciousness or something like that. I’m not a believer in strong emergence at the fundamental level. So, to me, what the emergence of space-time means is that space-time itself is like, the fluid mechanics. It’s like gas temperature and pressure and things like that. It’s just a coarse-grained, high-level way of thinking about something more fundamental, which we’re trying to put our finger on.

**Strogatz** (15:34): Wow, as you’re describing the gas in a box, I happen to be sitting in a box. I’m in a studio that is kind of box-shaped. There is a gas in here, which is the air that I’m breathing.

So anyway, yeah, very vivid to me, the example you’re talking about. And it is amazing, isn’t it? That there are laws at that collective or emergent scale that work, that don’t — you know, like thermodynamics was oblivious to statistical physics. In fact, was discovered first, and only later, the microscopic picture came out. And so, I guess you’re saying something like that would be happening now with space and time and gravity, that we have the macroscopic theory that’s Einstein’s.

**Carroll** (16:14): When I’m not spending my research time studying quantum mechanics and gravity, I’m studying emergence. I think that there’s a lot to be done here, to be sort of cleaned up and better understood, in a set of questions that spans from philosophy to physics to politics and economics, not to mention biology and the origin of life. So, I think that these are deep questions that we’ve been kind of messy and sloppy about addressing, but I don’t think that the emergence of space-time is difficult for that reason.

(16:45) So, when you talk about, is the United States emergent from its citizens? Or is Apple Computer Company emergent from something? Those are hard questions. Those are like, tricky, like “where do you draw the boundary?”, etc. But for space-time, I think it’s actually pretty straightforward. The lesson, the important take-home point for the podcast is, you don’t start with space-time and quantize it, okay? Just like when you have the gas in the box, you’re trying to get a better and better theory of the gas in the box, but you realize that it’s made of something fundamentally different. And I think that’s what I’m proposing, and other people are proposing for space-time as well, that the whole thing that used to work for electromagnetism and particles and the Higgs boson and the Standard Model, where you started with some stuff and quantized it, that’s not going to be the way it’s going to happen for gravity and space-time. You’re going to have something fundamentally different at the deep micro-level, and then you’re going to emerge into what we know of as space-time.

**Strogatz** (17:46): Shouldn’t we start talking about entanglement, at this point, maybe?

**Carroll** (17:49): Never too early to start talking about entanglement.

**Strogatz** (17:51): Let’s talk about it. What is it? I hear it a lot. I hear quantum people talking about it. Nowadays, especially, with quantum computing, we keep hearing about entanglement. Why don’t you just start with telling us what it means, where the idea came from?

**Carroll** (18:04): Yeah, I mean, let’s think about the Higgs boson. We discovered it a few years ago, it’s a real particle, and I wrote a book about it, *The Particle at the End of the Universe*. The Higgs boson — one of the reasons why it’s hard to detect is that it decays. It has a very, very short lifetime, right? So, you can imagine if someone put a Higgs boson right in front of you, it would generally decay into other particles in about one zeptosecond. That’s 10^{-21} seconds. Very, very quickly.

(18:31) One thing it can do, it can decay into an electron and a positron, an antielectron. So it can decay into two particles, electron and positron. Now remember quantum mechanics. So, you can predict roughly how long it will take the Higgs boson to decay, but when it spits out that electron and positron, you can’t predict the direction in which they’re going to move.

(18:54) I mean, that makes perfect sense because the Higgs boson itself is just a point. It has no directionality in space. So there’s some probability of seeing the electron, in a cloud chamber or whatever, moving in whatever direction you want. Likewise, for the positron, there’s some probability, seeing it moving in whatever direction you want. But you want momentum to be conserved. So you don’t want the Higgs boson sitting there, stationary, to decay into an electron and a positron both moving rapidly in the same direction. That would be a shift in the momentum, right?

(19:26) So, even though you don’t know what direction the electron is going to move in, and you don’t know what direction the positron is going to move in — sorry, I’m already, I’m being, I’m being the person who I make fun of, I’m speaking as if these are real. Even though you don’t know what direction you will measure the electron to be moving in, and you don’t know what direction you will measure the positron to be moving in, you know that if you measure them both, they will be back to back. Because they need to have equal and opposite momentum, for those to cancel out.

(19:54) So what that means is, if you believe all those things, right away, this is why we believe there’s only one wavefunction for the combined system of the electron and the positron. It’s not an independent question, what direction are you going to measure the electron in? What direction are you going to measure the positron in? It’s a statement you need to ask at the same time. That’s entanglement, right there. Entanglement is the fact that you cannot separately and independently predict what the observational outcome is going to be for the electron and the positron.

(20:26) And this is completely generic and everywhere in quantum mechanics. It’s not a rare, special thing. Many things are entangled with many other things. It’s the unique and fun and very useful time when things are not entangled with each other. It took a long time — like, Einstein and his friends — Einstein, Podolsky and Rosen, EPR — published a paper in 1935 that really pointed out the significance of entanglement. Because it was sort of there, already, implicit in the equations, but no one had really shone a flashlight on it, and that’s what Einstein did. And the reason why it bothered him is because when that Higgs boson decays and the positron and the electron move off in opposite directions, you can wait a long time, let’s say you wait a few years before you measure what direction the electron is moving in.

(21:14) So, both particles are very, very far away from each other. And now when you measure the location of one, supposedly the location of the other one is instantly determined. And there’s no limit of the speed of light or anything like that. So for obvious reasons, Einstein, very fond at the speed of light as a limit on things, he didn’t like that. He never really quite thought that that was the final answer, he was always searching for something better.

**Strogatz** (21:39): And the argument goes nowadays that it’s okay, it’s no violation of special relativity, because you can’t use this to transfer any information or something? Is that the statement?

**Carroll** (21:39): Yeah, well, you know, there’s, there’s a whole bunch of statements that one can make. But the one that we absolutely think is true, is the one that you just made. If you imagine these two particles moving back-to-back, and one person detects one, and there’s another one, you know, a light-year away, who’s going to detect the other one, the point is that they don’t know what your measurement outcome is, you would have to tell them.

So even though in the global point of view, now, the location where the other particle is going to be detected is known to God, or to the universe, it is not known to any particular person sitting at any location within the universe. It takes the speed of light time to take a signal that would let you know that there is some now new fact about the matter, where you’re going to observe the positron. So, you cannot actually use this for signaling, you just don’t know what has happened when your other observer has measured something. And you can actually prove that, under reasonable assumptions, in the theory as we know it.

(22:43) So it seems as if this is the tension, that the way the universe works involves correlations that travel faster than the speed of light, but in some well-defined sense, information does not travel faster than the speed of light. That should worry you, that we didn’t define any of these words. So you know, what does that mean? You’re not going to build a transporter beam or anything like that out of this stuff.

(23:09) But — but let me just add one other thought, which I think, again, is a result of my quirky way of thinking about these things, which is not entirely standard, which is, people really like locality. Like, locality is a central thing. Locality is just the idea that if I poke the universe at one point in space-time, the effects of that poke will happen at that point, and then they will ripple out. But they will ripple out to other points no faster than the speed of light, okay? There’s nothing I can do to poke the universe here that will change the state of the universe in a tangible way very, very far away. And you can see how this entanglement thing is kind of on the boundary of that, like, the description of the universe changes instantly far away, but no information is traveling.

(23:51) So then, if you believe that locality is fundamental like that, then you’re sort of asking this question, why does the universe almost violate that but seem to not quite? That’s the puzzle that we have. And this is — a lot of ink has been spilled in the foundations of quantum mechanics.

(24:06) I think about it entirely the other way around, because I think of the wavefunction as the fundamental thing, right? I think that’s what exists in reality. And the wavefunction, like the wavefunction of this positron and electron is utterly nonlocal. It just exists all — it’s a, it’s a feature of the universe as a whole right from the start. So, I also have a mystery to be explained, but my mystery is the opposite way. It’s not “why is locality approximately or, you know, seemingly violated by entanglement?” It’s “why is there locality at all?” Like, that’s the puzzle to me.

**Strogatz** (24:41): Okay, so with talking about entanglement and its discontents or its wonders, what does all this have to do with what we were saying earlier about space as emergent? Because there is some connection, right?

**Carroll** (24:52): That’s right. The aspiration is to say that we start with this abstract quantum wavefunction. So, what I mean by abstract is, it’s not a wavefunction of anything. The usual way of talking, because we’re human beings that start classically, is to say we have the wavefunction of the electron, of the harmonic oscillator, of the Standard Model of particle physics or whatever. No, that’s cheating. We don’t allow ourselves that. We just have an abstract quantum wavefunction and we’re asking, can we extract reality as we know it from the wavefunction? Space-time, quantum fields, all of those things, okay. So we don’t have a lot to work with.

(25:30) But what we can do is, we’re able to use clues from physics as we understand it in the real world. So, in the real world, we have, to a very good approximation, the world is run by what we call quantum field theory. Okay, so, the stuff of the world, the particles and the, you know, the forces, etc., all come from fields that spread all throughout space and time and have a quantum mechanical nature.

(25:55) So, there’s a field for the electron, there’s a field for the photon, a field for the gluon, a field for the Higgs boson, etc. A field for gravity. All of these things are quantum mechanical fields. Now, again, this is not what I’m proposing, this is just our current best approximation, right? This is what seems to fit the data. And you can ask questions about what that looks like in practice.

And so, the important thing about field theory is that even in empty space, there are still fields there. Space is not completely empty, it’s not just like, an empty vessel. There are fields that, as we say, are in their ground state. They’re in their lowest-energy state. So they’re — classically, you just say the field has value zero. Like you could say, there’s something called the magnetic field, but at this particular point in space, it’s zero. It’s still — there is a field, but its value is zero. Quantum mechanically, it’s more complicated than that, but you can still say it’s in its lowest-energy state. That’s something you’re allowed to say.

(26:49) And then what you can do is take two different points of space-time, at some distance between them, and because there’s still things there, because there still are fields even in empty space, you can say, is there entanglement between these two points of space? Because of the fields there. Are the — is the quantum state of the fields at these two points in space, is it entangled? And the answer is yes, it is always going to be entangled.

And in fact, more than that, if the points are nearby, the fields will be highly entangled with each other. And if the fields are far away, the entanglement will be very, very low. Not zero, but very, very low. So in other words, there is a relationship between the distance between two points and their amount of entanglement in the lowest-energy state of a conventional quantum field theory.

(27:38) And what we say is, look, we start with an abstract quantum wavefunction. We don’t have any such words like distance, or fields, for that matter, right? But we do have the word “entanglement.” We can figure out, if you divide up the wavefunction into this bit and this bit, are those two bits entangled? There’s mathematical ways to measure them using the mutual information, etc. So you can quantify the amount of entanglement between different pieces of the wavefunction. And then, rather than saying “the more distance, the less entanglement,” you turn that on its head. You say, “Look, I know what the entanglement is.” Let me assume, let me put out there as an *ansatz* [a mathematical assumption], that when the entanglement is strong, the distance is short. And I’m going to define something called the distance. And it’s a small number when the entanglement is large, it’s a big number when the entanglement is small.

(28:26) So what you’re doing is, in this big space in which the wavefunction lives, you’re dividing it up into little bits, you’re relating them — Steve, you will be happy about this. You’re drawing a network, a graph. You have different parts of Hilbert space. Those are nodes in the graph, and then they have edges, and the edges are the amount of entanglement. And there’s a function of those amount of entanglement which says, invert it, roughly speaking, and get a distance. So now you have a graph of nodes with distances between them. And you can ask, do those nodes fit together to approximate a smooth manifold? And if you pick the right kind of laws of physics, they will.

(29:06) And then you can ask, if I perturb it a little bit, so I poke it, so it’s not in its lowest-energy state, it has a little bit of energy in it. Well, that’s going to be dynamical. That’s going to stretch space-time, that’s going to change the amount of entanglement. We can interpret that as a change in the geometry of space. Is there an equation that that obeys?

And the answer is, you know, under many assumptions that are not entirely solid yet, but seem completely plausible, the geometry of that emergent space obeys Einstein’s equation of general relativity. Not completely as surprising and dramatic as it sounds, because there’s not a lot of equations it could have obeyed. But the point is that if we follow our nose, if we say we start not with space, but with entanglement, how should it behave? How should it interact? We get to a place where it’s not at all surprising that it has dynamics, that it changes, that it responds to what you and I would notice as energy, and the kind of response is the kind that Einstein had there in general relativity.

(30:03) So, you can imagine an alternative theory of physics — history of physics. Where Einstein did not invent general relativity. Where we invented quantum mechanics first, and we understood it. And we really thought about it very deeply, and at some point, someone said, you know, if you really take this seriously, the emergent geometry of space should be dynamical and curved, I’m going to call it general relativity. That’s not what happened. But that’s what we’re hoping to work out when we’re all done.

**Strogatz** (30:28): There’s this big story about this awful acronym, AdS/CFT correspondence, that some people may have heard of. Some of our listeners may know that there’s some — some work that has a similar spirit to what you’re describing, where you derive gravity — not you, Juan Maldacena, I guess, and Lenny Susskind and other people — are trying to derive gravity from quantum field theories that don’t have gravity in them. Can you tell us about some of that and explain it to us?

**Carroll** (30:35): Right. It is very much close in spirit. And the idea is that you have this principle, called the holographic principle. It doesn’t really deserve the name of a principle because it’s a little bit vague. But the idea is that for a black hole, all of the information, all the quantum mechanical information inside a black hole, can in certain circumstances be thought of as spread out on the boundary of the black hole.

So if you think of the interior of the black hole as a three-dimensional region of space, and the boundary, the event horizon, as a two-dimensional boundary, somehow, you could think of all the information of the black hole as being located on the boundary. So that’s holography, because there’s only a two-dimensional boundary that is filling in the three-dimensional inside, much like shining a light on a two-dimensional hologram gives you a three-dimensional image.

(31:45) What Maldacena did was applied that not to black holes, but to a certain kind of cosmological space-time called anti-de Sitter space. So, in general relativity, in Einstein’s theory of gravity, if there’s nothing going on, if there’s no energy, no stuff, or anything like that, you can solve the equation, Einstein’s equation, and you find flat space-time, which we call Minkowski space-time, this is just the arena where special relativity lives.

(32:12) The next simplest thing you can do is add energy to it, but add only vacuum energy, energy of empty space itself. So there’s no particles or photons or anything like that. There’s just empty space, it has energy. We think that space does have energy now, we discovered this with the accelerating universe, in 1998. And the equations were solved back in 1917 by Willem de Sitter, a Dutch astrophysicist. So, if you have a positive amount of energy in empty space, you get a cosmological solution called de Sitter space. And that is basically where our real universe is evolving to, as we expand and the galaxies move further and further away.

(32:52) If you just flip the sign to make the vacuum energy have a negative amount, you’re allowed to do that. And it’s called anti-de Sitter space. It’s just a flip of the sign in the math. And the great news is that this anti-de Sitter space — again, it’s a pure — all that should drive home to you that this is not the real world. Not only is it empty, but the vacuum energy is negative rather than positive. It’s a completely thought-experiment kind of thing. But what Maldacena showed is that gravity, quantum gravity, string theory he was thinking of in particular, inside anti-de Sitter space can be related to a theory of quantum field theory without gravity, that you can think of as living on the boundary infinitely far away.

(33:36) So if there’s a boundary to anti-de Sitter space infinitely far away, it’s one dimension less. Because it’s kind of like, you know, the event horizon of a black hole, it’s wrapped around the anti-de Sitter space. It is itself flat space-time. There’s no gravity there, you can define quantum field theory on it, you have no conceptual issues with quantizing it. It’s good old, well-defined quantum field theory. And Maldacena argued that it is the same theory as quantum gravity in the interior, in what we call the bulk of anti-de Sitter space. There’s a relationship between these two theories that is a one-to-one correspondence. And it’s hard to prove that. But there’s an enormous amount of evidence that it’s true.

(34:14) And then, subsequent to that, people like Mark Van Raamsdonk and Brian Swingle and others pointed out that if you take the theory on the boundary, the theory that we understand, the quantum field theory without gravity, and all you do is you twiddle the amount of entanglement between different parts of the quantum field theory on the boundary, the geometry of the anti-de Sitter space inside responds. It changes in response to that. In some sense, the geometry of that emergent anti-de Sitter space, holographically emergent, is very sensitive to the amount of entanglement on the boundary. So this is the sense in which, in this case, geometry is emerging from entanglement.

(34:57) So, to compare that to what I’m doing, I am not in anti-de Sitter space. I’m here on Earth, both literally and conceptually. I am in the limit where space-time is almost flat, right, where gravity is weak. Like the solar system, even though the sun is very big, gravity is still weak, it’s nowhere near being a black hole. So there’s no holography, everything is pretty local, as we were talking about before, everything is, you know, bumping up against other things right next to each other, right here in space.

(35:24) The holographic limit, that’s kind of the opposite. Holography kicks in where gravity is strong, where you either have a black hole or a cosmological horizon or something like that. And that’s when the information seems like it’s in one dimension less. What you need in the full theory, which nobody has, is both at once.

(35:47) There’s a huge number of people working on AdS/CFT. CFT because the particular kind of field theory you have on the boundary is what is called a conformal field theory. So C-F-T, conformal field theory. So, there’s a huge number of people working on that. And it’s fun, and it’s well-defined, there’s a lot of math, there’s a lot of physics, full employment, whereas what I’m doing is much less well-defined, because we don’t have this well-defined boundary where everything doesn’t involve gravity, and therefore you can solve all your equations.

(36:16) But, you know, I think that you’re going to need both at the end of the day. I think that the AdS/CFT approach doesn’t really illuminate what goes on in the solar system very well. It illuminates what goes on cosmologically pretty well. So I think that they’re compatible ways of sort of coming at the problem from different approaches.

**Strogatz** (36:32): You know, I’m glad you mentioned Van Raamsdonk and Swingle, because that’s another very seminal paper in this whole, I want to say “space” of emergent space-time. Thinking, you know, looking ahead, these ideas of emergent space-time, do you think they’ll have impact on our current models in physics?

**Carroll** (36:50): Well, I think there’s still, certainly, a lot to do just in terms of understanding the proposal, right? I mean, really going from these incomplete ideas about entanglement and emergent geometry to a full theory like, “Oh, this is why things have three dimensions of space. This is the kind of laws of physics that let this happen in the first place,” you know, and so on. And so there’s just, like, a lot of very basic groundwork remaining to be done. The ideal thing, the wonderful thing that would be amazing, is to make an experimental prediction from all this.

(37:25) And it’s not completely wacky to imagine that is possible. For the following reason: You know, it goes back to what we said about space and time not being quite on an equal footing. We’re using them in different ways. So the technical term for this is we’re violating Lorentz invariance. It’s this symmetry that was handed down by Lorentz, a famous Dutch physicist, a mentor of Einstein’s, that says it doesn’t matter how you look at space and time, everyone’s perspective is equal.

That’s not quite right, in our point of view. It might not be right. So, it’s possible that there is an experimental prediction for a tiny violation of Lorentz invariance. And this might show up in, you know, how photons propagate across the universe or something like that, or some very delicate, precise laboratory experiment we can do here on Earth. We don’t know. I don’t have that prediction yet for you. But I think that is something that is plausible within this framework.

**Strogatz** (38:21): That’s a wild idea. Because a lot of people think of the Lorentz invariance, basically, this principle of relativity, taken very seriously, as a deep inviolable principle in physics, and you’re saying it may be itself an emergent-like approximation. It’s almost like a spurious symmetry that comes out from looking at the emergent theory rather than the fundamental theory.

**Carroll** (38:43): Yeah, that’s exactly right. And again, maybe, as we have both been saying. It’s a low-probability, high-impact question to ask. So I think — it’s worth spending some of your time on questions like that.

**Strogatz** (38:55): I feel like you’ve been a very brave and generous person in sharing these speculations with us. I mean, you’ve been so honest about the tentative nature of science, which for all of us who actually do science and math, know that that’s how it really is. But I think it’s, it’s very healthy for our listeners to appreciate this, that we’re all sticking our necks out all the time and we kind of like it, and it’s what makes it such an adventure.

**Carroll** (39:20): Well, I do think that and, you know, I think that there’s a school of thought that says that scientists should not talk about their results until they’re completely established and refereed and everyone agrees they’re right. And not only do I think that that’s implausible, because even results that are refereed and published could be wrong, I think it’s very antithetical to the spirit of how science is, you know, and I want to emphasize that science is not just a set of results that are handed down from on high, it’s a process. We could be wrong. We’re making suppositions and hypotheses and guesses, and we’re going to figure out whether or not they work. And that’s not a bug, it’s a feature. That’s, that’s how science works. So I’m very willing to talk about tentative things as long as I try to emphasize that they are tentative things.

**Strogatz** (40:09): Yep. Thank you and bravo. And that we are trying to be rational. We’re looking for evidence, we’re willing to admit when we’re wrong, when we are wrong.

**Carroll** (40:17): Yeah, actually, I think that it would increase trust in science if we were more honest about the fact that we can be wrong all the time. Because we are going to be wrong some of the time, and if we pretend that we’re never wrong, then it’s going to hurt our credibility when we’re wrong.

**Strogatz** (40:32): Okay, amen, Sean. Thank you so much for joining us in a really delightful conversation today.

**Carroll** (40:37): It’s my pleasure. Thanks very much for having me on.

**Announcer** (40:43): If you like *The Joy of Why*, check out the *Quanta Magazine Science Podcast*, hosted by me, Susan Valot, one of the producers of this show. Also, tell your friends about this podcast and give us a like or a follow where you listen. It helps people find *The Joy of Why* podcast.

**Strogatz** (41:05): *The Joy of Why* is a podcast from *Quanta Magazine*, an editorially independent publication supported by the Simons Foundation. Funding decisions by the Simons Foundation have no influence on the selection of topics, guests, or other editorial decisions in this podcast or in *Quanta Magazine*. *The Joy of Why* is produced by Susan Valot and Polly Stryker. Our editors are John Rennie and Thomas Lin, with support by Matt Carlstrom, Annie Melchor, and Leila Sloman. Our theme music was composed by Richie Johnson. Our logo is by Jackie King, and artwork for the episodes is by Michael Driver and Samuel Velasco. I’m your host, Steve Strogatz. If you have any questions or comments for us, please email us at quanta@simonsfoundation.org. Thanks for listening.

Scientific progress has been inseparable from better measurements.

Before 1927, only human ingenuity seemed to limit how precisely we could measure things. Then Werner Heisenberg discovered that quantum mechanics imposes a fundamental limit on the precision of some simultaneous measurements. The better you pin down a particle’s position, for instance, the less certain you can possibly be about its momentum. Heisenberg’s uncertainty principle put an end to the dream of a perfectly knowable world.

In the 1980s, physicists began to glimpse a silver lining around the cloud of quantum uncertainty. Quantum mechanics, they learned, can be harnessed to aid measurement rather than hinder it — the thesis of a growing discipline known as quantum metrology. In 2019, gravitational wave hunters used a quantum metrological technique called quantum squeezing to improve the sensitivity of the LIGO detectors by a whopping 40%. Other groups have employed the phenomenon of quantum entanglement to precisely measure weak magnetic fields.

But the most controversial and counterintuitive strategy for exploiting quantum mechanics to boost precision is called postselection. In this approach, researchers take photons, or particles of light, that carry information about some system of interest and filter some of them out; the photons that survive this filtering enter a detector. Over the past 15 years, experiments using postselection have measured distances and angles remarkably precisely, suggesting that discarding photons is somehow beneficial. “The community still debates how useful it is and whether [postselection is] a genuinely quantum phenomenon,” said Noah Lupu-Gladstein, a graduate student at the University of Toronto.

Now, Lupu-Gladstein and six co-authors have pinpointed the source of the advantage in postselected measurements. In a paper accepted for publication in *Physical Review Letters*, they trace the advantage to negative numbers that arise in calculations because of Heisenberg’s uncertainty principle — ironically, the same rule that constrains measurement precision in other contexts.

Researchers say that the new understanding forges links between disparate areas of quantum physics and that it could prove useful in experiments that use sensitive photon detectors.

The paper is “quite exciting,” said Stephan De Bievre, a mathematical physicist at the University of Lille in France who was not involved in the research. “It links this negativity, which is a sort of abstract thing, to a concrete measurement procedure.”

To measure a quantity very precisely, physicists often look for a shift in the peaks of a wave, called a phase shift. Suppose, for example, they want to determine the changing distance between two mirrors, signifying that a passing gravitational wave has briefly warped space-time. They’ll first send in a laser beam that bounces back and forth between the mirrors. A displacement of one mirror will shift the peaks of the laser light; physicists then measure this phase shift by detecting the light leaving the system.

But light is composed of individual photons that only collectively behave like a wave. Each photon that physicists detect will deliver imperfect information about the light’s phase shift (and thus the mirror displacement). So a precise estimate requires averaging many measurements of individual photons. The goal of quantum metrology is to reduce the workload by increasing the information gained per photon.

How postselection achieves this has been a mystery. The new paper shows how.

In quantum mechanics, the equations defining a particle don’t say exactly where it is, or exactly how fast it’s going. Instead, they give a probability distribution of positions at which you might observe the particle, and another probability distribution for possible values of its momentum. But recall that Heisenberg’s uncertainty principle prevents precise simultaneous measurements of position and momentum (and other pairs of properties). This means you can’t multiply the two probability distributions to get the “joint probability distribution” representing the likelihood of different combinations of position and momentum, the way you can in classical probability theory. “If you try to define joint probabilities of two observables, then all hell breaks loose,” said De Bievre.

Instead, quantum probabilities combine in a more complicated way. One approach, derived independently by the American physicist John Kirkwood in 1933 and the British physicist Paul Dirac in 1945, defines the probabilities of different combinations of quantum properties by breaking the usual rule that probabilities must be positive numbers. In the Kirkwood-Dirac “quasiprobability” distribution, it’s as if some combinations of properties have a negative chance of happening.

In 2020, David Arvidsson-Shukur at the University of Cambridge, Nicole Yunger Halpern, now at the University of Maryland, and four other theorists developed a framework for describing quantum metrology experiments using the Kirkwood-Dirac distribution. This allowed them to explore how a quantum advantage might arise during postselection.

Arvidsson-Shukur and Yunger Halpern then teamed up with experimentalists in Toronto to develop their model further. In the new paper, they derived a quantitative relationship between the negativity of the Kirkwood-Dirac distribution and the information gained per detected photon in experiments with postselection. They showed that without negativity — that is, when measured properties of the photon aren’t related by the uncertainty principle, and their Kirkwood-Dirac distribution thus stays positive — postselection offers no advantage. But when there’s a high degree of negativity, the information gain shoots up: You can in principle resolve any phase shift, no matter how small, using only a single postselected photon.

To test this idea in an experiment, the researchers sent a laser through a thin quartz slab, which rotated the photons’ polarization by an amount dependent on the slab’s angle. The goal was to precisely estimate that angle. The physicists used polarization-sensitive optical components to filter photons, routing them into or away from a detector based on their polarization.

Like position and momentum, different directions of polarization are related by the uncertainty principle: The more precisely you measure how polarized a photon is along the *x*-axis, say, the less certain you can be of its polarization along the *y*-axis. By rotating the axes of the optical components relative to one another, the experimentalists could therefore change the amount of uncertainty in the measurement, and thus the negativity of the Kirkwood-Dirac distribution. The rotations also affected which photons got postselected.

By repeating the experiment in many different configurations, they showed that the information about the angle of the slab obtained from each detected photon increased linearly with the degree of negativity, just as their theory predicted.

Although maximizing the negativity makes individual photons more informative, it also means fewer photons get postselected. The probability that a photon survives postselection depends on the sum of the elements of the Kirkwood-Dirac distribution; in a distribution with high negativity, negative and positive quasiprobabilities nearly cancel out, and few photons make it to the detector. This trade-off between increased information per detected photon and fewer such photons guarantees that postselection doesn’t increase the total amount of information carried by all photons in an experiment. “We’re not getting free lunch,” Lupu-Gladstein said, “but we’re getting the lunch that we paid for.”

Still, some experiments benefit from using postselection to concentrate all relevant information into a handful of photons. State-of-the-art detectors often overload when exposed to too many photons at once. Postselection can serve to juice up the weak light that these detectors can handle.

Michael Raymer, a quantum physicist at the University of Oregon, said that “the study provides new insights into the sensitivity” of optical measurements. He cautions, though, that there may be other ways to interpret the origin of postselection’s advantage.

Recently, Yunger Halpern and other theorists showed that Kirkwood-Dirac negativity also underlies quantum behavior in contexts besides metrology, including quantum thermodynamics and fast information scrambling in black holes. The researchers say that bridges between these domains could foster further insights, or metrological advantages.

“One of my main hopes for this work is that it now opens the floodgates for people who are studying black holes to maybe say something about metrology,” said Lupu-Gladstein.

For nearly two centuries, all kinds of researchers interested in how fluids flow have turned to the Navier-Stokes equations. But mathematicians still harbor basic questions about them. Foremost among them: How well do the equations adhere to reality?

A new paper set to appear in the *Annals of Mathematics* has chipped away at that question, proving that a once-promising class of solutions can contain physics-defying contradictions. The advance is another step toward understanding the discrepancy between Navier-Stokes and the physical world — a mystery that underlies one of math’s most famous open problems.

“It’s very impressive,” said Isabelle Gallagher, a mathematician at the École Normale Supérieure in Paris and Université Paris Cité. “I mean, it’s the first time you really have [these] solutions which are not unique.”

Fluids are inherently difficult to describe, as their constituent molecules don’t move as one. To account for this, the Navier-Stokes equations describe a fluid using “velocity fields” that specify a speed and direction for each point in 3D space. The equations describe how a starting velocity field evolves over time.

The big question that mathematicians want to answer: Will the Navier-Stokes equations always work, for any starting velocity field into the arbitrarily distant future? The issue is considered so important that the Clay Mathematics Institute made it the subject of one of their famed Millennium Prize Problems, each of which carries a $1 million bounty.

In particular, mathematicians wonder whether a solution that starts out smooth — meaning its velocity fields don’t change abruptly from one nearby point to another — will always remain smooth. It’s possible that after a while, sharp spikes that represent infinite speed might pop up. This outcome, which mathematicians call blow-up, would deviate from the behavior of a real-life fluid. To claim the $1 million prize, a mathematician would have to either prove that blow-up will never happen, or find an example where it does.

Even if the equations can blow up, perhaps not all is lost. A secondary question is whether a blown-up fluid will always keep flowing in a well-defined, predictable way. More precisely: Is there only a single solution to the Navier-Stokes equations, no matter the initial conditions?

This feature, called uniqueness, is the subject of the new paper by Dallas Albritton and Elia Bruè of the Institute for Advanced Study and Maria Colombo of the Swiss Federal Institute of Technology Lausanne.

The non-quantum world works in this way. The laws of physics determine how a system evolves from one moment to the next, with no room for guesswork or randomness. If the Navier-Stokes equations can really describe real-life fluids, their solutions should obey the same rules. “If you don’t have uniqueness, then the model is [probably] incomplete,” said Vladimír Šverák, a professor at the University of Minnesota who was Albritton’s doctoral adviser. “It’s simply not possible to describe fluids by the Navier-Stokes equations as people had thought.”

In 1934, the mathematician Jean Leray discovered a novel class of solutions. These solutions could blow up, but just a little bit. (Technically, parts of the velocity field become infinite, but the fluid’s total energy remains finite.) Leray was able to prove that his non-smooth solutions can go on indefinitely. If these solutions are also unique, then they could help make sense of what happens after blow-up.

The new paper, however, has discouraging news. The three authors show that a single Leray starting point can be consistent with two very different outcomes, meaning their tether to reality is weaker than researchers hoped for.

Mathematicians suspected this about Leray solutions, and the last several years saw a steady accumulation of evidence. The new result “was somehow the cherry on top,” said Vlad Vicol, a professor at New York University’s Courant Institute.

Albritton, Bruè and Colombo entered the picture in the fall of 2020 when they joined a study group at IAS. The purpose of the group was to read two papers the mathematician Misha Vishik had posted online in 2018. While the most sought-after answers are about the Navier-Stokes equations in three-dimensional space, two-dimensional versions of the equations also exist. Vishik had proved that non-uniqueness occurs in a modified version of these 2D equations.

Yet two years after Vishik posted the papers, the details of his work were still hard to understand. The seven-person study group met regularly for about six months to work through the papers. “With all of us contributing, we were able to see what was going on,” said Albritton.

Vishik’s proof used an external force. In a real-world setting, a force might be due to splashing, wind, or anything else with the ability to change a fluid’s trajectory. But Vishik’s force was a mathematical construct. It wasn’t smooth, and didn’t represent any particular physical process.

With that force in place, Vishik had been able to find two distinct solutions to the two-dimensional equations. His solutions were based off of a vortex-like flow.

“It’s essentially creating a fluid flow that’s just swirling you around,” said Albritton.

Albritton and Colombo — later joined by Bruè — realized they could use Vishik’s vortex as the foundation for two distinct solutions in three dimensions as well.

“The strategy is actually very innovative,” said Vicol, who advised Albritton during the latter’s postdoctoral fellowship at NYU.

To prove non-uniqueness, the three authors constructed a doughnut-shaped “vortex ring” solution to the three-dimensional equations. At first, their fluid is completely still, but a force propels it into motion. This force, like Vishik’s, is not smooth, ensuring that the vortex ring will not be smooth either. As the fluid gains momentum, it flows along the vortices, circling through the doughnut hole and back up around the outside.

The authors then showed that this vortex ring solution can degenerate into a different solution.

The effect was something like dropping a stone into a lake. Typically, you’ll see a few waves that dissipate after a short time. Those waves show up in the Navier-Stokes equations as a “perturbation” added to the velocity field. You can play with the size of that perturbation by dropping the stone more or less gently; if you drop it very carefully from a point close to the surface, it might barely affect the lake at all.

But if you drop a stone into the flow that Albritton, Bruè and Colombo created, the perturbation will never disappear. Even if you drop the stone from effectively zero height, that vanishingly tiny disturbance would grow into something much more formidable. That creates a second distinct solution from the same initial conditions.

“You have one solution, and instead of making a finite disturbance, you make an infinitesimally small disturbance,” said Albritton. “And then, instantly the solutions are driven apart.”

The new paper does not definitively settle whether Leray solutions are unique. Its conclusions rely on an external force crafted specifically to make non-uniqueness occur. Mathematicians would prefer to avoid the addition of a force altogether and prove that some set of initial conditions leads to non-uniqueness without any outside influence. That question is now perhaps a stone’s throw closer to being answered.

*Editor’s note: Dallas Albritton has received funding from the Simons Foundation, which also funds this **editorially independent magazine**.*

On October 9, 2009, a two-ton rocket smashed into the moon traveling at 9,000 kilometers per hour. As it exploded in a shower of dust and heated the lunar surface to hundreds of degrees, the jet-black crater into which it plummeted, called Cabeus, briefly filled with light for the first time in billions of years.

The crash was no accident. NASA’s Lunar Crater Observation and Sensing Satellite (LCROSS) mission aimed to see what would be kicked up from the lunar shadows by the impact. A spacecraft trailing the rocket flew through the dust plume to sample it, while NASA’s Lunar Reconnaissance Orbiter observed from afar. The results of the experiment were astonishing: Scientists detected 155 kilograms of water vapor mixed into the dust plume. They had, for the first time, found water on the moon. “It was absolutely definitive,” said Anthony Colaprete of NASA’s Ames Research Center, the principal investigator of LCROSS.

The moon isn’t an obvious reservoir of water. “It’s really weird when you stop to think about it,” said Mark Robinson, a planetary scientist at Arizona State University. Its lack of atmosphere and extreme temperatures should cause any water to almost instantly evaporate. Yet about 25 years ago, spacecraft began to detect signatures of hydrogen around the moon’s poles, hinting that water might be trapped there as ice. LCROSS proved this theory. Scientists now think there’s not just a bit of water ice on the moon; there are 6 trillion kilograms of it.

Most of this ice resides in peculiar features at the moon’s poles called permanently shadowed regions (PSRs). These are craters like Cabeus into which the sun can’t reach, because of the geometry of the moon’s orbit. “They’re in permanent darkness,” said Valentin Bickel, a planetary scientist at the Max Planck Institute for Solar System Research in Germany.

PSRs are of immense interest to scientists. Inside, temperatures can drop below minus 170 degrees Celsius. “Some PSRs are colder than the surface of Pluto,” said Parvathy Prem, a planetary scientist at the Johns Hopkins University Applied Physics Laboratory in Maryland. This means ice on or below the lunar surface in PSRs won’t necessarily melt; instead it might have survived there for billions of years. Studying the ice’s chemical composition should reveal how it was delivered to the moon, in turn illuminating the origin of water on Earth, or indeed any rocky world around any star. It could also be a resource for future human activities on the moon.

Lunar QuickMap

Studies so far have provided a tantalizing glimpse at best. But that’s about to change. Next year, robotic vehicles will enter the bewildering icy depths of PSRs for the first time, revealing what the interiors of these shadowed craters look like. By the decade’s end, NASA plans to send humans to explore in person.

On the eve of this new era of moon landings, a slew of fresh studies of PSRs have revealed that these shadowed regions are even stranger than scientists imagined. What will we find lurking in the shadows?

“I don’t know what we’re going to see,” said Robinson, the lead scientist for next year’s robotic mission. “That’s the coolest thing.”

Speculation about PSRs dates back to 1952, when the American chemist Harold Urey first hypothesized their existence on the moon. “Near its poles there may be depressions on which the sun never shines,” he wrote. He observed that, whereas Earth orbits the sun with its rotational axis tilted by 23.5 degrees, the moon orbits at a mere 1.5-degree tilt. This means the sun’s rays strike its poles nearly horizontally, and the rims of polar craters will block light from directly reaching their depths. However, Urey believed that any ice in these sunless locations would have been “rapidly lost” because of the moon’s lack of atmosphere.

The American chemist Harold Urey won the 1934 Nobel Prize in Chemistry for discovering deuterium. He also worked on the Manhattan Project and did pioneering research on the origin of life, paleoclimatology, and the origin and properties of the moon.

Then in 1961, the geophysicist Kenneth Watson of Lawrence Berkeley National Laboratory theorized that ice could persist inside PSRs. Nightside temperatures on the moon were known to plunge to minus 150 degrees Celsius; Watson and two colleagues argued that this meant ice would get trapped in the coldest places, despite the exposure to space. “There should still be detectable amounts of ice in the permanently shaded areas of the moon,” they wrote.

Scientists debated the possibility of ice in PSRs until the early 1990s, when radar instruments detected signs of ice at the poles of Mercury, which was also thought to have permanently shadowed craters. In 1994, using a radar instrument on NASA’s Clementine spacecraft, scientists detected an enhanced signal over the moon’s south pole that was consistent with the presence of water ice. The hunt was on.

In 1999, Jean-Luc Margot at Cornell University and colleagues pinpointed PSRs on the moon that could contain ice. They used a radar dish in the Mojave Desert in California to make topographic maps of the lunar poles. “We simulated the direction of sunlight and used our topographic maps to identify regions that were permanently shadowed,” Margot said.

They located just a handful of PSRs, but subsequent studies have identified thousands. The largest measure tens of kilometers across inside giant craters, such as Shackleton crater at the lunar south pole, which is twice as deep as the Grand Canyon. The smallest span mere centimeters. At the Lunar and Planetary Science Conference held in Houston in March, Caitlin Ahrens, a planetary scientist at NASA’s Goddard Space Flight Center, presented research suggesting that some PSRs may grow and shrink slightly as temperatures on the moon fluctuate. “These are very dynamic cold regions,” Ahrens said in an interview. “They are not stagnant.”

Patrick O’Brien and a colleague recently identified double-shadowed regions on the moon that are cold enough to keep exotic ices frozen.

New research indicates that some craters also contain double-shadowed regions, or “shadows within shadows,” said Patrick O’Brien, a graduate student at the University of Arizona, who presented evidence for the idea in Houston. While PSRs don’t experience direct sunlight, most receive some reflected light bouncing off the crater’s rim, and this can melt ice. Double-shadowed regions are secondary craters inside PSRs that don’t get reflected light. “Temperatures can be even colder than the permanent shadows,” said O’Brien; they reach as low as minus 250 degrees Celsius.

The double-shadowed regions are cold enough to freeze more exotic ices, like carbon dioxide and nitrogen, should any exist there. Scientists say the chemical composition of these and of the water ice inside PSRs could reveal how water got to the moon — and, more importantly, to Earth, and to rocky worlds in general. “Water is essential to life as we know it,” said Margaret Landis, a planetary scientist at the University of Colorado, Boulder. The question is, she said, “When and how did the conditions favorable for life on Earth form?” Whereas Earth’s past has been scrambled by geological processes, the moon is a museum of the solar system’s history; its ice is thought to have remained mostly untouched since its arrival.

There are three predominant theories about how water got to the moon. The first is that it arrived via asteroid or comet impacts. In this scenario, when the solar system formed, water molecules in the hot inner solar system were vaporized and blown away by the solar wind; only water in the frigid outskirts could condense and accumulate into icy bodies. These bodies subsequently bombarded the inner solar system, including the moon, delivering water. The second theory is that volcanic eruptions on the moon sometime in its middle age formed a thin, temporary lunar atmosphere that engendered ice formation at the poles. Or solar wind could have transported hydrogen to the moon that mixed with oxygen to form ice.

In February, a re-analysis of the LCROSS plume published in *Nature Communications* indicated that the ice in Cabeus crater is most likely of cometary origin. Analyzing the amount of nitrogen, sulfur and carbon frozen into the ice along with water, Kathleen Mandt of the Johns Hopkins University Applied Physics Laboratory and colleagues found that “the best explanation was comets,” said Mandt. “The nitrogen-to-carbon ratio was way beyond what was reasonable for volcanoes to have delivered.”

If the moon’s ice was delivered exclusively by comets, the same might have been true for Earth. That could mean rocky worlds must experience such impacts to accumulate the water necessary for life to flourish. However, Landis says it’s too early to say whether Mandt’s research holds true for all ice on the moon. “The community needs more time to digest it,” she said.

The planetary scientist Kathleen Mandt’s recent study of water vapor kicked up from a lunar crater suggests that a comet delivered the water to the moon.

If some lunar ice is determined to be of volcanic origin, this would suggest worlds hold an innate ability to generate water from their interiors rather than relying on impacts. “It might be not all solar systems have lots of comets or asteroids,” said Landis, “but solar systems that form rocky planets might have this ability to have [volcanic] eruptions rear up water.”

Aside from looking for exotic ice in PSRs, scientists also want to measure the water ice’s proportion of deuterium, a heavier isotope of hydrogen. Substantial deuterium is more consistent with what’s found in comets (although rates vary), whereas less of it would point to solar wind. A volcanic origin would fall somewhere in the middle. Other elements will be informative, too; for instance, ice originating from volcanoes should contain abundant sulfur drawn up from the lunar interior, said Paul Hayne, a planetary scientist at the University of Colorado, Boulder.

No previous foray to the moon has ventured into its permanent shadows; the Apollo landings took place near the moon’s equator at a time when knowledge of PSRs was in its infancy. In 2019, China’s Chang’e-4 lander and rover touched down at the south pole, but it did not target PSRs.

In 2017, however, President Trump signed a directive to NASA to return humans to the moon, an initiative later named Artemis. Ahead of the first crewed Artemis landings in the mid-2020s, which may include the first sorties into the moon’s permanently shadowed craters, NASA is paying commercial companies to conduct initial robotic exploration.

Houston-based Intuitive Machines will be the first of these companies to explore a PSR, albeit briefly. Their Nova-C lander, scheduled to launch by the end of this year on a SpaceX rocket, will touch down on a ridge near Shackleton crater, a possible target for subsequent human exploration. The lander will then deploy a suitcase-size vehicle called the Micro-Nova Hopper. Intuitive Machines revealed details of the excursion at the Lunar and Planetary Science Conference: The Hopper will use thrusters to jump across the lunar surface, up to hundreds of meters at a time; in three hops, it will reach the edge of the 100-meter-wide Marston crater, which contains a PSR. Then the Hopper will fire itself above Marston and descend into the pitch-black depths.

The lander has cameras and lights, but it’s unclear what it will see. Sheets of surface ice are possible, said Robinson, the mission’s lead scientist, but he says it’s more likely that the vehicle’s lights will reflect off ice crystals mixed in with the lunar soil. Or if there’s minimal ice on the surface, it may not definitively show up in images at all. Whatever the case, the view will be historic.

The Hopper’s dip into Marston will last no more than 45 minutes, and the scientific return will be limited — the primary goal is simply to demonstrate that the hopping approach works. But we won’t have long to wait for a more thorough dive into the lunar abyss.

This summer, the inaugural launch of NASA’s new Space Launch System rocket (which will propel Artemis missions to the moon) will carry several small spacecraft that will study PSRs from lunar orbit. A Korean orbiter launching in August, meanwhile, will carry ShadowCam, a purpose-built NASA instrument designed to image PSRs.

The defining moment in robotic PSR exploration, however, will come in late 2023, when a golf cart-size rover called VIPER (Volatiles Investigating Polar Exploration Rover) will head to the moon on a SpaceX Falcon Heavy rocket. Upon exiting its landing vehicle, VIPER will drive into three of the moon’s permanently shadowed regions and drill into the ground.

Operating for up to 10 hours at a time before exiting to recharge its solar-powered batteries, the rover will drill up to a meter deep for subsurface ice, or dig into any exposed ice on the surface. “If there’s a block of ice we’re going to know right away, because of how hard it is to get through,” said Kris Zacny of Honeybee Robotics in Colorado, which designed the drill. The team expects to perform up to 50 drilling sessions.

NASA / Bridget Caswell, Alcyon Technical Services

VIPER will “revolutionize” our knowledge of these regions, said Landis. It will use spectrometers to analyze any ice that’s found, revealing the ratio of deuterium to hydrogen and looking for hints of carbon dioxide or nitrogen. VIPER may provide conclusive insight into where the moon’s ice comes from, and the general conditions under which ice might be found on rocky bodies. “We will have a quantum leap in our understanding,” said Colaprete, VIPER’s project scientist.

The scientific advances will come on the coattails of a different project. If ice is accessible on or near the surface in PSRs, NASA hopes that astronauts could use it as either drinking water or fuel. NASA is currently planning for the first crewed Artemis landing in 2025 to touch down near a PSR so that the astronauts can see for themselves how viable such an idea might be.

“This is not the Apollo program; we’re planning to stay there for a whole month,” said Jim Green, NASA’s former chief scientist. He added, “The concept of acquiring materials and having habitats on the moon is viable.”

Various proposals for how to extract and utilize water ice are under development, said Kevin Cannon, a space resource expert at Colorado School of Mines. “People are looking at mechanical systems like diggers, backhoes and excavators,” he said. Concentrated sunlight or an oven would then be used to extract the water from the excavated lunar soil. Another idea is to “skip the excavation step and just directly heat the ground in some kind of tent,” Cannon said.

Confirmation that there is indeed accessible ice on the moon could come by the start of next year, with the first images from inside a permanently shadowed lunar crater. By the end of 2023 we may know for sure how it got there.

“There are so many fundamental things we don’t yet understand,” said Prem. “We really are at the beginning.”