These are notes and reflections on interesting ideas I've come across in podcasts, videos, articles, and books. They’re not curated, in fact they are barely readable. I just hope that posting them here becomes a reason to put my thoughts into writing more often. Mention does not imply endorsement.

Index

Stephen Wolfram

Physicist, founder of Wolfram Alpha. Interesting takes on computation.

On the observer

The observer is the measurer of the quantity that is to be measured. It defines the dimension and quality of the input/output. In the case of a pressure measurement, for example, the underlying phenomenon is that a bunch of molecules are hitting a surface. The trajectory of the individual molecules is irrelevant, and there are many different configurations of such molecules that are equivalent to observer. All that matters is their aggregate effect.

Computational irreducibility

Our capacity to reduce computation for predicting or representing certain outcomes has to do with the equivalence of states. As observers, we don't care about a specific quantic configuration of the state but just about some aggregate properties, so we can model the phenomenon. If we want to reproduce the exact same system, we have to run the system, and predictive calculation is impossible

Reinforcement learning is everywhere

With computation and technology, we only select topics that are technologically interesting to us (= have practical applications). Science and computation can span much a much broader set of fields than the ones that are interesting for humans, but we only nourish and finance the projects that mean something to us. Example: materials behavior is largely uninteresting until it has a technological application, i.e. high temp superconductor.

Rulial Space

How the world is experienced and computed depends on the observer and its interests, as already stated. Different systems (man, cat, computer) have different representations of the universe and sets of possible actions. The set of all the possible representations and actions define all that's computationally possible: the rulial space. Two different humans have different representations of the world and interests, but they are very close in rulial space, which makes it fairly easy for them to communicate. A man and a cat are more distant in rulial space, and communication is limited. A man and weather are so distant in the rulial space that communication is, insofar, completely impossible.

Computational Thinking vs Computer Science

As the programming languages climb up higher and higher towards natural language, the skill of programming becomes obsolete.

What becomes more relevant is the skill of specifying well what you want the computational system to do: computational thinking. Quite surprisingly, LLMs seem to react well to prompts that are well formulated in a way in which humans would understand them well. Good communicators will be good prompt engineers.

Unlike human communication, however, computational communication should be focused on statements that are objectively true to be reliable (and that's hard). "Someone won a Nobel prize" is an objective concept, "Someone is a good person" is not.

Computational thinking will probably bring us to an understanding of "laws of thought" that goes beyond natural language, on even higher levels. This is what ChatGPT is demonstrating: it shows some reasoning on prompts that it has not seen before, which means it is capable of reproducing implicit patterns of thought on a level that is higher than language.

What is Computational Thinking?

It's the skill of formalizing the world through a computationally viable representation: formalizing an image, a color, a smell, and being able to handle them in computation.

Enthropy law and the observer

The phenomenon of the observer described above derives from the fact that we are computationally bounded entities observing computationally irreducible phenomena. In principle, we can predict the evolution of order into chaos (2nd principle of thermodynamics), but we are computationally unable to do so.

Vitalik Buterin

Ethereum founder. Interesting takes on distributed systems and crypto.

Metastability of distributed systems

Isn't it weird that money isn't backed by anything? Distributed systems are stable by an inherent stickiness of the status quo and difficulty of coordinated change.

Imagine after Elon acquired Twitter all the employees decided to quit and start Twitter 2.0. It's theoretically possible and yet it doesn't happen. If it did, Twitter's value would be zero. Its worth of $44B is backed by the inherent stickiness of distributed systems.

When systems change on a large scale, that's called a revolution. It's a messy business which doesn't happen too often. Inertia is also key: the incentives for the first 10% of the population to start a revolution are very different from those in the 55th percentile.

The origin of Cryptocoins

See the Byzantine generals problem.

In the 80s-90s it was demonstrated that a distributed computing system can work without being hacked. Easy to demonstrate if there's a limited number of actors with known identities (N coordinating corporate actors), more difficult for a larger number of anonymous actors.

Someone under the pseudonym of Satoshi Nakamoto proposed that an anonymous system could work if you limit the creation of tokens by bonding it to the use of economic resources. The economic resources would be the computational energy required to solve known mathematical problems (proof of work).

With such a system, you can run a distributed computing system, aka a computer that keeps working even if a large portion of its units break or becomes adversarial.

The program they wanted to run on this computer is the currency system: analyze transactions, verify that the sender has coins and execute the transaction.

Blockchain

In this distributed computer the nodes occasionally publish a "block" describing the current status of the system, based on a previous known block and a set of transactions.

Attack: there are block 1 and block 2, and then a malicious actor creates block 2' to reverse block 2. (?)

This can also happen by accident. If multiple blocks are being reversed at once, the system only follows the longest chain, so block 1-2-3-4 will be follow instead of block 1-2-4'.

Quadratic Funding

A way to democratize public funding for enterprises that provides public goods, remitting the decision to the people instead of a committee. Individual contributors pay a certain sum, and the recipient gets paid by contributors + centralized entity (government) an amount S that is the square of the sum of the square roots of the individual contributions.

Neil Gershenfeld

MIT professor. Interesting takes on the future of fabrication.

Bits and atoms

So far computation has mostly evolved in the bits space, but this is not how biology works. Biology has elements of physical computation and coordinated intelligence on multiple levels (molecules to macro systems). It is time for fabrication to evolve into a sophisticated interplay of stacks: molecular robots that build robots that build structures. Moreover, we should use machines to do the design themselves based on the ultimate goal of the project. Provide the context and let the system optimize, rather than micromanaging the structure. Bringing fabrication to be biology-like.

Design information

The genome is the design plan of complex organisms. Instead of planning the final structure, it plans the growth incentives. Nowhere in the human genome there's the number 5 for fingers or toes: it's just the best solution to a problem of conflicting incentives.

One reason is information storage: managing the incentives is computationally simpler than designing all the cells in the body. The deeper reason is search space: random small deviations can result in more effective structures which stack over generations, leading to evolution. Fins originally thought for swimming can transform into fingers or wings. This is at the heart of success of AI (along with computation capabilities scaling): in the bits world, AI has found a way to navigate the search space in an effective and interesting way.

Once we have scales of self reproducing robotic assemblers that can work on a scale going from molecular to a macroscopic level (placing 10^18 parts), we will need to fundamentally change the design approach from structural to an evolutionary one. It doesn't mean to randomly change features and see what survives like biological evolution does, but to successfully navigate the features space like AI does.

Molecular Intelligence

Just before dying, Turing was studying morphogenesis: how the genome gives rise to form.

How can such a limited amount of information (DNA) not only describe you, but describe you in such a way that you can exist, live, reproduce and evolve. That's molecular intelligence, and it's a blend in which you can't separate communication, computation and fabrication, or computer science and physical science, or hardware and software.

Resources

MIT Center for Bits and Atoms
FabLabs

Daniel Kahneman

Nobel laureate. Talks about psychology. Thinking, Fast and Slow.

On good and evil

Talking about the holocaust. Good and evil are not innate in humans, but tribalism is. Our sense of good and evil is actually a sense of in-group (my people) and out-group (the inhuman others). This distinction is deeply rooted in human nature.

On speed of learning (human vs AI)

Critiques AI mentioning that children just need a couple examples to learn something, while machines need massive datasets.

I have 2 arguments against that.

1. Evolution: children are models that are pre-trained biologically on millions of years of evolutionary adaptations.

2. Richness of the experience: maybe learning is multisensorial and a physical embodiment might make machines faster.

Finiteness is anothere element. But Lex makes a good point that we are very human-centric in our judgement. For a machine to be grounded in the world could mean something different.

On evaluation of difficulties

We are very biased when evaluating what is easy and what is difficult. Go is intricate but limited, driving seems easy but it's extremely difficult. People thought that perceiving was easy and reasoning complicated, but it's really the other way around (LLMs vs robotics)

Experiencing and remembering self

The self that experiences things and the one that remembers them act as two very different people. It's a difficult concept to grapple with because the experience is a specific instance (hard to pinpont) and the memory of it is a reconstruction.

If you were told that at the end of your vacation you wouldn't remember anything, the vacation you would pick would be different.

The remembering self is also magnified through social media: people build experiences so they can post them on social media. Existentialism and buddhism are founded on the idea that the core of life is in the experience, not to build a "memory score".

Kahneman abandoned this field because while he thought that the experience was to be optimized to achieve happiness, but this it is not how people behave. You can't build a theory of happiness without taking the desires of the remembering self into account.

But the remembering self is a feature not a bug: without remembering you wouldn't be able to intelligently look forward.

On meaning

Lex argues that meaning is key. Mentions Men in Search for Meaning by Victor Franco (book about finding meaning in concentration camps). Argues that finding purpose in life is key to achieve happiness.

K: there's a survivor bias: he says that he survived because of that but we can't know. K says that while people think that having a purpose is key, they don't behave like it, and they build their sense of meaning retrospectively based on their experiences.

On changing your mind

People essentially don't change their minds. There are ideas that are acceptable and you welcome them, and others that are foreign to you. An interesting aspect of that is that people do change their mind if the leaders they look up to do. The acceptance mentioned above is tribal. It is not about evidence but about stories, and about our differential trust towards others.

Intelligence is not only the ability to reason, it is also the ability to find relevant material in memory and to deploy attention when needed.

George Hotz

Hacker, founder of comma.ai.

Dangers of AI

Imagine a TikTok that is so good you can't look away from it. We risk amusing ourselves to death, or just stop having children. The machine is just a machine, but there are plenty of humans with access to the machine interested in manipulating you. Not with a plan for destroying humanity, but maybe overlooking the risk of destroying society as we know it.

Superiority of biology: Robustness

The bio stack is slow and robust, the silicon stack is fast and fragile. Humans haven't changed at all since the 80s, computers have. However, if AI gets us extinct, it will then die. We'll build super intelligence before we can build robustness of the same type that lets nature reproduce itself into systems.

It's not (only) about building a repair shop for robots. Nature can reproduce itself at every level of the carbon stack. The silicon stack is nowhere near that. How big does a system have to be to reproduce itself? Can civilization reproduce itself? Seed a colony on Mars and prevent it from dying out? Not really. Civilization is extra fragile.

Consciousness

Are LLMs conscious? It does feel like something to be alive and to experience things. We can also somehow relate to some fellow animals (mammals in particular), but that's because we share biology and evolutionary history with them. What does it feel like to be a web server? We have no idea. So the claim about LLMs being conscious the way we are says more about our tendency to anthropomorphize than about the LLMs. Think about it: it's already hard enough to imagine how a bee or an octopus feel like, and they share most of the our evolutionary history. The LLMs are learning to mimic our behavior. But then, is consciousness real or are we also just trying to mimic each others?

Lex: Consciousness is the capacity to suffer. George: We call consciousness whatever is most similar to human behavior (very anthropocentric definition)

Tim Urban

Blogger: Wait But Why.

On leadership

To me leadership is the ability to move things in the direction that the cultural forces are not already taking things. Real leadership is when someone changes the shape of the wave.

Tim on technology

As we become capable of more, the good times get better but the bad times get worse too. Imagine nuclear bombs could be manufactured with sand and a microwave. What would be the chances of us nuking ourselves back to the stone age?

Idea labs vs echo chambers

"This is the best boxer in the world", "Cool, who has he fought?", "No one", and "Are you crazy punching my boxer? This is violence". No sacred ideas, everything is approached with an open mind and subject to scrutiny. Political division doesn't tribalize.

Axes of judgement in an idea lab: decency and agreement. axis of judgement in an echo chamber: asshole to non-asshole.

Law and culture with regards to freedom

The freedom granted by the law has to be matched by the freedom granted by the culture. Totalitarian regimes block every freedom except some, democratic regime just block the freedom that would prevent others to be as free. If the culture is such that you get socially banned for a controversial opinion in your country or local community, you are in a totalitarian state.

Chamath Palihapitiya

Venture capitalist.

Mistakes

In poker, business, life it's all about your mistakes vs other people's mistakes. Two-faced god. In competitive games how well you do is proportional to the subtraction of the other players' mistakes minus your mistakes. Counterintuitively however, mistakes are what provides the learning experience (not only in competitive games), and the only way to learn to play is by making mistakes. A mistake is a deviation from GTO (game theory optimal), but it's not negative because it allows you to explore the map and master the game

Strong positions weakly held

Capacity of exploring both sides of an argument and bring them to battle, without being attached to them or even actually believing them.

Application in real life: exploring an idea and its consequences in case it proves right, place sensors in the form of investments, monitor closely and be ready to react in case the trend is right, keep a protection in case you are wrong, be ready for learning and revise your knowledge tree. Preserve optionality and regulate your reactions to things.

Success

If your precondition is to build something successful you've already failed, cause now you're playing someone else's game. What success means is not clear. You're walking into the woods. It's murky, it's dark, it's wet, it's raining, there's all these animals. There's no comfort there, you better really like hiking. There's no way to shortcut that. There's a very basic definition of success that's inside in, but that's mot what it is. I know people that are far richer than I am, that are completely broken.

Christof Koch

Neuroscientist.

On religion

Doesn't believe in God because he doesn't see any evidence, but the night is dark and full of wonders.

Buddhism resonates with him most because it assumes consciousness in all creatures and tries to diminish their suffering.

Consciousness is the internal state and perception of the world that in the subjective experience transcends physics. It can be accessed through meditation or activities that put you in the zone (rock climbing).

Simulation theory: it's the modern version of the philosophical classic that we live inside the mind of God.

Free will

Quantum physics frees us from a predetermined universe, and gives us the freedom to act out the creation. The freedom of our consciousness is not the same as for quantum particles (Heisenberg doesn't apply) and we can certainly be heavily influenced by external factors in our decisions, but if we bring our whole being to it, we ultimately have the freedom to choose.

Claustrum: region of the brain underneath the cortex. Unique structure that takes signals from and gives signals to all cortical regions. Is it the orchestra conductor and locus of consciousness?

Miscellaneous

Eliezer Yudkowski

On biology: Natural selection is stupid. Species compete for relative expansion of their genes in the next generation. They are greedy and don't look far into the future: if a smarter-than-instinctive behavior is needed, i.e. resources consumption restriction, the population rarely adopts them. They overconsume and get extinct. Mothers eat children if they have to.

On conscience: He defines consciousness as having a person that you care about looking out at the universe and wondering about it and appreciating it.

On intelligence: They mention a "Kasparov vs everyone" game, where he beat large groups led by good players, and state that AGI would be something like that. I agree, but not every task is like that, and I wonder where the difference lies

Steven Pinker

On AGI: He thinks that if we're smart enough to build the system, we're also smart enough to give it the right set of incentives, and that's enough to avoid AI takeover. Dominance is a trait of human nature, not of pure intelligence.

On risk: We are very bad at evaluating risk. The US govt spent trillions in fighting terrorism and zero on autonomous driving when in fact traffic kills many more people than terrorists.

Edward Frenkel

Is the world computation? There was a french literary movement in which authors artificially imposed constraints on their own writings. One of them wrote a 300 pages novel without ever using the letter "e". Shows craftsmanship, but what would we think if at the end of the book he claimed "e" doesn't exist? Just because we understand computation, we're not allowed to say that it explains everything.

Mark Tedmark

Truth-seeking AI: Metaculus. People bet on their predictions. Some people are better at predicting. Would love to create an AI that works like that: people trust it because it was right a lot of times.

How to trust an AI that is smarter than you: verifying a demonstration is much easier than coming up with one, so we can ask AI to prove it's gonna do good before we deploy it and just verify.

Stephen Kotkin

On how power corrupts: Speed of execution (absolute power) vs quality of the decisions (checks and balances). Luck streak but doesn't last. The attractiveness of speed of execution might bring someone craving absolute power to the position where they can exert it, but using absolute power methods (which may work initially) yields diminishing returns. You are ever more tempted to use them again, and the number of people benefiting from it diminish over time until it's just the dictator and those around him. The rest of the people grow resentment and a great deal of energy is spent in the internal power struggle.

Matthew MacDougall (Neuralink surgeon)

On great teams: There's a sweet spot where people e adisagrend forcefully speak their minds and passionately defend their positions and yet are still able to accept information from others and change their ideas when they're wrong. Analogy of how you polish rocks: put hard things in a hard container and spin it, people bash against each others and the outcome is a more refined product.

To make a good team at Neuralink we try to find people that are not afraid to defend their ideas passionately and strongly disagree, but are also ready to let their ideas die for the best idea to come up on top.

It's not something that is built into the primate brain by default. If we passionately put all the chips on a position and then walk away from it, part of our brain tells us that that we're losing power, face, standing in the community and you're a Z chimp now. You have to recognize that that voice in your head is maladaptive and it's not helping the team win.