Reading Notes: The Master Algorithm (Part 2)

For a moment in the 1980s, it seemed that knowledge engineering was about to take over the world.

Marvin Minsky, an MIT professor and AI pioneer, is skeptical of any unifying ideas in AI.

Cyc project is the most notorious failure in the history of AI.

Fred Jelinek once said: every time I fire a linguist, the recognizer’s performance goes up.

Learning and knowledge are intertwined in surprisingly subtle ways. The two camps often talk past each other. They speak different languages: ML speaks probability, and knowledge engineering speaks logic.

Learning algorithms are quite capable of accurately predicting rare, never-before-seen events.

Human intuition cannot replace data. Intuition is what you use when you don’t know the facts, and since you often don’t, intuition is precious.

Because of the influx of data, the boundary between evidence and intuition is shifting rapidly.

Listen to your customers, not to the HiPPO (highest paid person’s opinion). If you want to be tomorrow’s authority, ride the data, don’t fight it.

Stories of falling apples notwithstanding, deep scientific truths are not low-hanging fruit.

Science goes through three phases: Brahe, Kepler, and Newton.

ML has a delightful history of simple algorithms unexpectedly beating very fancy ones.

In ML the complexity is in the data. All master algorithm has to do is assimilate it.

The human hand is simple – four fingers, one opposable thumb – and yet it can make and use an infinite variety of tools. The master algorithm is to algorithms what the hand is to pens, swords, screwdrivers, and forks.

As Isaiah Berlin noted, some thinkers are foxes, they know many small things, and some are hedgehogs, they know one big thing.

Before we can discover deep truths with ML, we have to discover deep truths about ML.

If the best we can do is combine many different learners, each of which solves only a small part of the AI problem, we will soon run into the complexity wall.

A universal learner is a phenomenal weapon against the complexity monster.

ML is simple at heart. We just need to peel away the layers of math and jargon to reveal the innermost Russian doll.

No matter how good the learning algorithm is, it’s only as good as the data it gets. He who controls the data controls the learner.

The power of a theory lies in how much it simplifies our description of the world.

Heraclitus said, you never stop in the same river twice.

AND, OR, and NOT can all be implemented using NOR gates, so NOR can do everything.


  • All intelligence can be reduced to manipulating symbols
  • Use some initial knowledge
  • Incorporate pre existing knowledge into learning
  • Combine different pieces of knowledge on the fly
  • Master algorithm is inverse deduction:
    • Figure out what knowledge is missing in order to make a deduction go through


  • Reverse engineer the brain
  • Adjust the strengths of connections between neurons
  • Figure out which connections are to blame for which errors and changing them accordingly
  • Master algorithm is backgropagation


  • Believe in natural selection
  • Simulate natural selection on computer
  • Key problem is to learn structure, not just adjusting parameters
  • Create the brain that those adjustments can fine-tune
  • Master algorithm is genetic programming
    • Mate and evolve computer programs in same way that nature mates and evolves organisms.


  • Care above all with uncertainty
  • All learned knowledge is uncertain
  • Learning itself is a form of uncertain inference
  • Key is how to deal with noisy, incomplete, and even contradictory info without falling apart
  • Solution is probabilistic inference
  • Master algorithm is bayes’ theorem and its derivatives


  • Key is to recognize similarities, thereby infer other similarities
  • Key is to judge how similar two things are
  • Master algorithm is SVM

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s