Reading Notes: The Master Algorithm (Part 1)

“The Master Algorithm” by Dr. Pedro Domingos is a nice book. I enjoyed reading it.

Learners program themselves.

Learning algorithms are artifacts that design other artifacts.

Hundreds of new learning algorithms are invented every year, but they are all based on the same few basic ideas.

Some key questions:

  • How do we learn?
  • Is there a better way?
  • What can we predict?
  • Can we trust what we’ve learned?

Five tribes:

  • Symbolists view learning as inverse of deduction, take ideas from philosophy, psychology, and logic.
  • Connectionists reverse engineer the brain, inspired by neuroscience and physics.
  • Evolutionaries simulate evolution on computer and draw on genetics and evolutionary biology.
  • Bayesians believe learning is a form of probabilistic inference, and have roots in statistics.
  • Analogizers learn by extrapolating from similarity judgements and are influenced by psychology and math optimization.

We live in the age of algorithms.

Every algorithm, no matter how complex, can be reduced to just three operations: AND, OR, and NOT.

Computers are all about logic. Numbers and arithmetic are made of logic.

Michelangelo said that all he did was see the statue inside the block of marble and carve away the excess stone until the statue was revealed.

In any area of science, if a theory cannot be expressed as an algorithm, it’s not entirely rigorous.

Scientists make theories. Engineers make devices. Computer scientists make algorithms, which are both theories and devices.

A programmer, someone who creates algorithms and codes them up, is a minor god, creating universes at will.

There is a serpent in the Eden, it is called the complexity monster. Like Hydra, the complexity monster has many heads:

  • Space complexity
  • Time complexity
  • Human complexity

The power of ML is best explained by a low-tech analogy: farming:

  • Learning algorithms are seeds
  • Data is the soil
  • Learned programs are the grown plants
  • ML expert is like a farmer
    • Sowing the seeds
    • Irrigating and fertilizing the soil
    • Keeping an eye on health of crop but otherwise staying out of the way

ML is a sword with which to slay the complexity monster. Like the Hydra, the monster sprouts new heads as soon as we cut off the old ones, but they start off smaller and take a while to grow, so we still get a big leg up.

Some learners learn knowledge, and some learn skills. In ML:

  • Knowledge is often in the form of statistical models. Knowledge is complex.
  • Skills are often in the form of procedures. Procedures are simple

In the information-processing ecosystem, learners are the superpredators:

  • Herbivores:
    • Patiently munging on endless fields of data
    • Databases, crawlers, indexers, etc
      • A crawler is like a cow, the web is its worldwide meadow, each page is a blade of grass
      • Database is like elephant, is big and heavy and never forget
      • Learners eat up those info, digest it, and turn it into knowledge
  • Predators:
    • Statistical algorithm, online analytical processing, etc

ML experts are an elite priesthood even among computer scientists.

Computer science has traditionally been all about thinking deterministically. ML requires thinking statistically.

It turns out that:

  • Industrial Revolution automated manual work.
  • Information Revolution did the same for mental work.
  • ML automates automation itself.

Whoever has the best algorithms and the most data wins.

President Obama hired Rayid Ghani, a ML expert, as chief scientist of his campaign. Ghani did this:

  • Consolidate all voter info into a single database
  • Combined it with info from social networking, marketing, etc
  • Predicted 4 things for each voter:
    • How likely he or she was to support Obama
    • How likely he or she shows up at the pools
    • How likely he or she responds to campaign’s reminders
    • How likely he or she changes their mind about election based on a conversation about a specific issue
  • Run 66000 simulations of the election every night
  • Use the simulated results to direct its army of volunteers to find:
    • Whom to call
    • Which doors to knock on
    • What to say

One of the greatest talents a politician can have is to understand voters, individually or in small groups, and speak directly to them.

Evolution is an algorithm. God created not species but the algorithm for creating species.

A problem is in P if we can solve it efficiently, and it is in NP if we can efficiently check its solution. The P=NP question is whether every efficiently checkable problem is also efficiently solvable.
Every conceivable problem that can be solved by logical deduction can be solved by a Turing machine.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s