1. Provide a clear set of learning examples using Tensorflow, Kubernetes, and Google Cloud Platform for establishing Reinforcement Learning pipelines on various hardware accelerators.

2. Reproduce the methods of the original DeepMind AlphaGo papers as faithfully as possible, through an open-source implementation and open-source pipeline tools.

3. Provide our data, results, and discoveries in the open to benefit the Go, machine learning, and Kubernetes communities.

An explicit non-goal of the project is to produce a competitive Go program that establishes itself as the top Go AI. Instead, we strive for a readable, understandable implementation that can benefit the community, even if that means our implementation is not as fast or efficient as possible.

While this product might produce such a strong model, we hope to focus on the process. Remember, getting there is half the fun. 🙂

We hope this project is an accessible way for interested developers to have access to a strong Go model with an easy-to-understand platform of python code available for extension, adaptation, etc.

If you’d like to read about our experiences training models, see RESULTS.md.

To see our guidelines for contributing, see CONTRIBUTING.md.

The Hitchhiker’s guide to python has a good intro to python development and virtualenv usage. The instructions after this point haven’t been tested in environments that are not using virtualenv.

pip3 install virtualenv
pip3 install virtualenvwrapperCopy the code

You may want to use a cloud project for resources. If so set:

PROJECT=foo-projectCopy the code

Then, running

source cluster/common.shCopy the code

will set up other environment variables defaults.

All commands are compatible with either Google Cloud Storage as a remote file system, or your local file system. The examples here use GCS, but local file paths will work just as well.

To use GCS, set the BUCKET_NAME variable and authenticate via gcloud login. Otherwise, all commands fetching files from GCS will hang.

For instance, this would set a bucket, authenticate, and then look for the most recent model.

export BUCKET_NAME=your_bucket;
gcloud auth application-default login
gsutil ls gs://minigo/models | tail -3Copy the code

Which might look like:

gs://$BUCKET_NAME/models/000193-trusty.data-00000-of-00001
gs://$BUCKET_NAME/models/000193-trusty.index
gs://$BUCKET_NAME/models/000193-trusty.meta
Copy the code

These three files comprise the model, and commands that take a model as an argument usually need the path to the model basename, e.g. gs://$BUCKET_NAME/models/000193-trusty

You’ll need to copy them to your local disk. This fragment copies the latest model to the directory specified by MINIGO_MODELS

MINIGO_MODELS=$HOME/minigo-models
mkdir -p $MINIGO_MODELS
gsutil ls gs://minigo/models | tail -3 | xargs -I{} gsutil cp "{}" $MINIGO_MODELSCopy the code

Minigo uses the GTP Protocol, and you can use any gtp-compliant program with it.

# Latest model should look like: /path/to/models/000123-something
LATEST_MODEL=$(ls -d $MINIGO_MODELS/* | tail -1 | cut -f 1 -d '.')
BOARD_SIZE=19 python3 main.py gtp -l $LATEST_MODEL -r $READOUTS -v 3
Copy the code

(If no model is provided, it will initialize one with random values)

After some loading messages, it will display GTP engine ready, at which point it can receive commands. GTP cheatsheet:

genmove [color] # Asks the engine to generate a move for a side play [color] [coordinate] # Tells the engine that a move  should be played for `color` at `coordinate` showboard # Asks the engine to print the board.Copy the code

One way to play via GTP is to use gogui-display (which implements a UI that speaks GTP.) You can download the gogui set of tools at gogui.sourceforge.net/. See also documentation on interesting ways to use GTP.

gogui-twogtp -black 'python3 main.py gtp -l gs://$BUCKET_NAME/models/000000-bootstrap' -white 'gogui-display' -size 19 - 7.5 verbose komi - autoCopy the code

Another way to play via GTP is to watch it play against GnuGo, while spectating the games

BLACK="gnugo --mode gtp"
WHITE="python3 main.py gtp -l path/to/model"
TWOGTP="gogui-twogtp -black \"$BLACK\" -white \"$WHITE\" -games 10 \
  -size 19 -alternate -sgffile gnugo"
gogui -size 19 -program "$TWOGTP" -computer-both -auto
Copy the code

The following sequence of commands will allow you to do one iteration of reinforcement learning on 9×9. These are the basic commands used to produce the models and games referenced above.

The commands are

• bootstrap: initializes a random model
• selfplay: plays games with the latest model, producing data used for training
• gather: groups games played with the same model into larger files of tfexamples.
• train: trains a new model with the selfplay results from the most recent N generations.

This command starts self-playing, outputting its raw game data in a tensorflow-compatible format as well as in SGF form in the directories

gs://$BUCKET_NAME/data/selfplay/$MODEL_NAME/local_worker/*.tfrecord.zz
gs://$BUCKET_NAME/sgf/$MODEL_NAME/local_worker/*.sgf
Copy the code

python3 main.py selfplay gs://$BUCKET_NAME/models/$MODEL_NAME \
  --readouts 10 \
  -v 3 \
  --output-dir=gs://$BUCKET_NAME/data/selfplay/$MODEL_NAME/local_worker \
  --output-sgf=gs://$BUCKET_NAME/sgf/$MODEL_NAME/local_workerCopy the code

gs://$BUCKET_NAME/data/training_chunks/$MODEL_NAME-{chunk_number}.tfrecord.zz
Copy the code

The file gs://$BUCKET_NAME/data/training_chunks/meta.txtis used to keep track of which games have been processed so far. (more about this needed)

python3 main.py gather \
  --input-directory=gs://$BUCKET_NAME/data/selfplay \
  --output-directory=gs://$BUCKET_NAME/data/training_chunksCopy the code

As you might notice, playing games is fairly slow. One way to speed up playing games is to run Minigo on many computers simultaneously. Minigo was originally trained by containerizing these worker jobs and running them on a Kubernetes cluster, hosted on the Google Cloud Platform.

NOTE These commands will result in VMs being created and will result in charges to your GCP account! Proceed with care!

You’ll want to install the following command line tools

• gcloud
• gsutil (via gcloud components install gsutil)
• kubectl (via gcloud components install kubectl)
• docker

In order for each step to work, you’ll have to have the following permissions:

• storage.bucket.(create, get, setIamPolicy) (“Storage Admin”)
• storage.objects.(create, delete, get, list, update) (“Storage Object Admin”)
• iam.serviceAccounts.create (“Service Account Admin”)
• iam.serviceAccountKeys.create (“Service Account Key Admin”)
• iam.serviceAccounts.actAs (“Service Account User”)
• resourcemanager.projects.setIamPolicy (“Project IAM Admin”)
• container.clusters.create (“Kubernetes Engine Cluster Admin”)
• container.secrets.create (“Kubernetes Engine Developer”)

You’ll also want to activate Kubernetes Cluster on your GCP account (just visit the Kubernetes Engine page and it will automatically activate.)

The main way these jobs interact is through GCS, a distributed webservice intended to behave like a filesystem.

The selfplay jobs will find the newest model in the GCS directory of models and play games with it, writing the games out to a different directory in the bucket.

The training job will collect games from that directory and turn it into chunks, which it will use to train a new model, adding it to the directory of models, and completing the circle.

You will need a Docker image in order to initialize the pods.

If you would like to override the GCR Project or image tag, you can set:

The export PROJECT = my - PROJECT export VERSION = 0.1234Copy the code

Then make will produce and push the image!

CPU worker:

make image
make push
Copy the code

GPU worker:

make gpu-image
make gpu-push
Copy the code

• Getting a list of the selfplay games ordered by start time.

  kubectl get po --sort-by=.status.startTime
  Copy the code

• Attaching to a running pod (to check e.g. cpu utilization, what actual code is in your container, etc)

  kubectlc exec -it <pod id> /bin/bash
  Copy the code

• Monitoring how long it’s taking the daemonset to install the nvidia driver on your nodes

  kubectl get no -w -o yaml | grep -E 'hostname:|nvidia-gpu'
  Copy the code

If you’ve run rsync to collect a set of SGF files (cheatsheet: gsutil -m cp -r gs://$BUCKET_NAME/sgf/$MODEL_NAME sgf/), here are some handy bash fragments to run on them:

• Find the proportion of games won by one color:

  grep -m 1 "B+" **/*.sgf | wc -l
  Copy the code

  or e.g. “B+R”, etc to search for how many by resign etc.

• A histogram of game lengths (uses the ‘ministat’ package)

  find . -name "*.sgf" -exec /bin/sh -c 'tr -cd \; < {} | wc -c' \; | ministats
  Copy the code

• Get output of the most frequent first moves

  grep -oh -m 1 '^; B\[[a-s]*\]' **/*.sgf | sort | uniq -c | sort -nCopy the code

• Distribution of game-winning margin (ministat, again):

  find . -name "*.sgf" -exec /bin/sh -c 'grep -o -m 1 "W+[[:digit:]]*" < {} | cut -c3-'
  \; | ministat
  Copy the code

Also check the ‘oneoffs’ directory for interesting scripts to analyze e.g. the resignation threshold.