We propose a conceptually simple, flexible, and versatile framework for object instance segmentation. Our method can effectively detect the target in the image and generate a high quality segmentation mask for each instance. This aspect, called Mask R-CNN, is an extension of Faster R-CNN — adding a parallel branch for predicting target masks to its existing branch for bounding box recognition. Training for Mask R-CNN is simple, with only a little more computational overhead than Faster R-CNN, and runs at 5 FPS. In addition, Mask R-CNN can be easily generalized to other tasks, for example, allowing us to estimate human posture in the same framework. We got the best results on all three kinds of track in COCO puzzle kit, including instance segmentation, bounding box target detection and Person Keypoint detection. Using no other techniques, Mask R-CNN outperformed all available individual models on each mission, including the COCO 2016 Challenge winner. We hope that our simple and effective approach will serve as a solid foundation to help simplify future studies of instance level recognition. We will expose the code.

Mask_RCNN Keras

This is an implementation of Mask R-CNN based on Python 3, Keras and TensorFlow. This model generates bounding boxes and segmentation masks for each object instance in the image. It is implemented on top of Feature Pyramid Network (FPN) and ResNet101.



This project includes source code for Mask r-cnn built on FPN and ResNet101. – MS COCO Training code – Pre-trained MS COCO Weights – Use Jupyter Notebooks to visualize every step of the detection process – ParallelModel Class for multi-GPU training – MS COCO Indicator Evaluation (AP) – Examples for training your own data set

Code is organized and designed to be easily extended. If you use this code in your research, consider referencing this project. If you work in 3D vision, you may find our recently released Matterport3D dataset also useful

start

  • The easiest thing to do with Demo.ipynb is to start here. Here is a demonstration of how to segment your images with a pre-trained MS COCO model. It contains the code of object detection and entity segmentation for any picture.
  • Train_shapes. Ipynb demonstrates how to train Mask R-CNN on your own data set. The Notebook uses a simple simulation to show how it can be used on new data sets.
  • (model.py, utils.py, config.py): These files are the main implementation part of Mask RCNN
  • Inspect_data.ipynb. This notebook visualization is consisted in several different data processing steps.
  • Inspect_model. ipynb This notebook in-depth code demonstrates object detection and segmentation and provides a visualization of each step.
  • Inspect_weights. ipynb This notebook is used to observe the weights of the training model and check for special cases.

Step by step

Ipynb, inspect_model.ipynb, and inspect_weights. Ipynb. It also allows you to see the output of each point step by step. Here are some examples.

1. Anchor sorting and filtering

The first step is visualization of the candidate Region Proposal network and positive anchor points and negative anchor points along Anchor box Refinement.

2. Boundingbox streamline

This is an example of the final detection area (dashed line) and the improved result (solid line).

3. The mask is generated

Examples of generated masks. These masks are then scaled and placed in the appropriate areas of the image.

4. Visualization of activation values for each layer

It is often helpful to look at different layers of activation to solve some of the problems encountered in detection. (All zero, or random noise)

5. Histogram of weights

Another debugging trick is to look at the histogram of weights. This is inspect_weights.ipynb notebook. Give the demo effect.

6. TensorBoard visualization

Another very important debugging and visualization tool is TensorBoard. This model is configured to record changes in loss values and save weights after each epoch.

7. Combine to get the final result

Training on MS COCO

We provide pre-training weights to make it easier for you to get started. You can also use these weights to train your model for seven points. The training and verification code is in cox.py. You can import this module in notebook or run it directly from the command line.

Train a new model starting from pre-trained COCO weights python3 coco.py train --dataset=/path/to/coco/ --model=coco Train a new model starting from ImageNet weights python3 coco.py train --dataset=/path/to/coco/ --model=imagenet Continue training a model that you had trained earlier python3 coco.py train --dataset=/path/to/coco/ --model=/path/to/weights.h5 Continue training the last model you trained. This will find the last trained weights in the  model directory. python3 coco.py train --dataset=/path/to/coco/ --model=lastCopy the code


You can also run the COCO verification code like this:

The training optimization method, learning rate, and other parameters are set in coco-.py.

Python installs dependency packages

  • Python 3.4+ • TensorFlow 1.3+ • Keras 2.0.8+ • Jupyter Notebook • Numpy, skimage, scipy, Pillow

MS COCO requirements:

To train MS COCO, you will need the following:

  • pycocotools (installation instructions below)

  • MS COCO Dataset
  • Download the 5K minival and the 35K validation-minus-minival subsets. More details in the original Faster R-CNN implementation. If you use Docker, the code is already validated in a Docker Container.

The installation

  • Clone this repository

  • Download pre-trained COCO weights (mask_rcnn_coco.h5) from the releases page.

  • (Optional) To train or test on MS COCO install Pycocotools from one of these repos. They are forks of the original Pycocotools with fixes for Python3 and Windows (the official repo doesn’t seem to be active anymore).
  • Linux: https://github.com/waleedka/coco

  • Windows: https://github.com/philferriere/cocoapi. You must have the Visual C++ 2015 build tools on your path (see the repo for additional details)

More examples


  1. Keras [https://github.com/matterport/Mask_RCNN\]

  2. TensorFlow [https://github.com/CharlesShang/FastMaskRCNN]

  3. Pytorch [https://github.com/felixgwu/mask_rcnn_pytorch\]

  4. caffe [https://github.com/jasjeetIM/Mask-RCNN]

  5. MXNet [https://github.com/TuSimple/mx-maskrcnn]



Mask R-CNN Keras/TensorFlow/Pytorch code implementation