Lei Mao bio photo

Lei Mao

Machine Learning, Artificial Intelligence, Computer Science.

Twitter Facebook LinkedIn GitHub   G. Scholar E-Mail RSS

Google DeepDream API

Introduction

The introduction of Google DeepDream and its application in digital art can be found in my article “Google DeepDream in Python”.

Description

This API could add pattern features learned by Google DeepDream to your paintings.


The development of this API is still in progress. More functions will be added in the future.

Requirements

  • Python 3.6

Dependencies

  • tensorflow 1.3
  • numpy
  • PIL
  • os
  • sys
  • zipfile
  • six
  • argparse

Usage

Here I am using Georges Seurat’s “Sunday Afternoon On The Island Of La Grande Jatte” as an example to illustrate the usage of this Google DeepDream API.

$ python deepdream_api.py --help
usage: deepdream_api.py [-h]
                        [-l | -p layer_name channel_number output_filename | -pl layer_name channel_number output_filename | -r image_path layer_name channel_number output_filename | -rl image_path layer_name channel_number output_filename]

Designate function and keywords

optional arguments:
  -h, --help            show this help message and exit
  -l, --list            List the available layers and the number of channels
  -p layer_name channel_number output_filename, --preview layer_name channel_number output_filename
                        Preview the feature pattern of the neural network
  -pl layer_name channel_number output_filename, --previewlap layer_name channel_number output_filename
                        Preview the feature pattern of the neural network with
                        Laplacian Pyramid Gradient Normalization
  -r image_path layer_name channel_number output_filename, --render image_path layer_name channel_number output_filename
                        Render the image with the features from the neural
                        network
  -rl image_path layer_name channel_number output_filename, --renderlap image_path layer_name channel_number output_filename
                        Render the image with the features from the neural
                        network with Laplacian Pyramid Gradient Normalization

List the available layers and the number of channels

To check the available layer names and channel numbers for the deepdream program.


Input

$ python deepdream_api.py -l

Output

import/conv2d0_pre_relu/conv 64
import/conv2d1_pre_relu/conv 64
import/conv2d2_pre_relu/conv 192
import/mixed3a_1x1_pre_relu/conv 64
import/mixed3a_3x3_bottleneck_pre_relu/conv 96
import/mixed3a_3x3_pre_relu/conv 128
import/mixed3a_5x5_bottleneck_pre_relu/conv 16
import/mixed3a_5x5_pre_relu/conv 32
import/mixed3a_pool_reduce_pre_relu/conv 32
import/mixed3b_1x1_pre_relu/conv 128
import/mixed3b_3x3_bottleneck_pre_relu/conv 128
import/mixed3b_3x3_pre_relu/conv 192
import/mixed3b_5x5_bottleneck_pre_relu/conv 32
...

In the output, the layer name is on the left, the number of channels in the layer is on the right.

Preview the feature pattern of the neural network

To preview the feature pattern learned in a certain channel of a certain layer in the neural network. This is helpful for the user to select layers and channels used for image modification. While not cessary, you may also preview all the feature patterns learned in the neural network here. It should be noted that the high frequencies of the patterns might have been suppressed by Laplacian pyramid decomposition in those images.


Input

$ python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern.jpeg

Output

Feature Pattern Learned in the Neural Layers

Preview the feature pattern of the neural network

To preview the feature pattern learned in a certain channel of a certain layer in the neural network with Laplacian Pyramid Gradient Normalization. High frequency patterns were suppressed by Laplacian Pyramid Gradient Normalization.


It should be noted that the feature No.70 in “mixed4b_1x1_pre_relu” layer I generated in this example is almost the same as the one archieved in Google’s server.


Input

$ python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern_lap.jpeg

Output

Feature Pattern Learned in the Neural Layers

Render the image with the features from the neural network

Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided.


Input

$ python deepdream_api.py -r inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream.jpeg

Output

Render the image with the features from the neural network with Laplacian Pyramid Gradient Normalization

Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided.


Input

$ python deepdream_api.py -rl inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream_lap.jpeg

Output

More Functions

I also tried to add the features from a guide image to customize the pattern. However, it seems that my understanding of the optimzation objective is not correct (see my post on StackFlow). Therefore, I will hold this until I totally resolve my confusions.

Thought Updates

1/26/2018

I was looking at the Caffe source code, which contains the customization of patterns, this morning. I think I understand what they were really doing and will update the program once I got some casual time.

end = 'inception_3b/output'
h, w = guide.shape[:2]
src, dst = net.blobs['data'], net.blobs[end]
src.reshape(1,3,h,w)
src.data[0] = preprocess(net, guide)
net.forward(end=end)
guide_features = dst.data[0].copy()

def objective_guide(dst):
    x = dst.data[0].copy()
    y = guide_features
    ch = x.shape[0]
    x = x.reshape(ch,-1)
    y = y.reshape(ch,-1)
    A = x.T.dot(y) # compute the matrix of dot-products with guide features
    dst.diff[0].reshape(ch,-1)[:] = y[:,A.argmax(1)] # select ones that match best

_=deepdream(net, img, end=end, objective=objective_guide)

They first exacted and saved the neural network output of guided image at certain layer, and the dimension of this output tensor was, say, [64,20,20]. Then they extracted the neural network output of another image which they wished to modify at the same layer, and the dimension of this output tensor was, say, [64,30,30]. Then ch = 64. After reshaping, x becomes [64,400], and y becomes [64,900]. Therefore, A is a matrix of size [400,900]. A.argmax(1) returns an array of size [400], each element in the array ranges from 0 to 900. y[:,A.argmax(1)] is therefore a tensor of shape [64,400]. This will be the gradient to x[64,400] (If I understand it correctly, in Caffe, dst.diff specifies gradient of the last layer). Caffe could specify the gradient of the last layer directly, I need to figure out how to do it in TensorFlow or thinking of a loss function to optimize using TensorFlow autograd.


It’s really annoying that I do not have time to do these interesting things. So many things bothered me.

References

GitHub