### Lei Mao

Machine Learning, Artificial Intelligence, Computer Science.

### Introduction

The introduction of Google DeepDream and its application in digital art can be found in my article “Google DeepDream in Python”.

### Description

The development of this API is still in progress. More functions will be added in the future.

• Python 3.6

### Dependencies

• tensorflow 1.3
• numpy
• PIL
• os
• sys
• zipfile
• six
• argparse

### Usage

Here I am using Georges Seurat’s “Sunday Afternoon On The Island Of La Grande Jatte” as an example to illustrate the usage of this Google DeepDream API.

$python deepdream_api.py --help usage: deepdream_api.py [-h] [-l | -p layer_name channel_number output_filename | -pl layer_name channel_number output_filename | -r image_path layer_name channel_number output_filename | -rl image_path layer_name channel_number output_filename] Designate function and keywords optional arguments: -h, --help show this help message and exit -l, --list List the available layers and the number of channels -p layer_name channel_number output_filename, --preview layer_name channel_number output_filename Preview the feature pattern of the neural network -pl layer_name channel_number output_filename, --previewlap layer_name channel_number output_filename Preview the feature pattern of the neural network with Laplacian Pyramid Gradient Normalization -r image_path layer_name channel_number output_filename, --render image_path layer_name channel_number output_filename Render the image with the features from the neural network -rl image_path layer_name channel_number output_filename, --renderlap image_path layer_name channel_number output_filename Render the image with the features from the neural network with Laplacian Pyramid Gradient Normalization  #### List the available layers and the number of channels To check the available layer names and channel numbers for the deepdream program. Input $ python deepdream_api.py -l


Output

import/conv2d0_pre_relu/conv 64
import/conv2d1_pre_relu/conv 64
import/conv2d2_pre_relu/conv 192
import/mixed3a_1x1_pre_relu/conv 64
import/mixed3a_3x3_bottleneck_pre_relu/conv 96
import/mixed3a_3x3_pre_relu/conv 128
import/mixed3a_5x5_bottleneck_pre_relu/conv 16
import/mixed3a_5x5_pre_relu/conv 32
import/mixed3a_pool_reduce_pre_relu/conv 32
import/mixed3b_1x1_pre_relu/conv 128
import/mixed3b_3x3_bottleneck_pre_relu/conv 128
import/mixed3b_3x3_pre_relu/conv 192
import/mixed3b_5x5_bottleneck_pre_relu/conv 32
...


In the output, the layer name is on the left, the number of channels in the layer is on the right.

#### Preview the feature pattern of the neural network

To preview the feature pattern learned in a certain channel of a certain layer in the neural network. This is helpful for the user to select layers and channels used for image modification. While not cessary, you may also preview all the feature patterns learned in the neural network here. It should be noted that the high frequencies of the patterns might have been suppressed by Laplacian pyramid decomposition in those images.

Input

$python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern.jpeg  Output #### Preview the feature pattern of the neural network To preview the feature pattern learned in a certain channel of a certain layer in the neural network with Laplacian Pyramid Gradient Normalization. High frequency patterns were suppressed by Laplacian Pyramid Gradient Normalization. It should be noted that the feature No.70 in “mixed4b_1x1_pre_relu” layer I generated in this example is almost the same to the one archieved in Google’s server. Input $ python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern_lap.jpeg


Output

#### Render the image with the features from the neural network

Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided.

Input

$python deepdream_api.py -r inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream.jpeg  Output #### Render the image with the features from the neural network with Laplacian Pyramid Gradient Normalization Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided. Input $ python deepdream_api.py -rl inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream_lap.jpeg


Output

### More Functions

I also tried to add the features from a guide image to customize the pattern. However, it seems that my understanding of the optimzation objective is not correct (see my post on StackFlow). Therefore, I will hold this until I totally resolve my confusions.

#### 1/26/2018

I was looking at the Caffe source code, which contains the customization of patterns, this morning. I think I understand what they were really doing and will update the program once I got some casual time.

end = 'inception_3b/output'
h, w = guide.shape[:2]
src, dst = net.blobs['data'], net.blobs[end]
src.reshape(1,3,h,w)
src.data[0] = preprocess(net, guide)
net.forward(end=end)
guide_features = dst.data[0].copy()

def objective_guide(dst):
x = dst.data[0].copy()
y = guide_features
ch = x.shape[0]
x = x.reshape(ch,-1)
y = y.reshape(ch,-1)
A = x.T.dot(y) # compute the matrix of dot-products with guide features
dst.diff[0].reshape(ch,-1)[:] = y[:,A.argmax(1)] # select ones that match best

_=deepdream(net, img, end=end, objective=objective_guide)


They first exacted and saved the neural network output of guided image at certain layer, and the dimension of this output tensor was, say, [64,20,20]. Then they extracted the neural network output of another image which they wished to modify at the same layer, and the dimension of this output tensor was, say, [64,30,30]. Then ch = 64. After reshaping, x becomes [64,400], and y becomes [64,900]. Therefore, A is a matrix of size [400,900]. A.argmax(1) returns an array of size [400], each element in the array ranges from 0 to 900. y[:,A.argmax(1)] is therefore a tensor of shape [64,400]. This will be the gradient to x[64,400] (If I understand it correctly, in Caffe, dst.diff specifies gradient of the last layer). Caffe could specify the gradient of the last layer directly, I need to figure out how to do it in TensorFlow or thinking of a loss function to optimize using TensorFlow autograd.

It’s really annoying that I do not have time to do these interesting things. So many things bothered me.