Introduction
The introduction of Google DeepDream and its application in digital art can be found in my article “Google DeepDream in Python”.
Description
This API could add pattern features learned by Google DeepDream to your paintings.
The development of this API is still in progress. More functions will be added in the future.
Requirements
- Python 3.6
Dependencies
- tensorflow 1.3
- numpy
- PIL
- os
- sys
- zipfile
- six
- argparse
Usage
Here I am using Georges Seurat’s “Sunday Afternoon On The Island Of La Grande Jatte” as an example to illustrate the usage of this Google DeepDream API.
$ python deepdream_api.py --help
usage: deepdream_api.py [-h]
[-l | -p layer_name channel_number output_filename | -pl layer_name channel_number output_filename | -r image_path layer_name channel_number output_filename | -rl image_path layer_name channel_number output_filename]
Designate function and keywords
optional arguments:
-h, --help show this help message and exit
-l, --list List the available layers and the number of channels
-p layer_name channel_number output_filename, --preview layer_name channel_number output_filename
Preview the feature pattern of the neural network
-pl layer_name channel_number output_filename, --previewlap layer_name channel_number output_filename
Preview the feature pattern of the neural network with
Laplacian Pyramid Gradient Normalization
-r image_path layer_name channel_number output_filename, --render image_path layer_name channel_number output_filename
Render the image with the features from the neural
network
-rl image_path layer_name channel_number output_filename, --renderlap image_path layer_name channel_number output_filename
Render the image with the features from the neural
network with Laplacian Pyramid Gradient Normalization
List the available layers and the number of channels
To check the available layer names and channel numbers for the deepdream program.
Input
$ python deepdream_api.py -l
Output
import/conv2d0_pre_relu/conv 64
import/conv2d1_pre_relu/conv 64
import/conv2d2_pre_relu/conv 192
import/mixed3a_1x1_pre_relu/conv 64
import/mixed3a_3x3_bottleneck_pre_relu/conv 96
import/mixed3a_3x3_pre_relu/conv 128
import/mixed3a_5x5_bottleneck_pre_relu/conv 16
import/mixed3a_5x5_pre_relu/conv 32
import/mixed3a_pool_reduce_pre_relu/conv 32
import/mixed3b_1x1_pre_relu/conv 128
import/mixed3b_3x3_bottleneck_pre_relu/conv 128
import/mixed3b_3x3_pre_relu/conv 192
import/mixed3b_5x5_bottleneck_pre_relu/conv 32
...
In the output, the layer name is on the left, the number of channels in the layer is on the right.
Preview the feature pattern of the neural network
To preview the feature pattern learned in a certain channel of a certain layer in the neural network. This is helpful for the user to select layers and channels used for image modification. While not cessary, you may also preview all the feature patterns learned in the neural network here. It should be noted that the high frequencies of the patterns might have been suppressed by Laplacian pyramid decomposition in those images.
Input
$ python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern.jpeg
Output

Preview the feature pattern of the neural network
To preview the feature pattern learned in a certain channel of a certain layer in the neural network with Laplacian Pyramid Gradient Normalization. High frequency patterns were suppressed by Laplacian Pyramid Gradient Normalization.
It should be noted that the feature No.70 in “mixed4b_1x1_pre_relu” layer I generated in this example is almost the same to the one archieved in Google’s server.
Input
$ python deepdream_api.py -p mixed4b_1x1_pre_relu 70 pattern_lap.jpeg
Output

Render the image with the features from the neural network
Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided.
Input
$ python deepdream_api.py -r inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream.jpeg

Output

Render the image with the features from the neural network with Laplacian Pyramid Gradient Normalization
Apply feature pattern learned in a certain channel of a certain layer in the neural network to the image that the user provided.
Input
$ python deepdream_api.py -rl inputs/sunday_afternoon.jpg mixed4b_1x1_pre_relu 70 sunday_afternoon_deepdream_lap.jpeg

Output

More Functions
I also tried to add the features from a guide image to customize the pattern. However, it seems that my understanding of the optimzation objective is not correct (see my post on StackFlow). Therefore, I will hold this until I totally resolve my confusions.
Thought Updates
1/26/2018
I was looking at the Caffe source code, which contains the customization of patterns, this morning. I think I understand what they were really doing and will update the program once I got some casual time.
end = 'inception_3b/output'
h, w = guide.shape[:2]
src, dst = net.blobs['data'], net.blobs[end]
src.reshape(1,3,h,w)
src.data[0] = preprocess(net, guide)
net.forward(end=end)
guide_features = dst.data[0].copy()
def objective_guide(dst):
x = dst.data[0].copy()
y = guide_features
ch = x.shape[0]
x = x.reshape(ch,-1)
y = y.reshape(ch,-1)
A = x.T.dot(y) # compute the matrix of dot-products with guide features
dst.diff[0].reshape(ch,-1)[:] = y[:,A.argmax(1)] # select ones that match best
_=deepdream(net, img, end=end, objective=objective_guide)
They first exacted and saved the neural network output of guided image at certain layer, and the dimension of this output tensor was, say, [64,20,20]
. Then they extracted the neural network output of another image which they wished to modify at the same layer, and the dimension of this output tensor was, say, [64,30,30]
. Then ch = 64
. After reshaping, x
becomes [64,400]
, and y
becomes [64,900]
. Therefore, A
is a matrix of size [400,900]
. A.argmax(1)
returns an array of size [400]
, each element in the array ranges from 0
to 900
. y[:,A.argmax(1)]
is therefore a tensor of shape [64,400]
. This will be the gradient to x[64,400]
(If I understand it correctly, in Caffe, dst.diff specifies gradient of the last layer). Caffe could specify the gradient of the last layer directly, I need to figure out how to do it in TensorFlow or thinking of a loss function to optimize using TensorFlow autograd.
It’s really annoying that I do not have time to do these interesting things. So many things bothered me.
References
- Google Official DeepDream Tutorial in Caffe
- Google Official DeepDream Tutorial in TensorFlow
- Google Research Blog
- Siraj Raval’s Video Tutorial
- Siraj Raval’s Code
- Concept of Octave
- SIFT