Deep Learning (14) - Neural Style Transfer

Posted on
Edit


这是当下很流行的一个玩法。下面就是一个例子：


将原本的风景照片融入梵高星空画作的风格。还有时下国内很流行的脸萌，化妆的app，应该也是基于这个实现的。

算法步骤

要合成这样一张混合风格的图片的步骤大致如下：

  1. Initiate G randomly
  2. Use gradient descent to minimize J(G)

其中:

  • , 是超参数
  • C代表Content，S代表Style，G代表Generated

  • Say you use hidden layer l to compute content cost.
  • Use pre-trained ConvNet. (E.g., VGG network)
  • Let and be the activation of layer l on the images
  • If and are similar, both images have similar content

图片的风格往往隐含在通道中。我们先看看一个深层卷积神经网络，每一层激活函数输出的数据是什么样的。


上图是网络层有浅及深的数据片段。其中每9个格子代表一个神经元输出。如果放大看，可以发现，浅层网络关注的都是一些局部的线条或者图案。越往深层，图像越具体，3/4/5层已经可以看到具体的物体形态了。
那么，计算风格损失函数，就会是下面这样一个过程：

  1. 针对第l层，计算其风格损失函数

  1. 将该层激活输出的volume切片，计算Style Image与Generated Image的correlation
  2. 计算损失函数
  3. 总体的风格损失函数就是：
    其中是一个超参数，用来调节每一层的权重。

参考文献

Neural Style Transfer: Gatys et al., 2015, A neural algorithm of artistic style

%23%20Deep%20Learning%20%2814%29%20-%20Neural%20Style%20Transfer%0A@%28myblog%29%5Bdeep%20learning%2C%20machine%20learning%5D%0A%u8FD9%u662F%u5F53%u4E0B%u5F88%u6D41%u884C%u7684%u4E00%u4E2A%u73A9%u6CD5%u3002%u4E0B%u9762%u5C31%u662F%u4E00%u4E2A%u4F8B%u5B50%uFF1A%0A%21%5BAlt%20text%7C300x0%5D%28./1538285228968.png%29%0A%u5C06%u539F%u672C%u7684%u98CE%u666F%u7167%u7247%u878D%u5165%u68B5%u9AD8%u661F%u7A7A%u753B%u4F5C%u7684%u98CE%u683C%u3002%u8FD8%u6709%u65F6%u4E0B%u56FD%u5185%u5F88%u6D41%u884C%u7684%u8138%u840C%uFF0C%u5316%u5986%u7684app%uFF0C%u5E94%u8BE5%u4E5F%u662F%u57FA%u4E8E%u8FD9%u4E2A%u5B9E%u73B0%u7684%u3002%0A%0A%23%23%20%u7B97%u6CD5%u6B65%u9AA4%0A%u8981%u5408%u6210%u8FD9%u6837%u4E00%u5F20%u6DF7%u5408%u98CE%u683C%u7684%u56FE%u7247%u7684%u6B65%u9AA4%u5927%u81F4%u5982%u4E0B%uFF1A%0A1.%20Initiate%20G%20randomly%0A2.%20Use%20gradient%20descent%20to%20minimize%20J%28G%29%0A%0A%u5176%u4E2D%3A%0A%24%24J%28G%29%20%3D%20%5Calpha%20J_%7Bcontent%7D%28C%2CG%29%20+%20%5Cbeta%20J_%7Bstyle%7D%28S%2CG%29%24%24%0A-%20%24%5Calpha%24%2C%20%24%5Cbeta%24%u662F%u8D85%u53C2%u6570%0A-%20C%u4EE3%u8868Content%uFF0CS%u4EE3%u8868Style%uFF0CG%u4EE3%u8868Generated%0A%0A%23%23%20%24J_%7Bcontent%7D%28C%2CG%29%24%0A-%20Say%20you%20use%20hidden%20layer%20l%20to%20compute%20content%20cost.%0A-%20Use%20pre-trained%20ConvNet.%20%28E.g.%2C%20VGG%20network%29%0A-%20Let%20%24a%5E%7B%5Bl%5D%28C%29%7D%24%20and%20%24a%5E%7B%5Bl%5D%28G%29%7D%24%20be%20the%20activation%20of%20layer%20l%20on%20the%20images%0A-%20If%20%24a%5E%7B%5Bl%5D%28C%29%7D%24%20and%20%24a%5E%7B%5Bl%5D%28G%29%7D%24%20are%20similar%2C%20both%20images%20have%20similar%20content%0A%09-%20%24J_%7Bcontent%7D%28C%2CG%29%20%3D%20%5Cfrac%20%7B1%7D%7B2%7D%20%5Cbegin%7BVmatrix%7D%20a%5E%7B%5Bl%5D%28C%29%7D%20-%20a%5E%7B%5Bl%5D%28G%29%7D%5Cend%7BVmatrix%7D%5E2%24%0A%0A%23%23%20%24J_%7Bstyle%7D%28S%2CG%29%24%0A%u56FE%u7247%u7684%u98CE%u683C%u5F80%u5F80%u9690%u542B%u5728%u901A%u9053%u4E2D%u3002%u6211%u4EEC%u5148%u770B%u770B%u4E00%u4E2A%u6DF1%u5C42%u5377%u79EF%u795E%u7ECF%u7F51%u7EDC%uFF0C%u6BCF%u4E00%u5C42%u6FC0%u6D3B%u51FD%u6570%u8F93%u51FA%u7684%u6570%u636E%u662F%u4EC0%u4E48%u6837%u7684%u3002%0A%21%5BAlt%20text%5D%28./1538292610814.png%29%0A%u4E0A%u56FE%u662F%u7F51%u7EDC%u5C42%u6709%u6D45%u53CA%u6DF1%u7684%u6570%u636E%u7247%u6BB5%u3002%u5176%u4E2D%u6BCF9%u4E2A%u683C%u5B50%u4EE3%u8868%u4E00%u4E2A%u795E%u7ECF%u5143%u8F93%u51FA%u3002%u5982%u679C%u653E%u5927%u770B%uFF0C%u53EF%u4EE5%u53D1%u73B0%uFF0C%u6D45%u5C42%u7F51%u7EDC%u5173%u6CE8%u7684%u90FD%u662F%u4E00%u4E9B%u5C40%u90E8%u7684%u7EBF%u6761%u6216%u8005%u56FE%u6848%u3002%u8D8A%u5F80%u6DF1%u5C42%uFF0C%u56FE%u50CF%u8D8A%u5177%u4F53%uFF0C3/4/5%u5C42%u5DF2%u7ECF%u53EF%u4EE5%u770B%u5230%u5177%u4F53%u7684%u7269%u4F53%u5F62%u6001%u4E86%u3002%0A%u90A3%u4E48%uFF0C%u8BA1%u7B97%u98CE%u683C%u635F%u5931%u51FD%u6570%uFF0C%u5C31%u4F1A%u662F%u4E0B%u9762%u8FD9%u6837%u4E00%u4E2A%u8FC7%u7A0B%uFF1A%0A%21%5BAlt%20text%7C700x0%5D%28./1538292884877.png%29%0A1.%20%u9488%u5BF9%u7B2Cl%u5C42%uFF0C%u8BA1%u7B97%u5176%u98CE%u683C%u635F%u5931%u51FD%u6570%0A%21%5BAlt%20text%7C500x0%5D%28./1538292978497.png%29%0A2.%20%u5C06%u8BE5%u5C42%u6FC0%u6D3B%u8F93%u51FA%u7684volume%u5207%u7247%uFF0C%u8BA1%u7B97Style%20Image%u4E0EGenerated%20Image%u7684correlation%0A%09-%20%24G_%7Bkk%27%7D%5E%7B%5Bl%5D%28S%29%7D%20%3D%20%5CSigma_%7Bi%3D1%7D%5E%7Bn_H%5E%7B%5Bl%5D%7D%7D%20%5CSigma_%7Bj%3D1%7D%5E%7B%5Bl%5D%7D%20a_%7Bijk%7D%5E%7B%5Bl%5D%28S%29%7D%20a_%7Bijk%27%7D%5E%7B%5Bl%5D%28S%29%7D%24%0A%09-%20%24G_%7Bkk%27%7D%5E%7B%5Bl%5D%28G%29%7D%20%3D%20%5CSigma_%7Bi%3D1%7D%5E%7Bn_H%5E%7B%5Bl%5D%7D%7D%20%5CSigma_%7Bj%3D1%7D%5E%7B%5Bl%5D%7D%20a_%7Bijk%7D%5E%7B%5Bl%5D%28G%29%7D%20a_%7Bijk%27%7D%5E%7B%5Bl%5D%28G%29%7D%24%0A3.%20%u8BA1%u7B97%u635F%u5931%u51FD%u6570%0A%09%24%24%5Cbegin%7Balign%7D%20J_%7Bstyle%7D%5E%7B%5Bl%5D%7D%28S%2C%20G%29%20%26%3D%20%5Ccfrac%20%7B1%7D%7B%282n_H%5E%7B%5Bl%5D%7Dn_W%5E%7B%5Bl%5D%7Dn_C%5E%7B%5Bl%5D%7D%29%5E2%7D%20%5Cbegin%7BVmatrix%7D%20G%5E%7B%5Bl%5D%28S%29%7D%20-%20G%5E%7B%5Bl%5D%28G%29%7D%5Cend%7BVmatrix%7D_F%5E2%20%5C%5C%20%26%3D%20%5Ccfrac%20%7B1%7D%7B%282n_H%5E%7B%5Bl%5D%7Dn_W%5E%7B%5Bl%5D%7Dn_C%5E%7B%5Bl%5D%7D%29%5E2%7D%20%5CSigma_k%20%5CSigma_%7Bk%27%7D%28G_%7Bkk%27%7D%5E%7B%5Bl%5D%28S%29%7D%20-%20G_%7Bkk%27%7D%5E%7B%5Bl%5D%28G%29%7D%29%5E2%5Cend%7Balign%7D%24%24%0A4.%20%u603B%u4F53%u7684%u98CE%u683C%u635F%u5931%u51FD%u6570%u5C31%u662F%uFF1A%0A%24%24J_%7Bstyle%7D%28S%2CG%29%20%3D%20%5CSigma_l%5Clambda%5E%7B%5Bl%5D%7D%20J_%7Bstyle%7D%5E%7B%5Bl%5D%7D%28S%2CG%29%24%24%0A%u5176%u4E2D%24%5Clambda%5E%7B%5Bl%5D%7D%24%u662F%u4E00%u4E2A%u8D85%u53C2%u6570%uFF0C%u7528%u6765%u8C03%u8282%u6BCF%u4E00%u5C42%u7684%u6743%u91CD%u3002%0A%0A%23%23%20%u53C2%u8003%u6587%u732E%0ANeural%20Style%20Transfer%3A%20Gatys%20et%20al.%2C%202015%2C%20A%20neural%20algorithm%20of%20artistic%20style%0A%0A%0A%0A%0A