Inspired by cubism and various projects using genetic algorithms to paint the Mona Lisa here a method for teaching your computer to be an artist!

I bring you the artificial artist!

The idea is to use regression to "learn" what an image looks like and then draw the learned image. By choosing the parameters of the regressor carefully you can achieve some interesting visual effects.

If all this artsy talk is too much for you think of this as a way to compress an image. You could store the weights of the regressor instead of the whole image.

First some standard imports of things we will need later:

In [16]:

%matplotlib inline

In [17]:

from base64 import b64encode
from tempfile import NamedTemporaryFile

import numpy as np
import scipy
import matplotlib.pyplot as plt
from matplotlib import animation
from IPython.display import HTML
from sklearn.ensemble import RandomForestRegressor as RFR
from skimage.io import imread
from skimage.color import rgb2lab,lab2rgb
from skimage.transform import resize,rescale

from JSAnimation import IPython_display

Pretty pictures¶

First, some pictures to work with. The first one was taken during a trip to the Lake district in north England. The second one is a shot of the harbour in St. John's in Newfoundland.

In [18]:

lakes = imread('http://betatim.github.io/images/artificial-arts/lakes.jpg')
f,ax = plt.subplots(figsize=(10,6))
ax.xaxis.set_ticks([]); ax.yaxis.set_ticks([])
ax.imshow(lakes, aspect='auto')

Out[18]:

<matplotlib.image.AxesImage at 0x11e47a850>

In [19]:

newfie = imread('http://betatim.github.io/images/artificial-arts/newfie.jpg')
f,ax = plt.subplots(figsize=(10,6))
ax.xaxis.set_ticks([]); ax.yaxis.set_ticks([])
ax.imshow(newfie, aspect='auto')

Out[19]:

<matplotlib.image.AxesImage at 0x122aadf50>

The Artificial Artist: a new Kind of Instagram Filter¶

The artificial artist will be based on a decision tree regressor using the $(x,y)$ coordinates of each pixel in the image as features and the RGB values as the target. Once the tree has been trained we ask it to make a prediction for every pixel in the image, this is our image with the "filter" applied. All this is taken care of in the simple_cubist function. Once you see the first filtered image you will understand why it is called simple_cubist.

We also define a compare function which takes care of displaying several images next to each other or compiling them into an animation. This makes it easy to see what we just did.

In [20]:

def simple_cubist(img):
    w,h = img.shape[:2]
    img = rgb2lab(img)
    
    xx,yy = np.meshgrid(np.arange(w), np.arange(h))
    X = np.column_stack((xx.reshape(-1,1), yy.reshape(-1,1)))
                        
    Y = img.reshape(-1,3, order='F')
    
    min_samples = int(round(0.001 * len(X)))

    model = RFR(n_estimators=1,
                n_jobs=6,
                max_depth=None,
                min_samples_leaf=min_samples,
                random_state=43252,
                )
    model.fit(X, Y)
    art_img = model.predict(X)
    art_img = art_img.reshape(w,h,3, order='F')
    
    return lab2rgb(art_img)

def compare(*imgs, **kwds):
    """Draw several images at once for easy comparison"""
    animate = kwds.get("animate", False)
    
    if animate:
        fig, ax = plt.subplots(figsize=(8,5))
        ax.xaxis.set_ticks([])
        ax.yaxis.set_ticks([])
        anim = animation.FuncAnimation(fig, 
                                       lambda x: ax.imshow(imgs[x%len(imgs)],
                                                           aspect='auto'),
                                       frames=len(imgs),
                                       interval=1000)
        
        fig.tight_layout()
        return anim
    
    else:
        figsize = plt.figaspect(len(imgs)) * 1.5
        fig, axes = plt.subplots(nrows=len(imgs), figsize=figsize)
        for a in axes:
            a.xaxis.set_ticks([])
            a.yaxis.set_ticks([])

        for ax,img in zip(axes,imgs):
            ax.imshow(img)

        fig.tight_layout()
    
# Take the picture of the Lake District and apply our
# simple cubist filter to it
simple_lakes = simple_cubist(lakes)
compare(lakes, simple_lakes, animate=True)

Out[20]:

Click play on the animation to see the original image and the filtered one alternate. Do you recognise the scene in the second image? Looks like a blocky (or cubist!) version of the original.

To understand why we end up with rectangles we need to learn a bit about how a decision tree works. It will try and split the area of the image into regions (the blobs in the output image) which have a similar colour and for each pixel in a region it will predict the same colour.

The decision at each node of the tree can only be one of these two $x < z$ or $y < z$ where $x$ and $y$ are the coordinates of each pixel and $z$ is a value the learning algorithm can choose. As a result the tree can only make rectangular decision regions and we end up with a nice visual effect. A bit like a cubist painting.

Higher Powers¶

To get more complicated shapes for the decision regions (or blobs in the filtered image) we can provide more interesting features. The artistic function below will compute additional features like: $x^p \pm y^p$. By setting the powers parameter to zero, no new features are added. With $p=1$ the sum ($x^1+y^1 = x+y$) and difference ($x^1-y^1 = x-y$) of each pixel's coordinate is added as a feature. This will allow the tree to make decision regions with sloping sides. Larger values of $p$ add more complicated features, leading to more complicated shapes of the decision regions.

Let's give it a spin for the Newfoundland harbour image. The artistic function below allows you to cofigure the number of additional features (the powers parameter), as well as a few other things which we will get to later.

In [21]:

def artistic(img,
             frac=0.001,
             depth=None,
             n_estimators=1,
             powers=1):
    w,h = img.shape[:2]
    img = rgb2lab(img)
    
    xx,yy = np.meshgrid(np.arange(w), np.arange(h))
    X = np.column_stack((xx.reshape(-1,1), yy.reshape(-1,1)))
    X = np.column_stack((X,) +
                        tuple(np.power(X[:,0], p) + np.power(X[:,1], p)
                              for p in range(1,powers+1)) +
                        tuple(np.power(X[:,0], p) - np.power(X[:,1], p)
                              for p in range(1,powers+1))
                        )
                        
    Y = img.reshape(-1,3, order='F')
    
    min_samples = int(round(frac * len(X)))

    model = RFR(n_estimators=n_estimators,
                n_jobs=6,
                max_depth=depth,
                min_samples_leaf=min_samples,
                random_state=43252,
                )
    model.fit(X, Y)
    art_img = model.predict(X)
    art_img = art_img.reshape(w,h,3, order='F')
    
    return lab2rgb(art_img)

Let's see what the image looks like with an increasing number of features:

In [22]:

newfie_arts = [artistic(newfie, powers=n) for n in range(4)]
compare(newfie, *newfie_arts, animate=True)

Out[22]:

As you increase the number of features you get more complicated shaped blobs. Can you spot the tell tale shape of a parabola? We could also add features like $m\cdot x + n\cdot y$, these would give us slopes at an angle different from 45 degrees.

More or Less Details¶

Two of the remaining parameters influence the level of detail in the filtered image.

First, the fraction parameter. It determines the smallest number of pixels that have to be within one of the blobs before it can not be split any further. As the name suggests it is expressed as a fraction of the total number of pixels in the image. By making it smaller you get smaller patches, and if you keep going you will get one pixel per patch ... as in you just get back the original image.

The next animation compares the original image, with one rendered with fraction=0.001 and one with fraction=0.0001.

In [23]:

lakes_art2 = artistic(lakes, frac=0.0001, powers=0)
compare(lakes, simple_lakes, lakes_art2, animate=True)

Out[23]:

The second parameter you can tune is the depth of the decision tree. It limits how many decision steps the tree is allowed to have. So with a depth of 2 you can only get four regions (of probably different colours). The default value (None) allows the decision tree to decide itself when enough is enough.

Watch the tree come up with increasingly more complicated shapes as the depth is increased. If wondered about how a decision tree works this is quite a cool animation. You can see that it keeps refining the decisions regions by splitting them again and again. Below one animation for the harbour scene and one for the picture from the Lake District:

In [24]:

newfie_arts2 = [artistic(newfie, powers=1, depth=d) for d in (2,3,4,5,6,7,8,9,10,12,14,16,20)]
compare(newfie, *newfie_arts2, animate=True)

Out[24]:

In [25]:

lakes_arts2 = [artistic(lakes, powers=1, depth=d) for d in (2,3,4,5,6,7,8,9,10,12,14,16,20,24,28)]
compare(lakes, *lakes_arts2, animate=True)

Out[25]:

Smoothing¶

So far the boundaries between blobs in the image are abrupt, fear not, the artificial artist has a solution. So far we only trained one decision tree, and get sharp boundaries between the decision regions.

To get smoother boundaries you can train several decision trees and average them! There is some randomness involved with training a decision tree, as a result you will a slightly different decision tree every time you train a new one even if the input is axactly the same.

In [26]:

newfie_smoothed = [artistic(newfie, frac=0.001,
                            powers=2, depth=20, n_estimators=N) for N in (1,40)]
compare(newfie, *newfie_smoothed, animate=True)

Out[26]: