Texture Synthesis

What in this post is actually part of my computational photography homework, because I’m recently preparing for interviews, so for reviewing it, I re-implemented this method.


Texture synthesis is another very interesting application of image processing. What it does is, given a texture sample, generate new texture that is similar with the given sample, similar here means for human observer, the new generated texture appears to be the same kind of texture.

For example, if we have the following image.


what we want is to fill the black region using the same texture with middle part. A trivial solution is just copy the whole image (or part of it), and put it into the unfilled region, and the output image is something like this.


Actually we feel not that happy with this result, the blockiness is clearly perceivable, and looks like some tiles in the bathroom. We want the texture expands automatically.


This image looks better, although it is also kind of weird, but we can hardly tell that this image is made by tiling some tiles, it is more likely that to be a whole image. This is exactly what texture synthesis does.


function FindMatches(Template,SampleImage)
	ValidMask = 1s where Template is filled, 0s otherwise 
	GaussMask = Gaussian2D(WindowSize,Sigma)
	TotWeight = sum i,j GaussiMask(i,j)*ValidMask(i,j) 
	for i,j do
		for ii,jj do
			dist = (Template(ii,jj)-SampleImage(i-ii,j-jj))^2
			SSD(i,j) = SSD(i,j) + dist*ValidMask(ii,jj)*GaussMask(ii,jj)
		SSD(i,j) = SSD(i,j) / TotWeight 
	PixelList = all pixels (i,j)where SSD(i,j)<=min(SSD)*(1+ErrThreshold)
	return PixelList
function GrowImage(SampleImage,Image,WindowSize) 
	while Image not filled do
		progress = 0
		PixelList = GetUnfilledNeighbors(Image) 
		foreach Pixel in PixelList do
			Template = GetNeighborhoodWindow(Pixel) 
			BestMatches = FindMatches(Template, SampleImage) 
			BestMatch = RandomPick(BestMatches)
			if (BestMatch.error < MaxErrThreshold) then
				Pixel.value = BestMatch.value
		        progress = 1
		if progress == 0
			then MaxErrThreshold = MaxErrThreshold * 1.1 
	return Image

Something need to pay attention:

  1. Window size is manually selected, if the texture repeats, then we can use the unit size of the texture.
  2. Function GetUnfilledNeighbors() returns a list of all unfilled pixels that have filled pixels as their neighbors. The list is randomly permuted and then sorted by decreasing number of filled neighbor pixels.
  3. When calculating neighbors, we use the 8-direction neighbors.
  4. If the current pixel is at the edge of image (one or some of its neighbors doesn’t exist), then just flag the non-exist neighbors, and calculate regularly.
  5. GetNeigborhoodWindow() returns a window of size WindowSize around a given pixel (returns flagged windows when at edges).
  6. RandomPick() picks an element randomly from the list.
  7. Gaussian2D() generates a 2D Gaussian in a window of given a size centered in the center and with a given standard deviation (in pixels).
  8. The recommended constant values are: ErrThreshold = 0.1, MaxErrThreshold = 0.3, Sigma = WindowSize/6 (3sigma for each side).




Some results (including using former code)

1. radishes (100 * 100) VS expanded radishesss (111 * 111)

radishes   resultc1

2. black hole filling

   a4   a5

3. barbed wire fence (75 * 75) VS expanded baaaaarbed wire fence (225 * 225)

a6   a7

For testing this algorithm, you can use the Brodatz Textures, which includes 112 gray scale texture images.

Because the above github version code is today’s new implemented code, I didn’t include something like hole or foreground object removal or multi-channel image processing.

For implementing foreground object removal, just alter the map generating part, no matter hole removal or image expansion, all we need to do is set the part of we want to be zero in map.

For implementing 3-channel image texture synthesis, just split the image into 3 different channels, and do what we do with single channel image 3 times, and then merge the 3 channels result as the final result image.

Lastly, there are some people use image pyramid to do texture synthesis, although I never tried it, but I believe it is a good idea to synthesis the information of images on each layer of whatever pyramid, maybe I’ll try this the other day 🙂

This entry was posted in Algorithm, OpenCV, Vision and tagged , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.


  1. Posted January 23, 2016 at 6:53 pm | Permalink

    Currently it seems like Movable Type is the best blogging platform available right
    now. (from what I’ve read) Is that what you’re using on your blog?

  2. Valimo Ral
    Posted February 25, 2016 at 9:43 am | Permalink

    Hi. it was a very nice information. About pyramid based texture wrote by Heeger and Bergen…can you give more explanation about this:
    Match-Histogram (noise,texture)
    analysis-pyr = Make-Pyramid (texture)
    Loop for several iterations do
    synthesis-pyr = Make-Pyramid (noise)
    Loop for a-band in subbands of analysis-pyr
    for s-band in subbands of synthesis-pyr do
    Match-Histogram (s-band,a-band)
    noise = Collapse-Pyramid (synthesis-pyr)
    Match-Histogram (noise,texture)

    What do they mean by collapse pyramid?And also can we use Laplacian pyramid instead of steerable pyramid for test purpose only?

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>