Non-Parametric Data Synthesis
Objective (Part I)
This report presents the implementation and results of simulating the Efros & Leung (1999)
non-parametric texture synthesis algorithm to produce a texture image and to generate
MNIST digits using the same method. The goal is to reproduce texture growth based on
patch-based sampling and matching, and to explore how patch size impacts synthesis
quality both on texture and MNIST digits.
Firstly, we use assignment 2 file which has 12 texture pictures. We use 5X5, 7X7, and 9X9 to
synthesis the image.
A visual diagram process of synthesis (28X28 as example)
SSD formula:
With color images (RGB), each pixel has 3 values, so SSD needs to be computed per
channel and combined. That adds:
• Complexity
• More memory
• Higher chance of noisy or mismatched patches
➔ Therefore, we chose gray color to construct texture images.
The result as following:
Image 1
Image 2
Image 3
Image 4
Image 5
Image 6
Image 7
Image 8
Image 9
Patch Synthesis Comparison:
Patch Advantages Disadvantages Overall Effect
Size
5×5 Sharp detail, Lacks global Suitable for fine-
preserves local consistency, results in grained textures
texture well discontinuities and (e.g., noise, grains,
noisy artifacts fine sand)
7×7 Balanced detail Still at risk of boundary Natural appearance
and global discontinuities with some local
consistency variation, generally
stable
9×9 Continuous Can blur details, Suitable for strongly
patterns, synthesis may become structured textures,
consistent overly repetitive such as bricks or
structure, smooth stripes
synthesis
We also apply to an image we are interested in, and followed the steps below:
1. Initialize a blank canvas with the same size as the input (128x128)
2. Seed the center of the image with a random patch from the input
3. Grow the texture outward by:
o Finding unfilled pixels adjacent to filled regions
o Comparing the partially filled neighborhood with candidate patches from the
input
o Selecting the best-matching patch and copying its center pixel to the output
4. Repeat until the entire image is synthesized
Results:
Original Image
Original Grayscale Image vs 9x9 Patch Size
Original Grayscale Image vs 23x23 Patch Size
The visual comparison was conducted using two different patch sizes: 9x9 and 23x23.
Patch 9x9 captures more detail and better textures but contains slight noises due to
smaller context windows. Patch 23x23 produces similar results to the original one, with
smoother textures but lost at some small-scale local variation.
----------------------------------------------------------------------------------------------------------------
Objective (Part II)
To evaluate whether Efros & Leung’s non-parametric texture synthesis can be scaled to
generate digit-like structures from the MNIST dataset, using the same patch-based
sampling technique originally designed for textures.
Extract Image Patches from the one image
We do the same thing as before, but the dataset uses one image from MNIST.
Extracting from only one seed image, we extract image patches from the image from MNIST
dataset and try to generate the corresponding MNIST digits by the following:
Original image digital “7”
5x5 7x7 9x9
Conclusion: Effects of Patch Size on MNIST Digit Synthesis
Patch
Visual Outcome Synthesis Behavior
Size
Fragmented, blocky shapes that do Too small to capture structural context;
5×5
not resemble the original digit. synthesis is noisy and disjoint.
Partial digit structure appears, but Captures some local features but lacks
7×7
with inconsistent lines and gaps. enough context for continuity.
Best structural coherence due to larger
Clear resemblance to digit "7",
9×9 receptive field; good for shape
smooth and continuous strokes.
consistency.
Extract Image Patches from the Whole Dataset
Variating from extracting from only one seed image, we extract image patches from the
whole MNIST dataset and try to generate the corresponding MNIST digits by the following
method:
• Dataset: MNIST (28×28 grayscale handwritten digits)
• Patch Source: Thousands of 9×9 and 21×21 patches extracted from MNIST digits of
the same class (e.g., only “1” or “8”)
• Synthesis Strategy:
o Initialize a blank canvas with a small central seed patch
o Grow outward by matching partially known neighborhoods with database
patches
o Copy the center pixel of the best-matching patch into the unknown location
Interpretation
• Digit-Specific Training: Using class-specific patches (e.g., only “8”s) improves
structure preservation.
• Patch Size Trade-off:
o Smaller patches favor digit texture fidelity
o Larger patches improve structural continuity at the cost of detail
• Limitations:
o No label conditioning, so synthesized digits are generative in style, not class-
controlled
o Cannot learn stroke order or pen pressure dynamics, only pixel intensity
context
Results:
Different Patch Sizes for Digit “1”
Different Patch Sizes for Digit “2”
Different Patch Sizes for Digit “3”
Different Patch Sizes for Digit “4”
Different Patch Sizes for Digit “5”
Different Patch Sizes for Digit “6”
Different Patch Sizes for Digit “7”
Different Patch Sizes for Digit “8”
Different Patch Sizes for Digit “9”
From the above images, we can see that 5x5 patches don’t usually generate the similar
image as the original one. As the patches get bigger and bigger, the similarity also
increases. While Efros & Leung's approach is non-parametric and unsupervised, it can still
produce digit-like outputs from MNIST with surprising visual quality, especially when using
larger context windows like 21×21 patches.
Conclusion
This report demonstrates the implementation and evaluation of the Efros & Leung (1999)
non-parametric texture synthesis algorithm using PyTorch. Two distinct applications were
explored:
1. High-Resolution Texture Synthesis
First we apply images of textures from the given file. By applying the method to the
images, we observed that patch-based synthesis can recreate large-scale visual
patterns using only local pixel statistics. The choice of patch size has a profound
impact:
o Smaller patches (e.g., 9×9) capture local detail but introduce artifacts and
texture fragmentation.
o Larger patches (e.g., 23×23) preserve global coherence and smooth
transitions but may oversmoothed features.
2. MNIST Digit Generation
Scaling the method to MNIST demonstrates its ability to synthesize digit-like
structures from class-specific patch datasets.
o With 9×9 patches, the generated digits retain texture variation and pen stroke
density but often lack spatial consistency.
o With 21×21 patches, the output is smoother and more structurally
consistent, though at the cost of losing fine details.
These results show that non-parametric synthesis can be extended beyond
textures to simple image generation tasks, despite lacking explicit structural
modeling or supervision.
This work validates the versatility of non-parametric texture synthesis and opens the door
for experimental extensions, such as multi-scale patch fusion, color texture generation,
and stylized digit modeling.
Contributions:
Camille Chang:
• Reproduce texture generation on interested image
• Implementation of different patch sizes on textures
• Digit generation from the whole MNIST dataset
• Generated digit images comparison
• Report structuring and writing
Yating Chang:
• Reproduce texture generation on assignment 2 file
• Texture generation algorithm implementation
• Generated textures for images in assignment file
• Multi-scale comparison for single MNIST digit image extraction
• Report structuring and writing
