Play with it

I have set up a separate page where you can play with the compression. If you have trouble figuring out how to use it or what it’s about at all, please read the rest of this post as it might help to get started.

How it works

The mathematically inclined might want to refer to the Wikipedia article linked above on why fractal compression works at all, here I’ll only outline how I implemented it. As already said, we start with the square power-of-two image we want to find a fractal representation for. This image is subdivided into non overlapping domains of let’s say 16px² (one reason for the square & POT restriction is to make this possible without ugly corner cases). We also perform another subdivision of the image into ranges. The only real requirement for those ranges is that they must be strictly larger than the domains. In this implementation, the image is subdivided using a quadtree scheme, starting with the whole image and going down to ranges which are twice the size of the domains. These get scaled down to the domain size (using bilinear interpolation).

Now we have a rather big pool of range images, and the real work starts: We do an exhaustive search for range images which cover our domains as good as possible. We use the mean squared error to determine the similarity here. To improve the quality of our matches, two additional operations are performed on the ranges:

• The range images may be transformed to match the domains. The algorithm tries all eight symmetries of the square and chooses the best. This is the main reason for the “only squares” restriction.
• Also, we apply a brightness/contrast adjustment to our source (range) image. The nice thing about this step is, that by doing a linear regression analysis we get both, the optimal coefficients and the resulting error by looping over the pixel values only once.
• The heart of the encoding algorithm works only on gray-scale images. To support color images, their color information is converted from RGB to YUV color space, and the three resulting grey-scale channels are encoded in terms of each other.

Using this scheme, the pixel colors of every domain block are reduced to a block like this (in JSON notation):

{
   "pos": [3, 3, 2],
   "channel": 0,
   "transform": 6,
   "brightness": 80.87387183475751,
   "contrast": -0.837734338904282,
}

The meaning of these properties is as follows:

• pos defines the position of the range block in the source image, where an empty list means the whole source image is used for this domain block, [0] would mean the upper left quarter is used, [0,3] means the lower right quarter of the upper left quarter is used, and so on.
• channel is only really used for color images, and specifies the source channel where to extract from
• transform is in the range [0..7] and specifies which of the eight transformations to apply to this source image and finally
• brightness and contrast specify the value adjustment for the source image

You’re still reading… just start playing with the damn thing and if you care just post a questing in the comments section below.

Here’s a quick rundown of what’s new with this iteration:

• added support for triangles (and meshes thereof); indeed everything visible in the above scene consists solely of triangles
• implemented a simple BVH for intersection acceleration; the performance still leaves something to desire, but at least it makes rendering scenes with more than like 15 primitives bearable
• there were several cleanups and simplifications all around, but it’s been a while and I don’t remember exactly…

And here’s a level 3 Menger Sponge on a “brushed metal” ground, which take ages to render, mainly because of the inefficient sampling of the glossiness:

/**
 * An interface for objects that need to be disposed after usage. This
 * somehow adds a life cycle to objects which are created, in use and after
 * usage must be "disposed" (and should not be used beyond this point).
 *
 * @author Matthias Treydte <waldheinz at gmail.com>
 */
public interface Disposable {

    /**
     * Disposes this object. The general contract is that no instance methods
     * should be invoked on this object after disposal, unless stated otherwise
     * in the method's description.
     */
    public void dispose();
}

The idea is that every Object that has some cleaning up to be done after usage implements this interface and does the clean up in its dispose implementation. With this, we can just roll an AttributeHolder implementation which itself has an defined end-of-life (implements Disposable) and disposes all its contained attributes when this method is called. An straight-forward implementation might look like this:

/**
 * An abstract base class for implementing the {@link AttributeHolder}
 * interface which has a implementations of the attribute related methods and
 * forwards calls to its {@link #dispose()} method to all contained
 * {@link Disposable} instances.
 *
 * @author Matthias Treydte <waldheinz at gmail.com>
 */
public abstract class AbstractAttributeHolder
        implements AttributeHolder, Disposable {

    private final Object lock = new Object();
    private transient Map<Object, Object> attributes;

    public AbstractAttributeHolder() {
        this.attributes = new HashMap<Object, Object>();
    }

    public void replaceAttributes(Map<Object, Object> newAttr) {
        synchronized (getAttributeLock()){
            this.attributes = newAttr;
        }
    }

    @Override
    public Object getAttributeLock() {
        return this.lock;
    }

    @Override
    public final void putAttribute(Object key, Object value) {
        synchronized (getAttributeLock()) {
            attributes.put(key, value);
        }
    }

    @Override
    public final boolean hasAttribute(Object key) {
        synchronized (getAttributeLock()) {
            return attributes.containsKey(key);
        }
    }

    @Override
    public final Object getAttribute(Object key) {
        synchronized (getAttributeLock()) {
            return attributes.get(key);
        }
    }

    @Override
    public final Set<Object> getAttributes() {
        synchronized (getAttributeLock()) {
            return Collections.unmodifiableSet(
                    new HashSet<Object>(this.attributes.values()));
        }
    }

    @Override
    public void dispose() {
        synchronized (this.getAttributeLock()) {
            for (Object o : this.attributes.values()) {
                if (o instanceof Disposable) {
                    final Disposable d = (Disposable) o;
                    d.dispose();
                }
            }

            this.attributes.clear();
        }
    }
}

The interesting stuff happens in the dispose method where the contained map is scanned for instances of Disposable whose dispose method is then called in turn. This way concrete subclasses of AbstractAttributeHolder can be nested and when the outer scope is dispose, so are all nested scope and all the contained Disposable attributes.

Synchronization

The solution proposed in here is suitable for use in an multi-threaded environment, and like to elaborate on why I think it is so. The AttributeHolderScope basically binds some AttributeHolder to the current thread by the means of a ThreadLocal variable. But there is nothing that prevents any given AttributeHolder instance from being used by multiple threads concurrently. This is where the attributeLock() enters the stage. It is used to establish synchronization for modifying the HashMap which is the working horse of the AbstractAttributeHolder. Similarly, it also ensures the happens-before relation when reading from the map in the AttributeHolderScope.scope(..) method in case the map was modified by some other thread while the current thread is in the scope. Honestly, there is still some investigation needed here because I’m not entirely sure that this synchronization is sufficient when Guice has to build objects with a dependency tree with a depth > 1. So far it behaves nicely, though…

Sonium - 2006-10-24 18:25:30 : someone speak python here?
lucky - 2006-10-24 18:25:53 : HHHHHSSSSSHSSS
lucky - 2006-10-24 18:26:08 : SSSSS
Sonium - 2006-10-24 18:26:16 : the programming language

The script was written using the storage format analysis by Dmytry Lavrov. It works across at least Skype version 2.1.0.81 on Linux and some not-so-recent Windows version (don’t know exactly, sorry). The only command line parameter is the path to the directory where it will find the chatmsg*.dbb files and extract the relevant information. The main benefits of this script are

• that it decodes the obscure date/time format in the database (?) files, and
• uses this information to sort the log entries before printing them out, so you can follow conversations

Otherwise you can simply look at these files using your favorite hex editor as the messages themselves are stored in plain text format. You can download the script or copy-and-paste it from below.

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sys
from datetime import datetime

workDir = sys.argv[1]

sizes = [256, 512, 1024, 2048, 4096]

def readString(data, offset):
  result=''
  while (data[offset] != '\x00'):
    result += str(data[offset])
    offset += 1
  return result

class Message:
  def __init__(self, time, sender, message):
    self.time = time
    self.sender = sender
    self.message = message

  def __cmp__(self, other):
    return cmp(self.time, other.time)

  def __str__(self):
    return self.sender + " - " + str(self.time) +  " : " + self.message

def decodeTime(data, tsOffset):
  ts = ((ord(data[tsOffset+2]) & ~0x80) << 0) | \
    ((ord(data[tsOffset+3]) & ~0x80) << 7) | \
    ((ord(data[tsOffset+4]) & ~0x80) << 14) | \
    ((ord(data[tsOffset+5]) & ~0x80) << 21) | \
    ((ord(data[tsOffset+6]) & 0x0f) << 28)

  return datetime.fromtimestamp(ts)

allMessages = []

for size in sizes:
  try:
    f = open(workDir + '/chatmsg' + str(size) + '.dbb', 'rb')
    data = f.read()
    f.close()
  except IOError:
    continue

  pos=0
  while (1):
    membersOffset = data.find('\xe0\x03\x23', pos)

    if (membersOffset == -1): break
    # ignore member names (until ";")

    tsOffset = data.find('\xe5\x03', membersOffset)
    if (tsOffset == -1): break
    time = decodeTime(data, tsOffset)

    senderOffset = data.find('\xe8\x03', membersOffset + 4)
    if (senderOffset == -1): break
    sender = readString(data, senderOffset+2)

    msgOffset = data.find('\xfc\x03', senderOffset + 2 + len(sender))
    if (msgOffset == -1): break
    msg = readString(data, msgOffset + 2)

    allMessages.append(Message(time, sender, msg))

    pos=senderOffset

allMessages.sort()

for msg in allMessages:
  print msg

It works like this: the two smaller images are the endpoints of the animation, and when you press "start" the larger view gets interpolated between the two. If you click any of the small images it gets replaced with a fresh, random flame.

This thing works with current Chromium and Firefox but there's something wrong with Opera that I'm not willing to investigate. See below. Internet Explorer fails as usual. Thanks to pinetree it works with IE 9 beta now. Just for the record: I get around 10 fps for "Good" quality using Chromium 5.0.375.99 on my Core 2 Duo E8400 @ 3 GHz.

Update I checked what's up with Opera, and as it turns out Opera does not implement the createImageData method. But then, it supports the getImageData method, and I can paint the pixels black by myself. There is a page about the HTML 5 canvas at the Opera dev center. It states:

Note: not all browsers implement createImageData. On such browsers, you need to obtain your ImageData object using the getImageData method.

Well guys, just fix it and you can remove that ambiguous note. Geeez... And there is still trouble ahead: Opera does not saturate the color values you put in that ImageData object. That means whenever the accumulated value for a pixel gets bigger than 255 freaky colors appear. I could check for that in the JavaScript, but I do not want to sacrifice performance for other browsers.

The Attribute Holder Scope

• either we do not scope them at all (meaning the objects will be created again every time you require an instance)
• or they are singletons (which live forever within an given Injector).

/**
 * Implementors of this interface can serve as the backing store for
 * Objects that are scoped within an (subclass of) {@link AttributeHolderScope}.
 *
 * @author Matthias Treydte <waldheinz at gmail.com>
 */
public interface AttributeHolder {

    /**
     * Extracts the {@code Object} memorized for the specified key from this
     * {@code AttributeHolder}.
     *
     * @param key the identifier for the attribute to extract
     * @return the {@code Object} stored for the specified key, or {@code null}
     *      if either the {@code null} value was stored for this key or there
     *      is no attribute stored for the key
     * @see #hasAttribute(java.lang.Object) to discriminate the two reasons
     *      this method may return {@code null}
     */
    public Object getAttribute(Object key);

    /**
     * Decides if this {@code AttributeHolder} has an association for the
     * specified key.
     *
     * @param key the key to check if it's known to this {@code AttributeHolder}
     * @return if this key is known
     */
    public boolean hasAttribute(Object key);

    /**
     * Stores a new value in this {@code AttributeHolder}.
     *
     * @param key the key to identify the new attribute
     * @param value the new attribute
     */
    public void putAttribute(Object key, Object value);

    /**
     * Returns all values currently stored in this {@code AttributeHolder}. The
     * returned set can not be modified.
     *
     * @return all attributes of this holder
     */
    public Set<Object> getAttributes();

    /**
     * Returns an object on which to lock when access to multiple methods of
     * the {@code AttributeHolder} are to be made atomic.
     *
     * @return the {@code Object} to synchronize on
     */
    public Object getAttributeLock();
    
}

/**
 * A {@code Scope} that uses an {@link AttributeHolder} as the backing store
 * for it's scoped objects.
 * 
 * @author Matthias Treydte <waldheinz at gmail.com>
 */
public class AttributeHolderScope<AHT extends AttributeHolder>
        implements Scope, Provider<AHT> {
    
    private final ThreadLocal<AHT> holder;

    /**
     * Creates a new instance of {@code AttributeHolderScope}.
     */
    public AttributeHolderScope() {
        this.holder = new ThreadLocal<AHT>();
    }

    /**
     * Lets the current {@code Thread} enter this {@code Scope}.
     *
     * @param holder the {@link AttributeHolder} instance for the {@code Scope}
     * @throws IllegalStateException if the current {@code Thread} is already
     *      in this {@code Scope}
     */
    public void enter(AHT holder) throws IllegalStateException {
        if (holder == null)
            throw new NullPointerException();

        if (this.holder.get() != null)
            throw new IllegalStateException(
                    "already in " + getClass().getSimpleName() + " scope");
            
        this.holder.set(holder);
    }

    /**
     * Lets the current {@code Thread} leave this {@code Scope}.
     *
     * @throws OutOfScopeException if the current thread is not in
     *      this {@code Scope}
     */
    public void exit() throws OutOfScopeException {
        assertInScope();
        this.holder.remove();
    }

    /**
     * {@inheritDoc}
     *
     * @return {@docRoot}
     * @throws OutOfScopeException if the current thread is not in
     *      this {@code Scope}
     */
    @Override
    public AHT get() throws OutOfScopeException {
        assertInScope();
        return this.holder.get();
    }
    
    @Override
    public final <T> Provider<T> scope(
            final Key<T> key, final Provider<T> outer) {
        
        return new Provider<T>() {

            @Override
            public T get() {
                assertInScope();
                
                final AttributeHolder ah = holder.get();
                
                synchronized (ah.getAttributeLock()) {
                    T current = (T) ah.getAttribute(key);

                    if ((current == null) && !ah.hasAttribute(key)) {
                        current = outer.get();
                        ah.putAttribute(key, current);
                    }

                    return current;
                }
            }

            @Override
            public String toString() {
                return "Provider [scope=" + //NOI18N
                        AttributeHolderScope.this.getClass().getSimpleName() +
                        ", outer=" + outer.toString() + "]"; //NOI18N
            }
            
        };
    }

    private void assertInScope() throws OutOfScopeException {
        if (holder.get() == null) {
            throw new OutOfScopeException(
                    "not in " + getClass().getSimpleName()); //NOI18N
        }
    }
}

How to use it

public final class Feed implements AttributeHolder {
    private final URL feedUrl;
    private final Map<Object, Object> attributes;

    public Feed(URL feedUrl) {
        this.feedUrl = feedUrl;
        this.attributes = new HashMap<Object, Object>();
    }
   
    public URL getFeedUrl() {
        return this.feedUrl;
    }
    
    public Object getAttributeLock() {
        return this.attributes;
    }

    /* the rest of the AttributeHolder implementation delegates
     * to this.attributes and is omitted
     */
}

@Target({ TYPE, METHOD }) @Retention(RUNTIME) @ScopeAnnotation
public @interface FeedScoped {
    /* nothing */
}

public final class FeedScope extends AttributeHolderScope<Feed> {
    /* nothing */
}

public interface PostCache {
    public List<Post> getPosts();
    public void putPost(Post newPost);
}

public class TransientPostCache implements PostCache {
    private final List<Post> posts;
    /* implement the PostCache methods using this.posts */
}

public class FeedFetcher {
    @Inject
    public FeedFetcher(URL feedUrl, PostCache cache) {
        /* remember the parameters, ... */
    }

    public void doFetch() throws IOException {
        /* fetch the feed and add the posts to the cache */
    }
}

Injector inj = ...;
FeedScope fs = ...;

for (Feed feed : feedList) {
    fs.enter(feed);
    try {
        final FeedFetcher ff = inj.getInstance(FeedFetcher.class);
        ff.doFetch();
    } catch (IOException ex) {
        /* log the error, try again, whatever... */
    } finally {
        fs.exit();
    }
}

public final class FeedModule extends AbstractModule {
    protected void configure() {
        /* initialize the Feed scope */
        final FeedScope fScope = new FeedScope();
        bindScope(FeedScoped.class, fScope);
        bind(Feed.class).toProvider(fScope);
        bind(FeedScope.class).toInstance(fScope);

        bind(PostCache.class).
            to(TransientPostCache.class).
            in(FeedScoped.class);

        bind(FeedFetcher.class);
    }

    @Provides
    @FeedScoped
    URL provideUrl(Feed feed) {
        return feed.getFeedUrl();
    }
}

Was it worth all the trouble?

At work we have a web application where data URIs containing JPEG image data are generated on the server, transferred to the client and there the images are displayed by using these URIs as image sources. This approach works and it even performs nicely and all. Sadly, when trying this approach in WebKit based browsers (hello iPad...), we hit a bad memory leak.

As it turns out, all you have to do to trigger this is setting the "source" property of an image. Every time you do this, the image gets updated as expected, but as it seems the memory occupied by the last image that was displayed is never freed. I hacked a small demonstration of this so you can try yourself. But please be aware that this might crash your browser.

When you hit the "start" button the script will try to update the image every 50ms with a fresh, randomly-generated data URI which should result in an animated white noise (like your TV when not tuned to a station). And when using a current WebKit based browser, you will see the browsers unbounded memory usage. I highly recommend hitting "stop" before your OS starts swapping.

This bug is already known on the chromium issue tracker, but currently unresolved.

Affected Browsers

Here are some browsers that I tested. If you dare to, post your test results to the comments section and I'll update this list.

• Chrome 6 (Windows, 32 bit) : leaks
• Chromium 5.0.375.70 (Linux, 64 bit) : leaks
• Firefox 3.6.3 (Linux, 64 bit) : OK
• Firefox 3.6.3 (Windows, 32 bit) : OK
• Firefox 3.7a5 (Windows, 32 bit) : OK
• IE 8 (Windows, 32 bit): leaks
• IE 9 (Windows, 32 bit): OK
• iPhone 3.1 : leaks (most probably; crash)
• iPhone 4 : leaks (like 3.1)
• Konqueror 4.3.5 : leaks
• Opera 10.10 (Linux, 64 bit) : leaks
• Opera 10.* (Windows, 32 bit) : leaks
• Safari 5 (Windows, 32 bit) : leaks

To cut it short: Firefox works, all others fail in one or another way.

Workaround

Welcome to the world of canvas and manual pixel manipulation using JavaScript. Using canvas' capability to draw images is of no use here, as obviously you have to initialize the image-to-be-drawn using an... URI. *yuck*

render :: Int -> Image -> Scene -> Camera -> Integrator -> IO ()
render pass img scene cam int = do
   prng <- newStdGen
   img' <- return $! fromRand $ runRand prng (onePass img scene cam int)
   writeFile ("pass-" ++ (printf "%05d" pass) ++ ".ppm") (imageToPpm img')
   render (pass + 1) img' scene cam int

module Image(
   Image, ImageSample(..), 
   imageWidth, imageHeight, 
   imageToPpm, makeImage, addSample) where

import Data.Array.Diff

import Color

-- | places a @WeightedSpectrum@ in an @Image@
data ImageSample = ImageSample {
   samplePosX :: ! Float,
   samplePosY :: ! Float,
   sampleSpectrum :: ! WeightedSpectrum
   } deriving Show

-- | an image has a width, a height and some pixels
data Image = Image {
   imageWidth :: Int,
   imageHeight :: Int,
   _imagePixels :: (DiffUArray Int Float)
   }
   
-- | extracts the pixel at the specified offset from an Image
getPixel :: Image -> Int -> WeightedSpectrum
getPixel (Image _ _ p) o = (p ! o', s) where
   s = fromXyz (p ! (o' + 1), p ! (o' + 2), p ! (o' + 3))
   o' = o * 4
   
-- | puts an pixel to the specified offset in an Image
putPixel :: Image -> Int -> WeightedSpectrum -> Image
putPixel (Image w h p) o (sw, s) = seq p' Image w h p' where
   (sx, sy, sz) = toXyz s
   p' = p // [ (o', sw), (o' + 1, sx), (o' + 2, sy), (o' + 3, sz) ]
   o' = o * 4
   
-- | converts an image to ppm format
imageToPpm :: Image -> String
imageToPpm i@(Image w h _) = header ++ spixels 0
   where
      header = "P3\n" ++ show w ++ " " ++ show h ++ "\n255\n"
      spixels pos
         | pos == (w*h) = []
         | otherwise = (ppmPixel $ getPixel i pos) ++ spixels (pos + 1)

-- | applies gamma correction to an RGB triple
gamma :: Float -> (Float, Float, Float) -> (Float, Float, Float)
gamma x (r, g, b) = (r ** x', g ** x', b ** x') where
   x' = 1 / x

-- | converts a Float in [0..1] to an Int in [0..255], clamping values outside [0..1]
clamp :: Float -> Int
clamp v = round ( (min 1 (max 0 v)) * 255 )

-- | converts a @WeightedSpectrum@ into what's expected to be found in a ppm file
ppmPixel :: WeightedSpectrum -> String
ppmPixel ws = (toString . (gamma 2.2) .toRgb . mulWeight) ws
   where
      toString (r, g, b) = show (clamp r) ++ " " ++
         show (clamp g) ++ " " ++ show (clamp b) ++ " "

-- | converts a weighted spectrum to a plain spectrum by dividing out the weight
mulWeight :: WeightedSpectrum -> Spectrum
mulWeight (0, _) = black
mulWeight (w, s) = sScale s (1.0 / w)

-- | adds an sample to the specified image and returns the updated image
addSample :: Image -> ImageSample -> Image
addSample img@(Image w h _) (ImageSample sx sy (sw, ss))
   | isx > maxX || isy > maxY = img
   | otherwise = seq img' img'
   where
      img' = seq img seq newPixel putPixel img offset newPixel
      isx = floor sx
      isy = floor sy
      maxX = w - 1
      maxY = h - 1
      offset = isy * w + isx
      (oldW, oldS) = getPixel img offset
      newPixel = (oldW + sw, oldS + ss)

-- | creates a new all-black image of the specified width and height
makeImage :: Int -> Int -> Image
makeImage w h = Image w h pixels where
    pixels = listArray (0, pxCount - 1) (repeat 0.0)
    pxCount = w * h * 4 :: Int

data Ray = Ray {
   rayOrigin :: Point,
   rayDir :: Normal,
   rayMin :: Float,
   rayMax :: Float
   } deriving Show

type Camera = (Float, Float) -> Ray

-- | defines the view for projective camera models
data View = View {
   viewPos :: Point, -- ^ the position of the camera in world space
   viewLookAt :: Point, -- ^ the "look-at" point world space
   viewUp :: Normal, -- ^ the "up" vector
   viewFocalLength :: Float, -- ^ focal length
   viewAspect :: Float -- ^ aspect ratio of image plane
   }
   
-- | computes a point on the image plane
viewPoint :: View -> (Float, Float) -> Point
viewPoint (View pos la up dist aspect) (u, v) = \
      center `add` (scalMul right u') `add` (scalMul up' v') where
   center = pos `add` (scalMul dir dist)
   right =  normalize $ up `cross` dir
   up' = cross right dir
   dir = normalize $ sub la pos
   u' = u * aspect - 0.5
   v' = v - 0.5

-- | a simple "pinhole" camera
pinHoleCamera :: View -> Camera
pinHoleCamera view uv = Ray pv rd 0 infinity where
   rd = normalize $ sub pv $ viewPos view
   pv = viewPoint view uv