Glitching jpegs for no reason

I recently saw a neat project for "glitching" jpegs. Here's how I did something similar for my site.

Click/tap the image to try it out!

Getting the jpeg data

I opened the code, but struggled to make sense of its organization, so I turned to the older project it was based on. Luckily the older project was very simple, and I had my launch pad.

In essence, it uses a <canvas> to extract the jpeg data as a data url, and I used the same technique.

img.onload = () => {
  img.onload = null;
  const canvas = new Canvas(img.naturalWidth, img.naturalHeight);
  const ctx = canvas.getContext("2d");
  ctx.drawImage(img, 0, 0);
  const b64_jpeg = canvas.toDataURL("image/jpeg");
};

With jpeg bytes in hand, it's time to look into the jpeg file format to understand how glitching can work, without total loss of the image. I referenced this great illustration by Ange Albertini, wikipedia and my good friend the llm.

Key takeaways:

File segments are indicated by a 0xFF byte, then a byte for segment type
Image data is in "scan" segments of type 0xAD
In encoded data, 0xFF is followed by a 0x00 to indicate that it is not a new segment

The old code looked for the scan segment by searching for 0xFF, 0xAD, and then randomized some of the following bytes. The amount of "glitching" is determined by how many bytes are randomized.

This works reasonably well, but it sometimes corrupts the image too heavily. In an attempt to avoid near-total image loss, I identified some edge cases:

A byte is randomized to 0xFF, prematurely ending the scan
An existing (0xFF, 0x00) pair of bytes is changed, perhaps also ending the scan
The randomized byte is chosen as part of the scan segment metadata, corrupting the image more completely
The randomized byte is at the end of the file, and overwrites the end of image (probably not a real problem)

There's also progressive jpegs to consider, which contain multiple scans. In practice, I didn't observe this to be a problem, because I think that making a jpeg with Canvas.toDataURL will always produce a single scan jpeg.

Initial glitching

The main change I made was to read the jpeg segments, to only corrupt the actual encoded scan data, not any other file metadata. I also used atob and bota instead of manual b64 encoding.

Handling the 0xFF segment bytes in the encoded data here was easy with two rules

Do not replace a 0xFF or 0x00 byte
Do not generate a 0xFF byte

I set up an interactive page with the first image I had on hand to test it out. It was so slow. Each glitch was taking around 100ms to generate and render, from glitch() to load event. Well maybe that's not necessarily unreasonable, given that my source image was a 5000x5000 png lol.

Shrinking the image down to a reasonable size of 500px was enough of a speedup to get a relatively smooth animation. Great!

At this stage, what I had was a custom element <glitchy-img> that could be clicked to produce a glitched jpeg effect.

<glitchy-img src="X">
  <img src="X" />
  <canvas></canvas>
</glitchy-img>

Some accompanying css was needed to stack the <canvas> and <img> tags.

glitchy-img {
    position: relative;
}

glitchy-img > img {
    position: relative;
    width: 100%;
}

glitchy-img > canvas {
    position: absolute;
    width: 100%;
    top: 0;
    left: 0;
}

But all that b64 converting just didn't sit right with me, and some initial console.time logging indicated that all those b64 conversions were taking up most of the time.

Premature optimization

I think optimization without measurement is a little silly, but this is a silly project, so off we go.

I figured all of the b64 encoding/decoding could be avoided by randomizing the b64 characters in the string, instead of converting back and forth. Unfortunately, I realized that strings are immutable in js, so I would need to slice and concatenate.

Naively, the algorithm would slice the string for each byte to randomize. This would end up creating count copies of the source data.

glitch_jpeg(jpeg, count) {
  let glitched = jpeg.scan.slice()
  for (let i = 0; i < count; ++i) {
    const glitch_idx = rand_int(jpeg.scan_start, jpeg.scan_end);
    glitched =
      glitched.scan.slice(0, glitch_idx) +
      rand_b64() +
      glitched.scan.slice(glitch_idx + 1);

  }
  return glitched;
}

A bit better is to generate the indexes to randomize first and sort them. That allows slicing the source into non-overlapping chunks, so only a single copy is created.

glitch_jpeg(jpeg, count) {
  let idxs = Array.from({ length: count }, () =>
    rand_int(jpeg.scan_start, jpeg.scan_end),
  );
  idxs.sort((a, b) => a - b);
  idxs.push(jpeg.scan_end);

  const segments = [jpeg.scan.slice(0, idxs[0])];
  for (let i = 0; i < idxs.length - 1; ++i) {
    segments.push(rand_b64());
    segments.push(jpeg.slice(idxs[i] + 1, idxs[i + 1]))
  }
  return segments.join('');
}

But what about the 0xFF bytes? Recall the two rules

Do not replace a 0xFF or 0x00 byte
Do not generate a 0xFF byte

For (1), I had two ideas

decode the bytes around the glitched characters to check for 0xFF and 0x00
find all 0xFF bytes in a preprocessing step and refuse to randomize those indexes (and the following index)

I decided to preprocess, to avoid the finicky math around b64 index conversion.

For (2), what about not generating any b64 characters that could decode to 0xFF? Recall that in b64, 3 bytes are converted to 4 characters

bytes |101010 10|1010 1010|10 10 1010|
b64   |101010|10 1010|1010 10|10 1010|

If the first byte is 0xFF, then the first b64 character must be /. Same for the last byte and the last character. So that's easy, don't generate /.

If the middle byte is 0xFF, then the second b64 char is xx1111 (one of Pfv/), and the third char is 1111xx (one of 89+/). It's enough to ensure that one of these things can't happen.

Choosing the third char gives a new safe alphabet as the 60 chars A-Za-z0-7. The glitch effect should still look good with a little less randomness.

The new algorithm looks something like

glitch(count) {
  let idxs = [];
  while (idxs.length < count) {
    const idx = rand_int(this.jpeg.scan_start, this.jpeg.scan_end);
    if (!this.banned_indexes.has(idx)) {
      idxs.push(idx);
    }
  }
  idxs.sort((a, b) => a - b);
  idxs.push(this.jpeg.scan_end);

  const segments = [this.jpeg.scan.slice(0, idxs[0])];
  for (let i = 0; i < idxs.length - 1; ++i) {
    segments.push(rand_safe_b64());
    segments.push(this.jpeg.slice(idxs[i] + 1, idxs[i + 1]))
  }
  return segments.join('');
}

But one thing was still bothering me... that darn css!

`OffscreenCanvas` and `Blob`

Now I'm no css wizard, but doesn't it seem like such a shame to have both a <canvas> and an <img> and the accompanying css?

glitchy-img {
    position: relative;
}

glitchy-img > img {
    position: relative;
    width: 100%;
}

glitchy-img > canvas {
    position: absolute;  /* GROSS */
    width: 100%;
    top: 0;
    left: 0;
}

While browsing mdn for this project, I ran across OffscreenCanvas (docs). It seems to be very similar to a <canvas>, but it doesn't need to be rendered to the DOM. That would eliminate this ugly css.

Instead of toDataURL('image/jpeg'), with OffscreenCanvas we can use convertToBlob({type: 'image/jpeg'}). This gets the jpeg bytes directly, not as a b64 string. Then mdn helpfully leads to an example using object URLs to display images, using URL.createObjectURL(Blob)

This approach completely avoids base64. I was hoping to manipulate the Blob in-place for a very efficient implementation, but they are immutable. At least I get to reuse the slicing algorithm!

After all's said and done, the code is in a pretty good state

no vestigial <canvas> element and unique css requirements
- could even inherit the custom element from <img> at this point
no base64 bullshit
cool glitch effect that animates smoothly

Conclusion

Out of curiosity, I tried that 5000x5000 png again. On my desktop, it went from over 100ms down to about 12ms. I didn't measure along the way, or carefully benchmark, but it's obvious that there was a dramatic speedup somewhere in there.

Overall, I learned a bunch of modern JS stuff, got to poke the jpeg format a bit, and made something I like. I'll probably use this somewhere on my site, once I fine tune the effect.