The Strange Agency

By using a mask with the Stability API, we can create prompt-generated collages of arbitrary length. We can also subtly change the prompt being used throughout the collage.

Stitching

We need to start by creating a masking image that will be sent to the API along with a source image. The API will keep the part of our image where the mask is white, and it will fill in the black area. This was made in Photopea, an awesome alternative to Adobe.

After every generation, we want to keep the top 256 pixels of the returned image, and use the remaining 512 (i.e. 768 - 256) pixels for our next API call.

# set up transformation matrix
a, b, d, e = 1, 0, 0, 1
c = 0  # left / right
f = 256  # up / down

if artifact.type == generation.ARTIFACT_IMAGE:
    result = Image.open(io.BytesIO(artifact.binary))
    output.append(result)
    # now translate output for next round
    source = result.transform(result.size, Image.AFFINE, (a, b, c, d, e, f))

When we have generated all of our images, we stitch them together into a collage.

output_count = len(output)
collage = Image.new('RGB', (768, 256 + output_count * 256))

for i in range(1, output_count):
  collage.paste(im=output[i], box=(0, (i-1) * 256))

Scrolling

Finally, we can use ffmpeg to generate a scrolling animation of our collage. We calculate the height of our collage based on the length of the output array created in the main loop (though we could also query the height of the collage image created above). We set an arbitrary length of 32 seconds for the scroll and calculate a rate accordingly. Finally, we concoct a heinously complex ffmpeg command to get the party moving.

Note that we have an rm scroll.mp4 command in there, to clean up after a previous run. This will keep the ffmpeg from asking if you should erase the previously generated video. You may or may not want this feature. Also we use unsetopt nomatch to keep some shells from complaining about the [video] part of the command.

height = (1 + output_count) * 256
height = height - 432
rate = height / 32

! rm scroll.mp4
! unsetopt nomatch; ffmpeg -r 1 -loop 1 -t 33 -i collage.png -filter_complex "color=white:s=768x432, fps=fps=30[bg];[bg][0]overlay=y=-{height}+'t*{rate}':shortest=1[video]" -preset ultrafast -map [video] scroll.mp4

See the full source here. Now got out and scroll!