I trained DiffusionGemma to be an abstract expressionist
Reported by r/LocalLLaMA1 source ↓
A community post describes adapting DiffusionGemma, a text diffusion model, to generate recognizable images and perform image operations using a custom training and inference pipeline.
Key points
- The adaptation used fewer than 300 million parameters in total
- The model generated images and performed visual operations on supplied images
- The creator says a small amount of data and compute produced interpretable results
- Training used about 200 images to teach a model that generated targets for a larger dataset
Details
The creator built a text-to-image and image-manipulation pipeline for DiffusionGemma, which is based on a Gemma 4 backbone. The post says the adaptation used fewer than 300 million parameters and produced recognizable images, as well as visual operations on provided images.
The creator reports that generating images in the model’s native visual latent space, then feeding those representations back in blocks, improved quality and generalization on image-manipulation tasks. The method required patching an image-attention bug in Transformers, according to the post.
The training process included building a vision decoder and creating text targets to supervise the model. Rather than generate targets for every training image directly, the creator used descriptions from about 200 images to train a teacher-forced model that generated targets for the full dataset.
Written from coverage as of