Testing Q-Former Text-to-Image: Gradients through Frozen Backbones | CMU 10-423 Assignment 4
Use an original numerical test to check gradients, data, and evaluation alongside the original assignment specification.
#text-to-image
5 postsText-to-Image and VLMs: Generating Images versus Reading Them | CMU 10-423 Topic Guide
“Text plus images” does not specify the task. Text-to-image generation produces an image conditioned on text; a VLM may consume an image and produce an answer. Direction, output objective, and evaluation differ even when both use Transformers.
Text-to-Image AI, Principles and Practice — CLIP Reads the Prompt, Schedulers Decide Whether It Converges, LoRA Learns Only ΔW, and You Build a Web App with diffusers | NCCU Yen-Lung Tsai Generative AI Lecture 11
L11 fills in the rest of the Stable Diffusion diagram. CLIP is trained so that matching text and images get similar vectors, which turns a prompt into a 77×768 embedding. Schedulers compress 1,000 noising steps into twenty or thirty denoising steps, but ancestral samplers such as Euler a never settle: push to 100 steps and the subject changes jackets and seats. LoRA freezes the original W and learns only a ΔW factored into A·B. The hands-on part loads an SD 1.5-family model with diffusers, and the week 11 homework is your own image-generation web app.
Conditional Generative Models — cGAN, cVAE, and Classifier-Free Guidance | MIT 6.7960 Lecture 16
The key to conditional generation is 'feed y into the model': cGAN concatenates y into G/D; cVAE passes y to both encoder and decoder; in diffusion, Classifier Guidance uses gradients from an external classifier to push samples toward a class, while Classifier-Free Guidance trains conditional + unconditional together and linearly combines them at inference — the latter is the standard weapon behind Stable Diffusion and Imagen.
2022 AI Conference Guide: Computer Vision
2022 marked computer vision’s turn from recognition toward generation. Latent Diffusion Models appeared at CVPR and led to Stable Diffusion; NeRF research jumped from 25 papers in 2021 to more than 50 at CVPR alone; ConvNeXt mounted a compelling counterattack for CNNs; and ECCV in Tel Aviv set a record with 157 oral papers.