Faster Ideogram 4 with Less Prompt Compute

My earlier experiments focused on making LLaDA inference cheaper across both individual kernels and denoising steps. Looking for similar bottlenecks in image generation led me to Ideogram's NF4 path.

I profiled the open Ideogram 4 NF4 checkpoint at 1024², expecting the transformer to dominate runtime. Instead, the first major allocation happened before it even ran.

The pipeline constructed a conditioning tensor with shape (1, 4608, 53248). Only 512 rows contained text features; the remaining 4,096 represented image positions and were all zero. It also created a second (1, 4096, 53248) tensor for the negative branch that was entirely zero.

Both tensors passed through a large 4-bit projection before the image-position outputs were masked. Because the prompt stayed fixed, the same projection was repeated at every denoising step.

Removing the empty rows and caching the projected text reduced peak allocation by 2.14 GiB at 1024² and 9.46 GiB at 2048². My first implementation changed the final pixels, however. BitsAndBytes produced slightly different NF4 outputs when given fewer rows, so the exact version had to preserve the original matrix shape once per generation.

This post assumes basic familiarity with diffusion transformers.

Conditioning a Denoising Step

Ideogram begins from a noisy image latent and updates it over several denoising steps. The latent and timestep change each step.

A Qwen text encoder converts the prompt into one 53,248-dimensional feature vector per text position. For $T$ text positions, its output is

The image transformer uses a hidden width of 4,608, so a learned linear layer projects each Qwen row into that space:

At 1024², the transformer also processes 4,096 image positions. With the 512-token prompt from the main profile, the packed sequence has 4,608 positions:

The sequence length and the hidden width happen to be the same number here. They are not the same dimension. With a 1,941-token prompt the sequence grows to 6,037 positions while the hidden width stays at 4,608, which matters later.

Most of the Tensor Was Zero

The Qwen encoder produces features only for text positions. The Diffusers path still expanded the conditioning tensor to match the full text-and-image sequence:

where $0_I$ contains one zero row for each image position. At 1024², 4,096 of the 4,608 positive-branch rows were empty. Classifier-free guidance added a negative branch of another 4,096 zero rows.

The projection has a bias, so a zero row does not stay zero:

The original code masked the image positions after projection. It therefore built zero rows, projected them into identical bias vectors, and then discarded those outputs. Since the linear layer handles each row independently, the empty image rows cannot affect the projection of the real text rows.

The allocator trace matched the tensor sizes almost exactly. The wide tensors are built in fp32 before the cast to bf16, and in fp32 the two inputs occupy

Watching the allocator through one 1024² step on an L40S:

• static allocation, before conditioning: 14.998 GiB
• after the wide fp32 construction: 16.725 GiB active, 17.639 GiB peak
• after the cast to bf16 settles: 15.862 GiB
• after the transformer step: 18.189 GiB peak

Constructing the tensors temporarily added 1.727 GiB of GPU memory, while their final bf16 size matched the expected allocation.

Then the projection runs. The positive branch projects (1, 4608, 53248) in a median 9.969 ms and the negative branch projects (1, 4096, 53248) in 8.374 ms, and both happen again on the next step. A 20-step generation projects the same prompt twenty times.

I changed the path to keep only the real text rows, so the wide tensor is (1, 512, 53248). The fully masked negative conditioning became None instead of a materialized tensor. Nothing else moved: sampler, schedule, timesteps, latents, weights, guidance scale, attention implementation, and branch order all stayed as they were.

Moving the Projection Out of the Loop

The text features and projection do not change during generation, so the prompt only needs to be projected once before denoising begins.

For 512 text positions the cached bf16 tensor has shape (1, 512, 4608) and occupies about 4.5 MiB. Each step writes it into the packed hidden states with an indexed copy instead of rebuilding and reprojecting the Qwen-width tensors.

I first normalized and projected only the $T$ real rows. Mathematically, those outputs should match the first $T$ rows of the original $(T + I)$-row call. They did not match bitwise.

The NF4 Exactness Problem

The projection is a BitsAndBytes Linear4bit layer with NF4 weights. Its matrix multiplication has the form

where $K = 53248$, $N = 4608$, and $M$ is the input row count. The baseline used $M = T + I$. The smaller call used $M = T$.

BitsAndBytes chooses different kernels for different matrix shapes. Projecting only the 512 real rows therefore changed the order of dequantization and accumulation, even though the math was nominally the same.

The direct 512-row projection differed from the 4,608-row reference by up to 0.015625. After one denoising step, the guided velocity differed by as much as 0.546875. The outputs were still extremely close, with 0.9997085 cosine similarity, but no longer pixel identical.

I tried padding to a fixed row count, hoping there was a safe threshold. There wasn't: the row counts that matched the reference differed between prompts.

One 2,048-row version passed 12 of 18 complete generations and failed six. I could not find a fixed threshold or rounding rule that held across prompt lengths. The only row count I could justify for every prompt was the original one:

Preserving the Reference Shape

The working path builds a temporary scratch tensor with exactly $T + I$ rows. Its first $T$ rows contain the normalized text features. The remaining rows repeat those features until the scratch reaches the original packed length.

The filler values do not affect the real outputs, because the projection handles rows independently. They only make BitsAndBytes see the same GEMM shape as the baseline.

The layer runs once at that shape, in a median 14.762 ms. I keep the first $T$ projected rows, free the scratch and the unused outputs, and reuse the remaining (B, T, 4608) tensor throughout denoising.

real text rows
    -> repeat to the original packed row count
    -> run the NF4 projection once
    -> keep the first T outputs
    -> reuse them during denoising

The wide zero-padded conditioning disappears from the per-step path, while the one-time projection keeps the reference arithmetic.

Results

I measured batch-one runs on Modal with the prompt, seed, schedule, dtype, and branch order fixed between versions. Each latency is the median of three runs after an excluded warm-up at the exact shape. The stack is pinned: Diffusers 7685bffe, model revision 1874bc70, PyTorch 2.13.0, BitsAndBytes 0.50.0, weights all-resident on the card. Peak allocation is torch.cuda.max_memory_allocated, reset before each arm in the same process after warm-up, with both arms starting from the same 14.998254 GiB. Offloaded execution, later dependency versions, and batch sizes above one were not measured.

The 1024² A/B on an L40S separates the two halves of the change. Total is conditioning setup plus one denoising step, which deliberately charges the hoisted projection to a single step:

Hoisting alone saves almost nothing, because it still builds both wide inputs, and at one step it is 39.706 ms slower overall since the setup has nothing to amortize against. Removing the zeros is what frees the memory. Together they are 30.151 ms faster than baseline even at a single step, and a real generation runs 12 or 20 of them.

The saving grows with resolution, because the discarded Qwen-width rows are counting image positions:

Predicted from the retained bf16 zero rows alone, the savings would be 0.203125, 0.8125, and 3.25 GiB. The measured drops are larger because the fp32 construction and the per-step projection temporaries go away too. Runtime improves by only 1 to 3 percent, because the transformer still dominates each step.

The Outputs Are Identical

I set atol=0.01, rtol=0.01 before testing, but never needed the tolerance. Every comparison was exact.

Across 18 full 1024² generations, covering three prompt lengths, three seeds, and both Turbo-12 and Default-20:

• final latents were bitwise identical
• final images were bitwise identical, with SSIM 1.0 and infinite PSNR
• Tesseract text, word boxes, and diagnostics were identical

The OCR check matters because Ideogram 4 is built for rendering text. Text quality still varies by prompt and seed, as it does at baseline, but this optimization changes nothing: the output pixels are identical.

Full-generation peak memory drops by 0.792383 GiB on the short prompt and 1.620291 GiB on the near-maximum one, and the whole generation runs 2 to 3% faster.

References

• QLoRA: Efficient Finetuning of Quantized LLMs, which introduced NF4 (arXiv:2305.14314)
• ideogram-oss/ideogram4, the official reference implementation
• huggingface/diffusers, the pipeline this patch modifies

Measurements were taken on Modal L40S, A100-80GB, and A10 instances. The patch, the Modal scripts, and the raw result artifacts are at github.com/lsnchow/ideogram-zero-free-conditioning.