Stable Signer: Balancing Three Stakeholders’ Interests in Image Provenance

September 1, 2024Jun Yu

Contents

If a generated image is cropped, color-graded, and reposted, can we still trace where it came from?

Stable Signer changes selected parameters of a generative model so that its output carries a recoverable user identifier. Different models can share an extractor, while each model provider retains control over how the watermark enters its own model.

I designed the protocol around three questions: Would a model company adopt it? Can a third party trace images across models? And what quality, privacy, or false-accusation risks would users bear?

Try the demo · Source code

Start with the images: similar pictures, different messages

Each row below uses the same prompt and seed to produce one latent. We decode that latent with the original VAE decoder and with a message-conditioned decoder. The difference therefore comes from the decoder, not from two independent random generations.

Example	Original decoder	Watermarked decoder	Absolute pixel difference ×28
Observatory
Kingfisher
Ceramics

The messages are 48 bits long. The three examples recover 46, 48, and 48 bits, respectively. To show where the pixels changed, the difference images multiply absolute differences by 28 and clip them to the display range. They do not depict the actual visible distortion at its true intensity.

The demo lets you inspect these three precomputed examples, including image-level and model-level attacks, without waiting for inference. You can also enter your own prompt, seed, and message to generate an image live.

Why would anyone adopt it?

The obvious way to watermark an image is to run an embedder after generation. But someone who can run the generative model directly might skip that step. A model-level watermark moves the embedding mechanism into generation: images produced with the modified model parameters tend to carry the chosen message. Someone with access to the weights can still try to alter them. This changes the attack surface; it does not eliminate it.

Stable Signature demonstrated an instructive approach: fine-tune a latent diffusion model’s VAE decoder for a fixed message, then read that message from its images with a pretrained extractor. Our research reuses that extractor and asks two further questions. How can we assign different messages to different users without fine-tuning a decoder for each one? And how can different model providers participate in a common third-party provenance mechanism?

Framing the task only as “improve bit accuracy” misses constraints that determine whether anyone would use the system.

Stakeholder	What they need	Design consequence
Model provider	Keep its original weights and full architecture details private; avoid obvious quality loss; onboard users without retraining for each one	Keep model-specific parameter encoding on the provider side, make quality part of training, and condition a mapper on the message
Third-party provenance operator	Avoid maintaining a wholly different extraction procedure for every model	Share the extractor and message interface, but allow a separate mapper for each model
Ordinary user	Keep image quality and avoid being held responsible because of a statistical error	Separate extraction, matching, and identity claims; evaluate false positives and misattribution; define requirements for access, audits, and appeals

We call the third party a regulator in the research framework. It could be an independent organization; it need not be a government agency.

For a model company to join, I think one choice matters especially: the provider should decide exactly how its model is modified.

Which layers are selected, how they are grouped, what low-rank factorization is used, and how long the mapper’s output vector must be are all model-specific decisions. The third party needs a protocol for training and using the mapper. It does not need direct possession of the complete model weights to carry out that protocol.

This is what “shared” means here: a shared way to cooperate, not identical internals for every provider.

One interface, with different things kept on each side

The path from message to image and back is:

message m → model-specific mapper → compact vector ψ
                                        ↓ provider's parameter decoding rule
                                 message-conditioned model
                                        ↓
                                   output image y
                                        ↓ shared extractor
                                  recovered message m̂

The mapper and parameter decoding rule together can be viewed as a hypernetwork. Given a message, they produce changes to another network’s parameters instead of overlaying the message onto an image.

Party	Role in the research framework
Provider	Holds the original generative model; chooses the parameter encoding, layers, and rank; runs generation and model-side backpropagation
Third party	Holds the shared extractor; trains a mapper for each participating model; extracts messages and matches them against registered identifiers
Between them	Agree on vector dimensions and protocol; exchange the proxy, generated samples, and necessary gradients during joint training

A shared extractor is not a shared mapper. Each model still needs adaptation and training. Changing the selected layers, rank, or output dimension at deployment is not a harmless configuration edit: the mapper and the parameter decoding rule must remain compatible.

On the provider side, parameter encoding selects weights, compresses them spatially, applies a low-rank factorization, flattens the result, and permutes it to obtain the exchanged vector. Decoding uses the agreed layers and shapes to construct parameter changes from that vector. Because selection and compression occur first, this is not a reversible encoding of arbitrary model weights.

This is gray-box cooperation. The third party does not receive the original weights directly, but it still sees vector dimensions, training samples, and gradients. Reducing direct disclosure matters; it is not a proof that the third party can never infer model information. A permutation is not encryption.

How a message changes the decoder

Let θ denote the original model parameters, m a message, and φ the mapper parameters. A simplified expression of the process is:

\begin{aligned} \psi &= M_\phi(m), \\[2pt] \theta_m &= \operatorname{Decode}_{\theta}(\psi), \\[2pt] y_m &= G_{\theta_m}(x), \\[2pt] \hat m &= \mathbb{1}[E(y_m)>0]. \end{aligned}

The extractor E returns one score per bit. The final line thresholds those scores at zero. The θ subscript on Decode emphasizes its dependence on the original model held by the provider: ψ alone is not a copy of that model.

In the current SDXL-Turbo path, text conditioning and denoising first produce a latent. The watermark modulates the VAE decoder that turns this latent into an RGB image. We do not retrain the text encoder or denoising network for each message.

The modulation matrix is low-rank; the final weight difference need not be

For a selected convolutional layer, let W have shape C_out × C_in × k × k. We take two factors from the mapper’s output:

\begin{aligned} A_m &\in \mathbb{R}^{C_{\rm out}\times r}, \\ B_m &\in \mathbb{R}^{r\times C_{\rm in}}, \\ S_m &= \alpha A_m B_m, \\ W_m[o,i,u,v] &= W[o,i,u,v]\bigl(1+S_m[o,i]\bigr). \end{aligned}

Each input-output channel pair shares one modulation coefficient across the convolution kernel’s spatial positions. Unselected layers and other parameters stay unchanged.

This borrows the low-rank parameterization idea from LoRA, but uses a multiplicative update, W ⊙ (1 + αAB), rather than the usual additive update, W + αAB. The channel modulation matrix S has rank at most r. The actual weight difference, ΔW = W ⊙ S, generally does not.

The parameter-count intuition is simple. A 512 × 512 × 3 × 3 convolution has 2,359,296 weights. With rank 8, the two factors contain only 512×8 + 8×512 = 8,192 numbers. The trade-off is restricted expressiveness: this modulation cannot represent an arbitrary weight change.

The experiment below uses 48-bit 0/1 input, a mapper with no hidden layer, and 15 modulated VAE decoder convolutions. The rank is 8 and α is 0.01. The mapper emits a 149,536-dimensional vector, some partitions of which are unused. The exact partitioning is recorded in the configuration linked at the end.

Learn a message-to-modulation function instead of training once per message

Optimizing a separate decoder for every new user’s message would make each new user a new training job. Instead, we train a function M(m) by repeatedly sampling messages, so it learns to produce useful modulations for messages it has not seen before.

After training, a new 48-bit message only requires a mapper forward pass to construct its corresponding parameters. This saves repeated per-user training. It does not make all costs of adding users disappear: parameter construction, caching, model distribution, image generation, registration, and lookup still consume resources.

Training: image quality is a constraint

The mapper has to satisfy two objectives: the extractor must recover the message, and the image should remain close to the original model’s output. We can express the research objective as:

\begin{aligned} \min_\phi\quad &\mathbb{E}_{x,m}\bigl[ \mathcal{L}_{\rm msg}(E(y_m),m)\bigr] \\ \text{subject to}\quad &\mathbb{E}_{x,m}\bigl[ \mathcal{L}_{\rm perceptual}(y,y_m)\bigr]\leq\varepsilon. \end{aligned}

Training uses BCE for message recovery and Watson-VGG perceptual loss for image change. “How much quality loss can we accept?” is easier for a provider to reason about than the mixing weight between two losses. We therefore express the quality budget directly as a constraint ε.

Joint training proceeds as follows:

The third party samples a message, runs the mapper, and sends ψ to the provider.
The provider applies its parameter decoding rule, generates original and watermarked images from the same input, and computes perceptual loss.
If quality exceeds the budget, the provider returns a quality gradient with respect to ψ. The third party uses it to correct the mapper, with a cap on correction iterations.
The provider sends the watermarked image. The third party evaluates message loss with the extractor it keeps on its side, then returns the image gradient.
The provider backpropagates that gradient to ψ; the third party backpropagates from ψ to the mapper parameters.

The chain rule gives each side a computational and information boundary: the provider need not hold the extractor weights, and the third party need not hold the generative model weights. The arrangement also introduces sample and gradient transfer, synchronization between parties, and possible information leakage through the training interface.

The constraint applies to expected loss under the training samples, and quality correction has a finite iteration limit. It cannot guarantee the same quality budget for every new prompt and message.

What happens when images or models are changed?

We tested image transformations and parameter averaging on three fixed prompt, seed, and message combinations. For each combination, we used the first generated result, with no selection for recovery accuracy.

Image-level attacks: cropping, brightness, and a composite transform

Extraction runs on the final displayed RGB8 image, not on a floating-point tensor before saving. We apply ImageNet normalization and threshold the extractor’s output at zero for each bit. The attacks operate on those same displayed pixels.

Example	No attack	Center crop	Brightness ×1.5	Composite transform
Observatory, seed 23	46/48	47/48	47/48	33/48
Kingfisher, seed 41	48/48	48/48	48/48	37/48
Ceramics, seed 67	48/48	48/48	48/48	36/48

The center crop in this table retains about 90% of both the width and height, then resizes to 512×512 with Lanczos. It does not retain only 10% of the image. The composite transform retains the central 50% of each dimension, resizes to the original dimensions, multiplies brightness by 1.5, and JPEG-encodes with quality parameter 80. Keeping half the width and height retains only about a quarter of the area.

The mild transforms did not visibly damage recovery in these three examples, but the composite transform reduced recovery to 33–37 bits. The observatory recovered one more bit after a mild transform. A single sample’s thresholded result need not vary monotonically with attack strength; this does not show that cropping generally improves the watermark.

Three examples are not enough to estimate overall robustness.

Model-level attack: average two users’ models

If users receive watermarked model weights rather than only a generation API, image processing is not their only attack surface. Two users can cooperate to average their model parameters and try to dilute their identifiers.

The demo shows one fixed 50/50 average. For two signers sharing the same base W:

\begin{aligned} W_a &= W\odot(1+\alpha A_aB_a), \\ W_b &= W\odot(1+\alpha A_bB_b), \\ \frac{W_a+W_b}{2} &= W\odot\left(1+\frac{\alpha}{2} (A_aB_a+A_bB_b)\right). \end{aligned}

We average the complete parameter modulations, not A and B separately before multiplying them, and not two output images. The colluded model still performs a real decode of the same latent.

Signer A	Signer B	Averaged model

Example	Averaged output matching A’s message	Matching B’s message
Observatory	33/48	39/48
Kingfisher	34/48	38/48
Ceramics	20/48	28/48

For ceramics, matches fell to 20/48 and 28/48. Those two messages happen to be bitwise complements, so any extracted 48-bit string must have match counts summing to 48. The equation 20 + 28 = 48 is not, by itself, independent evidence about attack performance.

These results show why model-level threats need their own evaluation. They do not prove reliable identification of colluders, nor do they prove that every watermark was removed. Either claim needs a decision rule and more tests.

How much did the parameters move?

To see what averaging changed, we can inspect the layerwise modulations produced for two messages. Both plots below use the same mapper and VAE as the image experiments.

First, the relative perturbation strength of each layer:

q_\ell = \frac{\lVert W_{\ell,m}-W_\ell\rVert_F} {\lVert W_\ell\rVert_F}.

We use the FP32 weight differences actually executed by the model, with FP64 accumulation for the norms, and divide by the norm of the full convolution weight tensor. Across layers for signer A in the observatory example, the result ranges from roughly 6% to 42%. α = 0.01 therefore does not mean “each parameter changes by only 1%.” It multiplies a learned matrix product. Nor can parameter-change magnitude be converted directly into visible image distortion.

The second plot is a heatmap of the channel modulation matrix S. It shows A, B, and their average for up_blocks.1.resnets.0.conv1:

Each cell is a modulation coefficient for an output-input channel pair. It is not an image pixel, an original weight, or an absolute weight difference ΔW. The three panels share a symmetric-logarithmic color scale centered at zero, linear within ±1%, with no clipping of larger values. This makes both small changes and a few large modulations visible.

The plots help locate which layers move most and where different messages produce similar modulations. They are not a semantic explanation of model features: a red cell does not correspond to an object in the image. Establishing whether a layer is necessary for message recovery would require controlled ablations that disable, replace, or perturb that layer.

From reading 48 bits to naming a person

The most tempting mistake is to read 46/48 as “a 95.8% probability that this image came from this person.” It only says that 46 of the 48 extracted bit positions match one specified message.

At least three different questions are involved:

Message recovery: How many bits match a specified message? The bit match counts above answer this.
Watermark detection: Is the match unusual enough to reject the hypothesis that this image has no corresponding watermark? That needs a threshold and unmarked controls.
Attribution to a user: Among many registered messages, is there a sufficiently reliable and unambiguous match? This additionally depends on multiple comparisons, candidate conflicts, and the integrity of the registry.

For example, under an idealized null hypothesis where extracted bits from an unmarked image are independent and each is equally likely to be 0 or 1, the match count K against a fixed message follows Binomial(48, 1/2). With a decision threshold of at least t matching bits:

\Pr(K\geq t\mid H_0) = 2^{-48}\sum_{j=t}^{48}\binom{48}{j}.

If we compare against N registered messages, a union bound gives an upper bound of N · Pr(K ≥ t | H₀) on the overall false-positive probability, capped at 1. The bound does not require independence between comparisons, but the single-comparison formula relies on the bit-distribution assumption above. A real system would have to calibrate against unmarked images, outputs from other models, and its actual message codebook. Three examples and this idealized formula do not establish a measured false-positive rate. The Stable Signature project page also distinguishes detection, identification, and different kinds of error.

Thus, a 48-bit format is not proof that 2^48 users can be distinguished reliably. Extracting a message does not verify that the content is authentic, much less prove that a particular person did something. “Signer” here names a statistical watermarking mechanism, not a public-key digital signature.

What governance would this still need?

A common extractor reduces fragmentation in technical integration. It also concentrates power in the third party, a problem that higher recovery accuracy alone cannot solve.

The research threat model assumes a cooperating provider, keeps the extractor with the third party, and denies attackers access to a provenance query interface. It does not solve the case of a malicious provider deliberately turning off the watermark. For inspectability, the public demo exposes relevant code and extraction results. It cannot serve as a security proof under the original “secret extractor, no query access” assumptions.

For users, unresolved questions include: Who may query an identifier? Who may link it to an identity, and under what conditions? How can an error be appealed? Can registry entries be changed after the fact? Who audits the third party? A more plausible deployment would separate random identifiers from the identity mapping and authorize and log queries. These are requirements for a future system, not features implemented by the current demo.

I worry most about the concentration of provenance authority behind a common interface. How could the third party prove it followed the agreed rules? We have no cryptographic or algorithmic guarantee for that today. Calling the operator “trusted” does not answer the question.

Configuration and acknowledgments

The image results come from GPU inference with the trained 08R checkpoint. The parameter plots were computed offline on a CPU. The online demo runs inference centrally; it does not deploy the two parties in isolation or implement the confidentiality protocol.

Item	Version or protocol used here
Base model	SDXL-Turbo, revision `71153311d3dbb46851df1931d3ca6e939de83304`
Generation	FP32, 512×512, 4 steps, guidance scale 0
Mapper	Historical 08R artifact; binary input; rank 8; 15 active convolutional layers
Extraction	Displayed RGB8 pixels, ImageNet normalization, zero threshold for each bit
Example selection	Three fixed prompt/seed/message combinations, first result, no recovery-based filtering
Result index	Accompanying manifest with image hashes, messages, results, and upstream record locations

The full records are the image results manifest, parameter diagnostics manifest, and runtime artifact manifest. They record code revisions, configurations, and file hashes for the respective runs.

This research was conducted at Zhejiang University and has not yet been published. I primarily worked on framework and algorithm design, implementation, experiments and ablations, visualizations, and paper writing as a co-first author. I thank my adviser and collaborators.

Prior work. The extractor reuses pretrained weights released by Stable Signature; its 48-bit architecture follows the HiDDeN line of work. The low-rank parameterization and conditional parameter generation draw on LoRA and HyperNetworks, respectively. The base generative model is provided by Stability AI, and the runtime uses Hugging Face Diffusers.

Use and licenses. The online demo is for non-commercial research only. Powered by Stability AI. This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved. The adapted extractor code and converted weights are subject to CC BY-NC 4.0. See the NOTICE and licenses for modifications and full terms.