Feature Visualization

There’s a rising sense that neural networks have to be interpretable to people.
The sphere of neural community interpretability has shaped in response to those issues.
Because it matures, two main threads of analysis have begun to coalesce: characteristic visualization and attribution.

This text focuses on characteristic visualization.
Whereas characteristic visualization is a strong device, really getting it to work entails various particulars.
On this article, we study the main points and discover widespread approaches to fixing them.
We discover that remarkably easy strategies can produce high-quality visualizations. Alongside the way in which we introduce a couple of methods for exploring variation in what neurons react to, how they work together, and enhance the optimization course of.

Characteristic Visualization by Optimization

Neural networks are, usually talking, differentiable with respect to their inputs.
If we wish to discover out what sort of enter would trigger a sure conduct — whether or not that’s an inner neuron firing or the ultimate output conduct — we will use derivatives to iteratively tweak the enter
in direction of that purpose .

Whereas conceptually easy, there are refined challenges in getting the optimization to work. We are going to discover them, in addition to widespread approaches to sort out them within the part ”The Enemy of Feature Visualization″.

Optimization Goals

What do we wish examples of?
That is the core query in working with examples, no matter whether or not we’re looking by a dataset to search out the examples, or optimizing photos to create them from scratch.
We have now all kinds of choices in what we seek for:

If we wish to perceive particular person options, we will seek for examples the place they’ve excessive values — both for a neuron at a person place, or for a complete channel.
We used the channel goal to create many of the photos on this article.

If we wish to perceive a layer as a complete, we will use the DeepDream goal , trying to find photos the layer finds “attention-grabbing.”

And if we wish to create examples of output lessons from a classifier, we’ve got two choices — optimizing class logits earlier than the softmax or optimizing class possibilities after the softmax.
One can see the logits because the proof for every class, and the possibilities because the chance of every class given the proof.
Sadly, the best option to enhance the likelihood softmax provides to a category is commonly to make the options unlikely moderately than to make the category of curiosity possible .
From our expertise, optimizing pre-softmax logits produces photos of higher visible high quality.

Whereas the usual clarification is that maximizing likelihood doesn’t work very nicely as a result of you’ll be able to simply push down proof for different lessons, an alternate speculation is that it’s simply more durable to optimize by the softmax perform. We perceive this has typically been a problem in adversarial examples, and the answer is to optimize the LogSumExp of the logits as an alternative. That is equal to optimizing softmax however usually extra tractable. Our expertise was that the LogSumExp trick doesn’t appear higher than coping with the uncooked possibilities.

No matter why that occurs, it may be fastened by very sturdy regularization with generative fashions. On this case the possibilities is usually a very principled factor to optimize.

The targets we’ve talked about solely scratch the floor of potential targets — there are much more that one might attempt.
Of explicit notice are the targets utilized in model switch , which may train us in regards to the sorts of favor and content material a community understands,
and targets utilized in optimization-based mannequin inversion , which assist us perceive what info a mannequin retains and what it throws away.
We’re solely at first of understanding which targets are attention-grabbing, and there’s a lot of room for extra work on this space.

Why visualize by optimization?

Optimization can provide us an instance enter that causes the specified conduct — however why hassle with that?
Couldn’t we simply look by the dataset for examples that trigger the specified conduct?

It seems that optimization strategy is usually a highly effective option to perceive what a mannequin is basically searching for,
as a result of it separates the issues inflicting conduct from issues that merely correlate with the causes.
For instance, think about the next neurons visualized with dataset examples and optimization:

Optimization additionally has the benefit of flexibility.
For instance, if we wish to research how neurons collectively characterize info,
we will simply ask how a specific instance would have to be completely different for a further neuron to activate.
This flexibility may also be useful in visualizing how options evolve because the community trains.
If we had been restricted to understanding the mannequin on the fastened examples in our dataset, subjects like these ones could be a lot more durable to discover.

Then again, there are additionally vital challenges to visualizing options with optimization.
Within the following sections we’ll study strategies to get various visualizations, perceive how neurons work together, and keep away from excessive frequency artifacts.

Range

Do our examples present us the complete image?
Once we create examples by optimization, that is one thing we have to be very cautious of.
It’s totally potential for real examples to nonetheless mislead us by solely exhibiting us one “aspect” of what a characteristic represents.

Dataset examples have an enormous benefit right here.
By trying by our dataset, we will discover various examples.
It doesn’t simply give us ones activating a neuron intensely:
we will look throughout a complete spectrum of activations to see what prompts the neuron to completely different extents.

In distinction, optimization usually provides us only one extraordinarily constructive instance — and if we’re artistic, a really unfavourable instance as nicely.
Is there a way that optimization might additionally give us this variety?

Reaching Range with Optimization

A given characteristic of a community could reply to a variety of inputs.
On the category degree, for instance, a classifier that has been educated to acknowledge canines ought to acknowledge each closeups of their faces in addition to wider profile photos — though these have fairly completely different visible appearances.
Early work by Wei et al. makes an attempt to exhibit this “intra-class” variety by recording activations over your entire coaching set, clustering them and optimizing for the cluster centroids, revealing the completely different aspects of a category that had been realized.

A special strategy by Nguyen, Yosinski, and collaborators was to go looking by the dataset for various examples and use these as beginning factors for the optimization course of .
The concept is that this initiates optimization in numerous aspects of the characteristic in order that the ensuing instance from optimization will exhibit that aspect.
In newer work, they mix visualizing lessons with a generative mannequin, which they will pattern for various examples .
Their first strategy had restricted success, and whereas the generative mannequin strategy works very nicely — we’ll focus on it extra within the part on regularization underneath learned priors — it may be a bit difficult.

We discover there’s a quite simple option to obtain variety: including a “variety time period”

For this text we use an strategy primarily based on concepts from inventive model switch. Following that work, we start by computing the Gram matrix $G$ of the channels, the place $G_{i,j}$ to at least one’s goal that pushes a number of examples to be completely different from one another.
The variety time period can take quite a lot of types, and we don’t have a lot understanding of their advantages but.
One chance is to penalize the cosine similarity of various examples.
One other is to make use of concepts from model switch to pressure the characteristic to be displayed in numerous types.

In decrease degree neurons, a variety time period can reveal the completely different aspects a characteristic represents:

Numerous characteristic visualizations enable us to extra carefully pinpoint what prompts a neuron, to the diploma that we will make, and — by dataset examples — test predictions about what inputs will activate the neuron.

For instance, let’s study this straightforward optimization consequence.

Easy optimization

it in isolation one would possibly infer that this neuron prompts on the highest of canine heads, because the optimization reveals each eyes and solely downward curved edges.
Trying on the optimization with variety nonetheless, we see optimization outcomes which don’t embody eyes, and in addition one which incorporates upward curved edges. We thus should broaden our expectation of what this neuron prompts on to be principally in regards to the fur texture. Checking this speculation in opposition to dataset examples reveals that’s broadly appropriate. Be aware the spoon with a texture and colour related sufficient to canine fur for the neuron to activate.

The impact of variety will be much more putting in increased degree neurons, the place it could actually present us various kinds of objects that stimulate a neuron.
For instance, one neuron responds to completely different sorts of balls, though they’ve quite a lot of appearances.

This less complicated strategy has various shortcomings:
For one, the strain to make examples completely different could cause unrelated artifacts (equivalent to eyes) to look.
Moreover, the optimization could make examples be completely different in an unnatural means.
For instance, within the above instance one would possibly wish to see examples of soccer balls clearly separated from different forms of balls like golf or tennis balls.
Dataset primarily based approaches equivalent to Wei et al. can break up options aside extra naturally — nonetheless they might not be as useful in understanding how the mannequin will behave on completely different information.

Range additionally begins to brush on a extra elementary subject: whereas the examples above characterize a principally coherent thought, there are additionally neurons that characterize unusual mixtures of concepts.
Beneath, a neuron responds to 2 forms of animal faces, and in addition to automotive our bodies.

Examples like these recommend that neurons should not essentially the correct semantic items for understanding neural nets.

Interplay between Neurons

If neurons should not the correct option to perceive neural nets, what’s?
In actual life, mixtures of neurons work collectively to characterize photos in neural networks.
A useful means to consider these mixtures is geometrically: let’s outline activation area to be all potential mixtures of neuron activations.
We are able to then consider particular person neuron activations because the foundation vectors of this activation area. Conversely, a mixture of neuron activations is then only a vector on this area.

This framing unifies the ideas “neurons” and “mixtures of neurons” as “vectors in activation area”. It permits us to ask: Ought to we anticipate the instructions of the premise vectors to be any extra interpretable than the instructions of different vectors on this area?

Szegedy et al. discovered that random instructions appear simply as significant because the instructions of the premise vectors.
Extra not too long ago Bau, Zhou et al. discovered the instructions of the premise vectors to be interpretable extra usually than random instructions.
Our expertise is broadly in step with each outcomes; we discover that random instructions usually appear interpretable, however at a decrease charge than foundation instructions.

We are able to additionally outline attention-grabbing instructions in activation area by doing arithmetic on neurons.
For instance, if we add a “black and white” neuron to a “mosaic” neuron, we receive a black and white model of the mosaic.
That is harking back to semantic arithmetic of phrase embeddings as seen in Word2Vec or generative fashions’ latent areas.

These examples present us how neurons collectively characterize photos.
To raised perceive how neurons work together, we will additionally interpolate between them.
The optimization goal is a linear interpolation between the person channel targets. To get the interpolations to look higher, we additionally add a small alignment goal that encourages decrease layer activations to be related. We moreover use a mixture of separate and shared picture parameterizations to make it simpler for the optimization algorithm to trigger objects to line up, whereas nonetheless giving it the liberty to create any picture it must.
That is much like interpolating within the latent area of generative fashions.

That is solely beginning to scratch the floor of how neurons work together.
The reality is that we’ve got virtually no clue choose significant instructions, or whether or not there even exist notably significant instructions.
Unbiased of discovering instructions, there are additionally questions on how instructions work together — for instance, interpolation can present us how a small variety of instructions work together, however in actuality there are a whole bunch of instructions interacting.

The Enemy of Characteristic Visualization

If you wish to visualize options, you would possibly simply optimize a picture to make neurons fireplace.
Sadly, this doesn’t actually work.
As a substitute, you find yourself with a form of neural community optical phantasm — a picture stuffed with noise and nonsensical high-frequency patterns that the community responds strongly to.

These patterns appear to be the photographs form of dishonest, discovering methods to activate neurons that don’t happen in actual life.
Should you optimize lengthy sufficient, you’ll are inclined to see a few of what the neuron genuinely detects as nicely,
however the picture is dominated by these excessive frequency patterns.
These patterns appear to be carefully associated to the phenomenon of adversarial examples .

We don’t absolutely perceive why these excessive frequency patterns type,
however an vital half appears to be strided convolutions and pooling operations, which create high-frequency patterns within the gradient .

These high-frequency patterns present us that, whereas optimization primarily based visualization’s freedom from constraints is interesting, it’s a double-edged sword.
With none constraints on photos, we find yourself with adversarial examples.
These are definitely attention-grabbing, but when we wish to perceive how these fashions work in actual life, we have to by some means transfer previous them…

The Spectrum of Regularization

Coping with this excessive frequency noise has been one of many main challenges and overarching threads of characteristic visualization analysis.
If you wish to get helpful visualizations, you have to impose a extra pure construction utilizing some form of prior, regularizer, or constraint.

In truth, should you take a look at most notable papers on characteristic visualization, one among their details will often be an strategy to regularization.
Researchers have tried plenty of various things!

We are able to consider all of those approaches as residing on a spectrum, primarily based on how strongly they regularize the mannequin.
On one excessive, if we don’t regularize in any respect, we find yourself with adversarial examples.
On the alternative finish, we search over examples in our dataset and run into all the constraints we mentioned earlier.
Within the center we’ve got three most important households of regularization choices.

Three Households of Regularization

Let’s think about these three intermediate classes of regularization in additional depth.

Frequency penalization immediately targets the excessive frequency noise these strategies undergo from.
It might explicitly penalize variance between neighboring pixels (whole variation) , or implicitly penalize high-frequency noise by blurring the picture every optimization step .
If we take into consideration blurring in Fourier area, it’s equal to including a scaled L2 penalty to the target, penalizing every Fourier-component primarily based on its frequency.
Sadly, these approaches additionally discourage authentic high-frequency options like edges together with noise.
This may be barely improved through the use of a bilateral filter, which preserves edges, as an alternative of blurring .

(Some work makes use of related strategies to scale back excessive frequencies within the gradient earlier than they accumulate within the visualization .
These strategies are in some methods similar to the above and in some methods radically completely different — we’ll study them within the subsequent part, Preconditioning and Parameterization.)

Transformation robustness tries to search out examples that also activate the optimization goal extremely even when we barely remodel them.
Even a small quantity appears to be very efficient within the case of photos ,
particularly when mixed with a extra common regularizer for high-frequencies .
Concretely, which means we stochastically jitter, rotate or scale the picture earlier than making use of the optimization step.

Realized priors.
Our earlier regularizers use quite simple heuristics to maintain examples cheap.
A pure subsequent step is to truly study a mannequin of the true information and attempt to implement that.
With a powerful mannequin, this turns into much like looking over the dataset.
This strategy produces probably the most photorealistic visualizations, however it might be unclear what got here from the mannequin being visualized and what got here from the prior.

One strategy is to study a generator that maps factors in a latent area to examples of your information,
equivalent to a GAN or VAE,
and optimize inside that latent area .
Another strategy is to study a previous that offers you entry to the gradient of likelihood;
this lets you collectively optimize for the prior alongside along with your goal .
When one optimizes for the prior and the likelihood of a category, one recovers a generative mannequin of the info conditioned on that specific class.
Lastly, Wei et al. approximate a generative mannequin prior, at the least for the colour distribution, by penalizing distance between patches of the output and the closest patches retrieved from a database of picture patches collected from the coaching information.

Preconditioning and Parameterization

Within the earlier part, we noticed a couple of strategies that decreased excessive frequencies within the gradient moderately than the visualization itself.
It’s not clear that is actually a regularizer:
it resists excessive frequencies, however nonetheless permits them to type when the gradient constantly pushes for it.
If it isn’t a regularizer, what does remodeling the gradient like this do?

Remodeling the gradient like that is really fairly a strong device — it’s known as “preconditioning” in optimization.
You’ll be able to consider it as doing steepest descent to optimize the identical goal,
however in one other parameterization of the area or underneath a unique notion of distance.

Gradient blurring is equal to gradient descent in a unique parameterization of picture area, the place excessive frequency dimensions are stretched to make transferring in these instructions slower. Gradient Laplacian Pyramid normalization is a form of adaptive studying charge strategy in the identical area.

This modifications which course of descent will probably be steepest, and how briskly the optimization strikes in every course, however it doesn’t change what the minimums are.
If there are numerous native minima, it could actually stretch and shrink their basins of attraction, altering which of them the optimization course of falls into.
Consequently, utilizing the correct preconditioner could make an optimization drawback radically simpler.

How can we select a preconditioner that can give us these advantages?
first guess is one which makes your information decorrelated and whitened.
Within the case of photos this implies doing gradient descent within the Fourier foundation,

This factors to a profound reality in regards to the Fourier remodel.
So long as a correlation is constant throughout spatial positions — such because the correlation between a pixel and its left neighbor being the identical throughout all positions of a picture — the Fourier coefficients will probably be impartial variables.
To see this, notice that such a spatially constant correlation will be expressed as a convolution, and by the convolution theorem turns into pointwise multiplication after the Fourier remodel.

with frequencies scaled in order that all of them have equal vitality.

Be aware that we’ve got to watch out to get the colours to be decorrelated, too. The Fourier transforms decorrelates spatially, however a correlation will nonetheless exist between colours.
To handle this, we explicitly measure the correlation between colours within the coaching set and use a Cholesky decomposition to decorrelate them. Evaluate the instructions of steepest respectable earlier than and after decorrelating colours:





Correlated Colours





Decorrelated Colours

Let’s see how utilizing completely different measures of distance modifications the course of steepest descent.
The common L² gradient will be fairly completely different from the instructions of steepest descent within the L^∞ metric or within the decorrelated area:

All of those instructions are legitimate descent instructions for a similar goal,
however we will see they’re radically completely different.
Discover that optimizing within the decorrelated area reduces excessive frequencies,
whereas utilizing L^∞ will increase them.

Utilizing the decorrelated descent course ends in fairly completely different visualizations.
It’s exhausting to do actually honest comparisons due to hyperparameters, however the
ensuing visualizations appear lots higher — and develop quicker, too.

(Except in any other case famous, the photographs on this article had been optimizing within the decorrelated area and a collection of transformation robustness strategies.

Pictures had been optimized for 2560 steps in a color-decorrelated fourier-transformed area, utilizing Adam at a studying charge of 0.05.
We used every of following transformations within the given order at every step of the optimization:

• Padding the enter by 16 pixels to keep away from edge artifacts
• Jittering by as much as 16 pixels
• Scaling by an element randomly chosen from this checklist: 1, 0.975, 1.025, 0.95, 1.05
• Rotating by an angle randomly chosen from this checklist; in levels: -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5
• Jittering a second time by as much as 8 pixels
• Cropping the padding
)

Is the preconditioner merely accelerating descent, bringing us to the identical place
regular gradient descent would have introduced us if we had been affected person sufficient?
Or is it additionally regularizing, altering which native minima we get interested in?
It’s exhausting to inform for certain.
On the one hand, gradient descent appears to proceed enhancing as you exponentially enhance the variety of optimization steps — it hasn’t converged, it’s simply transferring very slowly.
Then again, should you flip off all different regularizers, the preconditioner appears to scale back high-frequency patterns.

Conclusion

Neural characteristic visualization has made nice progress over the previous couple of years.
As a group, we’ve developed principled methods to create compelling visualizations.
We’ve mapped out various vital challenges and located methods of addressing them.

Within the quest to make neural networks interpretable, characteristic visualization
stands out as one of the vital promising and developed analysis instructions.
By itself, characteristic visualization won’t ever give a totally passable
understanding. We see it as one of many elementary constructing blocks that,
mixed with further instruments, will empower people to know these methods.

There stays nonetheless plenty of vital work to be completed in enhancing characteristic visualization.
Some points that stand out embody understanding neuron interplay, discovering which items are most significant for understanding neural web activations, and giving a holistic view of the aspects of a characteristic.