Computing Receptive Fields of Convolutional Neural Networks

Whereas deep neural networks have overwhelmingly established state-of-the-art
leads to many synthetic intelligence issues, they’ll nonetheless be
tough to develop and debug.
Current analysis on deep studying understanding has targeted on
characteristic visualization ,
theoretical ensures ,
mannequin interpretability ,
and generalization .

On this work, we analyze deep neural networks from a complementary
perspective, specializing in convolutional fashions.
We’re inquisitive about understanding the extent to
which enter alerts might have an effect on output options, and mapping
options at any a part of the community to the area within the enter that
produces them. The important thing parameter to affiliate an output characteristic to an enter
area is the receptive area of the convolutional community, which is
outlined as the scale of the area within the enter that produces the characteristic.

As our first contribution, we
current a mathematical derivation and an environment friendly algorithm to compute
receptive fields of contemporary convolutional neural networks.
Earlier work
mentioned receptive area
computation for easy convolutional
networks the place there’s a single path from the enter to the output,
offering recurrence equations that apply to this case.
On this work, we revisit these derivations to acquire a closed-form
expression for receptive area computation within the single-path case.
Moreover, we prolong receptive area computation to trendy convolutional
networks the place there could also be a number of paths from the enter to the output.
To one of the best of our information, that is the primary exposition of receptive
area computation for such current convolutional architectures.

At the moment, receptive area computations are wanted in quite a lot of functions. For instance,
for the pc imaginative and prescient job of object detection, it will be significant
to signify objects at a number of scales as a way to acknowledge small and enormous cases;
understanding a convolutional characteristic’s span is usually required for that purpose
(e.g., if the receptive area of the community is small, it might not be capable of acknowledge giant objects).
Nevertheless, these computations are sometimes performed by hand, which is each tedious and error-prone.
It is because there are not any libraries to compute these parameters routinely.
As our second contribution, we fill the void by introducing an
open-source library
which handily performs the computations described right here. The library is built-in into the Tensorflow codebase and
could be simply employed to investigate quite a lot of fashions,
as offered on this article.

We anticipate these derivations and open-source code to enhance the understanding of complicated deep studying fashions,
resulting in extra productive machine studying analysis.

Overview of the article

We contemplate fully-convolutional neural networks, and derive their receptive
area dimension and receptive area places for output options with respect to the
enter sign.
Whereas the derivations offered listed here are common sufficient for any kind of sign used on the enter of convolutional
neural networks, we use pictures as a operating instance, referring to trendy laptop imaginative and prescient architectures when
applicable.

First, we derive closed-form expressions when the community has a
single path from enter to output (as in
AlexNet
or
VGG ). Then, we talk about the
extra common case of arbitrary computation graphs with a number of paths from the
enter to the output (as in
ResNet
or
Inception ). We contemplate
potential alignment points that come up on this context, and clarify
an algorithm to compute the receptive area dimension and places.

Lastly, we analyze the receptive fields of contemporary convolutional neural networks, showcasing outcomes obtained
utilizing our open-source library.

Downside setup

Contemplate a fully-convolutional community (FCN) with (L) layers, (l = 1,2,ldots
 ,L). Outline characteristic map (f_l in R^{h_ltimes w_ltimes d_l}) to indicate the
output of the (l)-th layer, with peak (h_l), width (w_l) and depth
(d_l). We denote the enter picture by (f_0). The ultimate output characteristic map
corresponds to (f_{L}).

To simplify the presentation, the derivations offered on this doc contemplate
(1)-dimensional enter alerts and have maps. For higher-dimensional alerts
(e.g., (2)D pictures), the
derivations could be utilized to every dimension independently. Equally, the figures
depict (1)-dimensional depth, since this doesn’t have an effect on the receptive area computation.

Every layer (l)’s spatial configuration is parameterized by 4 variables, as illustrated within the following determine:

• (k_l): kernel dimension (constructive integer)
• (s_l): stride (constructive integer)
• (p_l): padding utilized to the left facet of the enter characteristic map
  (non-negative integer)
  
  A extra common definition of padding may additionally be thought of: unfavorable
  padding, interpreted as cropping, can be utilized in our derivations
  with none modifications. To be able to make the article extra concise, our
  presentation focuses solely on non-negative padding.
  
• (q_l): padding utilized to the appropriate facet of the enter characteristic map
  (non-negative integer)

We contemplate layers whose output options rely regionally on enter options:
e.g., convolution, pooling, or elementwise operations comparable to non-linearities,
addition and filter concatenation. These are generally utilized in state-of-the-art
networks. We outline elementwise operations to
have a “kernel dimension” of (1), since every output characteristic will depend on a single
location of the enter characteristic maps.

Our notation is additional illustrated with the straightforward
community under. On this case, (L=4) and the mannequin consists of a
convolution, adopted by ReLU, a second convolution and max-pooling.

We undertake the conference the place the primary output characteristic for every layer is
computed by inserting the kernel on the left-most place of the enter,
together with padding. This conference is adopted by all main deep studying
libraries.

Single-path networks

On this part, we compute recurrence and closed-form expressions for
fully-convolutional networks with a single path from enter to output
(e.g.,
AlexNet
or
VGG ).

Computing receptive area dimension

Outline (r_l) because the receptive area dimension of
the ultimate output characteristic map (f_{L}), with respect to characteristic map (f_l). In
different phrases, (r_l) corresponds to the variety of options in characteristic map
(f_l) which contribute to generate one characteristic in (f_{L}). Be aware
that (r_{L}=1).

As a easy instance, contemplate layer (L), which takes options (f_{L-1}) as
enter, and generates (f_{L}) as output. Right here is an illustration:

It’s straightforward to see that (k_{L})
options from (f_{L-1}) can affect one characteristic from (f_{L}), since every
characteristic from (f_{L}) is instantly linked to (k_{L}) options from
(f_{L-1}). So, (r_{L-1} = k_{L}).

Now, contemplate the extra common case the place we all know (r_{l}) and need to compute
(r_{l-1}). Every characteristic (f_{l}) is linked to (k_{l}) options from
(f_{l-1}).

First, contemplate the state of affairs the place (k_l=1): on this case, the (r_{l})
options in (f_{l}) will cowl (r_{l-1}=s_lcdot r_{l} – (s_l – 1)) options
in in (f_{l-1}). That is illustrated within the determine under, the place (r_{l}=2)
(highlighted in purple). The primary time period (s_l cdot r_{l}) (inexperienced) covers the
complete area the place the
options come from, however it’ll cowl (s_l – 1) too many options (purple),
which is why it must be deducted.

As within the illustration under, be aware that, in some instances, the receptive
area area might include “holes”, i.e., a number of the enter options could also be
unused for a given layer.

For the case the place (k_l > 1), we simply want so as to add (k_l-1) options, which
will cowl these from the left and the appropriate of the area. For instance, if we
use a kernel dimension of (5) ((k_l=5)), there can be (2) further options used
on both sides, including (4) in whole. If (k_l) is even, this works as nicely,
for the reason that left and proper padding will add to (k_l-1).

As a result of border results, be aware that the scale of the area within the authentic
picture which is used to compute every output characteristic could also be completely different. This
occurs if padding is used, during which case the receptive area for
border options consists of the padded area. Later within the article, we
talk about the way to compute the receptive area area for every characteristic,
which can be utilized to find out precisely which picture pixels are used for
every output characteristic.

So, we receive the overall recurrence equation (which is

first-order,
non-homogeneous, with variable
coefficients

):

(start{align}
r_{l-1} = s_l cdot r_{l} + (k_l – s_l)
label{eq:rf_recurrence}
finish{align})

This equation can be utilized in a recursive algorithm to compute the receptive
area dimension of the community, (r_0). Nevertheless, we are able to do even higher: we are able to solve
the recurrence equation and acquire an answer by way of the (k_l)’s and
(s_l)’s:

start{equation}
r_0 = sum_{l=1}^{L} left((k_l-1)prod_{i=1}^{l-1}
s_iright) + 1 label{eq:rf_recurrence_final} finish{equation}

This expression makes intuitive sense, which could be seen by contemplating some
particular instances. For instance, if all kernels are of dimension 1, naturally the
receptive area can be of dimension 1. If all strides are 1, then the receptive
area will merely be the sum of ((k_l-1)) over all layers, plus 1, which is
easy to see. If the stride is bigger than 1 for a specific layer, the area
will increase proportionally for all layers under that one. Lastly, be aware that
padding doesn’t should be taken under consideration for this derivation.

Computing receptive area area in enter picture

Whereas it is very important know the scale of the area which generates one characteristic
within the output characteristic map, in lots of instances additionally it is vital to exactly
localize the area which generated a characteristic. For instance, given characteristic
(f_{L}(i, j)), what’s the area within the enter picture which generated it? This
is addressed on this part.

Let’s denote (u_l) and (v_l) the left-most
and right-most coordinates (in (f_l)) of the area which is used to compute the
desired characteristic in (f_{L}). In these derivations, the coordinates are zero-indexed (i.e., the primary characteristic in
every map is at coordinate (0)).

Be aware that (u_{L} = v_{L}) corresponds to the
location of the specified characteristic in (f_{L}). The determine under illustrates a
easy 2-layer community, the place we spotlight the area in (f_0) which is used
to compute the primary characteristic from (f_2). Be aware that on this case the area
consists of some padding. On this instance, (u_2=v_2=0), (u_1=0,v_1=1), and
(u_0=-1, v_0=4).

We’ll begin by asking the next query: given (u_{l}, v_{l}), can we
compute (u_{l-1},v_{l-1})?

Begin with a easy case: let’s say (u_{l}=0) (this corresponds to the primary
place in (f_{l})). On this case, the left-most characteristic (u_{l-1}) will
clearly be positioned at (-p_l), for the reason that first characteristic will probably be generated by
inserting the left finish of the kernel over that place. If (u_{l}=1), we’re
within the second characteristic, whose left-most place (u_{l-1}) is (-p_l
+ s_l); for (u_{l}=2), (u_{l-1}=-p_l + 2cdot s_l); and so forth. Normally:

(start{align}
u_{l-1}&= -p_l + u_{l}cdot s_l label{eq:rf_loc_recurrence_u}
v_{l-1}&= -p_l + v_{l}cdot s_l + k_l -1
label{eq:rf_loc_recurrence_v}
finish{align})

the place the computation of (v_l) differs solely by including (k_l-1), which is
wanted since on this case we need to discover the right-most place.

Be aware that these expressions are similar to the recursion derived for the
receptive area dimension eqref{eq:rf_recurrence}. Once more, we may implement a
recursion over the community to acquire (u_l,v_l) for every layer; however we are able to additionally
solve for (u_0,v_0) and acquire closed-form expressions by way of the
community parameters:

(start{align}
u_0&= u_{L}prod_{i=1}^{L}s_i – sum_{l=1}^{L}
p_lprod_{i=1}^{l-1} s_i
label{eq:rf_loc_recurrence_final_left}
finish{align})

This provides us the left-most characteristic place within the enter picture as a operate of
the padding ((p_l)) and stride ((s_l)) utilized in every layer of the community,
and of the characteristic location within the output characteristic map ((u_{L})).

And for the right-most characteristic location (v_0):

(start{align}
v_0&= v_{L}prod_{i=1}^{L}s_i -sum_{l=1}^{L}(1 + p_l –
k_l)prod_{i=1}^{l-1} s_i
label{eq:rf_loc_recurrence_final_right}
finish{align})

Be aware that, completely different from eqref{eq:rf_loc_recurrence_final_left}, this
expression additionally will depend on the kernel sizes ((k_l)) of every layer.

Relation between receptive area dimension and area.
You might be questioning that
the receptive area dimension (r_0) have to be instantly associated to (u_0) and
(v_0). Certainly, that is the case; it’s straightforward to point out that (r_0 = v_0 – u_0 +
1), which we depart as a follow-up train for the curious reader. To
emphasize, which means we are able to rewrite
eqref{eq:rf_loc_recurrence_final_right} as:

(start{align}
v_0&= u_0 + r_0 – 1
label{eq:rf_loc_recurrence_final_right_rewrite}
finish{align})

Efficient stride and efficient padding.
To compute (u_0) and (v_0) in follow, it
is handy to outline two different variables, which rely solely on the paddings
and strides of the completely different layers:

• efficient stride
  (S_l = prod_{i=l+1}^{L}s_i): the stride between a
  given characteristic map (f_l) and the output characteristic map (f_{L})
• efficient padding
  (P_l = sum_{m=l+1}^{L}p_mprod_{i=l+1}^{m-1} s_i):
  the padding between a given characteristic map (f_l) and the output characteristic map
  (f_{L})

With these definitions, we are able to rewrite eqref{eq:rf_loc_recurrence_final_left}
as:

(start{align}
u_0&= -P_0 + u_{L}cdot S_0
label{eq:rf_loc_recurrence_final_left_effective}
finish{align})

Be aware the resemblance between eqref{eq:rf_loc_recurrence_final_left_effective}
and eqref{eq:rf_loc_recurrence_u}. By utilizing (S_l) and (P_l), one can
compute the places (u_l,v_l) for characteristic map (f_l) given the placement at
the output characteristic map (u_{L}). When one is inquisitive about computing characteristic
places for a given community, it’s helpful to pre-compute three variables:
(P_0,S_0,r_0). Utilizing these three, one can receive (u_0) utilizing
eqref{eq:rf_loc_recurrence_final_left_effective} and (v_0) utilizing
eqref{eq:rf_loc_recurrence_final_right_rewrite}. This permits us to acquire the
mapping from any output characteristic location to the enter area which influences
it.

It is usually doable to derive recurrence equations for the efficient stride and
efficient padding. It’s easy to point out that:

(start{align}
S_{l-1}&= s_l cdot S_l label{eq:effective_stride_recurrence}
P_{l-1}&= s_l cdot P_l + p_l label{eq:effective_padding_recurrence}
finish{align})

These expressions will probably be helpful when deriving an algorithm to resolve the case
for arbitrary computation graphs, offered within the subsequent part.

Heart of receptive area area.
It is usually attention-grabbing to derive an
expression for the middle of the receptive area area which influences a
specific output characteristic. This can be utilized as the placement of the characteristic in
the enter picture (as performed for current
deep learning-based native options , for
instance).

We outline the middle of the receptive area area for every layer (l) as
(c_l = frac{u_l + v_l}{2}). Given the above expressions for (u_0,v_0,r_0),
it’s easy to derive (c_0) (keep in mind that (u_{L}=v_{L})):

(start{align}
c_0&= u_{L}prod_{i=1}^{L}s_i
– sum_{l=1}^{L}
left(p_l – frac{k_l – 1}{2}proper)prod_{i=1}^{l-1} s_i nonumber &= u_{L}cdot S_0
– sum_{l=1}^{L}
left(p_l – frac{k_l – 1}{2}proper)prod_{i=1}^{l-1} s_i
nonumber &= -P_0 + u_{L}cdot S_0 + left(frac{r_0 – 1}{2}proper)
label{eq:rf_loc_recurrence_final_center_effective}
finish{align})

This expression could be in comparison with
eqref{eq:rf_loc_recurrence_final_left_effective} to look at that the middle is
shifted from the left-most pixel by (frac{r_0 – 1}{2}), which is sensible.
Be aware that the receptive area facilities for the completely different output options are
spaced by the efficient stride (S_0), as anticipated. Additionally, it’s attention-grabbing to
be aware that if (p_l = frac{k_l – 1}{2}) for all (l), the facilities of the
receptive area areas for the output options will probably be aligned to the primary
picture pixel and positioned at ({0, S_0, 2S_0, 3S_0, ldots}) (be aware that on this
case all (k_l)’s have to be odd).

Different community operations.
The derivations offered on this part cowl most elementary operations on the
core of convolutional neural networks. A curious reader could also be questioning
about different commonly-used operations, comparable to dilation, upsampling, and many others. You
can discover a dialogue on these in the appendix.

Arbitrary computation graphs

Most state-of-the-art convolutional neural networks at present (e.g.,
ResNet or
Inception ) depend on fashions
the place every layer might have a couple of enter, which
signifies that there is perhaps a number of completely different paths from the enter picture to the
remaining output characteristic map. These architectures are normally represented utilizing
directed acyclic computation graphs, the place the set of nodes (mathcal{L})
represents the layers and the set of edges (mathcal{E}) encodes the
connections between them (the characteristic maps stream by the sides).

The computation offered within the earlier part can be utilized for every of the
doable paths from enter to output independently. The state of affairs turns into
trickier when one needs to take into consideration all completely different paths to seek out the
receptive area dimension of the community and the receptive area areas which
correspond to every of the output options.

Alignment points.
The primary potential challenge is that one output characteristic might
be computed utilizing misaligned areas of the enter picture, relying on the
path from enter to output. Additionally, the relative place between the picture areas
used for the computation of every output characteristic might differ. As a consequence,
the receptive area dimension might not be shift-invariant
 . That is illustrated within the
determine under with a toy instance, during which case the facilities of the areas used
within the enter picture are completely different for the 2 paths from enter to output.

On this instance, padding is used just for the left department. The primary three layers
are convolutional, whereas the final layer performs a easy addition.
The relative place between the receptive area areas of the left and
proper paths is inconsistent for various output options, which ends up in a
lack of alignment (this may be seen by hovering over the completely different output options).
Additionally, be aware that the receptive area dimension for every output
characteristic could also be completely different. For the second characteristic from the left, (6) enter
samples are used, whereas solely (5) are used for the third characteristic. This implies
that the receptive area dimension might not be shift-invariant when the community is just not
aligned.

For a lot of laptop imaginative and prescient duties, it’s extremely fascinating that output options be aligned:
“image-to-image translation” duties (e.g., semantic segmentation, edge detection,
floor regular estimation, colorization, and many others), native characteristic matching and
retrieval, amongst others.

When the community is aligned, all completely different paths result in output options being
centered persistently in the identical places. All completely different paths will need to have the
similar efficient stride. It’s straightforward to see that the receptive area dimension will probably be
the most important receptive area amongst all doable paths. Additionally, the efficient
padding of the community corresponds to the efficient padding for the trail with
largest receptive area dimension, such that one can apply
eqref{eq:rf_loc_recurrence_final_left_effective},
eqref{eq:rf_loc_recurrence_final_center_effective} to localize the area which
generated an output characteristic.

The determine under offers one easy instance of an aligned community. On this case,
the 2 completely different paths result in the options being centered on the similar
places. The receptive area dimension is (3), the efficient stride is (4) and
the efficient padding is (1).

Alignment standards
 . Extra exactly, for a community to be aligned at each
layer, we want each doable pair of paths (i) and (j) to have
(c_l^{(i)} = c_l^{(j)}) for any layer (l) and output characteristic (u_{L}). For
this to occur, we are able to see from
eqref{eq:rf_loc_recurrence_final_center_effective} that two circumstances have to be
glad:

(start{align}
S_l^{(i)}&= S_l^{(j)} label{eq:align_crit_1}
-P_l^{(i)} + left(frac{r_l^{(i)} – 1}{2}proper)&= -P_l^{(j)} + left(frac{r_l^{(j)} – 1}{2}proper)
label{eq:align_crit_2}
finish{align})

for all (i,j,l).

Algorithm for computing receptive area parameters: sketch.
It’s easy to develop an environment friendly algorithm that computes the receptive
area dimension and related parameters for such computation graphs.
Naturally, a brute-force method is to make use of the expressions offered above to
compute the receptive area parameters for every route from the enter to output independently,
coupled with some bookkeeping as a way to compute the parameters for the whole community.
This technique has a worst-case complexity of
(mathcal{O}left(left|mathcal{E}proper| occasions left|mathcal{L}proper|proper)).

However we are able to do higher. Begin by topologically sorting the computation graph.
The sorted illustration arranges the layers so as of dependence: every
layer’s output solely will depend on layers that seem earlier than it.
By visiting layers in reverse topological order, we make sure that all paths
from a given layer (l) to the output layer (L) have been taken under consideration
when (l) is visited. As soon as the enter layer (l=0) is reached, all paths
have been thought of and the receptive area parameters of the whole mannequin
are obtained. The complexity of this algorithm is
(mathcal{O}left(left|mathcal{E}proper| + left|mathcal{L}proper|proper)),
which is a lot better than the brute-force different.

As every layer is visited, some bookkeeping have to be performed as a way to maintain
observe of the community’s receptive area parameters. Specifically, be aware that
there is perhaps a number of completely different paths from layer (l) to the output layer
(L). To be able to deal with this example, we maintain observe of the parameters
for (l) and replace them if a brand new path with bigger receptive area is discovered,
utilizing expressions eqref{eq:rf_recurrence}, eqref{eq:effective_stride_recurrence}
and eqref{eq:effective_padding_recurrence}.
Equally, because the graph is traversed, it is very important verify that the community is aligned.
This may be performed by ensuring that the receptive area parameters of various paths fulfill
eqref{eq:align_crit_1} and eqref{eq:align_crit_2}.

Dialogue: receptive fields of contemporary networks

On this part, we current the receptive area parameters of contemporary
convolutional networks

The fashions used for receptive area computations, in addition to the accuracy reported on ImageNet experiments,
are drawn from the TF-Slim
image classification model library.

 , which have been computed utilizing the brand new open-source
library (script
here).
The pre-computed parameters for
AlexNet ,
VGG ,
ResNet ,
Inception
and
MobileNet
are offered within the desk under.
For a extra
complete record, together with intermediate community end-points, see
this
table.

As fashions developed, from
AlexNet, to VGG, to ResNet and Inception, the receptive fields elevated
(which is a pure consequence of the elevated variety of layers).
In the latest networks, the receptive area normally covers the whole enter picture:
which means the context utilized by every characteristic within the remaining output characteristic map
consists of the entire enter pixels.

We will additionally relate the expansion in receptive fields to elevated
classification accuracy. The determine under plots ImageNet
top-1 accuracy as a operate of the community’s receptive area dimension, for
the identical networks listed above. The circle dimension for every knowledge level is
proportional to the variety of floating-point operations (FLOPs) for every
structure.

We observe a logarithmic relationship between
classification accuracy and receptive area dimension, which suggests
that giant receptive fields are obligatory for high-level
recognition duties, however with diminishing rewards.
For instance, be aware how MobileNets obtain excessive recognition efficiency even
if utilizing a really compact structure: with depth-wise convolutions,
the receptive area is elevated with a small compute footprint.
Compared, VGG-16 requires 27X extra FLOPs than MobileNets, however produces
a smaller receptive area dimension; even when rather more complicated, VGG’s accuracy
is simply barely higher than MobileNet’s.
This means that networks which may effectively generate giant receptive
fields might take pleasure in enhanced recognition efficiency.

Allow us to emphasize, although, that the receptive area dimension is just not the one issue contributing
to the improved efficiency talked about above. Different components play a vital
position: community depth (i.e., variety of layers) and width (i.e., variety of filters per layer),
residual connections, batch normalization, to call just a few.
In different phrases, whereas we conjecture that a big receptive area is important,
in no way it’s enough.

Further experimentation is required to verify this speculation: for
instance, researchers might experimentally examine how classification
accuracy modifications as kernel sizes and strides differ for various
architectures. This may occasionally point out if, a minimum of for these architectures, a
giant receptive area is important.

Lastly, be aware {that a} given characteristic is just not equally impacted by all enter pixels inside
its receptive area area: the enter pixels close to the middle of the receptive area have extra “paths” to affect
the characteristic, and consequently carry extra weight.
The relative significance of every enter pixel defines the
efficient receptive area of the characteristic.
Current work
supplies a mathematical formulation and a process to measure efficient
receptive fields, experimentally observing a Gaussian form,
with the height on the receptive area heart. Higher understanding the
relative significance of enter pixels in convolutional neural networks is
an energetic analysis matter.

Fixing recurrence equations: receptive area dimension

The primary trick to resolve
eqref{eq:rf_recurrence} is to multiply it by (prod_{i=1}^{l-1} s_i):

(start{align}
r_{l-1}prod_{i=1}^{l-1} s_i& = s_l cdot r_{l}prod_{i=1}^{l-1} s_i + (k_l – s_l)prod_{i=1}^{l-1} s_i
nonumber & = r_{l}prod_{i=1}^{l} s_i + k_lprod_{i=1}^{l-1} s_i – prod_{i=1}^{l} s_i
label{eq:rf_recurrence_mult}
finish{align})

Then, outline (A_l = r_lprod_{i=1}^{l}s_i), and be aware that
(prod_{i=1}^{0}s_i = 1) (since (1) is the impartial aspect for
multiplication), so (A_0 = r_0). Utilizing this definition,
eqref{eq:rf_recurrence_mult} could be rewritten as:

start{equation}
A_{l} – A_{l-1} = prod_{i=1}^{l} s_i – k_lprod_{i=1}^{l-1} s_i
label{eq:rf_recurrence_adef}
finish{equation}

Now, sum it from (l=1) to (l=L):

(start{align}
sum_{l=1}^{L} left(A_{l} – A_{l-1} proper) = A_{L} – A_0 = sum_{l=1}^{L}
left(prod_{i=1}^{l} s_i – k_lprod_{i=1}^{l-1} s_i proper)
label{eq:rf_recurrence_sum_a}
finish{align})

Be aware that (A_0 = r_0) and (A_{L} = r_{L}prod_{i=1}^{L}s_i =
prod_{i=1}^{L}s_i). Thus, we are able to compute:

(start{align}
r_0&= prod_{i=1}^{L}s_i + sum_{l=1}^{L} left(k_lprod_{i=1}^{l-1} s_i
– prod_{i=1}^{l} s_i proper) nonumber &= sum_{l=1}^{L}k_lprod_{i=1}^{l-1}
s_i-sum_{l=1}^{L-1}prod_{i=1}^{l} s_i
nonumber &= sum_{l=1}^{L}k_lprod_{i=1}^{l-1} s_i-sum_{l=1}^{L}prod_{i=1}^{l-1}s_i
+ 1 label{eq:rf_recurrence_almost_final}
finish{align})

the place the final step is finished by a change of variables for the appropriate time period.

Lastly, rewriting eqref{eq:rf_recurrence_almost_final}, we receive the
expression for the receptive area dimension (r_0) of an FCN on the enter picture,
given the parameters of every layer:

start{equation}
r_0 = sum_{l=1}^{L} left((k_l-1)prod_{i=1}^{l-1} s_iright) + 1
finish{equation}

Navigate back to the main text

Fixing recurrence equations: receptive area area

The derivations are much like the one we use
to resolve eqref{eq:rf_recurrence}. Let’s contemplate the computation of (u_0).
First, multiply eqref{eq:rf_loc_recurrence_u} by (prod_{i=1}^{l-1} s_i).

(start{align}
u_{l-1}prod_{i=1}^{l-1} s_i& = u_{l} cdot s_lprod_{i=1}^{l-1} s_i – p_lprod_{i=1}^{l-1} s_inonumber
& = u_{l}prod_{i=1}^{l} s_i – p_lprod_{i=1}^{l-1} s_i
label{eq:rf_loc_recurrence_mult}
finish{align})

Then, outline (B_l = u_lprod_{i=1}^{l}s_i), and rewrite
eqref{eq:rf_loc_recurrence_mult} as:

start{equation}
B_{l} – B_{l-1} = p_lprod_{i=1}^{l-1} s_i
label{eq:rf_loc_recurrence_adef}
finish{equation}

And sum it from (l=1) to (l=L):

(start{align}
sum_{l=1}^{L} left(B_{l} – B_{l-1} proper) = B_{L} – B_0 =
sum_{l=1}^{L} p_lprod_{i=1}^{l-1} s_i label{eq:rf_loc_recurrence_sum_a}
finish{align})

Be aware that (B_0 = u_0) and (B_{L} = u_{L}prod_{i=1}^{L}s_i). Thus, we are able to
compute:

(start{align}
u_0&= u_{L}prod_{i=1}^{L}s_i – sum_{l=1}^{L}
p_lprod_{i=1}^{l-1} s_i
finish{align})

Navigate back to the main text

Different community operations

Dilated (atrous) convolution.
Dilations introduce “holes” in a convolutional kernel. Whereas the variety of
weights within the kernel is unchanged, they’re not utilized to spatially
adjoining
samples. Dilating a kernel by an element of (alpha) introduces striding of
(alpha) between the samples used when computing the convolution. This
signifies that the spatial span of the kernel ((okay>0)) is elevated to (alpha (k-1) + 1).
The above derivations could be reused by merely changing the kernel dimension
(okay) by (alpha (k-1) + 1) for all layers utilizing dilations.

Upsampling.
Upsampling is regularly performed utilizing interpolation (e.g., bilinear, bicubic
or nearest-neighbor strategies), leading to an equal or bigger receptive
area — because it depends on one or a number of options from the enter.
Upsampling layers usually produce output options which rely regionally on
enter options, and for receptive area computation functions could be
thought of to have a kernel dimension equal to the variety of enter options
concerned within the computation of an output characteristic.

Separable convolutions.
Convolutions could also be separable by way of spatial or channel dimensions.
The receptive area properties of the separable convolution are
similar to its corresponding equal non-separable convolution. For
instance, a (3 occasions 3) depth-wise separable convolution has a kernel
dimension of (3) for receptive area computation functions.

Batch normalization.
At inference time, batch normalization consists of feature-wise operations
which don’t alter the receptive area of the community. Throughout coaching,
nevertheless, batch normalization parameters are computed primarily based on all
activations from a particular layer, which signifies that its receptive area is
the entire enter picture.

Navigate back to the main text

Acknowledgments

We wish to thank Yuning Chai and George Papandreou for his or her cautious
assessment of early drafts of this manuscript.
Relating to the open-source library, we thank Mark Sandler for serving to with
the starter code, Liang-Chieh Chen and Iaroslav Tymchenko for cautious
code assessment, and Until Hoffman for enhancing upon the unique code launch.
Thanks additionally to Mark Sandler for help with mannequin profiling.

References

1. Visualizing higher-layer options of a deep community  [PDF]
   Erhan, D., Bengio, Y., Courville, A. and Vincent, P., 2009. College of Montreal, Vol 1341, pp. 3.
2. Visualizing and Understanding Convolutional Networks  [PDF]
   Zeiler, M.D. and Fergus, R., 2014. Proc. ECCV.
3. World Optimality in Neural Community Coaching  [PDF]
   Haeffele, B. and Vidal, R., 2017. Proc. CVPR.
4. On the World Convergence of Gradient Descent for Over-parameterized Fashions utilizing Optimum Transport  [PDF]
   Chizat, L. and Bach, F., 2018. Proc. NIPS.
5. Object Detectors Emerge in Deep Scene CNNs  [PDF]
   B. Zhou, A.Ok. and Torralba, A., 2015. Proc. ICLR.
6. Decoding Deep Visible Representations through Community Dissection  [PDF]
   Zhou, B., Bau, D., Oliva, A. and Torralba, A., 2018. IEEE Transactions on Sample Evaluation and Machine Intelligence.
7. In Search of the Actual Inductive Bias: On the Function of Implicit Regularization in Deep Studying  [PDF]
   Neyshabur, B., Tomioka, R. and Srebro, N., 2014.
8. Understanding Deep Studying Requires Rethinking Generalization  [PDF]
   Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2017. Proc. ICLR.
9. A Information to Receptive Area Arithmetic for Convolutional Neural Networks  https://distill.pub/2019/computing-receptive-fields
   Dang-Ha, T., 2017.
10. What are the Receptive, Efficient Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?  [PDF]
    Le, H. and Borji, A., 2017.
11. ImageNet Classification with Deep Convolutional Neural Networks  [PDF]
    Krizhevsky, A., Sutskever, I. and Hinton, G., 2012. Proc. NIPS.
12. Very Deep Convolutional Networks for Giant-Scale Picture Recognition  [PDF]
    Simonyan, Ok. and Zisserman, A., 2015. Proc. ICLR.
13. Deep Residual Studying for Picture Recognition  [PDF]
    He, Ok., Zhang, X., Ren, S. and Solar, J., 2016. Proc. CVPR.
14. Inception-v4, Inception-ResNet and the Impression of Residual Connections on Studying  [PDF]
    Szegedy, C., Ioffe, S., Vanhoucke, V. and Alemi, A., 2016.
15. Giant-Scale Picture Retrieval with Attentive Deep Native Options  [PDF]
    Noh, H., Araujo, A., Sim, J., Weyand, T. and Han, B., 2017. Proc. ICCV.
16. MobileNets: Environment friendly Convolutional Neural Networks for Cell Imaginative and prescient Purposes  [PDF]
    Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M. and Adam, H., 2017.
17. Understanding the Efficient Receptive Area in Deep Convolutional Neural Networks  [PDF]
    Luo, W., Li, Y., Urtasun, R. and Zemel, R., 2016. Proc. NIPS.

Updates and Corrections

In the event you see errors or need to recommend modifications, please create an issue on GitHub.

Reuse

Diagrams and textual content are licensed beneath Inventive Commons Attribution CC-BY 4.0 with the source available on GitHub, except famous in any other case. The figures which have been reused from different sources don’t fall beneath this license and could be acknowledged by a be aware of their caption: “Determine from …”.

Quotation

For attribution in tutorial contexts, please cite this work as

Araujo, et al., "Computing Receptive Fields of Convolutional Neural Networks", Distill, 2019.

BibTeX quotation

@article{araujo2019computing,
  writer = {Araujo, André and Norris, Wade and Sim, Jack},
  title = {Computing Receptive Fields of Convolutional Neural Networks},
  journal = {Distill},
  yr = {2019},
  be aware = {https://distill.pub/2019/computing-receptive-fields},
  doi = {10.23915/distill.00021}
}