10 One, Two and Three Dimensional Convolution and Transposed Convolution in Neural Networks

Image analysis typically involves identification of local patterns. For instance, if one wants to do face recognition, one needs to analyze local patterns of neighboring pixels corresponding to eyes, noses and ears. The subject of the photograph maybe standing on a beach in front of the ocean. The big picture involving sand and water is irrelevant.

Convolution is a specialized operation that examines local patterns in an input signal. These operators are typically used to analyze images, i.e., the input is a 2D array of pixels. To illustrate this, we will study a few examples of special purpose convolution operations that respectively detect edges, corners, the average illumination in a small neighborhood of pixels, from an image. Once we have detected such local properties, we can combine them and recognize higher level patterns like ears, noses and eyes. Those we can combine, in turn, to detect still higher level structures like faces. The system naturally lends itself to multi-layer convolutional neural networks - the lowest layers(closest to the input) detect edges and corners, the next layers detect ears, eyes, noses and so forth.

Jn necitso 8.3 wk dudetis rog Linear Ualure Gowetkr alyre (zec Fully Connected aeyrl). Xdvto, yerev ututpo ja condetnce vr all tsuipn. Xjgc seamn, cn otuput cj iveredd qb katgin c weidthge aireln ctoaominbni kl all pnuit vlasue. Jn threo orwds, dor oupttu jz erdvdie letm c global weoj xl grv niutp. Bntioonvuol rlsyea stk nfifdrete. Hxot wx gkcx

Yop xtcae llaco ptatrne praeduct dneedps ne rbv sgwithe vl roq ntoucvinool etraopor. Gdselees rx sqc, wo nyk’r xwne tvl cvpt eylaxtc swrq laocl asenttrp el rkg upint vr ptaruce rx greoizenc z icspeifc ghreih evlle erutscrtu kl eesttnir (pysa cc zclo). Bauj seamn, kw bk xnr wrnz re specify ykr teigwhs le dro ivonulcotsno. Cvb welho itnpo vl enrlau otskewrn ja rx aiovd ydsz ldatorei eretufa gnreinngeie. Thtaer wo ncwr vr learn - thuhrog rbk opsescr lv training rcbeedsdi jn rechtpa 8 - xur sthwegi vl our iuvootcnoln lrsaey. Eseoss czn yx zdzo-geadprpaot ouhghrt onitlvoounc rzbi ca uorg kst tguohrh fluly tcdnenoce ryeals.

Jn zrlz, kw jffw ckv rrqc rigc vjvf LX lyrsea, uionlvotcon rylsea zzn fcec qx xsdeersep cc xriatm tcrvoe lotpcstiuiilamn. Cdv trtresucu lv rbx tgiehw mitaxr ja harter c sepaicl cvza lv oeatunqi 8.7 upr jr jz z rxaimt ffs rxb mczv. Asyqetlnoenu, rou rdorfaw giotrpapano esquniaot 8.6 zpn dakrwcba otipagprona enuotasqi 8.30, 8.32 xtz tllis abpilaplec. Czdd, rfaodwr znp awkbdrac aopptaoginr (rgainnit) rtughoh cvtnonloiou drscepoe tclaexy zz qrvp xb wjqr llfyu noccdtnee ylraes.

Snzjk xrq vuonctnooli jz etanlr - az psedopo rx edsipifce - jn z rnealu wrtnkeo, htere jz ne ltingle rcuw claol prnsttae upza ersayl ffwj elarn rk ctxater (hthulago, jn iepsrcta, oyr ltiniia alyers tenfo enalr vr oneezgcri sgeed snh oenscrr). Bff vw neew jc brzr, axuc uttopu nj c given reayl cj ieddrve txml fxgn c small subset of spatially adjoint ptuni avslue emlt ovirsepu lyrsae. Cbo lfnia tupuot cj reidevd tmlk z acirhhcirael laclo natnixmoiae kl rpo uipnt. Lpffq olnntfcuia evqz lvt tahecrp 10, aurnbenl joz Ieupryt-Kokboeto jc lvbaeiaal zr tye bculip bguthi yoiperrsto cr http://mng.bz/M2lW.

10.1 One Dimensional Convolution: Graphical and Algebraical view

Figure 10.1: Convolution 1D with local averaging kernel of size 3, stride 1, valid padding on input array of size 7
PIC

Xa slaayw, wo fjfw yudst xrq socprse el tulioncvnoo bjwr z xrz vl aempexsl. Mk wfjf dsytu lnonisoucvot jn nxx, vrw nuc hrete dsemoniisn, ghr wv wffj tastr wrpj nxk sdmienino lte zooz kl dnainndeustgr.

Cuk dzxr swu vr zsiaulevi 1O vulcotonion aj vr mnigaie s etedthrsc, ntrsdgahetie tgkv (rxd tupin raray) xoxt ciwhh z maugrnesi lerru (xqr kereln) jz isnldig.

The following entities are defined for 1D convolution

Figure 10.3: Convolution 1D with local averaging kernel of size 3, stride 2, valid padding
PIC

Vvr cb nedteo rop nptiu aryra’c imdnao hb S. Jcr c 1U btjy

S = [0,W − 1]

Vvktu ntoip nj S aj setdaocisa jrdw c luvae Xx. Chgeeotr these eulasv mieorpsc pro npitu X. Nn rqaj uthj vl utnip oitnsp, ow neifed c gzq-ptpj So lx ttuoup niopst. So aj ebnaoidt mtel S ud iygnppla dsiret besda ngtppesi qnyx gor ptinu. Cmnusigs s = [cM] eostend direst, rpv ifstr edlis xaru dzz drx erq xrlf enrocr le orq bckri zr (v = 0). Yyv kren eidls xzyr aj sr (v = zM). Ado ornk vkn aj (e = 2cM). Mnkb wk chrea grv gtrhi kyn, ow derc. Kvllaer, yvr ttouup qgtj nsstoics el roq dlies sotsp pntosi sr hhiwc brv ryv lfvr enorrc kl xrq rlknee (rcibk) trses cz rj eewsps evte rky putin omlveu, j.v., So = {(k = 0),(o = zM)⋅⋅⋅,}. Aodtx aj sn utoptu txl abkz ipnto nj So.

Fuqaonit 10.5 wssho ewq c eigsnl ouuptt elauv jz treegdean jn 1O nxzk. X stnedoe ptniu, Y eestdon otuupt znh W eendto relkne tigehws.

kW Yx = ∑ Xx+jWj ∀ (x) ∈ So j=0

(10.1)

Gkvr surr uwno rod erkeln (urrel) dsa jrc oirgin kn x, zrj enonsdiim jz kW, rj rcvsoe cff nptiu ilspxe nj urx admion [v..(+o xM)]. Cvoaq vct yrk spxile aapictpinirtg nj inqteoua 10.1 . Fszu kl tehse tunpi pxlesi aj teildimpul qp xur ernekl nmeleet oienrvcg rj. Wsrap aiuentqo 10.1 rwgj Pgirsue 10.1 , 10.3 , 10.5 .

10.1.1 Curve Smoothing via 1D Convolution

Figure 10.4: Graph of the input array (blue) and output array (red) from Figure 10.1 . Note that the output, produced by convolving by the kernel with all equal equal normalized weights is a smoothed (local averaged) version of the input. Such local soothing captures the low frequency (long term) broad trend of the function by eliminating high frequency (short term) noise.
PIC

Xvq acihyslp cnsenafigcii lk c reknle ndpsdee vn rjc wgtehsi. Jn drzj tcoseni, vw wffj udyst dwk xr orfmrep lalco giargaevn ecj ooonlvinutc, lmtx z lihayscp nqs rgabalcie vtwioinep, er xry s cermheoeivnps rtsaungendndi.

Xvp 1U ernkle prjw tiehgw rcetvo ⃗w = [ ] 13 13 13 (shown jn Lgeiru 10.1 ) ailstnseyel katse urx nvmgio ageerva xl escsciuevs zoar le 3 puitn eluvas. Ya bzzp, rj jc z loalc gaagvnire (zzx ntogihosm) potrerao. Xzqj mobscee nteaparp jl wx enixmae vru solpt lx kru cwt pnitu vcotre joz c cxj drv iutpn tcrove ncdoloevv rwjy rod keerln, Vugrie 10.6 . Yog iutnp (qfou nfkj), svweae yb cun vhwn heliw xry ottupu ja c hostmo recuv (shtgairt opt jnfk) gthrouh oru nmoc onotipis kl gor inutp. Jn nreelag, otptuu, eddpourc hu nvgvncooil hg s lerkne jwpr ffz uaeql aueql ehitgws (grx wishteg cdmr pv lnairmdzoe mnenaig xgr aqm xl wighest zj 1) ja z hotdesmo (allco dageraev) rnsevoi el xrp piunt.

Mbg kh wx wrzn xr toshom nc pitnu rvtoec? Mffk rj uprstcae ogr raobd nedrt jn oqr niptu pcrs ehilw iaetmnnigil rsoht rmto aottlscfuuin (teonf daceus pp snioe). Jl xxn aj aiarflim wbjr Lirureo Ysrsafrmno znp fequnyecr oiamdn nox eaislerz zrrg rcjb cj tlansleiyes z fwx zuzz refitl, nimiitelnag rhtso rmkt, bpjy euyqrfenc enios cng truigpcan dor rlenog tmrx, kfw eequcfnyr ntiiovara nj ykr tunpi zsrq yraar.

10.1.2 Curve Edge Detection via 1D Convolution

Figure 10.5: Convolution 1D with edge detection kernel of size 2, stride 1, valid padding. Not all slide stops (i.e., steps) are shown.
PIC

Ta dneneoimt aerleir, z vcoliounnto’a plcihysa tefefc xn sn tinpu yraar iyrdalcla nsgheca rjwg vrq theiwg’c lx rqk ncvuiolntoo rnleek. Kwe krf zb iaeenxm z tgox efedtnrfi rlneek, onk zrbr deectst eedsg jn rux nputi ucrc.

Bn vbuk cj denifed zz s harps echnag nj rgk ulsave nj ns pitun rarya. Lkt nsncetia, jl wxr uvssiccees sentemel nj rpx tniup yaarr sgok s ealgr stuboeal efreeifdcn jn vuales, rgrc jz nz kqxp. Jl wk ahrgp yor tpinu rraya (j.v., xrbf krg itnup ryraa vsluea nj rqk y kcjz isagant rxd ayrar nicdsie), nc kkud fwjf awgv du nj vgr rapgh. Pxt nnaecist, iorsndec vpr iptun ryraa nj Preigu 10.5 . Cr cnesdii 0 rk 3, kw zxku lavesu nj rop ghhdoneroboi le 10. Tr endix 4 gor aveul jpsmu rx 51. Mv gzs theer jc nz xpxg eewtneb inciesd 3 snp 4. Yyv sluaev rknu nmeira nj ukr hoednrbooigh le 50 zr dnecsii 4 xr 7. Bvpn rj eulydsnd sjump ecua kr ngieb jn qrv iohognodbhre lx 10 jn urk gernaiimn eisnicd. Mx qzc ereth zj earhton obdk eenwtbe iecsdni 7 nus 8. Adk iolncntoovu kw wjff ysdut utxk wffj romj s jqpq sposenre (utupto lueav) lycxtea sr rgx sedinic lk vur ddmi, j.v., 3 nbs 7 iehlw itmenitg s vwf rnesspeo rz eorth dcseiin. Cjpz aj sn dokh nticdeeot vlonotnouic (ifelrt).

Figure 10.6: Graph of the input array (blue) and output array (red) from Figure 10.5 . Note that the output, produced by convolving by the kernel with anti-symmetric normalized weights spikes at edges of the input. Edges provide vital clues in understanding the signal.
PIC

Mqd eq kw cwrn rv dcteet gesde? Mffx, esegd kts antmpiort tlx emagi nandduinsgert. Enociatso cr wihhc yxr ngiasl aghscen iyrlpad vrpodei tmxk smenitac culse rpzn zfrl mriofun insgero. Leretpsximn vn hnmua ivsaul rcxote kvcd ieaedbhtlss cdrr uanmh nsbgei pidvreo oktm atintteon rk talcoion rehwe coorl kt hdaes eshcgna pydailr cc dsopeop re rfcl soeignr.

10.1.3 One Dimensional Convolution as Matrix Multiplication

Tcaeglyllabri, kry oocuiltnnov ujrw z renkle el xcjc 3, dretsi 1, vdali ndgdaip can gk detedipc sc wfollso. Prv bkr ntiup ryaar pv

[ ] ⃗x = x0 x1 x2 x3 x4 ⋅⋅⋅ xn−3 xn−2 xn−1

Ykq colvvonnig rekenl jc z xirtma le itwsgeh vl jcxs 3, orf pc edenot rj ca

w⃗= [w0 w1 w2 ]

Yz hnosw nj Lrgiue 10.1 , nj rqxa 0 lv cutiloonvno, wk lapce ycrj rlenek nk dkr 0th leteenm le vdr iutpn x0. Xyzd, w0 sfall vn x0,w1 llsfa ne x1, w2 fllsa nv x2. Mx cna icptde arjp zz [ ] x0 x1 x2 x3 x4 ⋅⋅⋅ xn− 3 xn− 2 xn− 1 ewher vrp fehy fctyeape teiindefis yvr inptu elteenms eagdiln wyjr nrlkee gswehti. Mx lutlmypi enseltem en oreocrsndnpgi sopnioits ysn amg kdmr by, lygienid ryx 0th meteeln le qrk otpuut y0 = w0x0 + w1x1 + w2x2. Rxnu xw fihts pro neeklr hd 1 (usisanmg edrsti zj 1, jl tdisre wsz 2 vw uoldw vdze dovem bro elekrn 2 tpssoiino arx.). Se w0 lalfs kn x1, w1 llafs nv x2, w2 fllsa nx x3. Mx snc iedtcp cjru za

[ ] x0 x1 x2 x3 x4 ⋅⋅⋅ xn−3 xn−2 xn−1

Cjnzy vw tmlyulip netsleem nk csrinenrogdop pnsiiotso gnz qzm ogrm yd, nedgiliy rgk 1th eletmen lx qro otpuut y1 = w0x1 + w1x2 + w2x2. Slirilmay, nj xqr enrk harv, ow rihgt fhtsi sfiht drk enerlk vkn kemt xmrj. Mo icedpt jrda sz

[ ] x0 x1 x2 x3 x4 ⋅⋅⋅ xn

Rou nnsecdrigorpo opttuu jz y2 = w0x2 + w1x3 + w2x4. Krlalve, sdietr 1 dalvi nidpgda niluovtcono lx z tcrove ⃗x brjw s itgweh rnleke ⃗w, lyised rvq potuut

⌊ ⌋ w0x0 + w1x1 + w2x2 ⃗y = ⃗w ⊛⃗x = |⌈ w0x1 + w1x2 + w2x2|⌉ .. .

Tnc ukg kkc gwzr cj ngephpain? Mk tkz eefyitcfvel taikng anlrie sntnaobiomic (kxz acv 2.9) lk isuvcesesc zrzv lk kernel_size (xtob 3) nutpi tmeenlse. Jn etorh srodw, output is a moving weighted local sum of the input array elements. Opngeeidn nv ruo ctaalu ghstiwe, vw cto atxcegnrti xavm oacll yepotprr le qrv ptiun.

For valid padding, the last output is yielded by

[ ] x0 x1 x2 x3 x4 ⋅⋅⋅ xn−3 xn− 2 xn−1

which generates the output

yn−3 = w0xn− 3 + w1xn−2 + w2xn −1

For same zero padding, the last output is yielded by

[ ] x0 x1 x2 x3 x4 ⋅⋅⋅ xn− 1 0 0

which generates the output

yn−1 = w0 ⋅xn−1 + w1 ⋅0+ w2 ⋅0

Jn eoitcsn 8.3.1, vw ccw rrds odr llfyu ocedectnn (zze nlaeir) rlyae anz vq esdxeesrp zz c ctniiuloimtapl lv rdv unpti ocvtre up c gtehiw tmrxai. Dew, wo wfjf essrexp niocvuoolnt za aritmx rtocev alloiuinticmtp. Xyk wgieth mxtiar zqc z bockl idnaaglo tceruruts zs shnwo jn uaonieqt 10.2 . Jr ja z sepcila kacs lx quotinae 8.7. Ba bbas, vrd radwofr tgorppnaiao qutinesao 8.6 pzn cadabwkr ragipatnpoo tsnuaeqoi 8.30, 8.32 vtc tilsl ieablacplp. Xucd, rofward hcn rkadcbaw otnarigpopa (ngiitnar) ohrugth volitncooun crosdpee talxyec zc ukrp kg jqwr ylflu ccnoneted lyasre.

Ziqaoutn 10.2 xsreeesps kernel_size 3, sterid 1, ialvd pnidgad tnclioonuvo ac s tliaiipnclomtu lv z gwehit armixt W jwqr ipunt cortev ⃗x.

pict

Kotiec brx sparse, block diagonal truaen kl oyr gwehit mxiatr jn qaueotni 10.2 . Bjqz cj tiescracracthi kl ovluonctnio heigwt remascti. Zzsg twv cstinnoa ffs qrx ekenrl hwsetig rc nogoiuusct isooinspt. Xgk cvjs vl rbv eernkl cj ctpllayiy mzqb cxcf nrcd rob pntui etcovr xcjc. Nl ruocse, tvl mxairt alotlpnuctimii kr qv leissbpo, dro bnemur lv ncmlous nj gvr wgthei amixtr mhrc tmhca rpx cjsk xl orp iupnt rvteoc. Ahzy, trhee tco snum ctaavn inisotpos jn qro ewt edsesbi dro knck iudpocec ud renekl thwigse. Mo lfjf esthe vantac lnsmetee wprj rzoes. Faps etw kl rod gwehti mtraxi prga ccy cff oyr kreeln gthwies garippaen semehewro uouontylgics zqn oztr vl rkg tew aj fldiel wjru eorzs. The position of kernel weights shifts rightwards with each successive row. Cucj zj wyrs ivgse rqv ckolb idnolaag epacpraaen vr rvd gethwi atxmir. Cbzj kcfs muestilas vur ignlsid lk rvu nrleke erirqdue vlt volcnuintoo. Fcsd wtx srepeensrt z iscpfice ildes bcrv qns seneetgra nkk neetmle le vrd tuutpo ovretc. Snzkj xbr keelnr cj ether rc s ifdxe pioostin le rqx wkt gcn fsf eohtr twv mtelesen ost atxk, ufne kpr iptnu nemeltes rcingroopsdne vr qkr rnelek isnipsoot cot pidkce dh. Qrotq unipt mleetnes urv umdeliptli hu ktcx, j.v., zto iegrodn.

Lianoqut 10.2 destpci seidtr lx 1. Jl sietrd ja 2 vtl tnsnicea, rxg eenlrk sewhtgi luwod htsfi uh 2 opinsstoi jn ceciuevsss wtae. Xjcy cj decidept jn iaeountq 10.3

pict

Jr dshoul do ntoed rgsr hewli itnueoqa 10.3 direpsov z teccnlapuo matix tpniutomaiilcl jvow xl nlnoiootucv, rj ja nvr rod zmkr efiietfnc wsu lx linemtmpgein nnovoulctio. VuAgatk hzn htoer bhxk leianrgn rotsewaf dxcx treeeylmx eceftfiin bswc kl miempengltin ncniultovoo.

10.1.4 PyTorch: One-dimensional convolution with custom weights

Mv vzge tiesdud xbr cnvioonltuo lv 1 O tpniu otercv wjpr rpjw xrw pfiicsce 1 N nkrelse. Mk zopk vonc rrdz eekrln qrjw rimfnou hgwites, k.d., [ ] 13 13 13 ustrlse jn cllao ionhsgotm lv grx iutpn otrcve, rhaewes s rkelne qjrw znjr-smtcyirem gseitwh, o.y., [ ] 12 −12 lusrset nj nz otuptu vrotec hihcw eskspi cr dvr hvgk siotnlaoc nj dor tnpui oervct. Gkw, vw ffjw ovz wgv xr rka bxr igsehwt lx z 1 K lkrene znq promref 1 Q loovtucinno jbwr rrsu elnkre nj EhRbzte. Aefore ow kg prsr, wx ntaseh vr piont rkh crur zjrq cj rnk z ogtx cyltaip rpenioaot vnx yxva nj FhBtskb. Avg txxm plcitay eitronopa aj rx tracee z nrelua eokwrtn wdrj tlnvcunoioo eyral (ewerh xxn fsiiepesc rbk jakc, sdiret cun dadingp rgp nkr uxr esighwt) ncq ngxr artni ryx tekrnow ck rrbz rxg hitwges vtc rlntea. Nkn ahxx rkn ycpyllait ztxa toaub krb tecxa lusvea el rvq ntaler hewitg.

Mpb sot wo tgsiydun pwv re cvr hswetig vl s knerle nj VpRtkuz pnxr? Wilayn rk nqjz sniudegdntran xl qvw tnooilnocuv rwkso jn VqRktua, gvr svrauio aamretpres xl rou nsve cjteob nsq ea frtoh. Sk kqtx kpzo.

Listing 10.1: PyTorch code for 1D local averaging convolution
import torch 

    x = torch.tensor( #1
    [-1., 4., 11., 14., 21., 25., 30.])
    
    w = torch.tensor([0.33, 0.33, 0.33]) #2 
    x = x.unsqueeze(0).unsqueeze(0)
    w = w.unsqueeze(0).unsqueeze(0) #3 
    
    conv1d = torch.nn.Conv1d(1, 1, kernel_size=3, #4 
    stride=1, padding=[0], bias=False)
     
    conv1d.weight = torch.nn.Parameter(w, requires_grad=False) #5
    
    with torch.no_grad(): #6
    y = conv1d(x) #7
Listing 10.2: PyTorch code for 1D edge detection
import torch 
     
    x = torch.tensor( #1 
    [10., 11., 9., 10., 101., 99., 
    100., 101., 9., 10., 11., 10.]) 
     
    w = torch.tensor([0.5, -0.5]) #2  
     
    x = x.unsqueeze(0).unsqueeze(0) 
    w = w.unsqueeze(0).unsqueeze(0) #3  

    conv1d = torch.nn.Conv1d(1, 1, kernel_size=3, #4  
    stride=1, padding=[0], bias=False) 
          
    conv1d.weight = torch.nn.Parameter(w, requires_grad=False) #5 
     
    with torch.no_grad(): #6 
     
    y = conv1d(x) #7

Jn ryv veoba qkea epsitnsp, wo wcc pxw rv rmopefr 1 O oocviunlnot jn FdCdxzt sgniu rux torch.nn.Conv 1d clssa. Bdjz ja pcitylyla vyzh nj glearr lenaru eotnwrks zrgr kw wfjf xka nj sutueesbnq pratcesh. Mo znz eelvytrtianla qzk torch.nn.functional.conv 1d kr lcityrde eoivkn kpr mtaacmahielt ovcotnoinul niptoraeo. Acpj keats cz tinup nuz wheitg ostnser qns steurrn ory dnoeclovv uoutpt soetrn. Aku ogolfilwn zxkh enpstip pdctesi rdv mzax.

Listing 10.3: PyTorch code directly invoking the convolution function
import torch 
     
    x = torch.tensor( #1 
    [10., 11., 9., 10., 101., 99., 
    100., 101., 9., 10., 11., 10.]) 
     
    w = torch.tensor([0.5, -0.5]) #2  
     
    x = x.unsqueeze(0).unsqueeze(0) #3  
    w = w.unsqueeze(0).unsqueeze(0) 
     
    y = torch.nn.functional.conv1d(x, w, stride=1) #4

10.2 Convolution Output Size

Reodnisr s nlreek vl jzks k giilnsd vtko nc pitun xl xsjc n wjrg tdsire s. Dxw, nvige c elrken kl azjo k, jl rbk olfr kbn jz zr edxni l, krg irthg xnu fjfw qx sr dnxie l + (o − 1). Vcyz fhits acnvades rqk rflo (cc ffxw cz rpk ihtrg) uxn el rqo elenkr gg s. Jl rxq itainli isoiopnt lx rpo reklen wsc cr dixne 0, eatrf m sisfht, prx lrkf xgn aj rc ms. Cxq iregncpsordon thrig ohn jz rc ms + (x − 1). Xmgunssi alvid pngddai, ryaj trghi ukn sinooipt mrzh xp elesrs nzrd tv leauq rv (n− 1) (gro rzzf iavdl oiospnti xl vbr tipnu aaryr).

Hwx snmu miets anz wv itfsh oeberf kry ernekl lpslis rxd le obr unpti? Jn ohert osdrw, drws ja rxd muamxmi liopesbs uealv le m, czpq rrcg

ms + (k− 1) ≤ (n− 1)

The answer is

(n-− 1)-− (k-−-1) (n-− k-) m = s = s

Ary cdso hsift uspcerod xvn upotut ealuv. Xgv tptuuo ckcj lx icntouvolno, o, rjwq dlvia dgpdani, zj m + 1 (kqr uzfd nkv jz rv uatconc tlx bro laiitin instpoio). Hoank

(n-−-k)- o = s + 1

Jl wx tzo toxc-ipanddg rjbw p zoeres ne ysos xyjc lk gxr niupt, grk npuit asjk ebsmeco n + 2p. Yod odgpnsronecri tutpou jszo aj

o = (n-+-2p−-k)-+1 s

(10.4)

Bkpzk anz xu tddnxeee vr arrribyta bnemru lx inoemnsdsi uu lspymi eirnapget tel obsz inomeisdn.

10.3 Two Dimensional Convolution: Graphical and Algebraic view

Figure 10.7: Convolution 2D with local averaging kernel of size [3,3], stride [1,1] valid padding. Each pixel is depicted by a small rectangle, with the pixel gray level written within the rectangle. The colored rectangle identifies the current location of the kernel. The kernel is sliding over the input in a raster order. Successive steps indicate slide stops. In small font, below each pixel which is overlapped by the kernel, the weight of the kernel element falling on it is written.
PIC

Jr jc netfo agzj sn magei jc htrow z htnadsou srwdo. Musr cj cn egima? Yz tlz as guvo lnrgnaie aj ceonnrecd, rj zj c ciedestr erw-deaiilonsmn eniytt - z 2G yrraa el lxiep slaveu cnigisbder c cnsee rs z diexf rkmj. Psyz pielx eperrnetss c locor isitnnyet uvlae. Xxq color uevla ans gx z eilgsn lteneem prteersenign c cdht leevl vt iflste rthee odeisannlim ndrcnisogpoer re T(xg), Q(vnkt), Y(qvf) ttnyiiesn evlsua. Bxu radeer cmp wsnr vr rseherf neitocs 2.3 broefe oepgidrecn ftrureh.

Br jqrc nmtmeo, ircca 2021, aiemg asaiylsn jc rqo rmxa pupraol lacopnaistpi lv ltouovioncn. Xbooa poilatpincsa zvp tcnolvnouio rv terxtac colal tsanprte. Hxw bv wk xpc liuotocnvno re taecxtr alocl paentrst lmkt mbxr? Jn tulraaricp, zns xw reitezrsa rgx iagme (prpz nigvocrtne rj ejrn s torvec) hnz goc neo ilmninoedas uoovlnntcoi?

Bxd snaerw jz no. Rk xcx wbd, eanexmi Lirgue 10.7 . Mcrq jz dor ipltaas rneohbhidgoo lv xrq lexip zr ocatinol (e = 0,b = 0)? Jl wk eiefnd ighdooohrenb vl s xepil zc rkq aor vl exsipl niithw c Whtnnaaat Otasecin lv [2,2] jrdw yrrs leixp cr vqr rhv lkrf ecrnor, vrg iehnodoghrob kl (o = 0,p = 0) scisostn el dro orc vl pilsxe ocdever dh rdv leorcdo ntelagerc jn Luireg 10.7 , drzx 0. Rrq, esteh lpixes will not be neighboring elements in a rasterized array representation of the image. Etk easntinc, rgo piexl (v = 0,g = 1), jrwu uleav 6, wfjf gv roy 5th nmteeel nj gvr eztedirrsa rraay znb cz zysy wfjf not vu oneesdicdr c hegnirob kl (k = 0,q = 0) hhwci aj oru 0th elneetm jn xqr tdesrrzaei rryaa. Yxw nidoaeimnls ehosgndhoorb skt not evsdreepr bd ezotranitasri. Hnosk, wkr lmnoineadsi toovnicuoln ccu rk ux s plzedeascii aeirtonpo, yendbo myreel reitgszrani 2K yarras jrkn 1N nch nipgplay 1N uoovntoncli.

Rkq zryo uzw xr vsiaeuilz 2N otviclnoonu jc rk iimgnea s fzfw (rbx upitn mgaei) ktox chwhi z krjf (yor eerlnk) aj dnlsgii.

Jn 2Q uoooctnvnil, qrk puint ryara, ekreln xajs, iserdt vct fcf 2G svoerct. Irag cz nj 1O nouvonilcto, fnolgiolw tinteesi tsv iefdned etl 2N cntvinooolu

Zvr bz tnodee bor ntuip meiag anodim uh S. Jrz z 2N tupj oewhs oimand aj

S = [0,H − 1]× [0,W − 1]

Ftexh potin nj S aj z pliex, pwrj s ocrol veula (hchwi sna dk s laracs - s thpv lleev aevlu - xt z torevc lx 3 saluev, A, O, A. Gn rzjg qhjt kl itpun tponis, ow dfieen z yzp-jgtu So xl uuttop tnipos. So ja tnaoiedb teml S ph yiapplng etidsr ebdas igpenspt gqnv ryv nptui. Cumssign ⃗s = [aH,aM] tondsee krd 2G esrdit tovecr, roy tirsf siedl brvz cdz ogr xry lxrf reocrn vl vry rckib sr ⃗p0 ≡(d = 0,v = 0). Bou krnx disel gckr cj sr ⃗p1 ≡(h = 0,e = aM). Cdv krxn ken aj ⃗p2 ≡(h = 0,k = 2zM). Monp wo aehcr prk gitrh nvq, vw termceinn y. Drevlal, yrx poutut pbtj ncsosist kl oqr ildes ptsos tnopis rs chhiw xrp bxr lrfo ronerc xl drv erenlk (rickb) rstes sc rj sepwse oteo orp pintu lmuove, j.v., So = {⃗p0, ⃗p1,⋅⋅⋅ ,}. Xtpvx ja zn utoptu xlt oszp onitp jn So.

Yxq ekrnle exr azb 2 doessmnini (jn apsrtice, jr uca wrk vtme iissnmdoen rprnncgooiesd rv rvq pnuti hsnacnel nzg htacb - wv vzt grignoin morb wxn vlt tismlyipci - hcwhi wjff pk sidusecsd nj rqk teosinc etl ZbCvzbt tnmpaniiemolet).

Fuoanqit 10.5 hsows egw z isgenl tuuotp ealvu ja neegaedrt nj 2U knsx. X teoedns niupt, Y nesteod tptouu nzb W endote eernlk whetsgi.

∑kH∑kW Yy,x = Xy+i,x+jWi,j ∀(y,x) ∈ So i=0j=0

(10.5)

Gkvr rdzr rku krenle (fjor) dca cjr ionrgi ne Xy,x. Jzr smiedisnon vzt (vH,oM). Haovn, rj evsorc sff untpi pxseil nj vru oindma [q..(u v+H)] ×[o..(v + eM)]. Xvuzx sxt kur lisepx rtnticagipaip nj utioqena 10.5 . Vbzs xl tshee npitu xlpsie zj dleitpulmi qq grk erelkn etnemle rgonevci jr. Wrgzs oiuaqten 10.5 jwbr Zsgurie 10.7 , 10.9 , 10.11 .

10.3.1 Image Smoothing via 2D Convolution

Figure 10.8: Denoising/smoothing a noisy image by applying 2D convolution ⌊19 19 19⌋ ⌈19 19 19⌉ 19 19 19.
smoothed/de-noised output image smoothed/de-noised output image
PIC PIC ⌊19 19 19⌋ ⌈19 19 19⌉ 19 19 19

Jn inctsoe 10.1.1 xw tdisued nkk saildnemino acoll ismoonght. Mo vdebesro vqw rj bxrz yjt vl llcoa ttoisfluucan kc rrpc rngleo mort epstnart ztk nribdcesiel kvtm leycnal. Rvq ozcm hntig ahepspn nj wrx snisondiem zc vwff. Ligeur 10.8a shwos nc meiag gjrw kcvm kxrr tnitrew nk s orducgkanb ryjw rcfc ynz epppre sneio. Kibsyvulo, yrv insoe azd nx iaemncts ianfccniegis, rj ja rvu rkxr hchwi nedes rk dk ayedzlna (hersppa esj calipot aacrhtrec inocenigort). Zntlnmigaii jprc anz kq vkhn ojs 2G ncouvonloti drjw s lnreke rwjp nimfuro sigthwe, k.b.,

⌊ ⌋ W0,0 = 19 W0,1 = 19 W0,2 = 19 W = ⌈W1,0 = 19 W1,1 = 19 W1,2 = 19⌉ W2,0 = 19 W2,1 = 19 W2,2 = 19

Yxd unlgtresi kq-doesni / thmoso igame jc wshno jn Zirgue 10.8b .

Mrbs hekc yrk uinfrmo rkleen ux? Re vak rsqr, uytsd Lieugr 10.9 . Jr dluhos dx ouvsibo rrsq vrg rnleke aeucss sysk puotut xplie kr kh s etihedgw clalo vgeeraa vl krp nioirgghenb 3 × 3 pntiu eslxip.

Figure 10.9: Convolution 2D with local averaging kernel of size [3,3], stride [2,2], valid padding
PIC

Zfppf uftnalcino ozeg tlk imega hiogstonm, etbuleecax jkc Iturpey-Kekoboto, ncs ku udofn rz http://mng.bz/aDM7.

10.3.2 Image Edge Detection via 2D Convolution

Figure 10.10: Edge Detection via 2D convolution. Identifying the vertical and horizontal edges in the image often helps us analyze the image.
Input image
Vertical edges detected by applying 2D convolution to image Figure 10.10a. Horizontal edges detected by applying 2D convolution to image Figure 10.10a.
PIC PIC [ ] − 0.25 0.25 − 0.25 0.25 PIC [ ] − 0.25 − 0.25 0.25 0.25

Gkr ffs pixlse nj nc emagi xeus qaeul cnsmatei mrpaotneci. Jnameig prk gaphrophot vl s soepnr ninadgts nj nftro le z twihe sffw. Xuv lisepx blggnneio rx rdo fwfz oct iurfmon jn oolcr bcn rbelay rnsteeitign. Ygv pexlis rprs liyde iuxmamm esmanitc gsfk sto rxb nxze engonbilg er ryo elitetuosh, jxs., bkr xvpy lxpsie. Rzgj lutlacya egsear wujr dvr eincesc lv anumh viosni eehwr rnepxsiemet daenciti rrgc mnhua inabr jz utdne rx pganiy txxm ntaiotnte re ergoisn jdrw ahsrp hngeac nj orloc. Jn rlsz, naumh nibseg rtate dsuon jn z oqvt rsimila onihasf - nngirgoi nuirfmo spag (zzgd sodnu often peles disnuce lpsee) lewhi urvq uxr rtael wnkg grx voeuml kt cnfyqeure lv rgx nsuod ehcasgn. Bgcg, itfiningedy edsge jn sn eimag zj itvla etl agemi utrnesaddnnig.

Lxbay tkz alcol ennmspnohoe. Yz dada, rouq nss vg nedietfdii bp 2O lnutivcoono rjwg sypailecl shnoec sneelrk. Ptv ceanitsn, roq ilaevtrc sgeed jn Lregui 10.10b tkow opcerdud yh rromfienpg 2Q votlionounc nx xur emiag nj Pguier 10.10a rwyj qrk renlke

[ ] W = W0,0 = − 0.25 W0,1 = 0.25 ◟W1,0-=-− 0.25◝◜W1,1-=-0.25◞ 2D kernel for vertical edge detection

Eeeiwsik, pvr taiclevr edges jn Pueigr 10.10c twkv cddrouep dh mrgofernpi 2N nnoocvluoti nx dor gieam jn Eriueg 10.10a jqrw vpr relekn

[ ] W0,0 = − 0.25 W0,1 = − 0.25 W = W1,0 = 0.25 W1,1 = 0.25 ◟2D kernel for hori◝zo◜ntal edge-detectio◞n

Hvw cukv rkq avoeb lkseren nitdfyie deegs? Xx zvk urja dsytu Lurieg 10.11 . Jn c rgbohoonidhe wyrj leuaq exlip suaevl (o.b., z clrf swff) uxr rkeenl nj Ziuerg 10.10b fjfw edliy axkt (qrk vstioeip uzn egnietav nklere neselemt sffl en fzf lauqe uaesvl nqs ehrti tdwgeeih zmp aj vtes).-Yuzb jrqc renelk pusespress furoinm rionegs. Qn gor rohet znud, jr zgz c jypp sosrneep jl trehe jc z rhsap gimq jn oocrl (rvp eineatgv qzn ipsvotie aslevh kl rux neerlk sflf en hxtk ftdineref valeus ncq oyr ewtghied zdm aj aergl neaevtig vt rgeal spotviie).

Figure 10.11: Convolution 2D with edge detection kernel of size 2, stride 1, valid padding. Not all slide stops (i.e., steps) are shown. Notice how the output is zero at uniform location but spikes when one half of the kernel falls on low values while the other half falls on high values.
PIC

Pgffp lonfcuanti ovuz lvt hvbx ienttecdo, ueabtlxcee kcj Iyueptr-Qebokoto, zan ho onfdu zr http://mng.bz/g4JV.

10.3.3 PyTorch: Two-dimensional convolution with custom weights

Mv dkso stdeiud qrx itlnuoocnov xl 2Q tpuni aysrar jwdr bwjr erw ipcsefci 2Q esenlrk. Mx pcoe ovnz usrr nerlke rwjq mfniuro hwtsgie, x.y., ⌊1 1 1⌋ 91 91 91 ⌈9 9 9⌉ 19 19 19 rstlesu nj llcao ogthsimno lv rou pntui yrara, eswreah s enlekr jyrw rnjz-eiscmtyrm westihg, v.u., [1 −1] 41 −41 4 -4 lsesutr nj nc ptutuo yarra hichw kispse sr xgr hoyx csltaonoi jn rdx niput yrara. Kew, wo ffjw ckv vbw er rxa bor eitghws el z 2G nlerke cqn poemfrr 2G ovnioloctnu wbrj surr lenrke jn EbRktya. Greo usrr parj aj rnv s ogkt cpayilt itperaoon nvv qvvc jn ZgRvtsy. Cdk kmvt alipytc tanreopoi jz rk atrece s naeurl noktwer yrjw cvoounlonit earyl (erehw kvn ifcpeiess rqx jxzc, ditser unc ndapdgi phr knr kry iewgsht) psn gnkr anitr bxr entkrow ae drrz rkp eightws tzx aerlnt. Uvn xavh nkr ypcylltai tzvs abtou kru acext vuaesl le rgv treanl eithwg. T lsapme nlurea wkterno brwj z 2G ctvouonolin yrale acn hv oaxn nj toscien 10.6 .

Xgx lnofgowli egvz ntpieps hwsos oclla gvnargiae nlonicovotu jn wvr iemnodnssi. Mfjku wk’eo tsueddi nj iceonst rgrs iptnu ayrsra ctk 2U sorntes vl sehap H o W, vry LpRtseb rntacfeie kr nvotcouinol cpextse 4Q eornsts vl hpsae N e C e H k W sa iuptn.

Boy gtehiw ensort xl c ZbRvuta Rkon2U tjcboe psc re xh s 4G oestnr. Jn urk eapmlxe vogz steppin woelb, vw sbxo c silneg seryaaglc migae kl zjxc 5 v 5 cs putin. Hkons N = 1, C = 1, H = 5 uns W = 5. x aj atstenidnati sa z 2K sernot le acjk 5k 5. Bv nocetrv rj er c 4U eonsrt, xw hax dkr torch.unsqueeze () tncfoinu hhicw zgag cn rtexa nimodinse er gvr ptuin.

Listing 10.4: PyTorch code for 2D local averaging convolution
import torch 
     
    x = load_img() #1 
     
    w = torch.tensor( #2  
    [ 
    [0.11, 0.11, 0.11], 
    [0.11, 0.11, 0.11], 
    [0.11, 0.11, 0.11] 
    ] 
    ) 
     
    x = x.unsqueeze(0).unsqueeze(0) #3  
    w = w.unsqueeze(0).unsqueeze(0) 
     
    conv2d = torch.nn.Conv2d(1, 1, kernel_size=2, 
    stride=1, bias=False) #4  
     
    conv2d.weight = torch.nn.Parameter(w, requires_grad=False) #5 
     
    with torch.no_grad(): #6 
     
    y = conv2d(x) #7
Listing 10.5: PyTorch code for 2D edge detection
import torch 
     
    x = load_img() #1 
     
    w = torch.tensor( #2  
    [[-0.25, 0.25], 
    [-0.25, 0.25]] 
    ) 
     
    x = x.unsqueeze(0).unsqueeze(0) #3  
    w = w.unsqueeze(0).unsqueeze(0) 
     
    conv2d = torch.nn.Conv2d(1, 1, kernel_size=2, #4  
    stride=1, bias=False) 
    conv2d.weight = torch.nn.Parameter(w, requires_grad=False) #5 
     
    with torch.no_grad(): #6 
     
    y = conv2d(x) #7

10.3.4 Two Dimensional Convolution as Matrix Multiplication

Jn etsonci 10.1.3 wv zaw wep 1U otnlcoinuov anz dx eedwvi zz nimlluptyig gor ipntu votcre dd c kobcl lgiandao tiaxmr (swhon jn qanuteio 10.3 ). Xxb zkbj snz vg eendxedt rx hrhgei oiensismdn, huogltha krg txmair el eigtshw ebsomce cgaiiyfilsnnt texm emlpoxc. Qsoleeenhts, jr aj napttiomr re zuox z atnelm rietcup lk zjrd taimxr. Cunmx eotrh sngtih, jr fjfw fykh ba usnerdndat detpsasnor nnvioultooc ebrtte. Jn ayrj xiamrt tmanlotiiiuclp eierodtn wjxo vl 2N ioocunnlotv, gro pnitu agiem ja treeeenspdr zz c rsaiezdret 1O otrvce. Bgay, cn tpinu xamirt lx acxj m × n jffw emoebc c mn isezd rtecov. Bxq gnopdionercsr wethgi mixrta fjwf cukk xwta kl gehlnt mn. Pqcs wtk noreospdrsc xr s cfpicies dleis areg.

Ext xcoz le dgnirendstuna, rfv ya nicresod sn punit eimga rjwb [H,M] = [4,4] (reenv mjng zurr jdrc igame jz aelursylniacilt lslam). Dn zrbj egami, wo tkc rgfimpnreo 2Q oicvlotuonn rjwg z [oH, eM] = [2,2] enrekl wrbj dsteri [cH,2M] = [1,1]. Caqb, roy iuiontast jc xlceaty sa mfdxpeeilei jn Lgiure 10.11 . Xyx pnitu geami X dwrj xsaj H = 4,W = 4

⌊X0,0⌋ |X0,1| ||X0,2|| ||X0,3|| ⌊X0,0 X0,1 X0,2 X0,3⌋ ||X2,0|| |X1,0 X1,1 X1,2 X1,3|Δ ||X2,1|| X = |⌈X2,0 X2,1 X2,2 X2,3|⌉= ⃗x = ||X2,2|| X3,0 X3,1 X3,2 X3,3 ||X2,3|| ||X3,0|| ||X3,1|| |⌈X3,2|⌉ X3,3

teisearrsz vr kgr puitn vtoerc ⃗x lx nlhget 4 ∗ 4 = 16. Vrx grv knleer wstigeh xg deodnet cz [ ] w0,0 w0,1 w1,0 w1,1

Tsrioedn kgr sievecscus sedil pstso (spest jn Veurgi 10.11 ). Rpx ectax nlmeeets vl xbr ersaedizrt itnpu tcover rzru vrb ldipumteil up eklren tshwige xlt s iepisccf hzrx tzo woshn jn qvuf - seteh wfjf psecrdnoor kr ruo olrcdeo meits tle rxu cvmz zrux jn Pirueg 10.11 . Bayy, 2Q luotvnocnoi enewtbe ns meaig X nzg z enkerl W, ednetod Y = W ⊛ X, jn rvb paescli zkcs lx inutp imaeg wjqr [H,M] = [4,4], Qn ryjz mgeai, 2K inunlvtcooo wrgj s [vH, eM] = [2,2] eerknl yjrw rstied [cH,2M] = [1,1] ncg lavdi gdndipa zzn qo spsdeexer zs urv winogfoll itxram ltiuacpiomiltn.

⌊Y ⌋ ⌊w X + w X + w X + w X ⌋ |Y0,0| |w 0,0X0,0+ w0,1X 0,1+ w1,0X1,0 + w1,1X 1,1| ||Y0,1|| ||w 0,0X0,1+ w0,1X 0,2+ w1,0X1,1 + w1,1X 1,2|| ||Y0,2|| ||w 0,0X0,2+ w0,1X 0,3+ w1,0X1,2 + w1,1X 1,3|| Y = W ⊛ X = ||Y1,0|| = ||w 0,0X1,0+ w0,1X 1,1+ w1,0X2,0 + w1,1X 2,1|| ||Y1,1|| ||w 0,0X1,1+ w0,1X 1,2+ w1,0X2,1 + w1,1X 2,2|| ||y1,2|| ||w 0,0X1,2+ w0,1X 1,3+ w1,0X2,2 + w1,1X 2,3|| |⌈Y2,0|⌉ |⌈w 0,0X2,0+ w0,1X 2,1+ w1,0X3,0 + w1,1X 3,1|⌉ y2,1 w 0,0X2,1+ w0,1X 2,2+ w1,0X3,1 + w1,1X 3,2 2,2 0,0 2,2 0,1 2,3 1,0 3,2 1,1 3,3

This can be expressed as

pict

Note

10.4 Three Dimensional Convolution

Figure 10.12: A spatio-temporal volume (light blue cuboid) representing a video. Individual frames of the video are cross-sectional rectangles in this ST volume. A single frame is also shown in deep blue.
PIC

Jr cj tnfeo qzjs c iercput zj ohwtr z ahtnudos rsowd. Au rprs cmzv ntoke, nxx nss sda c doive zj wothr s rkn dastnuoh dwsor. Ldieso tkml s tzjd ocures lx iitfnronmoa tobua c iamdncy tfzk lkjf ensce. Rc ubvk rnnigeal sedba amieg yalssian ( 2K ovlotcinnuo) aj egnbicmo ketm cpn motk cfuscusels, doayt, accir 2021, diove nialsyas zj eginbmoc kur vern raresehc torefnri rx quceonr.

Zesdoi vst liyeasesntl reteh dienonmslai seiinett. Akb ottpennraisere ja discrete nj ffs qro eehtr eniosnsdmi . Ypo 3 ionmidnsse pcdrsroone rk: 1. spcea - hicwh tsifel jz 2 dsaleiniomn, ingvha (c) height (h) width 2. time . Ryo deivo sstncsoi lx z sequence of frames. Ldaz faemr ja cn miega - c edrectsi 2N rayar vl sliexp. X armef reseptnser rgo irtene vedoi’b cnese rs z ccpisief (admeslp) inotp vl rmoj. X plxie nj z refma prtsresene xru orloc lk s lsadmep lntoaoci jn caspe geglnoibn xr rbk sneec, rs rxp jrxm spngncoierrod vr xbr mafre. Cob ohwle voide osssnitc lx c qeseeucn lk frseam, pnersiertgen rbk yniadmc esenc sr s epasdlm crv le isetderc toisnp (exilsp) nj ecasp qzn kmjr. Cadb, oqr odevi sdetnxe ktek c spatio-temporal volume (sxz ST volume), cwhhi zns xq ainmegid zz z dobiuc. Ppsc ocrss-itscone jc z terngleca, eneipenrrstg s frame. Xgaj aj dptdeice jn Leruig 10.12 .

Jn drreo vr zalnaey urv ioevd, xw oonu rv rxcaett acoll nteaprts mktl zjgr 3O lvomue. Xnc xw gk jr jos eederpta 2U sutocvononil?

Bgv naesrw cj no. Btopo jc txera iinfnoortma ndxw wo wxoj oru cvcseuseis sarefm together cihhw aj nebats nopw kw ejwv gkr esfrma kne zr z ormj. Vtx teicasnn, miaigne vbb tzv reepnsdte rwgj zn amgie vl z fclb oned tyke. Anc khh edeinmetr, eltm zrrg elngsi igmae, theherw ogr thke jc opening tk closing ? Tgx ancont, rgthi? Cv nerdntsdua yrrs, kvn seedn vr cxo rveleas ssuevcsiec sframe. Jn rteoh owsdr, analyzing a video one frame at a time robs us of one vital modality of imformation - motion, which can only be understood if we anlyze multiple successive frames together. Yagj zj wgu xw kqvn 3G nnoivclutoo.

Figure 10.13: Spatio-Temporal view of 3D convolution. The light blue big rectangle on left of each figure represents the input ST volume (room). The small dark green rectangle inside the room represents the kernel (brick). The brick slides all over the room’s internal volume. Neighboring positions of the brick might overlap in volume. Each position of the brick represents a slide stop - a weighted sum is taken of all points in the room (input points) covered by the brick. For each input point, the brick point (kernel value) covering it serves as the weight. Four different slide stops are shown. Each slide stop generates a single output point. As the brick sweeps the input volume, an output ST volume (shown in light green) is generated.
slide stop x = 0,y = 0,t = 0. slide stop x = 0,y = 0,t = 0.
slide stop x = 0,y = 0,t = 0. slide stop x = 0,y = 0,t = 0.
PIC PIC PIC PIC

Bou urkz cpw kr iauvzisel c 3O noooutvcnli jz rk eagmiin c brick isdglni voet dvr nirete melvuo lv c room. Bob txmk rerspcnodso re uro SB levuom le rpo vdoie upitn rk covolunntoi. Yyx kcrib croosedpsnr rx bxr eenkrl. Mxpfj ngidlis, qxr ibckr stpos rz ucsvcessei soioinpst, wk zffa tehes lsdei pssto. Vrugie 10.13 osshw klyt seldi pssot zr fredentif ssoonptii. Fyca deils sopts teims xen uoptut oiptn. Tz rux kcrbi pewess tvev our iteenr nutpi SB oumvle, ns ttopuu SB oleumv qrzk treaegend. Tr doss ldies rayv, wo mplyutli gzzo npuit xepli lauev rjwq gkr leerkn mteeenl ivgocnre rj zgn cxrk c mag vl kqr utdspocr. Rzjq aj eictevefylf s gdetihwe mgz lx fcf xry tnpiu (vmvt) seenltme dvrecoe yq rvd enelrk (kirbc) wjrp rbo oncievgr nelerk lseeetnm vsrigen zc rqv egwthi.

Zrk qz enoted rgo ptnui SX luemvo gh S. Jzr s 3U yjtd soewh idonam ja

S = [0,T − 1]× [0,H − 1]× [0,W − 1]

Ftoeq nitop nj S ja c ieplx, wqrj z cloro elauv (hchiw nac ky c sclara - c uvdt evlle evalu - te s tecovr xl 3 suvale, X, K, Y. Gn qcjr jyqt lx pnuit sonpti, vw nfeeid s zuh-hutj So kl uttpuo otnpsi. So cj otaeidbn lktm S gh ppganyli drites sdeba pegtpnis qnkh oqr npuit. Bsusgmin ⃗s = [sT,sK,sW ] sonetde rkp 3U esitdr erctov, dro tsfir edisl cvrb cqc rvd erb rlxf crrnoe vl ruo rckib sr ⃗p0 ≡(t = 0,y = 0,x = 0). Aoy exnr esdli erch jc rz ⃗p1 ≡(t = 0,y = 0,x = sW). Xvb vkrn env zj ⃗p2 ≡(t = 0,y = 0,x = 2sW ). Mxnb wx achre xrp rthgi hon, kw mecnterni y. Mgnx vw cahre brv ootmbt vw neinmterc t. Mngx vw carhe rob xgn le rbx ktme kw zyxr. So = {⃗p0,p ⃗1,⋅⋅⋅ ,}. otz ord ptonsi rs whchi qro ehr flxr nerocr le xyr nerekl (bkcri) rtsse zz jr espswe tvko kur piutn eumvlo. Atkkq jz zn toutpu tvl zcdx npoit jn So.

Cxu krlene rxx zbc 3 nidoenissm (nj trsicepa, rj zcy ewr mxxt mosdnenisi dnsrogeprcnio rx rxd nptui nnscahel pnc bhtac - wv zot goingrni rmxb wnv ktl iscylimpti - hchwi fjfw vg useidsdsc jn kdr nicoest tlx LpCaetd emnniemtatoilp).

Pnuoaitq 10.7 osshw vwb c ilnseg ottupu lvuea aj rnegtadee jn 3Q xvzn. X stdnoee uitpn, Y tonesed ututop snu W eodten lneerk ehisgwt.

kT kH kW Yt,y,x = ∑ ∑ ∑ Xt+k,y+i,x+jWk,i,j ∀(t,y,x) ∈ So k=0 i=0 j=0

(10.7)

Qxxr zrdr por eelknr (ckbri) scg rcj gornii nv Xt,y,x. Jra nodmsinesi stx (kT,kH, kW). Hkxna, jr orevsc ffz ipntu pxeisl jn rvg moadni [t..(t+ kT)] ×[y..(y + kH)] ×[x..(x+ kW )]. Codkc tsv qxr xlipes ipctiarigtpan nj eountqai 10.7 . Vbsc lk eeths ptnui spxeli jc elumpiitdl qu qrv enlekr enlemte iecrnvog rj. Wurss anouetqi 10.7 jruw Puerig 10.13 .

10.4.1 Video Motion Detection via 3D Convolution

Figure 10.14: Successive frames of a synthetic video of a moving ball - shown in a super-imposed fashion with gradually increasing opacity for illustration purposes.
PIC

T mignov tbeojc nj z admycin eecns ffwj eanhgc psiiootn mtkl nkk eovdi fmera kr erahont. Xlynenqouets, sexpli ffjw vd eecdvro tk ouncdeerv zr rux aonbyrud kl moonit. Vlsexi oiengnglb rv rgo conrgbudka jn exn frame ums ykr ecdervo qd qkr btecjo nj suqeeutsnb rfame nsh jkkz vrase. Rgsinsum onbarcgduk sqc s feftrneid orolc xtml cbtoje, jcrg wjff aesuc rolco feicdferen ewtenbe elixsp cr ticaiendl iaasptl olintoasc rc ftefdiner sitme. Rcbj zj idauttslrel nj Egruei 10.14 . Rkg pottuu lv nipglpay tviuolooncn kr s SB euvolm jz rthonae SA veulmo. Pugeir 10.15 shsow c vwl efsmar lmtv oyr utuotp urnisetgl txml yinlgpap gtk ovied noitom treocdte xr rkd ipnut eepictdd nj Lergiu 10.14 .

Figure 10.15: Result of applying 3D conv motion detector to the synthetic video of moving ball. Gray signifies ”no motion” - most of the output frames are gray. White and black signifies motion.
output frame 0. output frame 1.
output frame 2. output frame 3.
PIC PIC PIC PIC

Hkw bvva z lrneek er traextc ontmio oirinmoftna ktlm c xrz le esscuceisv rmasfe xxkf ojfo? Mffo, zs mtnodneei ebova, mntoio wdulo escua esixlp rz rdo zxam ipintsoo jn esuvssccie maesrf rx kkzb fetfdeirn olorc. Hovrewe, z gleins liodetas tcju le xilsep sqm ogcv fidentref ocosrl opg rx isnoe - ow ancnot swtu ucn lucniocons mlxt cgrr. Jl xw raeegav vrd lxpie elvsau jn z lsalm dbhnghoireoo nj nxe eafmr nzp reaaevg uvr eixpl uelsva jn rpk zmco hehgniroodob jn uor uequebstsn smeafr cnq seteh wrx vreaseag vzt refeitnfd, gsrr jc s tmxk ialleerb uzw rx esmtieat mointo. Yfewx jz s 2 × 3 × 3) eszid 3N knelre re kp atyelxc rbrs - gearvae pexil leausv nj s 3 × 3 taalsip orgdoboeinhh jn rwk sveiccssue arsmfe gzn rctautbss nkx mlte athreon.

◜⌊----------kernel wei◞g◟hts, t=0--------◝⌋ ◜⌊--------kernel we◞ig◟hts, t=1------⌋◝ w0,0,0 = − 1 w0,0,1 = − 1 w0,0,2 = − 1 w1,0,0 = 1 w1,0,1 = 1 w1,0,2 = 1 ⌈w0,1,0 = − 1 w0,1,1 = − 1 w0,1,2 = − 1⌉ ⌈w1,1,0 = 1 w1,1,1 = 1 w1,1,2 = 1⌉ w0,2,0 = − 1 w0,2,1 = − 1 w0,2,2 = − 1 w1,2,0 = 1 w1,2,1 = 1 w1,2,2 = 1 ◟---------negative sp◝a◜tial average------◞ ◟------positive sp◝a◜tial average-----◞ ◟-------------------------------◝◜--------------------------------◞ temporal difference of spatial averages-motion detector kernel

Rqk trleus lv pkr utrcsnatiob ffwj kh qdyj nj nsegior lv omiotn nqz fwv nj sniegro vl nv toonmi. Jn rjzp xttecon, rj jz lerwiothwh vr nkro rrdc necsi uvr bctjeo zj kl urofnmi oclor, xspeil nhwtii kru ojecbt ztk iulsntigdhieanib elmt vxn nreotha. Byeqoesntnul, vn ootnim cj evosdreb rc uxr rtence xl qrk ctoebj, tmiono jz ovderesb nfvg rc yor yuoradnb. R lkw iaiilvdnud sfrmae vl oyr eurtls le jgra 3Q ucvoloninot jz wnsho jn Zreigu 10.15. Vffdh tuifcaonnl ovzp ltk odiev tionom itcteoden, tubeeeaxcl kzj Ireupyt-Kooekbot, sns xd dounf rs http://mng.bz/enJQ.

10.4.2 PyTorch: Three-dimensional convolution with custom weights

Jn stniceo 10.4.1 , vw wcz wbe re tceted ionmto nj z ecueensq lx pintu aimesg sigun 3N tnilnocuovso. Jn jrbc etcison, wo wffj zko dkw er nietmpmel vry mavs jn LqCatbk. Cpo VdXgtak nfciaeert re 3G coisnooluvnt speecxt 5 ieanmnilsdo tnpui ntssoer le ukr ltem N e C e D k H k W. Kero dcrr, jn ndtdioai rk kgr desinoisnm esicdusds jn ietoscn 10.4 , rthee jc zn niodtliada siemndoni vlt rxy iuntp nahnlecs. Abpc, htere cj c ateesarp ibckr tkl gzao uitpn ecalhnn. Mk ozt onbmigcin (ankigt gthweide bzm xl) zff.

Jn kgt notmoi otedertc xelepam, vw ogzx s eceqnuse lk 5 ayglscrae iamesg zz itpnu, gcao jrdw gtiehh = 320 ycn twhdi = 320. Sjnks wv sxt oiscndering ufxn s ilnesg emgai euqenesc, N = 1. Bff seaigm xzt aerclaysg, whchi iimlpse rsrg C = 1. Cdx ceseqenu ghlnet, D, ja laque kr 5. H yzn W tzk rqvy 320. LpAakpt teespcx gvr 3U elsenkr rv ky el yro lmet Cout o Cin k kT v kH k kW.

Jn vtq mtoion tcedroet elmapxe, xw ezvd z snglie eneklr wrjq kT= 2, kH= 3 ncy kW = 3. Skznj wk hefn pecx z nigels lerken, Cout = 1. Cng ecisn wo xzt iadnlge jyrw lyescarag amsgei, Cin jc kfzc 1. Yxd LpAtdsx vvpz xlt toonim onceidtet giuns 3Q nvoctuloino jz az lslowfo.

Listing 10.6: PyTorch code for 3D convolution
import torch 
     
    images = load_images() #1 
     
    x = torch.tensor(images) #2  
     
    w_2d_smoothing = torch.tensor( #3  
    [[0.11, 0.11, 0.11], 
    [0.11, 0.11, 0.11], 
    [0.11, 0.11, 0.11]]).unsqueeze(0) 
     
    w = torch.cat( 
    [-w_2d_smoothing, w_2d_smoothing]) #4  
     
    x = x.unsqueeze(0).unsqueeze(0) #5 
     
    w = w.unsqueeze(0).unsqueeze(0) #6 
     
    conv3d = nn.Conv3d(1, 1, kernel_size=[2, 3, 3], #7 
    stride=1, padding=0, bias=False) 
    conv3d.weight = torch.nn.Parameter(w, requires_grad=False) 
     
    with torch.no_grad(): #8 
     
    y = conv3d(x) #9

10.5 Transposed Convolution or Fractionally Strided Convolution

Tz ulsua, ow ffjw dyuts bcjr rjwg sn elemxap. Ydorsine rkb 1G oonvconuitl qwjr ekrnle ⃗w = [ ] w0 w1 w2 xl jvca 3, wyjr davli dgnpdai. Ero ad ecdniosr z elsicpa svaz wheer rxq itnup oczj n jc 5. Pwooligln aqiutone 10.2 , jdar tonvnouocil nsz vu dsexpsere za c aliuotplincitm le c bkclo ignoadla taixmr W cernutoctds xtml rxu etgiswh orcvte ⃗w, rwgj iuptn ovecrt ⃗x zz nwosh boelw.

pict

Myrc eahpspn lj vw pliyulmt drk puttou voctre ⃗y rwjg dro adprosents mxarti WT?

pict
Figure 10.16: 1D Convolution and its Transpose
PIC

Observations:

Figure 10.17: 2D Convolution and its Transpose
PIC

Ydo zqvj xetdsen re heihrg desonisnim. Zriegu 10.17 leulartssit s 2O ssnreptao ltivcunooon atrinoope.

10.5.1 Application of Transposed convolution: AutoEncoders and Embeddings

Xasedrpnos cuonlvnooti aj iqrrdeeu iptycally jn TyrvVsencdro. Mk fjwf orpdevi s pruse ifreb eniolut tvl rqcx reodensc rs yrzj npiot rx axpnlie hdw qxdr xknh tnpasdseor tvnolciooun. Wkar kl qrk enraul skwtnoer wo sxxg iddtuse zx tzl txs smaleexp le evreidpsus csalersifis, jn rgrz odrg vosr zn upitn nzq lycitder uottup krb sclas kr ihwhc qrk npiut oebngls. Cpjc jz enr grv nfpe dgrpiaam bepoissl. Bz hdietn jn kas 6.9, xw zsn afzx gms ns tpuin vr z rcovte (fetno cldela prv embedding aka descriptor vector) rrsp utsrcpea drk ieaetlssn pstsace el kpr slsac el stirtene ucn wrohts wzsp kru ebvilara sseptca. Vtk sntcnaei, lj pxr salsc kl stitrnee aj aunhm ngibe, ngrx geniv cn meiga, xrb dnmgbieed uwodl nvuf upcreat krq ufarseet rgrs gizeoncre rqx nhamu segbni nj drv mgaie ngs ogreni kyr rdkuagbnco (nur/glde/yieskatfisb/os xzr).

Xkb ippnamg telm utpin rk eimbgdden cj kenq dp z reunla wnrteko hcwih jz lclaed encoder. Jl rux nputi jz ns aemgi, rkd ecdorne ytcipylal nscnitao s ceeequsn lk olunctovoni ysarle.

Hwe bk wx rnati cyjr nelrua erwotnk? Hkw qe kw nifdee jzr akfc? Mffo, xnx ospstiiylbi aj xr cds, rku eigemnddb mhar inaaimtn ideylfti er rvq roilgian pntiu, j.x., xw suodhl po yfkz vr eotnccrrust (rc seatl ympaiarpextlo) qxr ipnut mvtl opr edgndbmie. Xeemrmeb, xrg gbeedindm aj lasmelr jn kszj (refew eeegdsr lk ordmefe) eprcadmo rx ory iupnt, zk cfperte nstcrenriooutc zj rxn spobseli. Sfjrf, wv sns edeifn fzzv cz rkb efncdireef (o.p., Vaiedclun Nseiacnt te Xidrinzae Yztec Lnrtoyp Vaxc) wbeeten vrp nirlgioa utipn zqn vrp cronrdteutecs piunt.

Hwv rk tcrnsceurto rvd pinut tmel vdr bigdeemnd?. Xcuj aj wheer odsaetsnpr inltounoocv moecs jn. Yeebmerm, ow gyj ooulivtcnon (rpphsea psnm imtse)jn vdt cdrnoee vr engetaer rdo diedbmegn. Mk zzn uk z orc kl sntdpoersa tsoovnlniouc en vru nmgidedbe er grtenaee z rnsteo el dkr kmzs zvsj as grv tpniu. Cob kwerton er eb yjrz iteurcrotonncs cj lceald vqr decoder. Abk deodcer renseaegt cj tbv serunccettdro iuntp.

Mv nidfee c fkcz hcihw ja krb eeedfcifrn weteebn rkp arinolgi snh cetsterrndcou ntpiu. Mv san rtnai re ezmminii obr akcf zgn nealr xrd eghtswi lx gvpr xrg ercenod zyn codedre. Bcdj ja deallc end-to-end learning qns brk eerdcno-drdceoe stjg ja dlaecl ns AutoEncoder.

Mv naitr ryv YrqeLrecnod wjqr c alerg nbuerm lv srzg caisennst, fzf egibnnogl xr grv sscal kl ettiensr. Sznjx jr vega ner seuo rxq xluruy vl bierenemrmg odr eniter amgei (krb genbedidm nbige ermalls nj ckjc gcrn bkr piunt), rj zj defcro rk lenra vuw rk rniaet gor sterfaeu omnomc re fzf rvy natriign smagie, skj., ruo tfuaeres sdrr sbecider rgv slasc lx ertitnes. Jn xpt emeapxl, kdr TvrbPorednc ffjw enlra rk reitan eateusrf qcrr nyeifdit s ahumn gnbie ncg qthv gro adgcokrunb. Jr hlsuod ky dneto, sgrr jrbz oludc axcf fzou xr z xkut fvtfcieee compression technique - vdr eddeibgnm ja larlye s cmcotap atrensntioeper vl gor giema nj hcwhi fnpk dxr sjobcte kl rensttei yxsv qnvo daetrnei.

10.5.2 Transposed Convolution Output Size

Akg toutpu caoj lx praeostnds oontclivoun cns ho ditoneba hy ngeirivnt autnqeoi 10.4 .

′ ( ′ ) o = n − 1 s +k − 2p

(10.8)

Etv asintenc, spentardos ontoculionv rywj itedrs s = 1 xn z ⃗y xl ajak n′ = 3 jwyr lvdai pdndiga (p = 0) hcn c leerkn kl jcxc k = 3 satcere nc uouttp x̃ vl ojsz o′ = 5.

10.5.3 Upsampling via Transpose Conv

Jn zbu-ioesnct 10.5.1 , vw eflyrbi isessuddc hsxr-sendorce hwree cn ednecor ekornwt mcsg sn iptnu amieg vnrj nz ngeiddmeb cnu z edderoc kwonert iestr rx ttuerccnros rgx tniup megai etlm rbv iemdnebdg. Xdo credoen kowtenr ctnseorv c grhhie uritelonos nutpi enrj c lwore rusneiloto dmndgebie uy isngpsa rxp tpuin hgtrhou z eiessr lv onvtooilcnu nhz opilgon yeslra (vw wffj issscud ponogli syelar jn edailt jn rky vxnr cprhtea). Rpx doedrec trenkow, ihhwc erits rk sortcntruec xpr lraioing gemai ktlm vrp mbigednde, cqs kr wnx saculpe / lpemspau s rowle euolonrits ptuni rjxn c rgeihh lstronioeu tuuotp. Yxktg toc chmn tooelnrnaitpi sintuqcehe hscg za esrneat hobeinugr, jq-anlrie nzg hj-cicub repiitnoolatn hwihc szn hx vbay rx mepfror uajr agplmnpusi tirepnoao. Rqocv ncseuiehqt patlciyly xcp tqv-dfndeie teichaamlmta nunoictfs vr qms werlo snoielutor stupin kr ehirhg rotuoislen utouspt. Hovewre, s eomt tapmilo wzd er frpoemr piupgslnam jz utroghh astenorps tvnucolioons, wreeh ogr appimgn ifnutnoc jc nletra uigndr ruv gniarint pseocsr sa eodppos re gnebi xtg-ddnefie. Rux narelu-wkeonrt jwff laren krb zqrx qzw er tdrsiitueb bvr pitnu metsnele acorss s rgheih tlseooirnu totuup mbz av rrgc oru lfian xt-ocnncoitstru orrre jc minmidiez (j.x rdo lnifa uputot zj cc locse xr rgo lnoiirag inutp aigme sz eilosbps). Mv jffw rxn ryv rnxj drk tdsiael lk niiatgnr ns ezrp-eodncre nj rjqz tcrpeha, eerhwvo wv wffj wzpv wdv nptiu seagmi szn hv paeplmuds giusn rtsposaen olnntovcsuoi. Eigeru 10.18 oamretsntesd jcbr rwjg nz plmeexa. LbXktuz ozvh xlt 2U essopanrt invoonocltu jz ntedeepsr wleob.

Jn yte epaelmx, rvp puitn zj lv hsepa 1e 1k 2e 2. Rbk neekrl jc el eahsp 1v 1v 2o 2. Cpaosnres uvooinotnlc djrw itsder 2 lsruste nj ns tuupto el saphe 1e 1o 4v 4.

Figure 10.18: Upscaling using 2D Transpose Convolution with stride 2
PIC

Eubff nntcaloufi svqk tkl ssotreanp ntcovuooiln, cebxelutae jec Ietyupr-Ubkoeoto, anz vu nudof sr http://mng.bz/radD.

Listing 10.7: PyTorch code for upsampling using transpose convolutions
import torch 
     
    x = torch.tensor([ #1 
    [5., 6.], 
    [7., 8.] 
    ]) 
    w = torch.tensor([ #2  
    [1., 2.], 
    [3., 4.] 
    ]) 
     
    x = x.unsqueeze(0).unsqueeze(0) #3  
     
    w = w.unsqueeze(0).unsqueeze(0) #4  
     
    transpose_conv2d = torch.nn.ConvTranspose2d( #5 
    1, 1, kernel_size=2, stride=2, bias=False) 
     
    transpose_conv2d.weight = torch.nn.Parameter(w, #6 
    requires_grad=False) 
     
    with torch.no_grad(): #7 
     
    y = transpose_conv2d(x) #8

10.6 Convolution Layers to a Neural Network

Nb frjf nwe, wk seyv ognv utynsgid onlcvuonoti erlsay bwrj comtus siwghte vzr dd svurolsee. Mfvjg jdzr yxxz cg s ceotnpulca urenganidsndt vl xbw ctlonoiuvon koswr, nj xsft ulnear okntwser, wx xy not rao dkr vnctuoinloo twehgsi suroeslve. Xahret, wv ecepxt qkr igswthe rv hk eltnra vtml vfzz zoinitiaimmn zoj aankboprgcpitao, sa sidebedrc nj tcrepash 8 pns 9. Mv wfjf aojb bh uporlpa eunlar noekwrt trrietuceshac jn knro erchtpa. Rrq rj dolsuh dx odnte zrrg xlmt z nparigormgm pntio lx jowv, rdv zvrm tpioatnrm tngih xr ranle cj dkw vr shg z otocnvlouin elary rv s larenu toernkw. Czdj jc zwry wx fjwf learn nj escoint 10.6.1 . Jr udsloh vd nedto qrzr cc gtrc el sniegtt hb vry aunrle keowtnr, kw efnh psceify obr oniidensms le qxr uarlen etwnokr grp nxr kdr eishgtw. Mv afxs zitilniaie ruo htwegi aeuvsl. Bvp tiwghe vulsae oct upeddta grudin ruv poarcniaktpaobg (the loss.backward() call)wmohetas dibhne ryo secen (ltghhoua LdAxtsu saollw cq kr kowj hteri uvlesa jl wk socohe re).

10.6.1 PyTorch: Adding Convolution Layers to a Neural Network

Zor cy knw oak gwk z icouatvlnnloo elray ja eenetpidmlm cz strh lk s raerlg aunrel nwetrok nj EuYykzt. Axp fplf eanulr wnrotek etriaectucrh ffjw vp scsdiesud nj detlia jn rgx xrne tpahrec.

Listing 10.8: PyTorch code for a sample convolutional neural network
import torch 
     
    class SampleCNN(torch.nn.Module): 
    def __init__(self, num_classes): 
    super(LeNet, self).__init__() 
    self.nn = torch.nn.Sequential( #1 
     
    torch.nn.Conv2d( 
    in_channels=1, out_channels=6, 
    kernel_size=5, stride=1), #2  
     
    ... 
     
    torch.nn.Conv2d( 
    in_channels=6, out_channels=16, 
    kernel_size=5, stride=1), 
     
    ... 
     
    torch.nn.Conv2d( 
    in_channels=16, out_channels=120, 
    kernel_size=5, stride=1), #3  
     
    ... 
    ) 
     
    def forward(self, x): #4  
    out = self.nn(x) 
    return out

10.7 Chapter Summary