一种带矩限制的批量自适应方差缩减算法 An Adaptive and Momental Bound Method for Stochastic Variance Reduced Gradient

设为首页加入收藏期刊导航网站地图

期刊菜单

文章导航

AdvancesinAppliedMathematics A^êÆ?Ð,2022,11(12),9026-9038

PublishedOnlineDecember2022inHans.//www.abtbus.com/journal/aam

https://doi.org/10.12677/aam.2022.1112952

˜ «‘Ý•›1þg·A•~Ž{

ÁÁÁ ???]]]

§§§ XXX[[[

2 ∗

ú ô“‰ŒÆ1•Æ §ú ô7u

ú ô“‰ŒÆ§ú ô7u

Â vFÏµ2022c1126F¶¹^FÏµ2022c1221F¶uÙFÏµ2022c1229F

Á ‡

Ã õÅìÆS.ÔöL§Œ±8(•¦)˜‡`z¯K§DÚFÝeüŽ{9ÙC« Â

ñ „Ý¿ØU4<÷¿"©3‘Å•~FÝŽ{Ä:þ§(Üg·AÆSÇÚÝ•›

 A: §¿|^1þgŽ§ÀŒ1þ“OÜ?1ÛFÝ OŽ§À1þ

? 1ëêS“•#§JÑ‘Ý •›1þg·A•~Ž{§¿éŽ{Âñ5? 1`

² "ÏLÄuMNISTêŠ¢§yTŽ{k5§¿&ÄÑ¢ëêéŽ{-½5

K •"

' …c

Å ìÆS§g·AÆSÇ§Ý•›§1þ§‘Å •~FÝ{

AnAdaptiveandMomentalBoundMethod

forStochasticVarianceReducedGradient

GuiyongZhu

,JialinLei

2 ∗

XingzhiCollege,ZhejiangNormalUniversity,JinhuaZhejiang

ZhejiangNormalUniversity,JinhuaZhejiang

Received:Nov.26

,2022;accepted:Dec.21

,2022;published:Dec.29

,2022

∗ ÏÕŠö"

DOI:10.12677/aam.2022.1112952

Á ?]§X[

Abstract

Thetrainingprocessofmanymachinelearningmodelscanbereducedtosolvingan

optimizationproblem,andtheconvergencespeedofthetraditionalgradientdescent

algorithmanditsvariantsisnotsatisfactory.Inthispaper,basedonthestochas-

ticvariancereductiongradientalgorithm,wecombinethecharacteristicsofadaptive

learningrateandmomentlimitation,andusethebatchideatoselectlargebatch

samplesinsteadofallsamplesforthecalculationofglobalgradient,andselectsmall

batch samples for the iterative update of parameters, then propose the batch adaptive

variancereductionalgorithmwithmomentlimitation,andillustratetheconvergence

ofthealgorithm.Theeﬀectiveness ofthealgorithmis veriﬁed throughMNIST-based

numericalexperiments,andtheinﬂuenceoftheexperimentalparametersonthesta-

bilityofthealgorithmisexplored.

Keywords

MachineLearing,AdaptiveLearningRate,MomentalBound,Mini-Batch,SVRG

 2022byauthor(s)andHansPublishersInc.

This work is licensed undertheCreative Commons Attribution InternationalLicense(CCBY4.0).

http://creativecommons.org/licenses/by/4.0/

1. Úó

• ÄXeÃå4z¯K[1]

min P( w) ,P( w) =

i =1

( w) ,(1)

Ù ¥ψ

: R

→ R,i=1,...n.ù«.3ÅìÆS¯K¥š~ÊH,'X3kiÒÆS¥,nL«

 êþ,ψ

L «©a½£8.¥~i›”¼ê.

é u±þ 8I¼ê,Ï~¦^FÝ eü•{?1¦)[2–5], FÝ eükn«CN,©O´

 FÝeüŽ{!‘ÅFÝeüŽ{Ú1þ‘ÅFÝeüŽ{,ÙØÓƒ?3u¦^õ5

DOI:10.12677/aam.2022.11129529027 A^êÆ?Ð

Á ?]§X[

O Ž8I¼êFÝ,ŠâêþØÓ,3ëê•#O(5Ú‰1•#¤Ižm ƒm ?1

 ï.

 FÝeüŽ{[6]I‡OŽ¤kFÝ5?1S“•#,U¼4ÐFÝeü••

¿ Jø•ªÂñ„Ý y.FGDØ U3‚•#.,=Ø U|^Ä~f,Ù•#5KXe

= w

t −1

−

i =1

∇ ψ

( w

t −1

) .(2)

FGD 3¡éŒêâ8ž¬ÑyP{OŽœ¹, Ï•§3•z‡ëê•#ƒcÑ¬éƒcƒ

Ó -#OŽ,‘ÅF ÝeüŽ{[7](StochasticGradientDescent,SGD)zg•I‡Àü

‡ OŽCqFÝ‰1ëê• #5žØù«P{

= w

t −1

− α

∇ ψ

( w

t −1

) ,(3)

SGD OŽ¤=•FGD 1 /n, Óž•Œ^u3‚ÆS, d uÀ‘Å5Ú\•,

 —8I¼ê3Âñ L§¥ÑyìÅÄ .

‘ ÑyU?1þ‘ ÅFÝeüŽ {[8](Mini-BatchStochasticGradientDescent,

Mini-BatchSGD)

= w

t −1

−

|B|

i ∈|B|

∇ ψ

( w

t −1

) ,(4)

(4) ª¿©(ÜFÝeüŽ{Ú‘ÅFÝeüŽ{A:, =3zgëê•#žl¥‘Å

À Ü©?1 FÝO,3˜½§ÝþQyFÝeü••,qü$‘ÅÀ •.

ƒ uFÝe üó,1þ‘ÅFÝe ü•U‘?Œ5ÅìÆS ?Ö‡¦,E,‘ÖØ

 ÙÂñ„Ý…ú ":.

(4) ª¿©(ÜFÝeüŽ{Ú‘ÅFÝeüŽ{A:, =3zgëê•#žl¥

‘ ÅÀÜ©?1FÝO,3˜½§ÝþQyFÝeü••,qü$‘ ÅÀ

• .ƒuFÝeü ó,1þ‘ÅFÝeü•U‘?Œ5ÅìÆS?Ö‡¦,E,‘

ÖØ ÙÂñ„Ý…ú":.

C c5,‘ÅFÝeüŽ{®¤•ÅìÆSAO´ÝÆS :,‘XéFÝeü••ÚS

“ Ú•Øä&¢,ÅìZyÑNõÄuSGDU?Ž{.MomentumŽ{[9]3DÚ‘ÅFÝe

ü Ž{Ä:þV\ Äþ‘,ÄuFÝÄþ\ÈgŽ¿(Ü{¤FÝ¦^•ê\ £Ä²þ5k

 ;•,\„%C•`),Nesterov\„FÝŽ{[10,11]K´3MomentumŽ{Ä:þéF

Ý ‘ëê‰˜gS “•#.

3 FÝeüL§¥,du`zëêé8I¼ê•6ˆØƒÓ,Ùð½ÆSÇ¬—ë

ê S“ÑyFÝuÑ½öÂ ñ…ú¯K,•d•Ä`zŽ{UÄg· A/N!ÆSÇŒ.g

· AFÝ{[12](AdaptiveGradient,AdaGrad)ÏLÅÚÚ•5ÆS;þ•ŠDÂ{[13](Root

DOI:10.12677/aam.2022.11129529028 A^êÆ?Ð

Á ?]§X[

MeanSquarePropagation,RMSprop) òÆSÇ©)¤²•FÝ•êP~²þ5››AdaGrad Ž

{ ÆSÇeüL¯¯K;g·AÝO{ [14](AdaptiveMomentEstimation,Adam)Œ±wŠ

Momentum Ž{ÚRMSProp Ž{(ÜN, |^FÝ˜ÝÚÝO5éÆSÇ?1å;

g ·AÝ•›{[15](AdaptiveMomentalBound,AdaMod)^•ÏPÁ•›LpÆSÇ,ü$

Adam Ž{éÆSÇ¯a5.

, ‘ÅFÝeüŽ{3‘Å) •¿‘XS“gêO\Øä\È,?

 Ã{ ˆ‚5Âñ.•d,ïÄö‚JÑ˜XÄu• ~‘ÅeüŽ{ ,~X‘Å²

þ FÝeü{[16](StochasticAverageGradient,SAG)±/#FÝ“OÎFÝ0•ª,¿©•

Ä {¤FÝ…ˆ~OŽþ8;‘Åéó‹Iþ,{[17](StochasticDualCoordinate

Ascent,SDCA ¤3ëê•#L§¥•;éóCþ, ü$FÝ•\„Âñ; ‘Å•~Ž{[18]

(StochasticVarianceReductionGradient,SVRG ¤ÙØ%gŽ´|^ÛFÝ&E^u.•#

 FÝ ?1?,nØ©ÛL²,ùA«•{3A½ ^‡eÑ U~•¿ˆ ‚5Âñ.

 ©12Ü ©Äug·AÆSÇŽ{ÚÚ•~Ž{Ä:þ,JÑ‘Ý•›1þg

· A•~Ž{(AnAdaptiveandMomentalBoundMethodforStochasticVarianceReduced

Gradient,AmbSVRG), (Üg·AÆSÇÚÝ•›A:, ¿|^1þgŽéëê?1S“•

# ;13Ü©‰ÑŽ{Âñ5 ©Û;14Ü©ÄuMNISTêâ8éŽ{k5 ?1 y,

X &ÄÑ¢ëêéŽ{-½5K•;15Ü©‰Ñ©o (.

2. Ž{0

2.1. g·AÆSÇŽ{

3 FÝeüL§¥À˜‡Ü·ÆSÇ´é(J,LÆSÇ¬—Âñ…ú,LŒ

 ÆSÇ¬ —ëê¬3•ŠNCÅÄ ½uÑ,,3¡é ØÓ êâ8ž,AÀØÓ ÆS

Ç l·Aêâ8A,3FÝeüŽ{L§¥,¤këê•#Ñ¦^´ð½ÆSÇ ,d

u z‡ëê•#žÂñ„ÝÑØ¦ƒÓ,I‡ŠâëêÂñœ¹©O˜ØÓÆSÇ,l

ˆ éŽ{\„8.

é u½ Ú• ëê•#úª•

= w

t −1

− αg

 é½Ú•,AdaGradòzg•#FÝ\\,¦ÆSÇ·Aëê,éDÕêâ8‰ 1

Œ •#!éÈ—êâ8‰1•#

= w

t −1

−

√

+ 

C þG

•; {¤FÝ²•Ú,´•“Ž©1•0,ÏLéÆSÇ¦±ù˜‡©1‘5?U

z ‡ëêS“L§¥ÆSºÝ,lžØÃÄNÆSÇI‡.AdaGradŽ{Ì‡":

DOI:10.12677/aam.2022.11129529029 A^êÆ?Ð

Á ?]§X[

´ duz˜‘G

Ñ ´ ,©1¬‘XÔöL§Øä O\,‡L5 —ÆSÇ5ªCu0,

l ªŽS“ .•)ûAdaGradÆSÇ:ìeü¯K,RMSPropÚ\•ê\²þê





= βE





t −1

+(1 −β) g

= w

t −1

−

E [g

]

+ 

Ù ¥β••êP~Xê,AdaGrad¥FÝ² •Ú•†¤FÝ² •P~² þÏ ".

Adam Óž¼AdaGrad ÚRMSProp `:,Adam Ø=XRMSProp @Äu˜ÝO

Ž ·A5ëêÆSÇ,§ Óž„¿©|^FÝÝþŠ.äNL«•

= β

t −1

+(1 −β

) g

= β

t −1

+(1 −β

) g

Ù ¥m

Ú v

´ FÝ˜ÝÚÝO,du§‚ÐÏ¬keéÄ¯K,¤±I‡é§‚‰

˜ ?

ˆ m

1 −β

ˆ v

1 −β

¦ ^ùëê 5S“•#Adam•#5K

= w

t −1

−

√

ˆ v

+ 

ˆ m

¦ +g·AÆSÇ•{3Nõœ¹eÑéÉ•H,Adam•@•´ÝÆSµe¥¦^%

@ Ž{,§E, ¬‘-½5¯KµØ-½Ú4àÆSÇ.AdaModæ^Ä>.5•›4

à ÆS„Ç,3Ôö ÓžOŽg·AÆSÇ •ê•Ï²þŠ,¿¦^T²þŠ5?}ÔöL

§ ¥LpÆSÇ

= α

ˆ v

+ ,

= β

t −1

+(1 −β

) η

ˆ η

= min( η

) ,

Ï L1‡•§½Âc²wŠÚL/•ÏPÁ0'X.w,,β

=0 ž,AdaMod 

 duAdam,3OŽÑ c²wŠ,3§Ú cAdamŽÑ ÆSÇη

¥ ÀÑ˜‡•Š ,l

 ;•ÑyLpÆSÇœ¹.

du SGD‘Å5Ú\•¦SGD3½Ú•œ¹e•Uˆg‚5Âñ„Ý,©z

3 [18]•Ñ,‘ÅŽ{°ÝÚæ•¤ƒ',•ªu0ž,Ž{•¬•0.XÛ3

‘ ÅŽ {¥~‘ ÅFÝ•,òSGD‰ÚFÝeü˜‚5ÂñQ?ïÄö‚qJÑ

 ˜a•~Ž{.

DOI:10.12677/aam.2022.11129529030 A^êÆ?Ð

Á ?]§X[

2.2. •~Ž{

C c5,Äu•~‘ÅFÝeüŽ{¤•`zŽ{ïÄ ¥9:¯K,ïÄ ö‚ƒU

J ÑSAGÚSDCAƒ'Ž{,Ù¥SAGŽ{•#5K•

d =d −y

+ ∇ψ

( w) ,

= w

t −1

−

SAG 3S•¥•z‡‘o˜‡ÎFÝ, ‘ÅÀJ˜‡5•#ëêd, •#‘¥d

´ ^#FÝO†ÎFÝ,¤±zgOŽ¥•I‡ 9ü‡FÝ.SDCAŸþÏLEéó

C þ?1•~,•ìéóC þwÚéóC þαƒm'X,w•#Œ±L «¤

= w

t −1

− γ(∇ f

( w

t −1

)+ λnα

) ,

‘ XS“?1,(w,α)ò¬Âñ(w

∗

,α

∗

), u´( ∇f

( w)+ λnα

) →0, ∇f

( w)+ λnα

 ••

¬ ªCu0.ùÒˆ•~8.ùü«Ž{ÑUˆ‚5Âñ„Ý,3ëê•#L§

¥ Ñ‡¦•;ÜFÝ£½éóCþ¤,éS•˜mäk˜½‡¦.SVRGŽÑSAGÚSDCA

é Œþ•;˜mI¦ ¯K

˜ µ=

i =1

∇ ψ

(˜ w),

= w

t −1

− α(∇ ψ

( w

t −1

) −∇ψ

(˜ w)+˜µ).

Ù Ø%gŽ´|^ÛFÝ&EéëêS“•#¥FÝ?1?,z²{˜ÓSÌ‚OŽ˜

g ÛFÝ,zg •#•õOŽüg FÝ.3SGDÂñ5©Û¥I‡bFÝ•´k~

ê þ.,,´ Ï•ù‡þ.—SGDÃ{‚5Â ñ,SVRG|^AÏFÝ•#‘¦

 •k˜ ‡Œ± Øä~þ.,lˆü$ •8,Ï d• Ò‰‚5Âñ.

1 þ‘Å•~FÝŽ{[19]£BatchingSVRG¤|^Ü©FÝCqÛFÝ,3

˜ ½§Ýþü$SVRGOŽþ.SVRGÚBatchingSVRG3S“L§¥Ñ¦^½ÆSÇ?

1 `z¦),À J·ÜÆ SÇé`zŽ{–'-‡.d,þ˜gS“ëêlcS“ëêå

l ,cëêl •`Š?Cž,SVRGÚBatchingSVRGq¬Lõ/|^{¤FÝ&E

Š écFÝeü••?1? ,— Ž{3•`Š ? ,ü$Ž{Âñ„Ý.

é uæ^½ ÆSÇÚLÝ |^{¤FÝ &E ¯K,©ÄuSVRG,ŠâFÝ ˜

Ý ÚÝ O,OŽØÓëêe‡Ng·AÆS„Ç,JÑ‘Ý •›1þg·A•

~Ž {(AnAdaptiveandMomentalBoundMethodforStochasticVarianceReducedGradient,

AmbSVRG).

2.3.AmbSVRG Ž{

é u¯K(1),©òAdaMod¥˜ÚÝO†BatchingSVRG¥FÝ •#•ª‰(

DOI:10.12677/aam.2022.11129529031 A^êÆ?Ð

Á ?]§X[

Ü ,JÑ‘Ý•›1þg·A•~Ž{.

AmbSVRG Ž{:

Ñ \µÛÆSÇα

, •#ªÇm, ÄþXêβ

, β

∈ [0,1);

Ú 0µÐ©z˜w

= 0, m

= 0, v

= 0, s

= 0;

Ú 1µéÌ‚s= 1,2...T,-˜w=˜w

s −1

;

Ú 2µÀJÛFÝ•#1þº€Œ|B

| =B,OŽÛFÝ:˜µ=

i =1

∇ ψ

(˜ w);

Ú 3µ-w

=˜ w,s:=s+1;

Ú 4µéSÌ‚t= 1,2,...m,ÀJë ê•#1þº€Œ|B

| =b,

O ŽFÝ‘:g

i =1

[ ∇ψ

( w

t −1

) −∇ψ

(˜ w)]+˜µ;

Ú 5µOŽ˜ÝÚÝ:

= β

t −1

+(1 −β

) g

= β

t −1

+(1 −β

) g

;

Ú 6µ?1?:ˆm

1 −β

, ˆv

1 −β

;

Ú 7µé•#‘?1•ê\²þ,•›LpÆSÇ:

= α

√

ˆ v

+ ,S

= β

t −1

+(1 −β

) η

, ˆη

= min( η

);

Ú 8µë êS“•#:w

= w

t −1

− ˆη

ˆ m

;

Ú 9µ-t:=t+1,et=Ú 4,et=m=Ú 1.

3 AmbSVRGŽ{¥,Ú2(ÜBatchingSVRGŽ{¥1þgŽ,^Ü©OŽFÝ

C qÛ FÝ,3˜½^‡e,U~Ž{OŽþ,\¯Âñ;3Ú4¥,ò1þgŽK

\ SVRGFÝ•#5K¥éFÝ‰ÃO,lü$S“L§¥FÝ•,O\ÂñL§

¥ -½5;3Ú5¥ q|^• ê\²þgŽ,é˜FÝÚFÝ\\Äþ,¿©|^FÝ

& E,¿3Ú6ém

Ú v

Ð ÏeéÄ¯K‰ ?,¦Ž{3¡éØÓêâ8žUgÄ

N ëê ÆSÇ,ŒŒJpŽ{ -½ 5ÚÔö„Ý ;3Ú7,OŽg·AÆSÇ •ê•Ï²

þ Š,¿À ÆSÇ“\ëêS“•§,³›LpÆSÇ),? ˜ÚO\Ž{

- ½5.

3. Âñ5©Û

 !?ØAmbSVRGŽ{Âñ5,ÄkÑ¯K(1)¤I‡÷vÄb.

b 1éÃå4z¯K(1),8I¼êψ(w)÷v

(1) ψ•à¼ê;

(2) ψ•34Š:w

∗

, =w

∗

= argmin

P (w );

DOI:10.12677/aam.2022.11129529032 A^êÆ?Ð

Á ?]§X[

(3) ψ•|ÊF]ëY, =

ψ (w )−ψ

( w

∗

) −

k w− w

∗

6 ∇ψ

( w

∗

)

( w−w

∗

) .(5)

Ù ¥L>0•|Ê F]~ê.

b 2P(w)´µ-rà¼ê

P (w )−P (w

∗

) −

k w− w

∗

> ∇P(w

∗

)

( w−w

∗

) .(6)

½ n3.1.3b1,b2^‡e,µ>0,bm¿©Œk

α =

µmR (b +2LR )

2 LR

b +2LR

< 1,

 E[P(˜w

) −P( w

∗

)] ≤α

E [P(˜w

) −P( w

∗

)] ,@oAmbSVRGŽ{ÒPk‚5Âñ„Ý.

y ²é?¿i,^g

L «1tÚ ž|¢••,Kk

i =1

[ ∇ψ

( w

t −1

) −∇ψ

(˜ w)]+

i =1

∇ ψ

(˜ w),

d ©z[18]Œ•

i =1

k∇ ψ

( w) −∇ψ

( w

∗

) k

6 2L[P(w)−P(w

∗

)] .(7)

é g

Ù Ï "

E kg

= Ek

i =1

[ ∇ψ

( w

t −1

) −∇ψ

(˜ w)]+

i =1

∇ ψ

(˜ w)k

6 2Ek

j =1

∇ ψ

( w

t −1

) −

i =1

∇ ψ

( w

∗

) k

+2 Ek

i =1

∇ ψ

(˜ w)−

i =1

( w

∗

) k

−∇ p(˜w)k

= 2 Ek

j =1

∇ ψ

( w

t −1

) −

i =1

∇ ψ

( w

∗

) kv

+2 Ek

i =1

∇ ψ

(˜ w)

−

i =1

( w

∗

) k

− E[∇ ψ

(˜ w)−∇ψ

( w

∗

)] k

6 2Ek

j =1

∇ ψ

( w

t −1

) −

i =1

∇ ψ

( w

∗

) k

+2 Ek

i =1

∇ ψ

(˜ w)−

i =1

∇ ψ

( w

∗

) k

6 4

[ P( w

t −1

) −P( w

∗

)+ P(˜ w)−P(w

∗

)] .(8)

DOI:10.12677/aam.2022.11129529033 A^êÆ?Ð

Á ?]§X[

1 ˜‡Ø ª¦^ka+bk

6 2kak

+2 kbk

, ^˜ µ=

i =1

∇ ψ

(˜ w)“O∇P(˜w),…∇P(˜w)=

E [∇ψ

(˜ w)−∇ψ

( w

∗

)]; 1‡Øª¦^é?¿ξ, ÷vEkξ−Eξk

= Ekξk

−k Eξk

6 Ekξk

;

1 n‡Ø ª¦^(7)ª.

d (8)ªŒ•kg

k ´kþ.,Ø”•G,3AmbSVRGŽ{Ú5¥,m

= β

t −1

(1 −β

) g

, m

= 0 ž, éªfÐmŒ

= (1 −β

)



+ β

t −1

+ β

t −2

+ ···+ β

t −1



¦ ÙÏ"•

E kˆm

= Ek

1 −β

(1 −β

)

( g

+ β

t −1

+ β

t −2

+ ···+ β

t −1

1 −β

(1 −β

)

E kg

+ β

t −1

+ β

t −2

+ ···+ β

t −1

1 −β

(1 −β

)

(1+ β

+ β

+ ···+ β

t −1

) G= G.(9)

d (9)ªŒ•kˆm

k k.,ÓnŽ{Ú6¥k ˆv

k ÚÚ7¥k ˆη

k •´kþ.,=•3R>0,¦

 kˆη

k 6R,k

E kw

− w

∗

= Ekw

t −1

− ˆη

ˆ m

− w

∗

= kw

t −1

− w

∗

+2 R( w

t −1

− w

∗

)

E [bm

]+ R

E kˆm

6 kw

t −1

− w

∗

+2 R( w

t −1

− w

∗

)

∇ P(w

t −1

)

4 LR

[ p( w

t −1

) −p( w

∗

)+ p(˜ w)−p(w

∗

)]

6 kw

t −1

− w

∗

− 2R[p(w

t −1

) −p( w

∗

)]

4 LR

[ p( w

t −1

) −p( w

∗

)+ p(˜ w)−p(w

∗

)]

= kw

t −1

− w

∗

− 2R(1+2

)[ p( w

t −1

) −p( w

∗

)]

4 LR

[ p(˜ w)−p(w

∗

)] ,

Ù ¥1˜‡ØªdE[bm

] 6E[ g

] = ∇P( w

t −1

) Œ, 1‡Øª¦^P( w) à5

p (w

t −1

) −p( w

∗

) 6( w

t −1

− w

∗

)

∇ P(w

t −1

) .

DOI:10.12677/aam.2022.11129529034 A^êÆ?Ð

Á ?]§X[

ò þãØªét= 1,2,...,m¦Ú,Œ

E kw

− w

∗

+2 mR(1+2

) E[ P(˜ w

) −P( w

∗

)]

6 Ekw

− w

∗

4 mLR

E [P(˜w)−P(w∗)]

= Ek˜ w−w

∗

4 mLR

E [P(˜w)−P(w

∗

)]

E [P(˜w)−P(w

∗

)]+

4 mLR

E [P(˜w)−P(w

∗

)]

= 2( µ

− 1

2 mLR

) E[ P(˜ w)−p(w

∗

)] ,

é1 ‡Ø ª¦^(7)ª,ddŒ± µ

E [P(˜w

) −P( w

∗

)] 6

µmR (b +2LR )

2 LR

b +2LR

E [P(˜w

s −1

) −P( w

∗

)] ,

l Ž{Âñ5µE[P(˜w

) −P( w

∗

)] ≤α

E [P(˜w

) −P( w

∗

)] y. 

d Ž{Âñ5Œ±wÑ,Ž{Âñ„Ý† ëêbÚmk ','u 1þëêbÚ•#ªÇ

4. êŠ¢

Ž {¢y,ëê•CPU:Intel(R)Core(TM)i5-6200U2.40GHz,S•:4.00GB.¦^Ì

6 êâ8MNISTéŽ{k5?1y,êâ8Ì‡ëê„L1µ

Table1. Datasetintroduction

L 1.êâ80

8 ã¡a.ã¡ŒÔö8êþÿÁ8êþ

MNIST •Ýã28*282000010000

´ Ž{ëê éŽ{-½5K•.

4.1. Ì6Ž{é'

Ä uPython3.7¤AmbSVRGŽ{?,òÙ†Ì6Ž{Mini-BatchSGD!Batching

SVRG !AdaGrad!RMSPropŽ{?1é'.

DOI:10.12677/aam.2022.11129529035 A^êÆ?Ð

Á ?]§X[

l ã1Œ±wÑµ

(2) lÂñL§þw, ØAdaGrad Ž{ÂñL§Ø²wƒ, Ù¦Ž{ÑU-½/Âñ

• `);lÿÁ8O (Çþ,·Ý5w,3S“ÐÏ,AmbSVRGþ,·Ý•Í,Œ„ÙÂñ„

Ý •¯.

: ,BatchingSVRGO(Çˆ85.7%,E'AdaGradÚMini-BatchSGDŽ{`D.

Figure1. Accuracycomparisonofmainstreamalgorithms

ã 1.Ì6Ž{O(Çé'

4.2. -½5&Ä

 ,“\•#ªÇ•10!30!50Ž{¥,ÑØÓëêœ¹eŽ{ÔöžmÚÿÁ8O(Ç,

¢ ëê 9éAµd•I„L2.•;•¢ ó,5,é¢ ê â83ƒÓëê ^‡e?1

n gÓ¢,¿n g¢²þŠŠ••ª(J.

Table2. Theinﬂuenceofexperimentalparametersontheconvergenceeﬀectofthealgorithm

L 2.¢ëêéŽ{ÂñJK•&Ä

• #ªÇm

b = 100b = 300b = 500

traintimetestaccuracytraintimetestaccuracytraintimetestaccuracy

m = 1019.34s85.2%22.26s88.9%23.86s91.4%

m = 3024.22s87.5%28.69s89.1%35.12s89.5%

m = 5032.05s86.0%36.71s87.2%46.89s86.7%

DOI:10.12677/aam.2022.11129529036 A^êÆ?Ð

Á ?]§X[

d L2Œ•,3Ôöžmþ,‘X1þº€ŒbÚ•#ªÇmO\,Ž{OŽ“d\

Œ ,ÙƒéÔöžm•ÅìO\;3ÿÁ8O(Çþ,^‡b= 500žŽ{ÿÁ8O(ÇoN‡

' b= 300Úb= 100œ¹e••Œ*.ddŒ±wÑ,1þº€ŒbéŽ{ÿ Á8O(Çk

† K•,…äk˜½ '~'X;3b†m'X&Äþ,Œ±wm=10Úm=30ž,

‘ XbO\,Ž {ÿÁ8O(Ç•ÅìO\,m=50ž,bO\ŒU¬‘5O(Çeü,

Œ UmO\—SÌ‚ëê•#FÝÚÌ‚ÛFÝƒCŒ,éÂñ„ÝÚÂñO(Ç

E ¤˜½K•.

µ ØÏL4.1!Œ±wÑ,AmbSVRGŽ{Âñ„ÝÚÂñ°ÝLyâÑ,ÙÂñ°ÝÚ

RMSProp Ž{ƒ, ¿…3ÐÏŽ{Âñ„Ý`uÙ¦Ž{. ÏL4.2 !, )1þº€Œ

b Ú•#ªÇm éŽ{Âñ„ÝÚÂñ°ÝK•,éŽ{ÂñJNëäk˜½ë•¿Â.

5. (Ø

Œ 1þ“ OÜ?1ÛFÝOŽ,À1þ?1ëêS“ •#,ŽÑ‘Å

F ÝeüŽ{duÀ‘Å5Ú\p•¯K ,±9F Ýeü{ð½ÆSÇ—ëê3

• `:5£!Âñ„Ý…ú":,Óžég·AÆSÇŽ{³›LpÆSÇ).Ï

d AmbSVRGŽ{äkûÐ-½5Ú¯Âñ„Ý,¿…¢(JL²,3†Ì6Ž{é'e,

AmbSVRG Ž{k5•y.

É ›u Ï,Ã{yŽ{3E,ä(þk5,Óž duëê ÀéŽ{

O ŽÇk˜½K•,ÏdXÛÀëêéJpŽ{O ŽÇ•äk˜½y¢¿Â.

ë •©z

[1]Bottou,L.(2010)Large-ScaleMachineLearningwithStochasticGradientDescent.In:

Lechevallier,Y.andSaporta,G.,Eds., ProceedingsofCOMPSTAT’2010,Physica-VerlagHD,

177-186.https://doi.org/10.1007/978-3-7908-2604-316

[2]Sebastian,R.(2016)AnOverviewofGradientDescentOptimizationAlgorithms. Computer

ScienceMachineLearning ,1-12.

[3]Lecun, Y.and Bottou,L.(1998)Gradient-BasedLearning Appliedto Document Recognition.

ProceedingsoftheIEEE ,86,2278-2324.https://doi.org/10.1109/5.726791

[4]Bubeck, S.(2015) Convex Optimization:Algorithms and Complexity. FoundationsandTrends

inMachineLearning ,8,231-357.https://doi.org/10.1561/2200000050

[5]Bottou,L., Curtis,F.E.andNocedal, J.(2018) OptimizationMethods forLarge-Scale Machine

Learning. SIAMReview,60,223-311.https://doi.org/10.1137/16M1080173

DOI:10.12677/aam.2022.11129529037 A^êÆ?Ð

Á ?]§X[

[6]Nesterov,Y.(2013)GradientMethodsforMinimizingCompositeFunctions. Mathematical

Programming ,140,125-161.https://doi.org/10.1007/s10107-012-0629-5

[7]Robbins, H. andMonro,S. (1951) AStochastic Approximation Method. AnnalsofMathemat-

icalStatistics ,22,400-407.https://doi.org/10.1214/aoms/1177729586

[8]Li,M.,Zhang,T.,Chen,Y., etal.(2014)EﬃcientMini-BatchTrainingforStochasticOp-

timization. Proceedingsofthe20thACMSIGKDDConferenceonKnowledgeDiscoveryand

DataMining ,NewYork,24-27August2014,661-670.

[9]Qian,N.(1999)OntheMomentumTerminGradientDescentLearningAlgorithms. Neural

Networks ,12,145-151.https://doi.org/10.1016/S0893-6080(98)00116-6

[10]Nesterov,Y.(1983)AMethodofSolvingaConvexProgrammingProblemwithConvergence

RateO(1/k

). SovietMathematicsDoklady, 27,372-376.

[11]Botev,A.,Lever,G.andBarber,D.(2017)Nesterov’sAcceleratedGradientandMomentum

asApproximationstoRegularisedUpdateDescent. 2017InternationalJointConferenceon

NeuralNetworks ,Anchorage,AK,14-19May2017,1899-1903.

[12]Duchi, J., Hazan,E. and Singer, Y. (2011) Adaptive Subgradient Methods for Online Learning

andStochasticOptimization. JournalofMachineLearningResearch,12,2121-2159.

[13]Tieleman, T. and Hinton, G. (2012)Lecture 6.5-RMSProp:Dividethe Gradient by a Running

AverageofItsRecentMagnitude. COURSERA:NeuralNetworksforMachineLearning,4,

26-30.

[14]Kingma,D.andBa,J.(2015)Adam:AMethodforStochasticOptimization. Proceedingsof

the3rdInternationalConferenceonLearningRepresentations ,San Diego,7-9May 2015, 1-13.

[15]Ding,J.,Ren,X.,Luo,R., etal.(2019)AnAdaptiveandMomentalBoundMethodfor

StochasticLearning.arXiv:1910.12249

[16]Schmidt,M.,LeRoux,N.andBach,F.(2017)MinimizingFiniteSumswiththeStochastic

AverageGradient. MathematicalProgramming,162,83-112.

[17]Shalev,S.andZhang,T.(2013)StochasticDualCoordinateAscentMethodsforRegularized

LossMinimization. JournalofMachineLearningResearch,14,567-599.

https://doi.org/10.1007/s10107-016-1030-6

[18]Johnson,R.andZhang,T.(2013)AcceleratingStochasticGradientDescentUsingPredictive

VarianceReduction. Proceedingsofthe27thNeuralInformationProcessingSystems,Lake

Tahoe,5-10December2013,315-323.

[19]Babanezhad,R., Ahmed, M.O., Virani,A., etal.(2015) Stop Wasting MyGradients:Practical

SVRG. Proceedingsofthe28thInternationalConferenceonNeuralInformationProcessing

Systems ,2,2251-2259.

[20]Qiu, X.P. (2020)NeuralNetworksandDeepLearning.ChinaMachine Press,Beijing,160-169.

DOI:10.12677/aam.2022.11129529038 A^êÆ?Ð