SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 LinearZ AlgebraZ andZ Optimization:Z AnZ Introduction 1
2 LinearZ Transformations Z andZ LinearZ Systems 17
3 Diagonalizable Z MatricesZ andZ Eigenvectors 35
4 OptimizationZBasics:ZAZMachineZLearningZView 47
5 OptimizationZ ChallengesZ andZ AdvancedZ Solutions 57
6 LagrangianZ RelaxationZ andZ Duality 63
7 SingularZ ValueZ Decomposition 71
8 MatrixZ Factorization 81
9 TheZ LinearZ AlgebraZ ofZ Similarity 89
10 TheZ LinearZ AlgebraZ ofZ Graphs 95
11 OptimizationZ inZ ComputationalZ Graphs 101
viii
,ChapterZ 1
LinearZAlgebraZandZOptimization:ZAnZIntroduction
1. ForZ anyZ twoZ vectorsZ xZ andZ y,Z whichZ areZ eachZ ofZ lengthZ a,Z showZ thatZ (
i)Z xZ−ZyZ isZorthogonalZtoZxZ+Zy,Z andZ(ii)Z theZdotZproductZofZxZ−Z3yZ andZ
xZ+Z3yZ isZ negative.
(i)ZTheZfirstZisZsimply
·Z −ZZx·Z xZ yZ yZusingZtheZdistributiveZpropertyZofZmatri
xZmultiplication.ZTheZdotZproductZofZaZvectorZwithZitselfZisZitsZsquaredZ
length.ZSinceZbothZvectorsZareZofZtheZsameZlength,ZitZfollowsZthatZtheZr
esultZisZ0.Z(ii)ZInZtheZsecondZcase,ZoneZcanZuseZaZsimilarZargumentZtoZsh
owZthatZtheZresultZisZa2Z−Z9a2,ZwhichZisZnegative.
2. ConsiderZ aZ situationZ inZ whichZ youZ haveZ threeZ matricesZ A,Z B,Z andZ C,Z ofZ s
izesZ 10Z×Z2,Z2Z×Z10,ZandZ 10Z×Z10,Zrespectively.
(a) SupposeZyouZhadZtoZcomputeZtheZmatrixZproductZABC.ZFromZanZeffici
encyZper-
Zspective,ZwouldZitZcomputationallyZmakeZmoreZsenseZtoZcomputeZ(AB)CZ
orZwouldZitZmakeZmoreZsenseZtoZcomputeZA(BC)?
(b) IfZyouZhadZtoZcomputeZtheZmatrixZproductZCAB,ZwouldZitZmakeZmoreZ
senseZtoZcomputeZ (CA)BZ orZ C(AB)?
TheZmainZpointZisZtoZkeepZtheZsizeZofZtheZintermediateZmatrixZasZs
mallZasZpossibleZ inZorderZtoZreduceZbothZcomputationalZandZspaceZre
quirements.ZInZtheZcaseZofZABC,ZitZmakesZsenseZtoZcomputeZBCZfirst.
ZInZtheZcaseZofZCAB ZitZmakesZsenseZtoZcomputeZCAZfirst.ZThisZtypeZ
ofZassociativityZpropertyZisZusedZfrequentlyZinZmachineZlearningZinZo
rderZtoZreduceZcomputationalZrequirements.
3. ShowZ thatZ ifZ aZ matrixZ AZ satisfies—Z AZ =
ATZ,Z thenZ allZ theZ diagonalZ elementsZ o
fZ theZmatrixZareZ0.
NoteZthatZAZ+ZATZ=Z0.ZHowever,ZthisZmatrixZalsoZcontainsZtwiceZthe
ZdiagonalZelementsZofZAZonZitsZdiagonal.ZTherefore,ZtheZdiagonalZele
mentsZofZAZmustZbeZ0.
4. ShowZthatZifZweZhaveZaZmatrixZsatisfying
— ZAZ=
1
, ATZ,ZthenZforZanyZcolumnZvectorZ
x,ZweZhaveZ x ZAxZ=Z0.
T
NoteZ thatZ theZ transposeZ ofZ theZ scalarZ xTZAxZ remainsZ unchanged.Z Therefore,
Z weZ have
xTZAxZ=Z(xTZAx)TZ =ZxTZATZxZ=Z−xTZAx.Z Therefore,Z weZ haveZ 2xTZAxZ
=Z0.
2
