SOLUTION MANUAL
Linear Algebra and Optimization for Machine
Learning
1st Edition by Charu Aggarwal. Chapters 1 – 11
vii
,Contents
1 LinearT AlgebraT andT Optimization:T AnT Introduction 1
2 LinearT TransformationsT andT LinearT Systems 17
3 Diagonalizable T MatricesT andT Eigenvectors 35
4 OptimizationTBasics:TATMachineTLearningTView 47
5 OptimizationT ChallengesT andT AdvancedT Solutions 57
6 LagrangianT RelaxationT andT Duality 63
7 SingularT ValueT Decomposition 71
8 MatrixT Factorization 81
9 TheT LinearT AlgebraT ofT Similarity 89
10 TheT LinearT AlgebraT ofT Graphs 95
11 OptimizationT inT ComputationalT Graphs 101
viii
,ChapterT 1
LinearTAlgebraTandTOptimization:TAnTIntroduction
1. ForT anyT twoT vectorsT xT andT y,T whichT areT eachT ofT lengthT a,T showT thatT (i
)T xT−TyT isTorthogonalTtoTxT+Ty,T andT(ii)T theTdotTproductTofTxT−T3yT andTxT
+T3yT isT negative.
(i)TTheTfirstTisTsimply
·T −TTx·T xT yT yTusingTtheTdistributiveTpropertyTofTmatri
xTmultiplication.TTheTdotTproductTofTaTvectorTwithTitselfTisTitsTsquaredTl
ength.TSinceTbothTvectorsTareTofTtheTsameTlength,TitTfollowsTthatTtheTre
sultTisT0.T(ii)TInTtheTsecondTcase,ToneTcanTuseTaTsimilarTargumentTtoTsho
wTthatTtheTresultTisTa2T−T9a2,TwhichTisTnegative.
2. ConsiderT aT situationT inT whichT youT haveT threeT matricesT A,T B,T andT C,T ofT si
zesT 10T×T2,T2T×T10,TandT10T×T10,Trespectively.
(a) SupposeTyouThadTtoTcomputeTtheTmatrixTproductTABC.TFromTanTefficie
ncyTper-
Tspective,TwouldTitTcomputationallyTmakeTmoreTsenseTtoTcomputeT(AB)CT
orTwouldTitTmakeTmoreTsenseTtoTcomputeTA(BC)?
(b) IfTyouThadTtoTcomputeTtheTmatrixTproductTCAB,TwouldTitTmakeTmoreTs
enseTtoTcomputeT (CA)BT orT C(AB)?
TheTmainTpointTisTtoTkeepTtheTsizeTofTtheTintermediateTmatrixTasTsm
allTasTpossibleT inTorderTtoTreduceTbothTcomputationalTandTspaceTreq
uirements.TInTtheTcaseTofTABC,TitTmakesTsenseTtoTcomputeTBCTfirst.TI
nTtheTcaseTofTCABTitTmakesTsenseTtoTcomputeTCATfirst.TThisTtypeTofT
associativityTpropertyTisTusedTfrequentlyTinTmachineTlearningTinTorde
rTtoTreduceTcomputationalTrequirements.
3. ShowT thatT ifT aT matrixT AT satisfies—T AT =
ATT,T thenT allT theT diagonalT elementsT o
fT theTmatrixTareT0.
NoteTthatTAT+TATT=T0.THowever,TthisTmatrixTalsoTcontainsTtwiceTtheT
diagonalTelementsTofTATonTitsTdiagonal.TTherefore,TtheTdiagonalTele
mentsTofTATmustTbeT0.
4. ShowTthatTifTweThaveTaTmatrixTsatisfying
— TAT=
1
, ATT,TthenTforTanyTcolumnTvectorTx
,TweThaveT x TAxT=T0.
T
NoteT thatT theT transposeT ofT theT scalarT xTTAxT remainsT unchanged.T Therefore,T
weT have
xTTAxT=T(xTTAx)TT =TxTTATTxT=T−xTTAx.T Therefore,T weT haveT 2xTTAxT=
T0.
2
