Markov Decision Processes Finals V2
A Markov Process is a process in which all states do not depend on previous actions. ✔️✔️True,
Markov means that you don't have to condition on anything past the most recent state. A Markov
Decision Process is a set of Markov Property Compliant states, with rewards and values.
Decaying Reward encourages the agent to end the game quickly instead of running around and
gathering more reward ✔️✔️True, as reward decays the total reward for the episode decreases, so
the agent is encouraged to maximize total reward by ending the game quickly.
R(s) and R(s,a) are equivalent. ✔️✔️True, it just happens that it's easier to think about one vs the
other in certain situations.
Reinforcement Learning is harder to compute than a simple MDP. ✔️✔️True, you can just use the
Bellman Equations for an MDP, but Reinforcement Learning requires that you make observations and
then summarize those observations as values.
An optimal policy is the best possible sequence of actions for an MDP. ✔️✔️True, with a single caveat.
The optimal policy is a policy that maximizes reward over an entire episode by taking the argmax of
resulting values of actions + rewards. But MDPs are memoryless, so there is no concept of "sequence"
for a policy.
Temporal Difference Learning is the difference in reward you see on subsequent time steps.
✔️✔️False, Temporal Difference Learning is the difference in value estimates on subsequence time
steps.
RL falls generally into 3 different categories: Model-Based, Value-Based, and Policy-Based. ✔️✔️True,
Model-Based is essentially using the Bellman Equations to solve a problem, Value-Based is Temporal
Difference Learning, and Policy-Based is similar to Value-Based, but it solves in a finite amount of time
with a certain amount of confidence (in Greedy it's guaranteed).
, TD Learning is defined by Incremental Estimates that are Outcome Based. ✔️✔️True, TD Learning
thinks of learning in terms of "episodes", which it uses to estimate the transition functions rather than
having a predefined model.
For a learning rate to guarantee convergence, the sum of the learning rate must be infinite, and the sum
of the learning rate squared must be finite. ✔️✔️True, this is called a contraction mapping and it
guarantees convergence.
All of the TD learning methods have set backs, TD(1) is inefficient because it requires too much data and
has high variance, TD(0) has a maximum likelihood estimate but is hard to calculate for long episodes.
✔️✔️True, this is why we use TD(Lambda), which has many of the benefits of TD(0) but is much more
performant. Empirically, lambdas between 0.3 and 0.7 seem to perform best.
To control learning, you simply have the operator choose actions in addition to learning. ✔️✔️True,
states are experienced as observations during learning, so the operator can influence learning.
Q-Learning converges ✔️✔️True, the Bellman Equation satisfies a Contraction Mapping where the
sum of all is infinite, but the sum of all squared is less than infinite. It always converges to Q*.
As long as the update operators for Q-learning or Value-iteration are non-expansions, then they will
converge. ✔️✔️True, there are expansions that will converge, but only non-expansions are
guaranteed to converge independent of their starting values.
A convex combination will converge. ✔️✔️False, it must be a fixed convex combination to converge. If
the value can change, like with the Boltzmann exploration, then it is not guaranteed to converge.
In Greedy Policies, the difference between the true value and the current value of the policy is less than
some epsilon value for exploration. ✔️✔️True
It serves as a good check for how long we run value iteration until we're pretty confident that we have
the optimal policy. ✔️✔️True
For a set of linear equations, the solution can be found in polynomial time. ✔️✔️True
A Markov Process is a process in which all states do not depend on previous actions. ✔️✔️True,
Markov means that you don't have to condition on anything past the most recent state. A Markov
Decision Process is a set of Markov Property Compliant states, with rewards and values.
Decaying Reward encourages the agent to end the game quickly instead of running around and
gathering more reward ✔️✔️True, as reward decays the total reward for the episode decreases, so
the agent is encouraged to maximize total reward by ending the game quickly.
R(s) and R(s,a) are equivalent. ✔️✔️True, it just happens that it's easier to think about one vs the
other in certain situations.
Reinforcement Learning is harder to compute than a simple MDP. ✔️✔️True, you can just use the
Bellman Equations for an MDP, but Reinforcement Learning requires that you make observations and
then summarize those observations as values.
An optimal policy is the best possible sequence of actions for an MDP. ✔️✔️True, with a single caveat.
The optimal policy is a policy that maximizes reward over an entire episode by taking the argmax of
resulting values of actions + rewards. But MDPs are memoryless, so there is no concept of "sequence"
for a policy.
Temporal Difference Learning is the difference in reward you see on subsequent time steps.
✔️✔️False, Temporal Difference Learning is the difference in value estimates on subsequence time
steps.
RL falls generally into 3 different categories: Model-Based, Value-Based, and Policy-Based. ✔️✔️True,
Model-Based is essentially using the Bellman Equations to solve a problem, Value-Based is Temporal
Difference Learning, and Policy-Based is similar to Value-Based, but it solves in a finite amount of time
with a certain amount of confidence (in Greedy it's guaranteed).
, TD Learning is defined by Incremental Estimates that are Outcome Based. ✔️✔️True, TD Learning
thinks of learning in terms of "episodes", which it uses to estimate the transition functions rather than
having a predefined model.
For a learning rate to guarantee convergence, the sum of the learning rate must be infinite, and the sum
of the learning rate squared must be finite. ✔️✔️True, this is called a contraction mapping and it
guarantees convergence.
All of the TD learning methods have set backs, TD(1) is inefficient because it requires too much data and
has high variance, TD(0) has a maximum likelihood estimate but is hard to calculate for long episodes.
✔️✔️True, this is why we use TD(Lambda), which has many of the benefits of TD(0) but is much more
performant. Empirically, lambdas between 0.3 and 0.7 seem to perform best.
To control learning, you simply have the operator choose actions in addition to learning. ✔️✔️True,
states are experienced as observations during learning, so the operator can influence learning.
Q-Learning converges ✔️✔️True, the Bellman Equation satisfies a Contraction Mapping where the
sum of all is infinite, but the sum of all squared is less than infinite. It always converges to Q*.
As long as the update operators for Q-learning or Value-iteration are non-expansions, then they will
converge. ✔️✔️True, there are expansions that will converge, but only non-expansions are
guaranteed to converge independent of their starting values.
A convex combination will converge. ✔️✔️False, it must be a fixed convex combination to converge. If
the value can change, like with the Boltzmann exploration, then it is not guaranteed to converge.
In Greedy Policies, the difference between the true value and the current value of the policy is less than
some epsilon value for exploration. ✔️✔️True
It serves as a good check for how long we run value iteration until we're pretty confident that we have
the optimal policy. ✔️✔️True
For a set of linear equations, the solution can be found in polynomial time. ✔️✔️True
