ISYE 6501 - Midterm 1 2024 with complete verified solutions.

What do descriptive questions ask? What happened? (e.g., which customers are most alike) What do predictive questions ask? What will happen? (e.g., what will Google's stock price be?) Brainpower Read More 0:05 / 0:15 What do prescriptive questions ask? What action(s) would be best? (e.g., where to put traffic lights) What is a model? Real-life situation expressed as math. What do classifiers help you do? differentiate What is a soft classifier and when is it used? In some cases, there won't be a line that separates all of the labeled examples. So we use a classifier that minimizes the number of mistakes. What does it mean when the classifier/decision boundary is almost parallel to the vertical x-axis? The horizontal attribute is all that is needed. What does it mean when the classifier/decision boundary is almost parallel to the horizontal y-axis? The vertical attribute is all that is needed. What is time-series data? The same data recorded over time often recorded at equal intervals What is quantitative data? Number with a meaning: higher means more, lower means less (e.g., age, sales, temperature, income) What is categorical data? Numbers w/o meaning (e.g., zip codes), non-numeric (e.g., hair color), binary data (e.g., male/female, yes/no, on/off) Which of these is time series data? A. The average cost of a house in the United States every year since 1820 B. The height of each professional basketball player in the NBA at the start of the season A Which of these is structured data? A. The contents of a person's Twitter feed B. The amount of money in a person's bank account B What is structured data? Data that can be stores in a structured way What is unstructured data? Data that is not easily described and stored (e.g., written text) A survey of 25 people recorded each person's family size and type of car. Which of these is a data point? A. The 14th person's family size and car type B. The 14th person's family size C.The car type of each person A. A data point is all the information about one observation The farther the wrongly classified point is from the line ___ The bigger the mistake we've made The term including the margin gets larger so the importance of a large margin out weights avoiding mistakes and classifying known data samples. As lambda gets larger That term also drops towards zero, so the importance of minimizing mistakes and classifying known data points outweighs having a large margin. As lambda drops towards zero What can SVMs be used for to find a classifier with maximum seperation or margin between the two sets of points? When to use SVM? If it's impossible to avoid classification errors, SVM can find a classifier that trades off reducing errors and enlarging the margin. Error for data point j What does this formula describe? Total error What does this formula describe ? To maximize the distance between the two lines what do we need to minimize? m_j 1 What value do we give for more costly errors Giving a bad loan is twice as costly as withholding a good loan? What does this mean in the context of giving a loan? m_j 1 What value do we give for less costly errors? Why is it important to scale our data when using SVM? We're looking to minimize the sum of the squares of the coefficients, but if our data has very different scales a small change in one could swamp a huge change in the other. what does it signify when a coefficient for a classifier is close to zero it means the corresponding attribute is probably not relevant What do kernel methods allow for in SVMs nonlinear classifiers What is the common range for scaled data? between 0 and 1 What is the formula for min-max scaling? find min and max for a factor what is common standardization and its formula? scaling to a normal distribution with a mean of 0 and standard deviation of 1. what is the formula for general scaling between b and a When do you use scaling? Data in a bounded range (e.g., neural networks, RGB values, SAT scores, batting averages) When do you use standardization? PCA or clustering When is KNN used? Used for solving classification problems in which there are more than two classes. How do you deal with attributes that might be more important than others in KNN? You weight each dimension's distance different. The larger the weight the higher the impact. A large value of K will lead to a large variance in predictios Setting a large value of k will ... lead to a large model bias. What are real effects? Real relationships between attributes and responses. They are the same in all data sets, What are random effects? They are random but look like real effects. They are different in all data sets. Why can't we measure a model's effectiveness on data it was trained on? The model's performance on its training data is usually too optimistic, the model is fit to both real and random pattenrs in the data, so it becomes overly specialized to the specific randomness in the training set, that doesn't exist in other data. If we use the same data to fit a model as we do to estimate how good it is, what is likely to happen? The model will appear to be better than it really is. The model will be fit to both real and random patterns in the data. The model's effectiveness on this data set will include both types of patterns, but its true effectiveness on other data sets (with different random patterns) will only include the real patterns When comparing models, if we use the same data to pick the best model as we do to estimate how good the best one is, what is likely to happen? The model will appear to be better than it really is. The model with the highest measured performance is likely to be both good and lucky in its fit to random patterns. What is a training set used for used to fit the models What is a validation set used for? used to choose best model Why would we use two sets? Reason to use two different sets is because if the first set, the training set, had unique random effects that the classifer was designed for, we wouldn't be counting those benefits when we measure effectiveness on the validation set. What effects does randomness have on training /validation performance? sometimes the randomness will make the performance look worse than it really is, and sometimes the randomness will make the performance look better than it really is how are high-performing models affected by randomness? They are often boosted by above average random effects making it look better what is a test data set used for? to estimate performance of chosen model When do we need a validation set? When we are choosing between multiple models. What are the data splits when working with one model? 70-90% training, 10-30% test What are the data splits when comparing models? 50-70% training, split the rest between validation and test What are two methods of splitting data? random and roation What is the rotation method of splitting data? You take turns selecting points. 5 data point rotation sequence: (Training - Validation - Training - Test - Training What is the advantage of rotation over randomness? We make sure each part of the data is equally separated. What is the disadvantage of using rotation? We have to make sure we aren't creating some other type of bias when we assign points. what is k-fold cross validation? split the training/validation data into k-parts; we train on k-1 parts and validate on the remaining part. What metric do you use for k-fold cross validation when comparing models? The average of all k evaluations. What do we use when important data only appears in the validation or test sets? cross-validation What do we do after we've performed cross-validation? We train the model again using all the data. what are the benefits of k-fold cross validation? better use of data, better estimate of model quality, and chooses model more effectively What can clustering be used for? grouping data points (e.g., market segmentation) and discovering groups in data points (e.g., personalized medicine Which should we use most of the data for: training, validation, or test? training In k-fold cross-validation, how many times is each part of the data used for training, and for validation? k-1 times for training, and 1 time for validation what is rectangular distance useful for? calculating driving distance when the city is mapped in a grid what is the value of p for euclidean distance 2 what is the general equation for p-norm distance 2-norm Straight-line distance corresponds to which distance metric? How do you find the distance of an infinity norm? You find the largest | x_i - y_i | What is a centroid the center of a cluster What are the steps of k means? 0. Pick k clusters within range of data. 1. Assign each data point to nearest cluster center 2. Recalculate cluster centers (centroids) 3. Repeat 1 and 2 until no changes How do we find the cluster centers? We take the mean of all the data points in cluster. Why is k-means an expectation-maximization finding the mean of all the points in cluster is similar to finding an expectation. Assigning data points to cluster centers is the maximization step. Really we are minimizing, but we could think of it as maximizing the negative of the distance to a cluster center What are some of the consequences of outliers in k-means? It will drag the cluster center artificially to one side. Because k-means is a heuristic and thus fast what can we do? run it several times choosing different clusters centers and choose the best one and we can choose different values of k how does bias/variance change as k changes in KNN the higher the k the higher the bias the lower the k the higher the variance. when K = 1 that is the most complex model and thus likely to overfit the data. How do we find the best value of k in k means? Elbow method: we calculate the total distance of each data point to its cluster center and plot it in two dimensions. We look for the kik in the graph. When clustering for prediction how do we choose the prediction? When we see a new point, we just choose whichever cluster center is closest. What is the difference between classification and clustering? With classification mdoels, we know each data point's attributes and we already know the right classification for the data points (supervised). In clustering (unsupervised) we know the attributes but we don't know what group any of these data points are in. What is the difference between supervised learning and unsupervised learning? Supervised - the response is known Unsupervised - response is not known. The k-means algorithm for clustering is a "heuristic" because... ...it isn't guaranteed to get the best answer but it will get to a solution quickly. A group of astronomers has a set of long-exposure CCD images of various distant objects. They do not know yet which types of object each one is, and would like your help using analytics to determine which ones look similar. Which is more appropriate: classification or clustering? clustering Suppose one astronomer has categorized hundreds of the images by hand, and now wants your help using analytics to automatically determine which category each new image belongs to. Which is more appropriate: classification or clustering? classification Which of these is generally a good reason to remove an outlier from your data set? A. The outlier is an incorrectly-entered data, not real data. B. Outliers like this only happen occasionally. A. If the data point isn't a true one, you should remove it from your data set. What is an outlier? A data point that is very different from the rest What graph or plot can we use to find outliers? box-and-whisker plot What are the parts of a box-and-whisker plot? The bottom and top of the box are the 25th and 75th percentile. The middle valu is the median. The whiskers stretch up and down to reasonable range of values (10 and 90th or 5th and 95 percentiles) Where would outliers exist in a box and whisker plot outside of the whiskers. What are some ways to deal with outliers that are bad data? Omit them or use imputation What can change detection be used for? Determining whether action might be needed, determining impact of past action, determining changes to help plan. What is Cumulative sum (CUSUM) used for detect in crease, decrease or both What is C used for in the Cusum formula Since we expect there to be some randomness, we include a value C to pull the running total down If we have a larger C ... the harder for S_t to get large and the less sensitive the method will be If we have a smaller C ... the more sensitive the method is because S_t can get larger faster What factors go into finding the right values of C and T? how costly it is if the model takes a long time to nice a change, and how costly it is if the model think it has found a change that really isn't there. Why are hypothesis tests often not sufficient for change detection? They often are slow to detect changes. Hypothesis tests generally have high threshold levels, which makes them slow to detect changes. In the CUSUM model, having a higher threshold T makes it... detect changes slower, and less likely to falsely detect changes. In the exponential smoothing equation S_t = alpha times x_t + (1-alpha) times S_{t-1} a value of closer to 1 is chosen if... There's less randomness, so we're more willing to trust the observation. We put more weight on the observation x_t than the previous estimate S_{t-1} A multiplicative seasonality, like in the Holt-Winters method, means that the seasonal effect is... Proportional to the baseline value. A multiplicative seasonality is larger when the baseline value is larger, because its effect is a multiple of the baseline In the exponential smoothing equation S_t = alpha times x_t + (1-alpha) times S_{t-1} only the current observation x_t is considered in calculating the estimate S_t. False. we consider all previous observations Is exponential smoothing better for short-term forecasting or long-term forecasting? Short-term Exponential smoothing bases its forecast primarily on the most-recent data points. For forecasts of the longer-term future, there aren't data points close to the time being forecasted In simple forecasting with basic exponential smoothing what is the value of F_{t+i} S_t What does autoregression mean? Previous values of the thing being estimated are used to calculate the estimate Why would we want to estimate the variance? Knowing the variance can help us estimate the amount of error Why is GARCH different from ARIMA and exponential smoothing? GARCH estimates variance ARIMA and exponential smoothing both estimate the value of an attribute; GARCH estimates the variance When would regression be used instead of a time series model? When there are other factors or predictors that affect the response. Regression helps show the relationships between factors and a response If two models are approximately equally good, measures like AIC and BIC will favor the simpler model. Simpler models are often better because... Simpler models are less likely to be over-fit, easier to understand, and easier to explain What is not a common use of regression? Prescriptive analytics: Determining the best course of action Regression is often good for describing and predicting, but is not as helpful for suggesting a course of action True or false: regression is a way to determine whether one thing causes another. False. Regression can show relationships between observations, but it doesn't show whether one thing causes another Suppose our regression model to estimate how tall a 2-year-old will be as an adult has the following coefficients: 0.56xFatherHeight + 0.51xMotherHeight - 0.02xFatherHeightxMotherHeight The negative sign on the coefficient of FatherHeightxMotherHeight means: People with two taller-than-average parents won't be as tall as the individual effects of father's height and mother's height add up to The negative coefficient for the interaction term brings down the overall estimate What does "heteroscedasticity" mean? The variance is different in different ranges of the data You might want to de-trend data before... ...using time-series data in a regression model Factor-based models like regression generally don't account for time-based effects like trend. Which of the following does principal component analysis (PCA) do? Transform data so there's no correlation between dimensions and rank the new dimensions in likely order of importance. If you use principal component analysis (PCA) to transform your data and then you run a regression model on it, how can you interpret the regression coefficients in terms of the original attributes? Each original attribute's implied regression coefficient is equal to a linear combination of the principal components' regression coefficients. This is equivalent to using the inverse transformation. True or false: In a regression tree, every leaf of the tree has a different regression model that might use different attributes, have different coefficients, etc. True. Each leaf's individual model is tailored to the subset of data points that follow all of the branches leading to the leaf. Tree-based approaches can be used for other models besides regression. True. For example, a classification tree might have a different SVM or KNN model at each leaf. It might even use SVM at some leaves and KNN at others (though that's probably rare). A common rule of thumb is to stop branching if a leaf would contain less than 5% of the data points. Why not keep branching and allow models to find very close fits to each very small subset of data? Fitting to very small subsets of data will cause overfitting. With too few data points, the models will fit to random patterns as well as real ones True or False: When using a random forest model, it's easy to interpret how its results are determined. False. Unlike a model like regression where we can show the result as a simple linear combination of each attribute times its regression coefficient, in a random forest model there are so many different trees used simultaneously that it's difficult to interpret exactly how any factor or factors affect the result. A logistic regression model can be especially useful when the response... ...is a probability (a number between zero and one) or is binary (either zero or one). wga A model is built to determine whether data points belong to a category or not. A "true negative" result is: A data point that is not in the category, and the model correctly says so. True' and 'false' refer to whether the model is correct or not, and 'positive' and 'negative' refer to whether the model says the point is in the category. True or False: The most useful classification models are the ones that correctly classify the highest fraction of data points. False. Sometimes the cost of a false positive is so high that it's worth accepting more false negatives, or vice versa. In exponential smoothing what is S_t the expected baseline response at time period t e.g., blood pressure at hour t In exponential smoothing what is x_t observed response. Observed blood pressure at t S_t = alpha times x_t + (1 - alpha)S_{t-1}. When alpha is closer to zero a lot of randomness in the system. the previous baseline is probably a good indicator of today's baseline S_t = alpha times x_t + (1 - alpha)S_{t-1}. When alpha is closer to 1 not much randomness in the system. If we observe a fluctuation today, it probably means today's baselines is close to the observed data What is T_t in S_t = alpha times x_t + (1 - alpha)(S_{t-1} + T_{t-1}) The trend at time t what is the initial condition for T? T_1 = 0 what is the initial condition for S_t? S_1 = x_1 What is L length of cycle. When we're taking daily observation then L is 7 What is C_t the multiplicative seasonality factor of time. It inflates or delates the observation When C is = 1.1 what does that mean? 10% higher just because of that interval of cycle When C is = 1 what does that mean? no effect How does the exponential smoothing formula weight more recent observations more than older ones? (1 - alpha) 1 The further in the future we go ... The more uncertainty thus the anticpated forecast error gets larger When using exponential smoothing for prediction/forecasting what value is used for x_{t+1}? S_t when does exponential smoothing work well when the data is stationary (i.e., mean, variance and other measures are all expected to be constant over time) explain ARIMA (auto regression integrated moving average) predicts the value based on other factors (regression), uses earlier values to predict (auto). ARIMA autoregresses on the differences. It uses p time periods of previous observations to predict d-th order differences and also incorporates the moving average by looking at q previous errors (x_t_hat - x_t) What is the ARIMA(p,d,q) model? pth order autoregression, dth order differences, q th order moving average. what is ARIMA(p,d,q) (0,0,0) white noise what is ARIMA (0,1,0) random walk what is ARIMA (p,0,0) Auto Regression model, only the auto regressive part is active what is ARIMA(0,0,q) Moving Average model - only the moving average part is active what is ARIMA (0, 1, 1) basic exponential smoothing How many data points do you need for ARIMA to work? 40 When is ARIMA better than exponential smoothing? When the data is less stable what is GARCH (Generalized AutoRegressive Conditional Heteroskedasticity)? model that estimates or forecasts the variance of something that we have time series data for What is the difference between GARCH and ARIMA Variances/squared errors not observations/linear errors. Raw variances not differences of variances or differences of observations like in ARIMA what are the parameters in a GARCH model? p and q when is variance estimation important traditional portfolio optimization: balances the expected return of a set of investment with the amount of volatility. Variance is a proxy for the amount of volatility or risk here What is variance in GARCH? an estimate of the amount of error What will Regression tell you? How systems work (descriptive questions) and what will happen in the future (prescriptive questions) How is the best fit regression line determined? It is the line that minimizes the sum of squared errors What does AIC (Akaike Information Criterion) do and some of its properties? Encourages fewer parameters k and higher likelihood. Works well with a lot of data points. How do you compare two AICs? If the relative likelihood is 8.2% what does that mean? Model 2 is 8.2% as likely as Mode 1 to be better When do you use BIC? when there are more data points than parameters What's the difference between AIC and BIC BIC encourages models with fewer parameters than AIC does When would you use Corrected AIC and not AIC when you have smaller data sets What does |BIC_1 - BIC_2| 10 mean? smaller BIC model is "very likely" better What does 6 |BIC_1 - BIC_2| 10 mean? smaller BIC model is "likely" better What does 2 |BIC_1 - BIC_2| 6 mean? smaller BIC model is "somewhat likely" better What does 0 |BIC_1 - BIC_2| 2 mean? smaller BIC model is "slightly likely" better When trying to answer questions about how a system works what is important the coefficients If you using regression to make forecasts the key answers are? The responses what is causation? One thing causes another What is correlation? two things tend to happen or not happen together When is there causation? Cause is before effect, idea of causation makes sense, no outside factors causing the relationship (hard to guarantee) Can we still make predictions with a model if the predictor and response are highly correlated yes. Even though we can use it for empirical predictions, it doesn't make sense to say that the model shows causation what is the R-squared value? estimates how much variability the model accounts for what is adjusted r-squared same as r^2 but favors simpler models by penalizing for using too many variables What is the T-statistic? the coefficient divided by its standard error; related to the p-value When you have higher p-values ... increase the possibility of including irrelevant factors when you have lower -pvalues increase the possibility of leaving out a relevant factor What happens to p-values when you have a lot of data? they get small when attributes are not all related to the response If you have 100 attributes with a p-value of 2% each what does that mean? we can expect 2 of them to be irrelevant. Which plots can we use to check for normality? Q-Q plot What does a Box-Cox transformation do? performs a logarithm transformation that stretches out the smaller range to enlarge its variability and shrinks the larger range to reduce its variability Why would we want to detrend data? because the trend in time series could mess up a factor based analysis what can you detrend? The response and predictors in factor-based models Name two factor-based models? SVM and regression How do you detrend data? Factor-by-factor. You fit a one-dimensional linear regression to the data and subtract What does PCA do? Removes correlations within the data and ranks coordinates dimension in order of the amount of variance Whey do you focus on the first n principal components? Reduces the effect of randomness and earlier principal components are likely to have higher signal to-noise ratios What is CART? Classification and Regression Trees How do you perform pruning? For every pair of leaves created by the same branch, we use the other half of the data to see whether the estimation error is actually improved by branching. If the branching does improve error, the branches stay, but if the branching actually makes the error gets or not change, we move the branches How do we build a tree? Start with half of the data and build a regression model on it. Then, wheenver there's a leaf we can branch from, we can calculate the variance of the response among all data points in the leaf. We test splitting on each factor to determine how much lower the total variance of the two branches would be compared to the least variance and choose the factor with the lowest variance. If the decrease in variance is more than some threshold data, and there would be enough data points in each branch, we make the split; otherwise, we assume there's not enough benefit to branching and the leaf remains as is What is the idea behind random forests? Introduce radomness. We generate many different trees. They will have different strengths and weaknesses. The average of all these trees is better than a single tree with specific strengths and weaknesses How are the steps in random forests? 1. Introduce randomness via bootstrapping. Branching: randomly choose a small number of factors, set X. The common number of factors to use is log(n). Choose the best factor within X to branch on. What is the benefit of Random Forests It has better overall estimates. while each tree might be over-fitting in one place or another they don't necessarily over-fit the same way. The average overall tree tends to fall those overreaction to random effects. What are the drawbacks of random forests? Harder to explain/interpret results. Can't give us a specific regression or classification model from the data. How is the prediction calculated in Random Forests when doing regression trees? use the average of the predicted response How is the prediction calculated in Random Forests when doing classification? use the mode -- the most common predicted response what are the similarities between Logistic Regression and Linear regression transformation of input data, consider interaction terms, variable selection, has trees differences Logistic Regression takes longer to calculate, has no closed-form solution, and difficult to understand model quality (no r-squared value) what is sensitivity the fraction of category members that are correctly classified TP / (TP + FN) what is specificity the fraction of non-category member that are correctly identified TN / (TN + FP) what does the roc curve plot sensitivity plotted against 1 - specificity what is the Area Under Curve probability that the model estimates a random "yes" point higher than a random "no" point what does it mean when the AUC = 0.5 we are just guessing What does ROC/AUC give you and what doesn't it gives a quick-and-dirty estimate of quality but does not differentiate between the coset of FN and FP what does TP mean? point in the category, correctly classified what does FP mean point not in category, model says it is what does TN mean? point not in category, correctly classified what does FN mean? point in the category model says no how do you do KNN regression? plot all the data. predict response by taking average response of k closest data points what are parametric methods? the form of the predictor (linear regression) what are non-parametric methods we don't force any specific form onto the predictor (knn) What is a spline? function of polynomials that connect to each other How does regression splines work? Fit different functions to different parts of the data set with smooth connections between the parts. What is the points where the different functions connect? they are called knots Why do connection have to be smooth? Otherwise you could have drastically different answers for very nearby points. How does Bayesian Regression work? Start with data and estimate of how regression coefficients and the random error is distributed. Then we use Bayes theorem to update estimate. When should you use Bayesian Regression? When there's not much data and want to combine expert opinion. If we have a classifier where one type of mistake is more costly where do we move the line? further away from that class In a classifier what range can a0 have it can be between -1 and 1 In knn, how can we remove unimportant attributes set the weight of that distance to 0` What is bias? Bias is an error from erroneous assumption in the learning algorithm High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting) what is variance? An error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended outputs (overfitting) what is a contextual outlier value isn't far from the rest overall but is far from the points nearby in time outlier data point that is very different from the rest collective outlier something is missing in a range of points but can't tell exactly where how could we detect outliers when there are multiple dimensions? we could fit a model and then determine the points with a large error What is an example of when we should keep outliers? When the magnitude of the model's error is part of the model's value What can removing outliers do? Paint an overly optimistic picture What are two ways to deal with outliers Have two models: Logistic Regression model to estimate the likelihood of outliers happening under certain conditions. Then a second model to predict with outliers and one without what type of data is used in change detection? time-series data What question does CUSUM answer? has the mean of the observed distribution gone above a critical level what method can we use to try to detect both increases and decreases a control chart What are some complexities with time series data? trends (increasing/decreasing) and cyclical patterns how do you calculate the observed trend? It is S_t - S_{t-1}. The difference between the two baselines what is triple exponential smoothing called holt or holt-winters method why is T_1 = 0 because it indicates no trend why is the first L value of C set to 1 that indicates so seasonality in multiplicative relationship where does the smoothing come from in exponential smoothing S_{t-1} * (1 - alpha) brings up low x_t and brings down high x_t where does the exponential part come from in exponential smoothing S_{t-1} include all the previous S_t's in exponential smoothing how are more recent observations weighted more than older ones? (1-alpha) will be less than one so as the exponent gets bigger so each term with a previous observation will be smaller breakdown the name of exponential smoothing jumps (peaks and valleys) in the data are smoothed out and there is exponential weighting of previous observations what are the key parts of ARIMA 1. often the data isn't stationary but the differences might be. 2. predicting the current value based on previous time periods' values 3. moving average why is exponential smoothing an infinite autoregression model because it uses all the data we have What does exponential smoothing autoregress on? What does ARIMA autoregress on? The observed observerations. The differences What do ARIMA models do? Help forecast or estimate a value. what is a common error measure for simple linear regression sum of squared error what makes up the best-fit regression line coefficients that minimize the sum of squared errors. what is likelihood the probability (probability density) of some observed outcomes given a set of parameter values maximum likelihood parameters that give the highest probability what is the maximum likelihood estimate the set of parameters that minimizes the sum of squared errors. what can extra parameters do? cause overfittign what does a smaller AIC encourage higher likelihood and less parameters How does BIC's penalty term compare to AIC's penalty term It's bigger. BIC encourages models with fewer parameters than AIC does if there is a strong relationship between a predictor and the response what will it's p-value be very low. how do you find the implied regression coefficients in PCR? you mul