Python?
Assignment 1 Miodrag Bolic Due: October 4, 2019 Total number of points is 10. Please solve 5 out of 6 problems. Bonus is additional 2.5 points. Instructions: Upload your answers in an ipython notebook to UOttawa Bright Space. Your individual submissions should use the following filenames: ELG_7172B_YOURNAME_HW2.ipynb Your code should be in code cells as part of your ipython notebook. Do not use a different format. *Do not just send your code. The homework solutions should be in a report style. Be sure to add comments to your code as well as markdown cells where you describe your approach and discuss your results. * Please submit your notebook in an executed status, so that we can see all the results you computed. However, we will still run your code and all cells should reproduce the output when executed. If you have multiple files (e.g. you've added code files or images) create a tarball for all files in a single file and name it: ELG_7172B_YOURNAME_HW2.tar.gz or ELG_7172B_YOURNAME_HW2.zip The following can help with the assignement: 1. M. D. Bloice, A. HolzingeA, Tutorial on Machine Learning and Data Science Tools with Python, pp. 435–462, Machine Learning for Health Informatics, Volume 9605, Lecture Notes in Computer Science, Springer, 2016. ISBN: 978-3-319-50477-3. https://www.researchgate.net/publication/311555646_A_Tutorial_on_Machine_Learning_and_Data_Science_Tools_with_Python Github link: https://github.com/mdbloice/MLDS 2. Old course Web site with some assignments and solutions: https://github.com/Health-Devices/Course-Uncertainty-Evaluation-in-Measurements-and-Machine-Learning 3. Please note that the textbook is on Brightspace. Python code for the book figures is at https://github.com/probml/pyprobml/tree/master/scripts PROBLEM 1 Variance of a sum a. Show that the variance of a sum is var [X + Y ] = var[X] + var [Y ] + 2cov [X, Y ] , where cov [X, Y ] is the covariance between X and Y. b. In Python, generate N=100 random variables X, Y from multivariate Gaussian. Assume fixed initial values of your choice for mean[X], mean[Y], var[X], var [Y ] and cov [X, Y]. Now, compute empirical variance and covariances (from your data) and show that above formula is correct. PROBLEM 2 Bayesian analysis of the exponential distribution A lifetime X of a machine is modeled by an exponential distribution with unknown parameter θ. The likelihood is p(x|θ) = θe−θx for x ≥ 0, θ > 0. a. Show that the MLE is , where b. Suppose we observe X1 = 5, X2 = 6, X3 = 4 (the lifetimes (in years) of 3 different iid machines). What is the MLE given this data? c. Assume that an expert believes θ should have a prior distribution that is also exponential What is the posterior? d. Is the exponential prior conjugate to the exponential likelihood? PROBLEM 3 The following labeled data is given: x={10.9, 12.4, 13.5, 14.6, 14.8, 15.6, 16.2, 17.5, 18.3, 18.6} and y={24.8, 30.0, 31.0, 29.3, 35.9, 36.9, 42.5, 37.9, 38.9, 40.5}. Write python code to perform linear and ridge regression. Show R-squared error and adjusted R-squared error and explain results. You can use the code linreg_poly_ridge.py and linreg_poly_vs_degree.py (Fig 8.1 from the textbook) https://github.com/probml/pyprobml/tree/master/scripts PROBLEM 4 Start with the code from Fig 6.11 and 6.12 from the text book. You can find the code here: https://github.com/probml/pyprobml/tree/master/scripts. For the parameters presented in Fig 6.12b compute standard error (see section 6.3.1.3) and predict probability of next data point to be equal one. PROBLEM 5 Consider samples x1,…, xn from a Gaussian random variable with known variance and unknown mean µ. We further assume a prior distribution (also Gaussian) over the mean, , with fixed mean m and fixed variance s2. Thus the only unknown is µ. 1. Calculate the MAP estimate . You can state the result without proof. Alternatively, with a lot more work, you can compute derivatives of the log posterior, set to zero and solve. 2. Show that as the number of samples n increase, the MAP estimate converges to the maximum likelihood estimate. 3. Suppose n is small and fixed. What does the MAP estimator converge to if we increase the prior variance s2? 4. Suppose n is small and fixed. What does the MAP estimator converge to if we decrease the prior variance s2? PROBLEM 6 Implement the following problem in Python by simulating multivariate normal and solve analytically. To generate multivariate Gaussian random numbers please use the code below from http://scipy-cookbook.readthedocs.io/items/CorrelatedRandomSamples.html Let ?=(?1,?2,?3)? have a N(µ, Σ) distribution where µ=(0,0,0)T and Find the joint distribution of the following pairs of random variables. Are they independently distributed? Provide explanations for your answers. (i) ?1 and ?2. (ii) ?2 and ?3X3. (iii) (?1, ?2) and ?3. (iv) (?1 + ?2)/2 and ?3. (v) ?2 and −2.5?1+?2−?3. Bonus Implement figure 6.23 from the textbook in Python (convert from Matlab to Python).