{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> All content here is under a Creative Commons Attribution [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) and all source code is released under a [BSD-2 clause license](https://en.wikipedia.org/wiki/BSD_licenses). \n", ">\n", ">Please reuse, remix, revise, and [reshare this content](https://github.com/kgdunn/python-basic-notebooks) in any way, keeping this notice." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Run this cell once, at the start, to load the notebook's style sheet.\n", "from IPython.core.display import HTML\n", "css_file = './images/style.css'\n", "HTML(open(css_file, \"r\").read())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "### Summarizing data visually and numerically (statistics)\n", "\n", "In the [prior module](https://yint.org/pybasic09) we covered:\n", "1. Box plots\n", "2. Bar plots (bar charts) \n", "3. Histograms\n", "\n", "while in this module we will cover:\n", "4. Data tables\n", "5. Time-series, or sequence plots\n", "6. Scatter plots\n", "7. Pointers to some other interesting plots\n", "\n", "In between, throughout the notes, we will also introduce statistical and data science concepts. This way you will learn how to interpret the plots and also communicate your results with the correct language." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Regression and scatter plots\n", "\n", "In fact, if there is one thing we can guarantee that you will see in your career being ***misused***, it will be this model. Misused by others when they interpret the values from the model, misused when they build this model, misused when making predictions.\n", "\n", "We already saw above that, counter-intuitively, we can calculate the $R^2$ value from a linear regression before even fitting the intercept and slope! That is because $R^2$ is just the square of the correlation coefficient, $r \\times r = R^2$. And since $r$ is just the correlation between two variables, the $R^2$ number just tells how strongly they are correlated.\n", "\n", "$R^2$ **never** indicates how ***well does the linear regression model fit my data***. The only way to see how well the model fits is to first fit it. And then calculate the residuals and see how small they are. \n", "\n", "Let's take a look, and along the way we introduce the Seaborn library's tools for regression plots." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Standard imports required to show plots\n", "%matplotlib inline\n", "import pandas as pd\n", "import seaborn as sns\n", "\n", "# Load the data\n", "grades = pd.read_csv('https://openmv.net/file/unlimited-time-test.csv')\n", "\n", "# Fit a regression model between Time and Grade, showing the model's prediction boundaries\n", "sns.regplot(x = \"Time\", y = \"Grade\", data = grades);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.lmplot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### TO DO\n", "\n", "* Regression plots for grades against time\n", "\n", "\n", "\n", "* MUST COVER: time-series of stability data from which a database was built on\n", "\n", "* Bubble plots from this notebook: https://nbviewer.jupyter.org/github/engineersCode/EngComp2_takeoff/blob/master/notebooks_en/2_Seeing_Stats.ipynb\n", "\n", "* regression: https://towardsdatascience.com/simple-and-multiple-linear-regression-in-python-c928425168f9\n", "\n", "* MUST COVER: qq-plot in Pandas\n", "\n", "* MUST COVER: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.probplot.html\n", "\n", "* Regression model:http://localhost:8888/notebooks/Notebooks/Thermocouple%20-%20linear%20regression.ipynb\n", "\n", "* Later: PCA loadings are orthogonal. Plot a scatter plot, and see the correlation is zero\n", "\n", "* PCA plots: https://jakevdp.github.io/PythonDataScienceHandbook/05.09-principal-component-analysis.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Some tips\n", "\n", "* One\n", "* Two" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "* Merging and filtering data\n", "* Learning about tuples??\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ ">***Feedback and comments about this worksheet?***\n", "> Please provide any anonymous [comments, feedback and tips](https://docs.google.com/forms/d/1Fpo0q7uGLcM6xcLRyp4qw1mZ0_igSUEnJV6ZGbpG4C4/viewform)." ] } ], "metadata": { "gist": { "data": { "description": "Module-10-interactive.ipynb", "public": true }, "id": "" }, "hide_input": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.1" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "221.984px" }, "toc_section_display": true, "toc_window_display": true }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }