+
+
+
+
+
+
+
+
+
Introduction to Regression - Lesson 1
+
+
Putting it into perspective
+
✅ There are many types of regression methods, and which one you pick
+depends on the answer you’re looking for. If you want to predict the
+probable height for a person of a given age, you’d use
+linear regression, as you’re seeking a numeric
+value. If you’re interested in discovering whether a type of
+cuisine should be considered vegan or not, you’re looking for a
+category assignment so you would use
+logistic regression. You’ll learn more about logistic
+regression later. Think a bit about some questions you can ask of data,
+and which of these methods would be more appropriate.
+
In this section, you will work with a small
+dataset about diabetes. Imagine that you wanted to test a treatment
+for diabetic patients. Machine Learning models might help you determine
+which patients would respond better to the treatment, based on
+combinations of variables. Even a very basic regression model, when
+visualized, might show information about variables that would help you
+organize your theoretical clinical trials.
+
That said, let’s get started on this task!
+
+
+
+
+
+
2. The diabetes dataset
+
In this exercise, we’ll put our regression skills into display by
+making predictions on a diabetes dataset. The diabetes
+dataset includes 442 samples of data around diabetes,
+with 10 predictor feature variables, age, sex,
+body mass index, average blood pressure, and
+six blood serum measurements as well as an outcome variable
+y: a quantitative measure of disease progression one year
+after baseline.
+
+
+
+
+
+
+
+
+
+
+| Number of predictors |
+First 10 columns are numeric predictive values |
+
+
+| Outcome/Target |
+Column 11 is a quantitative measure of disease progression one year
+after baseline |
+
+
+| Predictor Information |
+
+- age age in years
+- sex
+- bmi body mass index
+- bp average blood pressure
+- s1 tc, total serum cholesterol
+- s2 ldl, low-density lipoproteins
+- s3 hdl, high-density lipoproteins
+- s4 tch, total cholesterol / HDL
+- s5 ltg, possibly log of serum triglycerides level
+- s6 glu, blood sugar level
+ |
+
+
+
+
+🎓 Remember, this is supervised learning, and we need a named ‘y’
+target.
+
+
Before you can manipulate data with R, you need to import the data
+into R’s memory, or build a connection to the data that R can use to
+access the data remotely.
+
+
+The readr package, which
+is part of the Tidyverse, provides a fast and friendly way to read
+rectangular data into R.
+
+
Now, let’s load the diabetes dataset provided in this source URL: https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html
+
Also, we’ll perform a sanity check on our data using
+glimpse() and dsiplay the first 5 rows using
+slice().
+
Before going any further, let’s introduce something you will
+encounter quite often in R code: the pipe operator
+%>%
+
The pipe operator (%>%) performs operations in
+logical sequence by passing an object forward into a function or call
+expression. You can think of the pipe operator as saying “and then” in
+your code.
+
+
# Import the data set
+diabetes <- read_table2(file = "https://www4.stat.ncsu.edu/~boos/var.select/diabetes.rwrite1.txt")
+
+
+# Get a glimpse and dimensions of the data
+glimpse(diabetes)
+
## Rows: 442
+## Columns: 11
+## $ age <dbl> 0.038075906, -0.001882017, 0.085298906, -0.089062939, 0.005383060,~
+## $ sex <dbl> 0.05068012, -0.04464164, 0.05068012, -0.04464164, -0.04464164, -0.~
+## $ bmi <dbl> 0.061696207, -0.051474061, 0.044451213, -0.011595015, -0.036384692~
+## $ map <dbl> 0.021872355, -0.026327835, -0.005670611, -0.036656447, 0.021872355~
+## $ tc <dbl> -0.044223498, -0.008448724, -0.045599451, 0.012190569, 0.003934852~
+## $ ldl <dbl> -3.482076e-02, -1.916334e-02, -3.419447e-02, 2.499059e-02, 1.55961~
+## $ hdl <dbl> -0.043400846, 0.074411564, -0.032355932, -0.036037570, 0.008142084~
+## $ tch <dbl> -0.002592262, -0.039493383, -0.002592262, 0.034308859, -0.00259226~
+## $ ltg <dbl> 0.019908421, -0.068329744, 0.002863771, 0.022692023, -0.031991445,~
+## $ glu <dbl> -0.017646125, -0.092204050, -0.025930339, -0.009361911, -0.0466408~
+## $ y <dbl> 151, 75, 141, 206, 135, 97, 138, 63, 110, 310, 101, 69, 179, 185, ~
+
# Select the first 5 rows of the data
+diabetes %>%
+ slice(1:5)
+
+
+
+
glimpse() shows us that this data has 442 rows and 11
+columns with all the columns being of data type double
+
+glimpse() and slice() are functions in dplyr. Dplyr, part
+of the Tidyverse, is a grammar of data manipulation that provides a
+consistent set of verbs that help you solve the most common data
+manipulation challenges
+
+
Now that we have the data, let’s narrow down to one feature
+(bmi) to target for this exercise. This will require us to
+select the desired columns. So, how do we do this?
+
dplyr::select()
+allows us to select (and optionally rename) columns in a data
+frame.
+
# Select predictor feature `bmi` and outcome `y`
+diabetes_select <- diabetes %>%
+ select(c(bmi, y))
+
+# Print the first 5 rows
+diabetes_select %>%
+ slice(1:5)
+
+
+
+
+
+
3. Training and Testing data
+
It’s common practice in supervised learning to split the
+data into two subsets; a (typically larger) set with which to train the
+model, and a smaller “hold-back” set with which to see how the model
+performed.
+
Now that we have data ready, we can see if a machine can help
+determine a logical split between the numbers in this dataset. We can
+use the rsample
+package, which is part of the Tidymodels framework, to create an object
+that contains the information on how to split the data, and
+then two more rsample functions to extract the created training and
+testing sets:
+
set.seed(2056)
+# Split 67% of the data for training and the rest for testing
+diabetes_split <- diabetes_select %>%
+ initial_split(prop = 0.67)
+
+# Extract the resulting train and test sets
+diabetes_train <- training(diabetes_split)
+diabetes_test <- testing(diabetes_split)
+
+# Print the first 3 rows of the training set
+diabetes_train %>%
+ slice(1:3)
+
+
+
+
+
+
4. Train a linear regression model with Tidymodels
+
Now we are ready to train our model!
+
In Tidymodels, you specify models using parsnip() by
+specifying three concepts:
+
+Model type differentiates models such as linear
+regression, logistic regression, decision tree models, and so
+forth.
Model mode includes common options like
+regression and classification; some model types support either of these
+while some only have one mode.
Model engine is the computational tool which
+will be used to fit the model. Often these are R packages, such as
+"lm" or
+"ranger"
+
This modeling information is captured in a model specification, so
+let’s build one!
+
# Build a linear model specification
+lm_spec <-
+ # Type
+ linear_reg() %>%
+ # Engine
+ set_engine("lm") %>%
+ # Mode
+ set_mode("regression")
+
+
+# Print the model specification
+lm_spec
+
## Linear Regression Model Specification (regression)
+##
+## Computational engine: lm
+
After a model has been specified, the model can be
+estimated or trained using the fit()
+function, typically using a formula and some data.
+
y ~ . means we’ll fit y as the predicted
+quantity/target, explained by all the predictors/features ie,
+. (in this case, we only have one predictor:
+bmi )
+
# Build a linear model specification
+lm_spec <- linear_reg() %>%
+ set_engine("lm") %>%
+ set_mode("regression")
+
+
+# Train a linear regression model
+lm_mod <- lm_spec %>%
+ fit(y ~ ., data = diabetes_train)
+
+# Print the model
+lm_mod
+
## parsnip model object
+##
+##
+## Call:
+## stats::lm(formula = y ~ ., data = data)
+##
+## Coefficients:
+## (Intercept) bmi
+## 154.7 996.3
+
From the model output, we can see the coefficients learned during
+training. They represent the coefficients of the line of best fit that
+gives us the lowest overall error between the actual and predicted
+variable.
+
+
+
5. Make predictions on the test set
+
Now that we’ve trained a model, we can use it to predict the disease
+progression y for the test dataset using parsnip::predict().
+This will be used to draw the line between data groups.
+
# Make predictions for the test set
+predictions <- lm_mod %>%
+ predict(new_data = diabetes_test)
+
+# Print out some of the predictions
+predictions %>%
+ slice(1:5)
+
+
+
+
Woohoo! 💃🕺 We just trained a model and used it to make
+predictions!
+
When making predictions, the tidymodels convention is to always
+produce a tibble/data frame of results with standardized column names.
+This makes it easy to combine the original data and the predictions in a
+usable format for subsequent operations such as plotting.
+
dplyr::bind_cols() efficiently binds multiple data
+frames column.
+
# Combine the predictions and the original test set
+results <- diabetes_test %>%
+ bind_cols(predictions)
+
+
+results %>%
+ slice(1:5)
+
+
+
+
+
+
6. Plot modelling results
+
Now, its time to see this visually 📈. We’ll create a scatter plot of
+all the y and bmi values of the test set, then
+use the predictions to draw a line in the most appropriate place,
+between the model’s data groupings.
+
R has several systems for making graphs, but ggplot2 is
+one of the most elegant and most versatile. This allows you to compose
+graphs by combining independent components.
+
# Set a theme for the plot
+theme_set(theme_light())
+# Create a scatter plot
+results %>%
+ ggplot(aes(x = bmi)) +
+ # Add a scatter plot
+ geom_point(aes(y = y), size = 1.6) +
+ # Add a line plot
+ geom_line(aes(y = .pred), color = "blue", size = 1.5)
+
+
+✅ Think a bit about what’s going on here. A straight line is running
+through many small dots of data, but what is it doing exactly? Can you
+see how you should be able to use this line to predict where a new,
+unseen data point should fit in relationship to the plot’s y axis? Try
+to put into words the practical use of this model.
+
+
Congratulations, you built your first linear regression model,
+created a prediction with it, and displayed it in a plot!
+
+
+
LS0tDQp0aXRsZTogJ0J1aWxkIGEgcmVncmVzc2lvbiBtb2RlbDogR2V0IHN0YXJ0ZWQgd2l0aCBSIGFuZCBUaWR5bW9kZWxzIGZvciByZWdyZXNzaW9uIG1vZGVscycNCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDoNCiAgICBkZl9wcmludDogcGFnZWQNCiAgICB0aGVtZTogZmxhdGx5DQogICAgaGlnaGxpZ2h0OiBicmVlemVkYXJrDQogICAgdG9jOiB5ZXMNCiAgICB0b2NfZmxvYXQ6IHllcw0KICAgIGNvZGVfZG93bmxvYWQ6IHllcw0KLS0tDQoNCiMjIEludHJvZHVjdGlvbiB0byBSZWdyZXNzaW9uIC0gTGVzc29uIDENCg0KIyMjIyBQdXR0aW5nIGl0IGludG8gcGVyc3BlY3RpdmUNCg0K4pyFIFRoZXJlIGFyZSBtYW55IHR5cGVzIG9mIHJlZ3Jlc3Npb24gbWV0aG9kcywgYW5kIHdoaWNoIG9uZSB5b3UgcGljayBkZXBlbmRzIG9uIHRoZSBhbnN3ZXIgeW91J3JlIGxvb2tpbmcgZm9yLiBJZiB5b3Ugd2FudCB0byBwcmVkaWN0IHRoZSBwcm9iYWJsZSBoZWlnaHQgZm9yIGEgcGVyc29uIG9mIGEgZ2l2ZW4gYWdlLCB5b3UnZCB1c2UgYGxpbmVhciByZWdyZXNzaW9uYCwgYXMgeW91J3JlIHNlZWtpbmcgYSAqKm51bWVyaWMgdmFsdWUqKi4gSWYgeW91J3JlIGludGVyZXN0ZWQgaW4gZGlzY292ZXJpbmcgd2hldGhlciBhIHR5cGUgb2YgY3Vpc2luZSBzaG91bGQgYmUgY29uc2lkZXJlZCB2ZWdhbiBvciBub3QsIHlvdSdyZSBsb29raW5nIGZvciBhICoqY2F0ZWdvcnkgYXNzaWdubWVudCoqIHNvIHlvdSB3b3VsZCB1c2UgYGxvZ2lzdGljIHJlZ3Jlc3Npb25gLiBZb3UnbGwgbGVhcm4gbW9yZSBhYm91dCBsb2dpc3RpYyByZWdyZXNzaW9uIGxhdGVyLiBUaGluayBhIGJpdCBhYm91dCBzb21lIHF1ZXN0aW9ucyB5b3UgY2FuIGFzayBvZiBkYXRhLCBhbmQgd2hpY2ggb2YgdGhlc2UgbWV0aG9kcyB3b3VsZCBiZSBtb3JlIGFwcHJvcHJpYXRlLg0KDQpJbiB0aGlzIHNlY3Rpb24sIHlvdSB3aWxsIHdvcmsgd2l0aCBhIFtzbWFsbCBkYXRhc2V0IGFib3V0IGRpYWJldGVzXShodHRwczovL3d3dzQuc3RhdC5uY3N1LmVkdS9+Ym9vcy92YXIuc2VsZWN0L2RpYWJldGVzLmh0bWwpLiBJbWFnaW5lIHRoYXQgeW91IHdhbnRlZCB0byB0ZXN0IGEgdHJlYXRtZW50IGZvciBkaWFiZXRpYyBwYXRpZW50cy4gTWFjaGluZSBMZWFybmluZyBtb2RlbHMgbWlnaHQgaGVscCB5b3UgZGV0ZXJtaW5lIHdoaWNoIHBhdGllbnRzIHdvdWxkIHJlc3BvbmQgYmV0dGVyIHRvIHRoZSB0cmVhdG1lbnQsIGJhc2VkIG9uIGNvbWJpbmF0aW9ucyBvZiB2YXJpYWJsZXMuIEV2ZW4gYSB2ZXJ5IGJhc2ljIHJlZ3Jlc3Npb24gbW9kZWwsIHdoZW4gdmlzdWFsaXplZCwgbWlnaHQgc2hvdyBpbmZvcm1hdGlvbiBhYm91dCB2YXJpYWJsZXMgdGhhdCB3b3VsZCBoZWxwIHlvdSBvcmdhbml6ZSB5b3VyIHRoZW9yZXRpY2FsIGNsaW5pY2FsIHRyaWFscy4NCg0KVGhhdCBzYWlkLCBsZXQncyBnZXQgc3RhcnRlZCBvbiB0aGlzIHRhc2shDQoNCiFbQXJ0d29yayBieSBcQGFsbGlzb25faG9yc3RdKC4uLy4uL2ltYWdlcy9lbmNvdVJhZ2UuanBnKXt3aWR0aD0iNjMwIn0NCg0KIyMgMS4gTG9hZGluZyB1cCBvdXIgdG9vbCBzZXQNCg0KRm9yIHRoaXMgdGFzaywgd2UnbGwgcmVxdWlyZSB0aGUgZm9sbG93aW5nIHBhY2thZ2VzOg0KDQotICAgYHRpZHl2ZXJzZWA6IFRoZSBbdGlkeXZlcnNlXShodHRwczovL3d3dy50aWR5dmVyc2Uub3JnLykgaXMgYSBbY29sbGVjdGlvbiBvZiBSIHBhY2thZ2VzXShodHRwczovL3d3dy50aWR5dmVyc2Uub3JnL3BhY2thZ2VzKSBkZXNpZ25lZCB0byBtYWtlcyBkYXRhIHNjaWVuY2UgZmFzdGVyLCBlYXNpZXIgYW5kIG1vcmUgZnVuIQ0KDQotICAgYHRpZHltb2RlbHNgOiBUaGUgW3RpZHltb2RlbHNdKGh0dHBzOi8vd3d3LnRpZHltb2RlbHMub3JnLykgZnJhbWV3b3JrIGlzIGEgW2NvbGxlY3Rpb24gb2YgcGFja2FnZXNdKGh0dHBzOi8vd3d3LnRpZHltb2RlbHMub3JnL3BhY2thZ2VzLykgZm9yIG1vZGVsaW5nIGFuZCBtYWNoaW5lIGxlYXJuaW5nLg0KDQpZb3UgY2FuIGhhdmUgdGhlbSBpbnN0YWxsZWQgYXM6DQoNCmBpbnN0YWxsLnBhY2thZ2VzKGMoInRpZHl2ZXJzZSIsICJ0aWR5bW9kZWxzIikpYA0KDQpUaGUgc2NyaXB0IGJlbG93IGNoZWNrcyB3aGV0aGVyIHlvdSBoYXZlIHRoZSBwYWNrYWdlcyByZXF1aXJlZCB0byBjb21wbGV0ZSB0aGlzIG1vZHVsZSBhbmQgaW5zdGFsbHMgdGhlbSBmb3IgeW91IGluIGNhc2UgdGhleSBhcmUgbWlzc2luZy4NCg0KYGBge3IsIG1lc3NhZ2U9Riwgd2FybmluZz1GfQ0KaWYgKCFyZXF1aXJlKCJwYWNtYW4iKSkgaW5zdGFsbC5wYWNrYWdlcygicGFjbWFuIikNCnBhY21hbjo6cF9sb2FkKHRpZHl2ZXJzZSwgdGlkeW1vZGVscykNCmBgYA0KDQpOb3csIGxldCdzIGxvYWQgdGhlc2UgYXdlc29tZSBwYWNrYWdlcyBhbmQgbWFrZSB0aGVtIGF2YWlsYWJsZSBpbiBvdXIgY3VycmVudCBSIHNlc3Npb24uIChUaGlzIGlzIGZvciBtZXJlIGlsbHVzdHJhdGlvbiwgYHBhY21hbjo6cF9sb2FkKClgIGFscmVhZHkgZGlkIHRoYXQgZm9yIHlvdSkNCg0KYGBge3IgbG9hZF90aWR5X3ZlcnNlX21vZGVscywgbWVzc2FnZT1GLCB3YXJuaW5nPUZ9DQojIGxvYWQgdGhlIGNvcmUgVGlkeXZlcnNlIHBhY2thZ2VzDQpsaWJyYXJ5KHRpZHl2ZXJzZSkNCg0KIyBsb2FkIHRoZSBjb3JlIFRpZHltb2RlbHMgcGFja2FnZXMNCmxpYnJhcnkodGlkeW1vZGVscykNCg0KDQpgYGANCg0KIyMgMi4gVGhlIGRpYWJldGVzIGRhdGFzZXQNCg0KSW4gdGhpcyBleGVyY2lzZSwgd2UnbGwgcHV0IG91ciByZWdyZXNzaW9uIHNraWxscyBpbnRvIGRpc3BsYXkgYnkgbWFraW5nIHByZWRpY3Rpb25zIG9uIGEgZGlhYmV0ZXMgZGF0YXNldC4gVGhlIFtkaWFiZXRlcyBkYXRhc2V0XShodHRwczovL3d3dzQuc3RhdC5uY3N1LmVkdS9+Ym9vcy92YXIuc2VsZWN0L2RpYWJldGVzLnJ3cml0ZTEudHh0KSBpbmNsdWRlcyBgNDQyIHNhbXBsZXNgIG9mIGRhdGEgYXJvdW5kIGRpYWJldGVzLCB3aXRoIDEwIHByZWRpY3RvciBmZWF0dXJlIHZhcmlhYmxlcywgYGFnZWAsIGBzZXhgLCBgYm9keSBtYXNzIGluZGV4YCwgYGF2ZXJhZ2UgYmxvb2QgcHJlc3N1cmVgLCBhbmQgYHNpeCBibG9vZCBzZXJ1bSBtZWFzdXJlbWVudHNgIGFzIHdlbGwgYXMgYW4gb3V0Y29tZSB2YXJpYWJsZSBgeWA6IGEgcXVhbnRpdGF0aXZlIG1lYXN1cmUgb2YgZGlzZWFzZSBwcm9ncmVzc2lvbiBvbmUgeWVhciBhZnRlciBiYXNlbGluZS4NCg0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCAqKk51bWJlciBvZiBvYnNlcnZhdGlvbnMqKiB8ICoqNDQyKiogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKz09PT09PT09PT09PT09PT09PT09PT09PT09PT0rPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Kw0KfCAqKk51bWJlciBvZiBwcmVkaWN0b3JzKiogICB8IEZpcnN0IDEwIGNvbHVtbnMgYXJlIG51bWVyaWMgcHJlZGljdGl2ZSB2YWx1ZXMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCAqKk91dGNvbWUvVGFyZ2V0KiogICAgICAgICB8IENvbHVtbiAxMSBpcyBhIHF1YW50aXRhdGl2ZSBtZWFzdXJlIG9mIGRpc2Vhc2UgcHJvZ3Jlc3Npb24gb25lIHllYXIgYWZ0ZXIgYmFzZWxpbmUgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KfCAqKlByZWRpY3RvciBJbmZvcm1hdGlvbioqICB8IC0gICBhZ2UgYWdlIGluIHllYXJzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzZXggICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBibWkgYm9keSBtYXNzIGluZGV4ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBicCBhdmVyYWdlIGJsb29kIHByZXNzdXJlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzMSB0YywgdG90YWwgc2VydW0gY2hvbGVzdGVyb2wgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzMiBsZGwsIGxvdy1kZW5zaXR5IGxpcG9wcm90ZWlucyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzMyBoZGwsIGhpZ2gtZGVuc2l0eSBsaXBvcHJvdGVpbnMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzNCB0Y2gsIHRvdGFsIGNob2xlc3Rlcm9sIC8gSERMICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzNSBsdGcsIHBvc3NpYmx5IGxvZyBvZiBzZXJ1bSB0cmlnbHljZXJpZGVzIGxldmVsICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KfCAgICAgICAgICAgICAgICAgICAgICAgICAgICB8IC0gICBzNiBnbHUsIGJsb29kIHN1Z2FyIGxldmVsICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfA0KKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKw0KDQo+IPCfjpMgUmVtZW1iZXIsIHRoaXMgaXMgc3VwZXJ2aXNlZCBsZWFybmluZywgYW5kIHdlIG5lZWQgYSBuYW1lZCAneScgdGFyZ2V0Lg0KDQpCZWZvcmUgeW91IGNhbiBtYW5pcHVsYXRlIGRhdGEgd2l0aCBSLCB5b3UgbmVlZCB0byBpbXBvcnQgdGhlIGRhdGEgaW50byBSJ3MgbWVtb3J5LCBvciBidWlsZCBhIGNvbm5lY3Rpb24gdG8gdGhlIGRhdGEgdGhhdCBSIGNhbiB1c2UgdG8gYWNjZXNzIHRoZSBkYXRhIHJlbW90ZWx5LlwNCg0KPiBUaGUgW3JlYWRyXShodHRwczovL3JlYWRyLnRpZHl2ZXJzZS5vcmcvKSBwYWNrYWdlLCB3aGljaCBpcyBwYXJ0IG9mIHRoZSBUaWR5dmVyc2UsIHByb3ZpZGVzIGEgZmFzdCBhbmQgZnJpZW5kbHkgd2F5IHRvIHJlYWQgcmVjdGFuZ3VsYXIgZGF0YSBpbnRvIFIuDQoNCk5vdywgbGV0J3MgbG9hZCB0aGUgZGlhYmV0ZXMgZGF0YXNldCBwcm92aWRlZCBpbiB0aGlzIHNvdXJjZSBVUkw6IDxodHRwczovL3d3dzQuc3RhdC5uY3N1LmVkdS9+Ym9vcy92YXIuc2VsZWN0L2RpYWJldGVzLmh0bWw+DQoNCkFsc28sIHdlJ2xsIHBlcmZvcm0gYSBzYW5pdHkgY2hlY2sgb24gb3VyIGRhdGEgdXNpbmcgYGdsaW1wc2UoKWAgYW5kIGRzaXBsYXkgdGhlIGZpcnN0IDUgcm93cyB1c2luZyBgc2xpY2UoKWAuDQoNCkJlZm9yZSBnb2luZyBhbnkgZnVydGhlciwgbGV0J3MgaW50cm9kdWNlIHNvbWV0aGluZyB5b3Ugd2lsbCBlbmNvdW50ZXIgcXVpdGUgb2Z0ZW4gaW4gUiBjb2RlOiB0aGUgcGlwZSBvcGVyYXRvciBgJT4lYA0KDQpUaGUgcGlwZSBvcGVyYXRvciAoYCU+JWApIHBlcmZvcm1zIG9wZXJhdGlvbnMgaW4gbG9naWNhbCBzZXF1ZW5jZSBieSBwYXNzaW5nIGFuIG9iamVjdCBmb3J3YXJkIGludG8gYSBmdW5jdGlvbiBvciBjYWxsIGV4cHJlc3Npb24uIFlvdSBjYW4gdGhpbmsgb2YgdGhlIHBpcGUgb3BlcmF0b3IgYXMgc2F5aW5nICJhbmQgdGhlbiIgaW4geW91ciBjb2RlLlwNCg0KYGBge3IgbG9hZF9kYXRhc2V0LCBtZXNzYWdlPUYsIHdhcm5pbmc9Rn0NCiMgSW1wb3J0IHRoZSBkYXRhIHNldA0KZGlhYmV0ZXMgPC0gcmVhZF90YWJsZTIoZmlsZSA9ICJodHRwczovL3d3dzQuc3RhdC5uY3N1LmVkdS9+Ym9vcy92YXIuc2VsZWN0L2RpYWJldGVzLnJ3cml0ZTEudHh0IikNCg0KDQojIEdldCBhIGdsaW1wc2UgYW5kIGRpbWVuc2lvbnMgb2YgdGhlIGRhdGENCmdsaW1wc2UoZGlhYmV0ZXMpDQoNCg0KIyBTZWxlY3QgdGhlIGZpcnN0IDUgcm93cyBvZiB0aGUgZGF0YQ0KZGlhYmV0ZXMgJT4lIA0KICBzbGljZSgxOjUpDQoNCmBgYA0KDQpgZ2xpbXBzZSgpYCBzaG93cyB1cyB0aGF0IHRoaXMgZGF0YSBoYXMgNDQyIHJvd3MgYW5kIDExIGNvbHVtbnMgd2l0aCBhbGwgdGhlIGNvbHVtbnMgYmVpbmcgb2YgZGF0YSB0eXBlIGBkb3VibGVgDQoNCj4gZ2xpbXBzZSgpIGFuZCBzbGljZSgpIGFyZSBmdW5jdGlvbnMgaW4gW2BkcGx5cmBdKGh0dHBzOi8vZHBseXIudGlkeXZlcnNlLm9yZy8pLiBEcGx5ciwgcGFydCBvZiB0aGUgVGlkeXZlcnNlLCBpcyBhIGdyYW1tYXIgb2YgZGF0YSBtYW5pcHVsYXRpb24gdGhhdCBwcm92aWRlcyBhIGNvbnNpc3RlbnQgc2V0IG9mIHZlcmJzIHRoYXQgaGVscCB5b3Ugc29sdmUgdGhlIG1vc3QgY29tbW9uIGRhdGEgbWFuaXB1bGF0aW9uIGNoYWxsZW5nZXMNCg0KTm93IHRoYXQgd2UgaGF2ZSB0aGUgZGF0YSwgbGV0J3MgbmFycm93IGRvd24gdG8gb25lIGZlYXR1cmUgKGBibWlgKSB0byB0YXJnZXQgZm9yIHRoaXMgZXhlcmNpc2UuIFRoaXMgd2lsbCByZXF1aXJlIHVzIHRvIHNlbGVjdCB0aGUgZGVzaXJlZCBjb2x1bW5zLiBTbywgaG93IGRvIHdlIGRvIHRoaXM/DQoNCltgZHBseXI6OnNlbGVjdCgpYF0oaHR0cHM6Ly9kcGx5ci50aWR5dmVyc2Uub3JnL3JlZmVyZW5jZS9zZWxlY3QuaHRtbCkgYWxsb3dzIHVzIHRvICpzZWxlY3QqIChhbmQgb3B0aW9uYWxseSByZW5hbWUpIGNvbHVtbnMgaW4gYSBkYXRhIGZyYW1lLg0KDQpgYGB7ciBzZWxlY3QsIG1lc3NhZ2U9Riwgd2FybmluZz1GfQ0KIyBTZWxlY3QgcHJlZGljdG9yIGZlYXR1cmUgYGJtaWAgYW5kIG91dGNvbWUgYHlgDQpkaWFiZXRlc19zZWxlY3QgPC0gZGlhYmV0ZXMgJT4lIA0KICBzZWxlY3QoYyhibWksIHkpKQ0KDQojIFByaW50IHRoZSBmaXJzdCA1IHJvd3MNCmRpYWJldGVzX3NlbGVjdCAlPiUgDQogIHNsaWNlKDE6NSkNCmBgYA0KDQojIyAzLiBUcmFpbmluZyBhbmQgVGVzdGluZyBkYXRhDQoNCkl0J3MgY29tbW9uIHByYWN0aWNlIGluIHN1cGVydmlzZWQgbGVhcm5pbmcgdG8gKnNwbGl0KiB0aGUgZGF0YSBpbnRvIHR3byBzdWJzZXRzOyBhICh0eXBpY2FsbHkgbGFyZ2VyKSBzZXQgd2l0aCB3aGljaCB0byB0cmFpbiB0aGUgbW9kZWwsIGFuZCBhIHNtYWxsZXIgImhvbGQtYmFjayIgc2V0IHdpdGggd2hpY2ggdG8gc2VlIGhvdyB0aGUgbW9kZWwgcGVyZm9ybWVkLg0KDQpOb3cgdGhhdCB3ZSBoYXZlIGRhdGEgcmVhZHksIHdlIGNhbiBzZWUgaWYgYSBtYWNoaW5lIGNhbiBoZWxwIGRldGVybWluZSBhIGxvZ2ljYWwgc3BsaXQgYmV0d2VlbiB0aGUgbnVtYmVycyBpbiB0aGlzIGRhdGFzZXQuIFdlIGNhbiB1c2UgdGhlIFtyc2FtcGxlXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3JzYW1wbGUvKSBwYWNrYWdlLCB3aGljaCBpcyBwYXJ0IG9mIHRoZSBUaWR5bW9kZWxzIGZyYW1ld29yaywgdG8gY3JlYXRlIGFuIG9iamVjdCB0aGF0IGNvbnRhaW5zIHRoZSBpbmZvcm1hdGlvbiBvbiAqaG93KiB0byBzcGxpdCB0aGUgZGF0YSwgYW5kIHRoZW4gdHdvIG1vcmUgcnNhbXBsZSBmdW5jdGlvbnMgdG8gZXh0cmFjdCB0aGUgY3JlYXRlZCB0cmFpbmluZyBhbmQgdGVzdGluZyBzZXRzOg0KDQpgYGB7ciBzcGxpdCwgbWVzc2FnZT1GLCB3YXJuaW5nPUZ9DQpzZXQuc2VlZCgyMDU2KQ0KIyBTcGxpdCA2NyUgb2YgdGhlIGRhdGEgZm9yIHRyYWluaW5nIGFuZCB0aGUgcmVzdCBmb3IgdGVzdGluZw0KZGlhYmV0ZXNfc3BsaXQgPC0gZGlhYmV0ZXNfc2VsZWN0ICU+JSANCiAgaW5pdGlhbF9zcGxpdChwcm9wID0gMC42NykNCg0KIyBFeHRyYWN0IHRoZSByZXN1bHRpbmcgdHJhaW4gYW5kIHRlc3Qgc2V0cw0KZGlhYmV0ZXNfdHJhaW4gPC0gdHJhaW5pbmcoZGlhYmV0ZXNfc3BsaXQpDQpkaWFiZXRlc190ZXN0IDwtIHRlc3RpbmcoZGlhYmV0ZXNfc3BsaXQpDQoNCiMgUHJpbnQgdGhlIGZpcnN0IDMgcm93cyBvZiB0aGUgdHJhaW5pbmcgc2V0DQpkaWFiZXRlc190cmFpbiAlPiUgDQogIHNsaWNlKDE6MykNCg0KYGBgDQoNCiMjIDQuIFRyYWluIGEgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWwgd2l0aCBUaWR5bW9kZWxzDQoNCk5vdyB3ZSBhcmUgcmVhZHkgdG8gdHJhaW4gb3VyIG1vZGVsIQ0KDQpJbiBUaWR5bW9kZWxzLCB5b3Ugc3BlY2lmeSBtb2RlbHMgdXNpbmcgYHBhcnNuaXAoKWAgYnkgc3BlY2lmeWluZyB0aHJlZSBjb25jZXB0czoNCg0KLSAgIE1vZGVsICoqdHlwZSoqIGRpZmZlcmVudGlhdGVzIG1vZGVscyBzdWNoIGFzIGxpbmVhciByZWdyZXNzaW9uLCBsb2dpc3RpYyByZWdyZXNzaW9uLCBkZWNpc2lvbiB0cmVlIG1vZGVscywgYW5kIHNvIGZvcnRoLg0KDQotICAgTW9kZWwgKiptb2RlKiogaW5jbHVkZXMgY29tbW9uIG9wdGlvbnMgbGlrZSByZWdyZXNzaW9uIGFuZCBjbGFzc2lmaWNhdGlvbjsgc29tZSBtb2RlbCB0eXBlcyBzdXBwb3J0IGVpdGhlciBvZiB0aGVzZSB3aGlsZSBzb21lIG9ubHkgaGF2ZSBvbmUgbW9kZS4NCg0KLSAgIE1vZGVsICoqZW5naW5lKiogaXMgdGhlIGNvbXB1dGF0aW9uYWwgdG9vbCB3aGljaCB3aWxsIGJlIHVzZWQgdG8gZml0IHRoZSBtb2RlbC4gT2Z0ZW4gdGhlc2UgYXJlIFIgcGFja2FnZXMsIHN1Y2ggYXMgKipgImxtImAqKiBvciAqKmAicmFuZ2VyImAqKg0KDQpUaGlzIG1vZGVsaW5nIGluZm9ybWF0aW9uIGlzIGNhcHR1cmVkIGluIGEgbW9kZWwgc3BlY2lmaWNhdGlvbiwgc28gbGV0J3MgYnVpbGQgb25lIQ0KDQpgYGB7ciBsbV9tb2RlbF9zcGVjLCBtZXNzYWdlPUYsIHdhcm5pbmc9Rn0NCiMgQnVpbGQgYSBsaW5lYXIgbW9kZWwgc3BlY2lmaWNhdGlvbg0KbG1fc3BlYyA8LSANCiAgIyBUeXBlDQogIGxpbmVhcl9yZWcoKSAlPiUgDQogICMgRW5naW5lDQogIHNldF9lbmdpbmUoImxtIikgJT4lIA0KICAjIE1vZGUNCiAgc2V0X21vZGUoInJlZ3Jlc3Npb24iKQ0KDQoNCiMgUHJpbnQgdGhlIG1vZGVsIHNwZWNpZmljYXRpb24NCmxtX3NwZWMNCg0KYGBgDQoNCkFmdGVyIGEgbW9kZWwgaGFzIGJlZW4gKnNwZWNpZmllZCosIHRoZSBtb2RlbCBjYW4gYmUgYGVzdGltYXRlZGAgb3IgYHRyYWluZWRgIHVzaW5nIHRoZSBbYGZpdCgpYF0oaHR0cHM6Ly9wYXJzbmlwLnRpZHltb2RlbHMub3JnL3JlZmVyZW5jZS9maXQuaHRtbCkgZnVuY3Rpb24sIHR5cGljYWxseSB1c2luZyBhIGZvcm11bGEgYW5kIHNvbWUgZGF0YS4NCg0KYHkgfiAuYCBtZWFucyB3ZSdsbCBmaXQgYHlgIGFzIHRoZSBwcmVkaWN0ZWQgcXVhbnRpdHkvdGFyZ2V0LCBleHBsYWluZWQgYnkgYWxsIHRoZSBwcmVkaWN0b3JzL2ZlYXR1cmVzIGllLCBgLmAgKGluIHRoaXMgY2FzZSwgd2Ugb25seSBoYXZlIG9uZSBwcmVkaWN0b3I6IGBibWlgICkNCg0KYGBge3IgdHJhaW4sIG1lc3NhZ2U9Riwgd2FybmluZz1GfQ0KIyBCdWlsZCBhIGxpbmVhciBtb2RlbCBzcGVjaWZpY2F0aW9uDQpsbV9zcGVjIDwtIGxpbmVhcl9yZWcoKSAlPiUgDQogIHNldF9lbmdpbmUoImxtIikgJT4lDQogIHNldF9tb2RlKCJyZWdyZXNzaW9uIikNCg0KDQojIFRyYWluIGEgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWwNCmxtX21vZCA8LSBsbV9zcGVjICU+JSANCiAgZml0KHkgfiAuLCBkYXRhID0gZGlhYmV0ZXNfdHJhaW4pDQoNCiMgUHJpbnQgdGhlIG1vZGVsDQpsbV9tb2QNCmBgYA0KDQpGcm9tIHRoZSBtb2RlbCBvdXRwdXQsIHdlIGNhbiBzZWUgdGhlIGNvZWZmaWNpZW50cyBsZWFybmVkIGR1cmluZyB0cmFpbmluZy4gVGhleSByZXByZXNlbnQgdGhlIGNvZWZmaWNpZW50cyBvZiB0aGUgbGluZSBvZiBiZXN0IGZpdCB0aGF0IGdpdmVzIHVzIHRoZSBsb3dlc3Qgb3ZlcmFsbCBlcnJvciBiZXR3ZWVuIHRoZSBhY3R1YWwgYW5kIHByZWRpY3RlZCB2YXJpYWJsZS4NCg0KIyMgNS4gTWFrZSBwcmVkaWN0aW9ucyBvbiB0aGUgdGVzdCBzZXQNCg0KTm93IHRoYXQgd2UndmUgdHJhaW5lZCBhIG1vZGVsLCB3ZSBjYW4gdXNlIGl0IHRvIHByZWRpY3QgdGhlIGRpc2Vhc2UgcHJvZ3Jlc3Npb24geSBmb3IgdGhlIHRlc3QgZGF0YXNldCB1c2luZyBbcGFyc25pcDo6cHJlZGljdCgpXShodHRwczovL3BhcnNuaXAudGlkeW1vZGVscy5vcmcvcmVmZXJlbmNlL3ByZWRpY3QubW9kZWxfZml0Lmh0bWwpLiBUaGlzIHdpbGwgYmUgdXNlZCB0byBkcmF3IHRoZSBsaW5lIGJldHdlZW4gZGF0YSBncm91cHMuDQoNCmBgYHtyIHRlc3QsIG1lc3NhZ2U9Riwgd2FybmluZz1GfQ0KIyBNYWtlIHByZWRpY3Rpb25zIGZvciB0aGUgdGVzdCBzZXQNCnByZWRpY3Rpb25zIDwtIGxtX21vZCAlPiUgDQogIHByZWRpY3QobmV3X2RhdGEgPSBkaWFiZXRlc190ZXN0KQ0KDQojIFByaW50IG91dCBzb21lIG9mIHRoZSBwcmVkaWN0aW9ucw0KcHJlZGljdGlvbnMgJT4lIA0KICBzbGljZSgxOjUpDQpgYGANCg0KV29vaG9vISDwn5KD8J+VuiBXZSBqdXN0IHRyYWluZWQgYSBtb2RlbCBhbmQgdXNlZCBpdCB0byBtYWtlIHByZWRpY3Rpb25zIQ0KDQpXaGVuIG1ha2luZyBwcmVkaWN0aW9ucywgdGhlIHRpZHltb2RlbHMgY29udmVudGlvbiBpcyB0byBhbHdheXMgcHJvZHVjZSBhIHRpYmJsZS9kYXRhIGZyYW1lIG9mIHJlc3VsdHMgd2l0aCBzdGFuZGFyZGl6ZWQgY29sdW1uIG5hbWVzLiBUaGlzIG1ha2VzIGl0IGVhc3kgdG8gY29tYmluZSB0aGUgb3JpZ2luYWwgZGF0YSBhbmQgdGhlIHByZWRpY3Rpb25zIGluIGEgdXNhYmxlIGZvcm1hdCBmb3Igc3Vic2VxdWVudCBvcGVyYXRpb25zIHN1Y2ggYXMgcGxvdHRpbmcuDQoNCmBkcGx5cjo6YmluZF9jb2xzKClgIGVmZmljaWVudGx5IGJpbmRzIG11bHRpcGxlIGRhdGEgZnJhbWVzIGNvbHVtbi4NCg0KYGBge3IgdGVzdF9wcmVkLCBtZXNzYWdlPUYsIHdhcm5pbmc9Rn0NCiMgQ29tYmluZSB0aGUgcHJlZGljdGlvbnMgYW5kIHRoZSBvcmlnaW5hbCB0ZXN0IHNldA0KcmVzdWx0cyA8LSBkaWFiZXRlc190ZXN0ICU+JSANCiAgYmluZF9jb2xzKHByZWRpY3Rpb25zKQ0KDQoNCnJlc3VsdHMgJT4lIA0KICBzbGljZSgxOjUpDQpgYGANCg0KIyMgNi4gUGxvdCBtb2RlbGxpbmcgcmVzdWx0cw0KDQpOb3csIGl0cyB0aW1lIHRvIHNlZSB0aGlzIHZpc3VhbGx5IPCfk4guIFdlJ2xsIGNyZWF0ZSBhIHNjYXR0ZXIgcGxvdCBvZiBhbGwgdGhlIGB5YCBhbmQgYGJtaWAgdmFsdWVzIG9mIHRoZSB0ZXN0IHNldCwgdGhlbiB1c2UgdGhlIHByZWRpY3Rpb25zIHRvIGRyYXcgYSBsaW5lIGluIHRoZSBtb3N0IGFwcHJvcHJpYXRlIHBsYWNlLCBiZXR3ZWVuIHRoZSBtb2RlbCdzIGRhdGEgZ3JvdXBpbmdzLg0KDQpSIGhhcyBzZXZlcmFsIHN5c3RlbXMgZm9yIG1ha2luZyBncmFwaHMsIGJ1dCBgZ2dwbG90MmAgaXMgb25lIG9mIHRoZSBtb3N0IGVsZWdhbnQgYW5kIG1vc3QgdmVyc2F0aWxlLiBUaGlzIGFsbG93cyB5b3UgdG8gY29tcG9zZSBncmFwaHMgYnkgKipjb21iaW5pbmcgaW5kZXBlbmRlbnQgY29tcG9uZW50cyoqLg0KDQpgYGB7ciBwbG90X3ByZWQsIG1lc3NhZ2U9Riwgd2FybmluZz1GfQ0KIyBTZXQgYSB0aGVtZSBmb3IgdGhlIHBsb3QNCnRoZW1lX3NldCh0aGVtZV9saWdodCgpKQ0KIyBDcmVhdGUgYSBzY2F0dGVyIHBsb3QNCnJlc3VsdHMgJT4lIA0KICBnZ3Bsb3QoYWVzKHggPSBibWkpKSArDQogICMgQWRkIGEgc2NhdHRlciBwbG90DQogIGdlb21fcG9pbnQoYWVzKHkgPSB5KSwgc2l6ZSA9IDEuNikgKw0KICAjIEFkZCBhIGxpbmUgcGxvdA0KICBnZW9tX2xpbmUoYWVzKHkgPSAucHJlZCksIGNvbG9yID0gImJsdWUiLCBzaXplID0gMS41KQ0KICANCmBgYA0KDQo+IOKchSBUaGluayBhIGJpdCBhYm91dCB3aGF0J3MgZ29pbmcgb24gaGVyZS4gQSBzdHJhaWdodCBsaW5lIGlzIHJ1bm5pbmcgdGhyb3VnaCBtYW55IHNtYWxsIGRvdHMgb2YgZGF0YSwgYnV0IHdoYXQgaXMgaXQgZG9pbmcgZXhhY3RseT8gQ2FuIHlvdSBzZWUgaG93IHlvdSBzaG91bGQgYmUgYWJsZSB0byB1c2UgdGhpcyBsaW5lIHRvIHByZWRpY3Qgd2hlcmUgYSBuZXcsIHVuc2VlbiBkYXRhIHBvaW50IHNob3VsZCBmaXQgaW4gcmVsYXRpb25zaGlwIHRvIHRoZSBwbG90J3MgeSBheGlzPyBUcnkgdG8gcHV0IGludG8gd29yZHMgdGhlIHByYWN0aWNhbCB1c2Ugb2YgdGhpcyBtb2RlbC4NCg0KQ29uZ3JhdHVsYXRpb25zLCB5b3UgYnVpbHQgeW91ciBmaXJzdCBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCwgY3JlYXRlZCBhIHByZWRpY3Rpb24gd2l0aCBpdCwgYW5kIGRpc3BsYXllZCBpdCBpbiBhIHBsb3QhDQo=
+
+
+
+