Data-Science-For-Beginners/translations/pcm/5-Data-Science-In-Cloud/19-Azure

Data Science for Cloud: Di "Azure ML SDK" Way

Data Science for Cloud: Azure ML SDK - Sketchnote by @nitya

Table of contents:

• Data Science for Cloud: Di "Azure ML SDK" Way
  • Pre-Lecture Quiz
  • 1. Introduction
    • 1.1 Wetin be Azure ML SDK?
    • 1.2 Heart failure prediction project and dataset introduction
  • 2. How to train model wit Azure ML SDK
    • 2.1 Create Azure ML workspace
    • 2.2 Create compute instance
    • 2.3 Load di Dataset
    • 2.4 Create Notebooks
    • 2.5 Train model
      • 2.5.1 Setup Workspace, experiment, compute cluster and dataset
      • 2.5.2 AutoML Configuration and training
  • 3. Model deployment and endpoint consumption wit Azure ML SDK
    • 3.1 Save di best model
    • 3.2 Model Deployment
    • 3.3 Endpoint consumption
  • 🚀 Challenge
  • Post-lecture quiz
  • Review & Self Study
  • Assignment

Pre-Lecture Quiz

1. Introduction

1.1 Wetin be Azure ML SDK?

Data scientists and AI developers dey use Azure Machine Learning SDK to build and run machine learning workflows wit di Azure Machine Learning service. You fit use am for any Python environment, like Jupyter Notebooks, Visual Studio Code, or any Python IDE wey you like.

Di main tins wey di SDK dey do include:

• Explore, prepare and manage di lifecycle of your datasets wey you go use for machine learning experiments.
• Manage cloud resources for monitoring, logging, and organizing your machine learning experiments.
• Train models either for your local machine or use cloud resources, including GPU-accelerated model training.
• Use automated machine learning, wey go take configuration parameters and training data, then automatically test different algorithms and hyperparameter settings to find di best model for predictions.
• Deploy web services to turn your trained models into RESTful services wey any application fit use.

Learn more about di Azure Machine Learning SDK

For di previous lesson, we see how to train, deploy and use model in Low code/No code way. We use di Heart Failure dataset to create Heart failure prediction model. For dis lesson, we go do di same thing but dis time we go use Azure Machine Learning SDK.

1.2 Heart failure prediction project and dataset introduction

Check here for di Heart failure prediction project and dataset introduction.

2. How to train model wit Azure ML SDK

2.1 Create Azure ML workspace

To make am simple, we go work for jupyter notebook. Dis mean say you don already get Workspace and compute instance. If you don already get Workspace, you fit jump go section 2.3 Notebook creation.

If you never get, abeg follow di instructions for section 2.1 Create an Azure ML workspace for di previous lesson to create workspace.

2.2 Create compute instance

For di Azure ML workspace wey we create before, go di compute menu and you go see di different compute resources wey dey available.

Make we create compute instance to set up jupyter notebook.

1. Click di + New button.
2. Give name to your compute instance.
3. Choose your options: CPU or GPU, VM size and core number.
4. Click di Create button.

Congrats, you don create compute instance! We go use dis compute instance to create Notebook for di Creating Notebooks section.

2.3 Load di Dataset

If you never upload di dataset, check di previous lesson for section 2.3 Loading the Dataset.

2.4 Create Notebooks

NOTE: For di next step, you fit either create new notebook from scratch, or upload di notebook wey we create for your Azure ML Studio. To upload am, just click di "Notebook" menu and upload di notebook.

Notebook na very important part of di data science process. You fit use am do Exploratory Data Analysis (EDA), call compute cluster to train model, or call inference cluster to deploy endpoint.

To create Notebook, we need compute node wey go serve di jupyter notebook instance. Go back to di Azure ML workspace and click Compute instances. For di list of compute instances, you go see di compute instance wey we create before.

1. For di Applications section, click di Jupyter option.
2. Tick di "Yes, I understand" box and click di Continue button.
3. Dis go open new browser tab wit your jupyter notebook instance. Click di "New" button to create notebook.

Now we don get Notebook, we fit start to train di model wit Azure ML SDK.

2.5 Train model

First, if you get any doubt, check di Azure ML SDK documentation. E get all di information wey you need to understand di modules wey we go use for dis lesson.

2.5.1 Setup Workspace, experiment, compute cluster and dataset

You need to load di workspace from di configuration file wit dis code:

from azureml.core import Workspace
ws = Workspace.from_config()

Dis go return object of type Workspace wey represent di workspace. Then you go create experiment wit dis code:

from azureml.core import Experiment
experiment_name = 'aml-experiment'
experiment = Experiment(ws, experiment_name)

To get or create experiment from workspace, you go request di experiment wit di experiment name. Experiment name must be 3-36 characters, start wit letter or number, and fit only contain letters, numbers, underscores, and dashes. If di experiment no dey for di workspace, new experiment go dey created.

Now you need to create compute cluster for di training wit dis code. Note say dis step fit take few minutes.

from azureml.core.compute import AmlCompute

aml_name = "heart-f-cluster"
try:
    aml_compute = AmlCompute(ws, aml_name)
    print('Found existing AML compute context.')
except:
    print('Creating new AML compute context.')
    aml_config = AmlCompute.provisioning_configuration(vm_size = "Standard_D2_v2", min_nodes=1, max_nodes=3)
    aml_compute = AmlCompute.create(ws, name = aml_name, provisioning_configuration = aml_config)
    aml_compute.wait_for_completion(show_output = True)

cts = ws.compute_targets
compute_target = cts[aml_name]

You fit get di dataset from di workspace wit di dataset name like dis:

dataset = ws.datasets['heart-failure-records']
df = dataset.to_pandas_dataframe()
df.describe()

2.5.2 AutoML Configuration and training

To set di AutoML configuration, use di AutoMLConfig class.

As dem describe for di doc, e get plenty parameters wey you fit play wit. For dis project, we go use di following parameters:

• experiment_timeout_minutes: Di maximum time (for minutes) wey di experiment fit run before e go stop automatically and results go dey available.
• max_concurrent_iterations: Di maximum number of concurrent training iterations wey di experiment fit allow.
• primary_metric: Di main metric wey dem go use to check di experiment status.
• compute_target: Di Azure Machine Learning compute target to run di Automated Machine Learning experiment.
• task: Di type of task to run. E fit be 'classification', 'regression', or 'forecasting' depending on di type of automated ML problem wey you wan solve.
• training_data: Di training data wey di experiment go use. E suppose get both training features and label column (fit also get sample weights column).
• label_column_name: Di name of di label column.
• path: Di full path to di Azure Machine Learning project folder.
• enable_early_stopping: Whether to stop early if di score no dey improve for short term.
• featurization: Whether make di featurization step dey automatic or customized.
• debug_log: Di log file to write debug information.

from azureml.train.automl import AutoMLConfig

project_folder = './aml-project'

automl_settings = {
    "experiment_timeout_minutes": 20,
    "max_concurrent_iterations": 3,
    "primary_metric" : 'AUC_weighted'
}

automl_config = AutoMLConfig(compute_target=compute_target,
                             task = "classification",
                             training_data=dataset,
                             label_column_name="DEATH_EVENT",
                             path = project_folder,  
                             enable_early_stopping= True,
                             featurization= 'auto',
                             debug_log = "automl_errors.log",
                             **automl_settings
                            )

Now wey you don set your configuration, you fit train di model wit dis code. Dis step fit take up to one hour depending on your cluster size.

remote_run = experiment.submit(automl_config)

You fit run di RunDetails widget to show di different experiments.

from azureml.widgets import RunDetails
RunDetails(remote_run).show()

3. Model deployment and endpoint consumption wit Azure ML SDK

3.1 Save di best model

Di remote_run na object of type AutoMLRun. Dis object get method get_output() wey dey return di best run and di fitted model wey follow am.

best_run, fitted_model = remote_run.get_output()

You fit see di parameters wey dem use for di best model by just printing di fitted_model and see di properties of di best model by using di get_properties() method.

best_run.get_properties()

Now register di model wit di register_model method.

model_name = best_run.properties['model_name']
script_file_name = 'inference/score.py'
best_run.download_file('outputs/scoring_file_v_1_0_0.py', 'inference/score.py')
description = "aml heart failure project sdk"
model = best_run.register_model(model_name = model_name,
                                model_path = './outputs/',
                                description = description,
                                tags = None)

3.2 Model Deployment

Once you don save di best model, we fit deploy am wit di InferenceConfig class. InferenceConfig dey represent di configuration settings for custom environment wey dem go use for deployment. Di AciWebservice class dey represent machine learning model wey dem deploy as web service endpoint for Azure Container Instances. Di deployed service go be load-balanced, HTTP endpoint wit REST API. You fit send data to dis API and get di prediction wey di model return.

Di model dey deploy wit di deploy method.

from azureml.core.model import InferenceConfig, Model
from azureml.core.webservice import AciWebservice

inference_config = InferenceConfig(entry_script=script_file_name, environment=best_run.get_environment())

aciconfig = AciWebservice.deploy_configuration(cpu_cores = 1,
                                               memory_gb = 1,
                                               tags = {'type': "automl-heart-failure-prediction"},
                                               description = 'Sample service for AutoML Heart Failure Prediction')

aci_service_name = 'automl-hf-sdk'
aci_service = Model.deploy(ws, aci_service_name, [model], inference_config, aciconfig)
aci_service.wait_for_deployment(True)
print(aci_service.state)

Dis step fit take few minutes.

3.3 Endpoint consumption

You fit use your endpoint by creating sample input:

data = {
    "data":
    [
        {
            'age': "60",
            'anaemia': "false",
            'creatinine_phosphokinase': "500",
            'diabetes': "false",
            'ejection_fraction': "38",
            'high_blood_pressure': "false",
            'platelets': "260000",
            'serum_creatinine': "1.40",
            'serum_sodium': "137",
            'sex': "false",
            'smoking': "false",
            'time': "130",
        },
    ],
}

test_sample = str.encode(json.dumps(data))

And den you fit send dis input go ya model make e predict:

response = aci_service.run(input_data=test_sample)
response

Dis one go show '{"result": [false]}'. Dis mean say di patient input wey we send go di endpoint generate di prediction false, wey mean say dis person no likely get heart attack.

Congrats! You don use di model wey dem deploy and train for Azure ML wit di Azure ML SDK!

NOTE: When you don finish di project, no forget to delete all di resources.

🚀 Challenge

Plenty other things dey wey you fit do wit di SDK, but sadly, we no fit cover all for dis lesson. But beta news be say, if you sabi how to check di SDK documentation well, e go help you plenty. Go look di Azure ML SDK documentation and find di Pipeline class wey go allow you create pipelines. Pipeline na collection of steps wey fit run as workflow.

HINT: Go di SDK documentation and type keywords for di search bar like "Pipeline". You suppose see di azureml.pipeline.core.Pipeline class for di search results.

Post-lecture quiz

Review & Self Study

For dis lesson, you don learn how to train, deploy and use model to predict heart failure risk wit di Azure ML SDK for di cloud. Check dis documentation for more information about di Azure ML SDK. Try create ya own model wit di Azure ML SDK.

Assignment

Data Science project using Azure ML SDK

---

Disclaimer:
Dis document don use AI translation service Co-op Translator take translate am. Even though we dey try make e accurate, abeg sabi say automated translations fit get mistake or no correct well. Di original document for di native language na di main correct source. For important information, e good make una use professional human translation. We no go fit take blame for any misunderstanding or wrong interpretation wey fit happen because of dis translation.