127.0.0.1 - - [04/Jun/2021:02:38:26 +0000] "GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.0" 200 7 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 127.0.0.1 - - [04/Jun/2021:02:38:26 +0000] "GET /?a=fetch&content=%3Cphp%3Edie%28%40md5%28HelloThinkCMF%29%29%3C%2Fphp%3E HTTP/1.0" 200 7 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 2021-06-04 11:04:04,413 | root | INFO | 200 127.0.0.1 - - [04/Jun/2021:11:04:04 +0000] "GET /swagger.json HTTP/1.0" 200 12940 "-" "Go-http-client/1.1" 2021-06-04 11:04:05,847 | root | INFO | 200 127.0.0.1 - - [04/Jun/2021:11:04:05 +0000] "GET /swagger.json HTTP/1.0" 200 12940 "-" "Go-http-client/1.1" 2021-06-04 11:04:08,561 | root | INFO | 200 127.0.0.1 - - [04/Jun/2021:11:04:08 +0000] "GET /swagger.json HTTP/1.0" 200 12940 "-" "Go-http-client/1.1" 2021-06-04 11:04:44,217 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:04:44,220 studio.core INFO Handling http request - Start: 2021-06-04 11:04:44,220 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:04:44,220 studio.core INFO | Pre-processing - Start: 2021-06-04 11:04:44,220 studio.core INFO | Pre-processing - End with 0.0001s elapsed. 2021-06-04 11:04:44,221 studio.core INFO | Processing - Start: 2021-06-04 11:04:44,238 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,254 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,256 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,257 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,259 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,261 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,262 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,264 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,268 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,270 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,272 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,273 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,275 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,283 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,313 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,315 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:44,331 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:04:44,334 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:44,339 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:44,339 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:44,340 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:44,340 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:44,340 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:44,340 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:04:44,341 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | table = 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:44,341 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:44,342 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | table = 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:44,342 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:44,343 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:04:44,343 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:04:44,343 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:04:44,344 studio.core INFO | | | | DataTable.clone - End with 0.0009s elapsed. 2021-06-04 11:04:44,344 studio.module INFO | | | | Change columns element type 2021-06-04 11:04:44,346 studio.module INFO | | | | Change categorical columns 2021-06-04 11:04:44,346 studio.module INFO | | | | Change feature label columns 2021-06-04 11:04:44,346 studio.module INFO | | | | Change column names 2021-06-04 11:04:44,346 studio.core DEBUG | | | | return: 2021-06-04 11:04:44,347 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:44,347 studio.core INFO | | | MetadataEditorModule.run - End with 0.0062s elapsed. 2021-06-04 11:04:44,348 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0169s elapsed. 2021-06-04 11:04:44,348 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:04:44,349 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:44,354 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:44,354 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:44,354 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:04:44,355 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:44,355 studio.core DEBUG | | | | | table = 2021-06-04 11:04:44,355 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:44,355 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:44,356 studio.core DEBUG | | | | | table = 2021-06-04 11:04:44,358 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:44,358 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:04:44,360 studio.core DEBUG | | | | return: 2021-06-04 11:04:44,360 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:44,360 studio.core INFO | | | SelectColumnsModule.run - End with 0.0054s elapsed. 2021-06-04 11:04:44,360 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0127s elapsed. 2021-06-04 11:04:44,361 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:04:44,368 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:04:44,370 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:04:44,376 studio.core INFO | | | preprocess_transactions - End with 0.0059s elapsed. 2021-06-04 11:04:44,376 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:04:44,418 studio.core INFO | | | preprocess_features - End with 0.0419s elapsed. 2021-06-04 11:04:44,419 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:04:44,419 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:04:44,437 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:04:44,451 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:04:44,451 studio.core INFO | | | Update features for users - End with 0.0317s elapsed. 2021-06-04 11:04:44,451 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:04:44,452 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:04:44,452 studio.core INFO | | | Update features for items - End with 0.0001s elapsed. 2021-06-04 11:04:44,452 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:04:44,452 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:04:44,464 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:44,528 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:04:44,537 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:44,613 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:04:44,621 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:04:44,625 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:04:44,625 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:04:50,513 studio.core INFO | | | Generate predictions - End with 5.9003s elapsed. 2021-06-04 11:04:50,514 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.1532s elapsed. 2021-06-04 11:04:50,514 studio.core INFO | Processing - End with 6.2935s elapsed. 2021-06-04 11:04:50,514 studio.core INFO Handling http request - End with 6.2940s elapsed. 2021-06-04 11:04:50,514 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:04:50,522 | root | INFO | run() output is HTTP Response 2021-06-04 11:04:50,522 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:04:50 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:04:51,326 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:04:51,327 studio.core INFO Handling http request - Start: 2021-06-04 11:04:51,327 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:04:51,327 studio.core INFO | Pre-processing - Start: 2021-06-04 11:04:51,328 studio.core INFO | Pre-processing - End with 0.0002s elapsed. 2021-06-04 11:04:51,328 studio.core INFO | Processing - Start: 2021-06-04 11:04:51,332 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,350 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,356 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,358 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,360 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,362 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,363 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,365 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,366 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,368 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,369 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,371 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,373 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,381 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,413 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,415 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:51,421 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:04:51,425 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:51,430 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:51,430 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:51,431 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:51,431 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:51,431 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:51,431 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:04:51,431 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:51,431 studio.core DEBUG | | | | | table = 2021-06-04 11:04:51,431 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:51,431 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:51,432 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:51,432 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:51,432 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:51,432 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | table = 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:04:51,433 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:04:51,433 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:04:51,434 studio.core INFO | | | | DataTable.clone - End with 0.0008s elapsed. 2021-06-04 11:04:51,435 studio.module INFO | | | | Change columns element type 2021-06-04 11:04:51,436 studio.module INFO | | | | Change categorical columns 2021-06-04 11:04:51,437 studio.module INFO | | | | Change feature label columns 2021-06-04 11:04:51,437 studio.module INFO | | | | Change column names 2021-06-04 11:04:51,437 studio.core DEBUG | | | | return: 2021-06-04 11:04:51,437 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:51,437 studio.core INFO | | | MetadataEditorModule.run - End with 0.0059s elapsed. 2021-06-04 11:04:51,438 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0164s elapsed. 2021-06-04 11:04:51,438 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:04:51,439 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:51,444 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:51,444 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:51,445 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:04:51,445 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:51,445 studio.core DEBUG | | | | | table = 2021-06-04 11:04:51,445 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:51,445 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:51,446 studio.core DEBUG | | | | | table = 2021-06-04 11:04:51,446 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:51,446 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:04:51,447 studio.core DEBUG | | | | return: 2021-06-04 11:04:51,447 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:51,447 studio.core INFO | | | SelectColumnsModule.run - End with 0.0025s elapsed. 2021-06-04 11:04:51,448 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0095s elapsed. 2021-06-04 11:04:51,448 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:04:51,450 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:04:51,451 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:04:51,458 studio.core INFO | | | preprocess_transactions - End with 0.0068s elapsed. 2021-06-04 11:04:51,458 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:04:51,510 studio.core INFO | | | preprocess_features - End with 0.0521s elapsed. 2021-06-04 11:04:51,511 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:04:51,512 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:04:51,527 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:04:51,543 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:04:51,543 studio.core INFO | | | Update features for users - End with 0.0314s elapsed. 2021-06-04 11:04:51,543 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:04:51,543 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:04:51,544 studio.core INFO | | | Update features for items - End with 0.0002s elapsed. 2021-06-04 11:04:51,544 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:04:51,544 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:04:51,553 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:51,622 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:04:51,631 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:51,657 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:04:51,712 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:04:51,713 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:04:51,713 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:04:57,851 studio.core INFO | | | Generate predictions - End with 6.1418s elapsed. 2021-06-04 11:04:57,852 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.4040s elapsed. 2021-06-04 11:04:57,852 studio.core INFO | Processing - End with 6.5241s elapsed. 2021-06-04 11:04:57,852 studio.core INFO Handling http request - End with 6.5252s elapsed. 2021-06-04 11:04:57,852 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:04:57,858 | root | INFO | run() output is HTTP Response 2021-06-04 11:04:57,859 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:04:57 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:04:59,155 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:04:59,156 studio.core INFO Handling http request - Start: 2021-06-04 11:04:59,156 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:04:59,156 studio.core INFO | Pre-processing - Start: 2021-06-04 11:04:59,157 studio.core INFO | Pre-processing - End with 0.0003s elapsed. 2021-06-04 11:04:59,157 studio.core INFO | Processing - Start: 2021-06-04 11:04:59,161 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,188 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,190 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,191 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,193 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,195 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,197 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,198 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,200 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,202 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,204 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,205 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,207 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,218 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,224 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,227 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:04:59,233 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:04:59,237 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:59,243 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:59,243 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:59,243 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:59,243 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:59,244 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:04:59,244 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:04:59,244 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:59,244 studio.core DEBUG | | | | | table = 2021-06-04 11:04:59,244 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:59,245 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:59,245 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:59,245 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:59,245 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:59,245 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | table = 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | column_select = 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | new_field = 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:04:59,246 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:04:59,247 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:04:59,247 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:04:59,248 studio.core INFO | | | | DataTable.clone - End with 0.0009s elapsed. 2021-06-04 11:04:59,248 studio.module INFO | | | | Change columns element type 2021-06-04 11:04:59,250 studio.module INFO | | | | Change categorical columns 2021-06-04 11:04:59,250 studio.module INFO | | | | Change feature label columns 2021-06-04 11:04:59,250 studio.module INFO | | | | Change column names 2021-06-04 11:04:59,250 studio.core DEBUG | | | | return: 2021-06-04 11:04:59,250 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:59,251 studio.core INFO | | | MetadataEditorModule.run - End with 0.0067s elapsed. 2021-06-04 11:04:59,251 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0178s elapsed. 2021-06-04 11:04:59,251 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:04:59,253 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:04:59,258 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:04:59,258 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:04:59,259 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:04:59,259 studio.core DEBUG | | | | kwargs: 2021-06-04 11:04:59,259 studio.core DEBUG | | | | | table = 2021-06-04 11:04:59,259 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:59,259 studio.core DEBUG | | | | validated_args: 2021-06-04 11:04:59,260 studio.core DEBUG | | | | | table = 2021-06-04 11:04:59,260 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:04:59,260 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:04:59,261 studio.core DEBUG | | | | return: 2021-06-04 11:04:59,261 studio.core DEBUG | | | | | [0] = 2021-06-04 11:04:59,262 studio.core INFO | | | SelectColumnsModule.run - End with 0.0027s elapsed. 2021-06-04 11:04:59,262 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0104s elapsed. 2021-06-04 11:04:59,262 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:04:59,268 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:04:59,311 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:04:59,317 studio.core INFO | | | preprocess_transactions - End with 0.0051s elapsed. 2021-06-04 11:04:59,317 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:04:59,327 studio.core INFO | | | preprocess_features - End with 0.0103s elapsed. 2021-06-04 11:04:59,328 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:04:59,329 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:04:59,346 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:04:59,361 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:04:59,410 studio.core INFO | | | Update features for users - End with 0.0812s elapsed. 2021-06-04 11:04:59,410 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:04:59,410 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:04:59,410 studio.core INFO | | | Update features for items - End with 0.0001s elapsed. 2021-06-04 11:04:59,410 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:04:59,411 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:04:59,421 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:59,438 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:04:59,450 studio.common INFO | | | Process null values for features. 2021-06-04 11:04:59,524 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:04:59,526 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:04:59,527 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:04:59,528 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:05:05,355 studio.core INFO | | | Generate predictions - End with 5.8312s elapsed. 2021-06-04 11:05:05,356 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.0933s elapsed. 2021-06-04 11:05:05,356 studio.core INFO | Processing - End with 6.1987s elapsed. 2021-06-04 11:05:05,356 studio.core INFO Handling http request - End with 6.1999s elapsed. 2021-06-04 11:05:05,356 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:05:05,363 | root | INFO | run() output is HTTP Response 2021-06-04 11:05:05,410 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:05:05 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:05:07,721 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:05:07,721 studio.core INFO Handling http request - Start: 2021-06-04 11:05:07,721 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:05:07,722 studio.core INFO | Pre-processing - Start: 2021-06-04 11:05:07,722 studio.core INFO | Pre-processing - End with 0.0002s elapsed. 2021-06-04 11:05:07,722 studio.core INFO | Processing - Start: 2021-06-04 11:05:07,727 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,748 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,750 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,751 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,755 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,758 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,760 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,762 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,764 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,765 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,767 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,769 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,771 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,779 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,811 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,813 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:07,819 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:05:07,822 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:07,829 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:07,829 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:07,829 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:07,830 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:07,830 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:07,830 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:05:07,830 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:07,830 studio.core DEBUG | | | | | table = 2021-06-04 11:05:07,830 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:07,830 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:07,831 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:07,831 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:07,831 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:07,831 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:07,831 studio.core DEBUG | | | | | table = 2021-06-04 11:05:07,831 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:05:07,832 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:05:07,832 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:05:07,833 studio.core INFO | | | | DataTable.clone - End with 0.0009s elapsed. 2021-06-04 11:05:07,833 studio.module INFO | | | | Change columns element type 2021-06-04 11:05:07,835 studio.module INFO | | | | Change categorical columns 2021-06-04 11:05:07,835 studio.module INFO | | | | Change feature label columns 2021-06-04 11:05:07,835 studio.module INFO | | | | Change column names 2021-06-04 11:05:07,835 studio.core DEBUG | | | | return: 2021-06-04 11:05:07,836 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:07,836 studio.core INFO | | | MetadataEditorModule.run - End with 0.0058s elapsed. 2021-06-04 11:05:07,837 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0177s elapsed. 2021-06-04 11:05:07,837 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:05:07,838 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:07,843 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:07,843 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:07,843 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:05:07,844 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:07,844 studio.core DEBUG | | | | | table = 2021-06-04 11:05:07,844 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:07,844 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:07,844 studio.core DEBUG | | | | | table = 2021-06-04 11:05:07,844 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:07,844 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:05:07,846 studio.core DEBUG | | | | return: 2021-06-04 11:05:07,846 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:07,846 studio.core INFO | | | SelectColumnsModule.run - End with 0.0024s elapsed. 2021-06-04 11:05:07,846 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0093s elapsed. 2021-06-04 11:05:07,846 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:05:07,849 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:05:07,850 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:05:07,854 studio.core INFO | | | preprocess_transactions - End with 0.0044s elapsed. 2021-06-04 11:05:07,855 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:05:07,864 studio.core INFO | | | preprocess_features - End with 0.0087s elapsed. 2021-06-04 11:05:07,911 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:05:07,912 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:05:07,928 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:05:07,943 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:05:07,944 studio.core INFO | | | Update features for users - End with 0.0318s elapsed. 2021-06-04 11:05:07,944 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:05:07,944 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:05:07,944 studio.core INFO | | | Update features for items - End with 0.0001s elapsed. 2021-06-04 11:05:07,944 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:05:07,944 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:05:07,953 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:08,020 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:05:08,029 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:08,056 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:05:08,058 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:05:08,059 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:05:08,059 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:05:14,248 studio.core INFO | | | Generate predictions - End with 6.1919s elapsed. 2021-06-04 11:05:14,248 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.4018s elapsed. 2021-06-04 11:05:14,249 studio.core INFO | Processing - End with 6.5262s elapsed. 2021-06-04 11:05:14,249 studio.core INFO Handling http request - End with 6.5274s elapsed. 2021-06-04 11:05:14,249 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:05:14,256 | root | INFO | run() output is HTTP Response 2021-06-04 11:05:14,256 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:05:14 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:05:18,513 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:05:18,514 studio.core INFO Handling http request - Start: 2021-06-04 11:05:18,514 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:05:18,514 studio.core INFO | Pre-processing - Start: 2021-06-04 11:05:18,514 studio.core INFO | Pre-processing - End with 0.0002s elapsed. 2021-06-04 11:05:18,514 studio.core INFO | Processing - Start: 2021-06-04 11:05:18,518 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,536 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,538 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,540 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,542 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,543 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,546 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,548 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,549 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,551 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,553 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,555 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,557 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,565 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,614 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,615 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:18,622 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:05:18,626 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:18,632 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:18,632 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:18,633 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:18,633 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:18,633 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:18,633 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:05:18,633 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:18,634 studio.core DEBUG | | | | | table = 2021-06-04 11:05:18,634 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:18,634 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:18,634 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:18,634 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:18,635 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:18,635 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:18,635 studio.core DEBUG | | | | | table = 2021-06-04 11:05:18,635 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:18,635 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:18,636 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:18,636 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:18,636 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:18,636 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:05:18,636 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:05:18,636 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:05:18,638 studio.core INFO | | | | DataTable.clone - End with 0.0009s elapsed. 2021-06-04 11:05:18,638 studio.module INFO | | | | Change columns element type 2021-06-04 11:05:18,640 studio.module INFO | | | | Change categorical columns 2021-06-04 11:05:18,640 studio.module INFO | | | | Change feature label columns 2021-06-04 11:05:18,640 studio.module INFO | | | | Change column names 2021-06-04 11:05:18,640 studio.core DEBUG | | | | return: 2021-06-04 11:05:18,640 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:18,641 studio.core INFO | | | MetadataEditorModule.run - End with 0.0070s elapsed. 2021-06-04 11:05:18,641 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0189s elapsed. 2021-06-04 11:05:18,642 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:05:18,643 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:18,648 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:18,648 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:18,649 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:05:18,649 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:18,649 studio.core DEBUG | | | | | table = 2021-06-04 11:05:18,649 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:18,649 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:18,649 studio.core DEBUG | | | | | table = 2021-06-04 11:05:18,649 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:18,650 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:05:18,651 studio.core DEBUG | | | | return: 2021-06-04 11:05:18,651 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:18,652 studio.core INFO | | | SelectColumnsModule.run - End with 0.0029s elapsed. 2021-06-04 11:05:18,652 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0101s elapsed. 2021-06-04 11:05:18,652 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:05:18,654 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:05:18,656 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:05:18,661 studio.core INFO | | | preprocess_transactions - End with 0.0054s elapsed. 2021-06-04 11:05:18,661 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:05:18,715 studio.core INFO | | | preprocess_features - End with 0.0537s elapsed. 2021-06-04 11:05:18,717 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:05:18,717 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:05:18,733 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:05:18,748 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:05:18,748 studio.core INFO | | | Update features for users - End with 0.0316s elapsed. 2021-06-04 11:05:18,749 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:05:18,749 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:05:18,749 studio.core INFO | | | Update features for items - End with 0.0001s elapsed. 2021-06-04 11:05:18,749 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:05:18,749 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:05:18,758 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:18,825 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:05:18,834 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:18,857 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:05:18,911 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:05:18,913 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:05:18,913 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:05:24,727 studio.core INFO | | | Generate predictions - End with 5.8701s elapsed. 2021-06-04 11:05:24,728 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.0752s elapsed. 2021-06-04 11:05:24,728 studio.core INFO | Processing - End with 6.2132s elapsed. 2021-06-04 11:05:24,728 studio.core INFO Handling http request - End with 6.2141s elapsed. 2021-06-04 11:05:24,728 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:05:24,735 | root | INFO | run() output is HTTP Response 2021-06-04 11:05:24,735 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:05:24 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:05:33,846 | root | INFO | Scoring Timer is set to 60.0 seconds 2021-06-04 11:05:33,847 studio.core INFO Handling http request - Start: 2021-06-04 11:05:33,847 studio.azureml.designer.serving.dagengine.request_handler INFO | Run: is_classic = False, with_details = False, verbose = True 2021-06-04 11:05:33,847 studio.core INFO | Pre-processing - Start: 2021-06-04 11:05:33,847 studio.core INFO | Pre-processing - End with 0.0002s elapsed. 2021-06-04 11:05:33,847 studio.core INFO | Processing - Start: 2021-06-04 11:05:33,851 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,869 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,870 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,872 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,873 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,875 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,876 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,878 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,879 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,881 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,883 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,885 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,887 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,894 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,897 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,899 studio.common WARNING | | 0 and empty string will be converted into False 2021-06-04 11:05:33,916 studio.core INFO | | Executing node 1: Edit Metadata - Start: 2021-06-04 11:05:33,920 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:33,926 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:33,926 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:33,926 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:33,926 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:33,926 studio.modulehost INFO | | | Parse enum parameter 2021-06-04 11:05:33,927 studio.core INFO | | | MetadataEditorModule.run - Start: 2021-06-04 11:05:33,927 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:33,927 studio.core DEBUG | | | | | table = 2021-06-04 11:05:33,927 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:33,927 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:33,927 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:33,927 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:33,928 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | table = 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | column_select = 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | new_data_type = 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | new_categorical = 2021-06-04 11:05:33,928 studio.core DEBUG | | | | | new_field = 2021-06-04 11:05:33,929 studio.core DEBUG | | | | | new_column_names = None 2021-06-04 11:05:33,929 studio.core DEBUG | | | | | date_time_format = None 2021-06-04 11:05:33,929 studio.core DEBUG | | | | | time_span_format = None 2021-06-04 11:05:33,929 studio.core INFO | | | | DataTable.clone - Start: 2021-06-04 11:05:33,930 studio.core INFO | | | | DataTable.clone - End with 0.0009s elapsed. 2021-06-04 11:05:33,930 studio.module INFO | | | | Change columns element type 2021-06-04 11:05:33,932 studio.module INFO | | | | Change categorical columns 2021-06-04 11:05:33,932 studio.module INFO | | | | Change feature label columns 2021-06-04 11:05:33,932 studio.module INFO | | | | Change column names 2021-06-04 11:05:33,932 studio.core DEBUG | | | | return: 2021-06-04 11:05:33,932 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:33,932 studio.core INFO | | | MetadataEditorModule.run - End with 0.0055s elapsed. 2021-06-04 11:05:33,933 studio.core INFO | | Executing node 1: Edit Metadata - End with 0.0164s elapsed. 2021-06-04 11:05:33,933 studio.core INFO | | Executing node 2: Select Columns in Dataset - Start: 2021-06-04 11:05:33,934 studio.common DEBUG | | | Load schema successfully. 2021-06-04 11:05:33,939 studio.modulehost INFO | | | Return without parsing 2021-06-04 11:05:33,939 studio.modulehost INFO | | | Parse ColumnSelection parameter 2021-06-04 11:05:33,939 studio.core INFO | | | SelectColumnsModule.run - Start: 2021-06-04 11:05:33,939 studio.core DEBUG | | | | kwargs: 2021-06-04 11:05:33,940 studio.core DEBUG | | | | | table = 2021-06-04 11:05:33,940 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:33,940 studio.core DEBUG | | | | validated_args: 2021-06-04 11:05:33,940 studio.core DEBUG | | | | | table = 2021-06-04 11:05:33,940 studio.core DEBUG | | | | | feature_list = 2021-06-04 11:05:33,940 studio.module INFO | | | | Select column indexes from Dataset 2021-06-04 11:05:33,941 studio.core DEBUG | | | | return: 2021-06-04 11:05:33,942 studio.core DEBUG | | | | | [0] = 2021-06-04 11:05:33,942 studio.core INFO | | | SelectColumnsModule.run - End with 0.0023s elapsed. 2021-06-04 11:05:33,942 studio.core INFO | | Executing node 2: Select Columns in Dataset - End with 0.0089s elapsed. 2021-06-04 11:05:33,942 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - Start: 2021-06-04 11:05:33,944 studio.module INFO | | | Init all items recommendation scorer. 2021-06-04 11:05:33,947 studio.core INFO | | | preprocess_transactions - Start: 2021-06-04 11:05:33,952 studio.core INFO | | | preprocess_transactions - End with 0.0048s elapsed. 2021-06-04 11:05:33,952 studio.core INFO | | | preprocess_features - Start: 2021-06-04 11:05:33,961 studio.core INFO | | | preprocess_features - End with 0.0086s elapsed. 2021-06-04 11:05:33,962 studio.core INFO | | | Update features for users - Start: 2021-06-04 11:05:33,962 studio.common INFO | | | | Check features compatibility with existing feature metas 2021-06-04 11:05:34,026 studio.common INFO | | | | Found 31 features to update 2021-06-04 11:05:34,040 studio.common INFO | | | | Updated features have 109 rows, and fixed id vocab has 107 rows. 2021-06-04 11:05:34,041 studio.core INFO | | | Update features for users - End with 0.0781s elapsed. 2021-06-04 11:05:34,041 studio.core INFO | | | Update features for items - Start: 2021-06-04 11:05:34,041 studio.common INFO | | | | Update feature metas with None features 2021-06-04 11:05:34,041 studio.core INFO | | | Update features for items - End with 0.0001s elapsed. 2021-06-04 11:05:34,041 studio.common DEBUG | | | Init recommendation input function builder. 2021-06-04 11:05:34,041 studio.common INFO | | | Build 31 features for User ids. 2021-06-04 11:05:34,050 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:34,113 studio.common INFO | | | Build 41 features for Item ids. 2021-06-04 11:05:34,122 studio.common INFO | | | Process null values for features. 2021-06-04 11:05:34,149 studio.core INFO | | | Generate predictions - Start: 2021-06-04 11:05:34,151 studio.module INFO | | | | Model is expected to be fed with features: ['feature_user_feature_25', 'feature_item_feature_40', 'feature_user_feature_6', 'feature_item_feature_14', 'feature_user_feature_26', 'feature_user_feature_7', 'feature_item_feature_15', 'feature_user_feature_27', 'feature_user_feature_8', 'feature_item_feature_16', 'feature_user_feature_28', 'feature_user_feature_9', 'feature_item_feature_17', 'feature_user_feature_29', 'feature_user_feature_10', 'feature_user_feature_30', 'feature_item_feature_18', 'feature_user_feature_11', 'feature_item_feature_0', 'feature_item_feature_19', 'feature_user_feature_12', 'feature_item_feature_20', 'feature_item_feature_1', 'feature_item_feature_21', 'feature_user_feature_13', 'feature_item_feature_22', 'feature_item_feature_2', 'feature_item_feature_23', 'feature_user_feature_14', 'feature_item_feature_24', 'User', 'feature_item_feature_3', 'feature_item_feature_25', 'feature_item_feature_4', 'feature_user_feature_15', 'feature_item_feature_26', 'Item', 'feature_item_feature_5', 'feature_user_feature_16', 'feature_item_feature_27', 'feature_item_feature_38', 'feature_item_feature_6', 'feature_user_feature_17', 'feature_item_feature_28', 'feature_user_feature_0', 'feature_item_feature_7', 'feature_user_feature_18', 'feature_item_feature_29', 'feature_user_feature_1', 'feature_item_feature_8', 'feature_user_feature_19', 'feature_item_feature_30', 'feature_item_feature_31', 'feature_item_feature_9', 'feature_user_feature_20', 'feature_item_feature_32', 'feature_user_feature_2', 'feature_item_feature_10', 'feature_item_feature_33', 'feature_user_feature_3', 'feature_user_feature_21', 'feature_item_feature_34', 'feature_item_feature_11', 'feature_user_feature_22', 'feature_user_feature_4', 'feature_item_feature_35', 'feature_user_feature_23', 'feature_item_feature_36', 'feature_item_feature_12', 'feature_item_feature_37', 'feature_user_feature_5', 'feature_user_feature_24', 'feature_item_feature_13', 'feature_item_feature_39'] Using config: {'_model_dir': 'studiomodelpackage/Resources/0/model/checkpoints', '_tf_random_seed': 42, '_save_summary_steps': None, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-04 11:05:34,152 studio.module INFO | | | | Get 1 test instances, split into 1 batches. 2021-06-04 11:05:34,152 studio.module INFO | | | | Rebuild model: Epochs: 15 Batch size: 32 Wide optimizer: OptimizerSelection.Adagrad Wide learning rate: 0.1 Deep optimizer: OptimizerSelection.Adagrad Deep learning rate: 0.1 Hidden units: (256, 128) Activation function: ActivationFnSelection.ReLU Dropout: 0.8 Batch norm: True Crossed dimension: 1000 User embedding dimension: 16 Item embedding dimension: 16 Categorical feature embedding dimension: 4 Calling model_fn. 2021-06-04 11:05:40,232 studio.core INFO | | | Generate predictions - End with 6.0829s elapsed. 2021-06-04 11:05:40,232 studio.core INFO | | Executing node 3: Score Wide and Deep Recommender - End with 6.2899s elapsed. 2021-06-04 11:05:40,232 studio.core INFO | Processing - End with 6.3848s elapsed. 2021-06-04 11:05:40,233 studio.core INFO Handling http request - End with 6.3857s elapsed. 2021-06-04 11:05:40,233 studio.azureml.designer.serving.dagengine.request_handler ERROR Run: Server internal error is from Module Score Wide and Deep Recommender : Error occurs when executing node 3 with module Score Wide and Deep Recommender. Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 139, in _execute node_outputs = node.execute(node_global_params) > node_outputs = {'2:Results_dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > node_global_params = {} > node = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag_node.py", line 88, in execute global_parameters=global_params > global_params = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/module_host.py", line 112, in execute return self.module_instance.run(**StandardCustomModuleHost._deepcopy_args(normalized_kwargs)) > self = > normalized_kwargs = {'trained_wide_and_deep_recommendation_model': ModelDirectory(meta={'type': 'ModelDirectory', 'extension': {}, 'model': 'model_spec.yaml', 'registerModel': True, 'modelOutputPath': 'trained_model_outputs'}), 'dataset_to_score': DataFrameDirectory(met... (omitted 1572 chars) ...terpreter_path': 'python', 'user_managed_dependencies': 'false', 'conda_dependencies': '{"name": "project_environment", "channels": ["defaults"], "dependencies": ["pip=20.2", "python=3.6.8", {"pip": ["azureml-designer-recommender-modules==0.0.31"]}]} | ', 'docker_enabled': 'true', 'base_docker_image': 'mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04:20210301.v1', 'base_dockerfile': None, 'target': 'BKRUstyianovych'} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/common/entry_utils.py", line 44, in wrapper return func(obj, **kwargs) > func = > obj = > kwargs = {'trained_wide_and_deep_recommendation_model': , 'dataset_to_score': , 'recommended_item_selection': , 'maximum_number_of_items_to_recommend_to_a_user': 5, 'whether_to_return_the_predicted_ratings_of_the_items_along_with_the_labels': True} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/score_wide_and_deep_recommender.py", line 151, in run scored_data_df = scorer.score() > scorer = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/score/wide_and_deep_scorers.py", line 72, in score for predictions in self.learner.predict(input_function_builder=self.input_function_builder): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/modules/recommendation/dnn/wide_and_deep/common/wide_n_deep_model.py", line 226, in predict for predictions in self.estimator.predict(input_fn=input_fn, yield_single_examples=False): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 622, in predict features, None, ModeKeys.PREDICT, self.config) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1148, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) > self = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > kwargs = {'labels': None, 'mode': 'infer', 'config': } File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 1089, in _model_fn linear_sparse_combiner=linear_sparse_combiner) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn_linear_combined.py", line 182, in _dnn_linear_combined_model_fn_v2 mode=mode)) File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/dnn.py", line 509, in _dnn_model_fn_builder_v2 logits = dnn_model(features, mode) > dnn_model = > features = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > mode = 'infer' File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__ outputs = call_fn(cast_inputs, *args, **kwargs) > call_fn = > cast_inputs = {'User': , 'feature_user_feature_0': , 'feature_user_feature_1': , 'feature... (omitted 5936 chars) ...ature_38': , 'feature_item_feature_39': , 'feature_item_feature_40': } > args = ('infer',) > kwargs = {} File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in converted code: relative to /azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages: tensorflow_estimator/python/estimator/canned/dnn.py:356 call * net = self._input_layer(features) tensorflow_core/python/keras/engine/base_layer.py:847 __call__ outputs = call_fn(cast_inputs, *args, **kwargs) tensorflow_core/python/feature_column/dense_features.py:133 call self._state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3176 get_dense_tensor transformation_cache, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3774 get_sparse_tensors transformation_cache.get(self, state_manager), None) tensorflow_core/python/feature_column/feature_column_v2.py:2608 get transformed = column.transform_feature(self, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3752 transform_feature return self._transform_input_tensor(input_tensor, state_manager) tensorflow_core/python/feature_column/feature_column_v2.py:3729 _transform_input_tensor prefix='column_name: {} input_tensor'.format(self.key)) tensorflow_core/python/feature_column/utils.py:58 assert_string_or_int '{} dtype must be string or integer. dtype: {}.'.format(prefix, dtype)) ValueError: column_name: feature_user_feature_0 input_tensor dtype must be string or integer. dtype: . The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/request_handler.py", line 70, in handle_request response = processor.run(raw_data) > raw_data = b'{"Inputs":{"WebServiceInput0":[{"entrantID":"110","ScholarshipWillingnessBool":true,"Budget":15000,"Sex":"Чоловіча","EntrantAge":16,"Region":"Львівська область","EduType":"Гімназія","FavoriteSubject":"Історія"... (omitted 530 chars) ..."DesiredDirection":"Природничі науки","EnglishSkills":"A1 (Elementary)","WillingnessToStudyInUkraine":true,"SecondLanguage":"Відсутня","SecondLanguageBool":"false","InternetForStudyBool":true,"AverageCertificateScore":11.7}]}}' > processor = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/processor.py", line 24, in run webservice_output, name2schema = self.dag.execute(webservice_input, global_parameters) > webservice_input = {'WebServiceInput0': defaultdict(, {'entrantID': ['110'], 'ScholarshipWillingnessBool': [True], 'Budget': [15000], 'Sex': ['Чоловіча'], 'EntrantAge': [16], 'Region': ['Львівська область'], 'EduType': ['Гімна... (omitted 671 chars) ...€Ð¸Ñ€Ð¾Ð´Ð½Ð¸Ñ‡Ñ– науки'], 'EnglishSkills': ['A1 (Elementary)'], 'WillingnessToStudyInUkraine': [True], 'SecondLanguage': ['Відсутня'], 'SecondLanguageBool': ['false'], 'InternetForStudyBool': [True], 'AverageCertificateScore': [11.7]})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 171, in execute dynamic_outputs = self._execute(graph_inputs, global_parameters) > graph_inputs = {'1:Dataset': DataFrameDirectory(meta={'type': 'DataFrameDirectory', 'visualization': [{'type': 'Visualization', 'path': '_data.visualization'}], 'extension': {}, 'format': 'Parquet', 'data': '_data.parquet'})} > global_parameters = {} > self = File "/azureml-envs/azureml_01485dcda8a2cb90a21934821bfb7bfc/lib/python3.6/site-packages/azureml/designer/serving/dagengine/dag.py", line 142, in _execute raise DagNodeExecutionError(node_index, node.module_name) from e > node_index = '3' > node = DagNodeExecutionError: Error occurs when executing node 3 with module Score Wide and Deep Recommender. 2021-06-04 11:05:40,240 | root | INFO | run() output is HTTP Response 2021-06-04 11:05:40,240 | root | INFO | 500 127.0.0.1 - - [04/Jun/2021:11:05:40 +0000] "POST /score?verbose=true HTTP/1.0" 500 6292 "-" "Go-http-client/1.1" 2021-06-04 11:06:17,214 | root | INFO | 200