ML Studio Data Labeling On-Demand ML Assisted Labeling failing with ImportError

Question

ML Studio Data Labeling On-Demand ML Assisted Labeling failing with ImportError

Steve Mandras 81

We are trying to kick off an on-demand ML assissted labeling against a ML Studio Data Labeling project. Getting the following error (using AzureML-ai-ml-automl-dnn-text-gpu:27, auto-selected by ML Studio). I see there is a version 36 that may fix this issue. How can we use the latest environment for our ML assisted labeling needs? I believe this is a package version mismatch that may be fixed in newer versions.

Error (std_log.txt):

WARNING:root:user_name does not exist.
Cleaning up all outstanding Run operations, waiting 300.0 seconds
1 items cleaning up...
Cleanup took 0.06811404228210449 seconds
Traceback (most recent call last):
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1968, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/transformers/training_args.py", line 76, in <module>
    from accelerate.state import AcceleratorState, PartialState
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/__init__.py", line 16, in <module>
    from .accelerator import Accelerator
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/accelerator.py", line 36, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/utils/__init__.py", line 193, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/utils/bnb.py", line 29, in <module>
    from ..big_modeling import dispatch_model, init_empty_weights
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/big_modeling.py", line 24, in <module>
    from .hooks import (
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/hooks.py", line 31, in <module>
    from .utils.other import recursive_getattr
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/utils/other.py", line 29, in <module>
    from ..commands.config.default import write_basic_config  # noqa: F401
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/commands/config/__init__.py", line 19, in <module>
    from .config import config_command_parser
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/commands/config/config.py", line 25, in <module>
    from .sagemaker import get_sagemaker_input
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/accelerate/commands/config/sagemaker.py", line 35, in <module>
    import boto3  # noqa: F401
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/session.py", line 29, in <module>
    import botocore.credentials
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/credentials.py", line 34, in <module>
    from botocore.config import Config
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/awsrequest.py", line 24, in <module>
    import botocore.utils
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/utils.py", line 32, in <module>
    import botocore.httpsession
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/botocore/httpsession.py", line 10, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/urllib3/util/ssl_.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "hd_text_classification_dnn_driver.py", line 1, in <module>
    from azureml.automl.dnn.nlp.classification.multiclass import runner
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/azureml/automl/dnn/nlp/__init__.py", line 12, in <module>
    from azureml.automl.dnn.nlp.common._utils import _load_pyfunc  # noqa: F401
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/azureml/automl/dnn/nlp/common/_utils.py", line 37, in <module>
    from transformers import TrainingArguments
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1956, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1970, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.training_args because of the following error (look up to see its traceback):
cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/azureml-envs/azureml-automl-dnn-text-gpu/lib/python3.9/site-packages/urllib3/util/ssl_.py)

Sridhar M 2,995 Reputation points Microsoft External Staff Moderator

2025-12-02T01:23:56.1233333+00:00

Hi Steve Mandras

Thank you for bringing this to our attention. We are currently checking the details and working on resolving the issue. We’ll update you as soon as we have more information.
Steve Mandras 81 Reputation points

2025-12-02T14:35:01.8466667+00:00

So you are essentially saying that a Data Labeling Manager cannot run an on-demand ML-assisted labeling job in the Data Labeling project? Not sure where we'd even run those commands- on the clusters?
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-04T11:56:18.8433333+00:00

Hi Steve Mandras,

You can run an on-demand ML-assisted labeling job from a Data Labeling project — that part is fully supported. The issue you're hitting is not a limitation of Data Labeling Manager itself. The actual problem is that the built-in environment that Data Labeling uses today is fixed to an older image (v27), and that image contains outdated dependency versions that trigger the DEFAULT_CIPHERS ImportError.

At this moment:

The Data Labeling service does not allow selecting or overriding the environment image.

The job will always use the pinned azureml-automl-dnn-text-gpu:27 environment, even if newer images (v36) exist in the workspace.

So, the limitation is with the environment version that Data Labeling is wired to, not with your ability to trigger ML-assisted labeling.

Because the environment cannot be changed manually (“running commands on clusters” will not help), the only viable options are:

Use the built-in ML-assisted labeling only for tasks that work with the pinned environment. Text-based tasks that rely on HuggingFace/transformers will fail until the environment is updated by the platform.

If ML-assisted labeling is required immediately, run the model training/inference pipeline outside the Data Labeling project using your own environment (v36), then upload the predictions back into the labeling project. This is the only way today to use the newer environment.

What is actually happening

The error is a known dependency mismatch issue inside environment v27.

The Data Labeling product team needs to update the pinned image internally.

Until they release that updated image, the built-in ML-assisted flow cannot use newer packages.

Thank you!
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-05T06:21:37.3366667+00:00

Hi Steve Mandras,

Did you get any chance to review the above response. Do let me know if you have any further queries.

Thank you!
Steve Mandras 81 Reputation points

2025-12-05T16:12:22.29+00:00

Before we go down this path, how long will it take Azure to start using the newer environments that fix this problem? Are they aware that it is a probelm with their back-end and needs to be updated? Can we get someone from that team to chime in?
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-08T11:30:25.98+00:00

Hi Steve Mandras,

I will coordinate with the backend team and update you shortly. Thank you for your patience.
Steve Mandras 81 Reputation points

2025-12-08T16:40:15.05+00:00

Thanks @SRILAKSHMI C
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-10T12:54:59.44+00:00

Hi Steve Mandras,

There is a known issue with the image you’re using, and our engineering team is already working on a fix. The current expected timeline for resolution is by next week.

The problem is related to recent upgrades in the urllib3 library, which introduced this behavior.

We’ll keep you updated as soon as the fix is deployed.

Thank you!
Steve Mandras 81 Reputation points

2025-12-11T20:08:32.5+00:00

Looks like we had a successful ML Assisted model training and registration in our data labeling test project. They may have fixed things on the back-end in our region faster (kudos). Thanks for your assistance @SRILAKSHMI C
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-12T06:17:33.3533333+00:00

Hi Steve Mandras,

Great to hear that the ML-assisted training and model registration are now working successfully! It’s possible the backend fix for your region rolled out sooner than expected glad everything is running smoothly again.

I’ve also converted my earlier comment into an answer, could you please take a moment to mark it as Accepted? This helps others in the community with the same question find the solution more easily.

Thank you!

Answer accepted by question author

0 additional answers

Your answer

Sridhar M 2,995 Reputation points Microsoft External Staff Moderator

2025-12-02T01:23:56.1233333+00:00

Hi Steve Mandras

Thank you for bringing this to our attention. We are currently checking the details and working on resolving the issue. We’ll update you as soon as we have more information.
Steve Mandras 81 Reputation points

2025-12-02T14:35:01.8466667+00:00

So you are essentially saying that a Data Labeling Manager cannot run an on-demand ML-assisted labeling job in the Data Labeling project? Not sure where we'd even run those commands- on the clusters?
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-04T11:56:18.8433333+00:00

Hi Steve Mandras,

You can run an on-demand ML-assisted labeling job from a Data Labeling project — that part is fully supported. The issue you're hitting is not a limitation of Data Labeling Manager itself. The actual problem is that the built-in environment that Data Labeling uses today is fixed to an older image (v27), and that image contains outdated dependency versions that trigger the DEFAULT_CIPHERS ImportError.

At this moment:

The Data Labeling service does not allow selecting or overriding the environment image.

The job will always use the pinned azureml-automl-dnn-text-gpu:27 environment, even if newer images (v36) exist in the workspace.

So, the limitation is with the environment version that Data Labeling is wired to, not with your ability to trigger ML-assisted labeling.

Because the environment cannot be changed manually (“running commands on clusters” will not help), the only viable options are:

Use the built-in ML-assisted labeling only for tasks that work with the pinned environment. Text-based tasks that rely on HuggingFace/transformers will fail until the environment is updated by the platform.

If ML-assisted labeling is required immediately, run the model training/inference pipeline outside the Data Labeling project using your own environment (v36), then upload the predictions back into the labeling project. This is the only way today to use the newer environment.

What is actually happening

The error is a known dependency mismatch issue inside environment v27.

The Data Labeling product team needs to update the pinned image internally.

Until they release that updated image, the built-in ML-assisted flow cannot use newer packages.

Thank you!
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-05T06:21:37.3366667+00:00

Hi Steve Mandras,

Did you get any chance to review the above response. Do let me know if you have any further queries.

Thank you!
Steve Mandras 81 Reputation points

2025-12-05T16:12:22.29+00:00

Before we go down this path, how long will it take Azure to start using the newer environments that fix this problem? Are they aware that it is a probelm with their back-end and needs to be updated? Can we get someone from that team to chime in?
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-08T11:30:25.98+00:00

Hi Steve Mandras,

I will coordinate with the backend team and update you shortly. Thank you for your patience.
Steve Mandras 81 Reputation points

2025-12-08T16:40:15.05+00:00

Thanks @SRILAKSHMI C
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-10T12:54:59.44+00:00

Hi Steve Mandras,

There is a known issue with the image you’re using, and our engineering team is already working on a fix. The current expected timeline for resolution is by next week.

The problem is related to recent upgrades in the urllib3 library, which introduced this behavior.

We’ll keep you updated as soon as the fix is deployed.

Thank you!
Steve Mandras 81 Reputation points

2025-12-11T20:08:32.5+00:00

Looks like we had a successful ML Assisted model training and registration in our data labeling test project. They may have fixed things on the back-end in our region faster (kudos). Thanks for your assistance @SRILAKSHMI C
SRILAKSHMI C 11,215 Reputation points Microsoft External Staff Moderator

2025-12-12T06:17:33.3533333+00:00

Hi Steve Mandras,

Great to hear that the ML-assisted training and model registration are now working successfully! It’s possible the backend fix for your region rolled out sooner than expected glad everything is running smoothly again.

I’ve also converted my earlier comment into an answer, could you please take a moment to mark it as Accepted? This helps others in the community with the same question find the solution more easily.

Thank you!

Answer 1

Hi Steve Mandras,

Welcome to Microsoft Q&A and Thank you for reaching out.

I understand that you’re running into an ImportError when starting an on-demand ML Assisted Labeling job in Azure ML Studio, and the logs indicate a package/version mismatch inside the auto-selected environment (azureml-automl-dnn-text-gpu:27). The issue is related to outdated dependencies in that environment, and newer versions (such as v36) include updates that resolve the urllib3 / DEFAULT_CIPHERS import failure.

At the moment, the built-in environment used by ML Assisted Labeling is pinned to older versions and cannot automatically pick up newer images unless manually overridden. To address this, here are the available options:

Use the Updated Environment (v36)

If version 36 is available in your workspace, switching the ML Assisted Labeling job to this environment resolves the import issue. The ImportError is caused by outdated urllib3 and botocore packages, which v36 fixes.

If you are manually configuring environments, ensure that the newer environment is installed:

pip install azureml-ai-ml-automl-dnn-text-gpu==36

Once installed, verify its packages:

pip show azureml-ai-ml-automl-dnn-text-gpu

Confirm Dependency Compatibility

The error clearly shows that the older environment attempts to load modules (transformers, accelerate, botocore) that depend on newer SSL/urllib3 components. Updating the environment ensures all dependent libraries load correctly.

Trigger a Test Run

After switching to the updated environment, run a small test labeling job to confirm the import modules load without errors and everything resolves cleanly.

Please refer this

I Hope this helps. Do let me know if you have any further queries.

Thank you!

Share via

ML Studio Data Labeling On-Demand ML Assisted Labeling failing with ImportError

0 additional answers

Your answer