Model access error during training on SageMaker
I am trying to run a training job with my own data on SageMaker using HugginFace estimator.
I have accepted T&C on the model page,
I do a hugging face login
from huggingface_hub import notebook_login
notebook_login()Run fit method
git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.26.0'}
huggingface_estimator = HuggingFace(
entry_point='run_clm.py',
source_dir='./examples/pytorch/language-modeling',
instance_type='ml.p3.2xlarge',
instance_count=1,
role=role,
git_config=git_config,
transformers_version='4.26.0',
pytorch_version='1.13.1',
py_version='py39',
hyperparameters = hyperparameters
)
huggingface_estimator.fit()
But i get access token error.
HTTPError: 401 Client Error: Unauthorized for url:
Cannot access gated URL repo for URL https://huggingface.co/bigcode/starcoder/resolve/main/config.json.
OSError: bigcode/starcoder is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' . If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.
I have tried to pass the auth token in HuggingFace estimator but has not worked, tried to pass it as hyperparameter as a hub_token but that didnt work.
'hub_token': HfFolder.get_token()
How can i successfully run the fit method?
Try doing this:
from huggingface_hub.hf_api import HfFolder
HfFolder.save_token('your_hf_token')