Hello everyone,
I'm currently facing a challenge with Cloud Composer. We want to install a custom Python package from a private repository, which is why we're using a personal access token. I found a guide from Google suggesting that we should have a pip.conf file in the /dags/config/pip directory. The file should contain the following configuration:
[global]
extra-index-url = https://<username>:<personal_access_token>@github.com/<repository_owner>/<repository_name>.git
(Note: I've replaced sensitive information with placeholders to avoid leaking personal data.)
https://cloud.google.com/composer/docs/composer-3/install-python-dependencies?hl=de#install-public-r...
Despite following these instructions, the package doesn't seem to install when I execute a DAG that requires it. The error indicates that the package isn't installed.
I've also tried creating a requirements.txt file and placed it in the DAGs folder with the following content:
git+https://<username>:<personal_access_token>@github.com/<repository_owner>/<repository_name>.git
Unfortunately, this approach hasn't worked either. Additionally, I've noticed that sometimes the contents of these files get deleted unexpectedly.
I've already verified that Cloud Composer can access the public internet, so connectivity shouldn't be the issue.
Has anyone experienced similar issues or have any insights on how to resolve this? Any help would be greatly appreciated!
Thank you!
LinkedIn
Twitter
