This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

GCE Cloud-Init util.py return error for all apt commands

Posted on 11-08-2023 05:01 PM
NickK
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

I've been using the following cloud-init config when building some GCE VMs for two years and within the last week they started to fail with a python traceback for apt update and installs.  Has anybody else seen similar behavior?

 

#cloud-config

# Install APT sources
apt:
  sources:
    gcsfuse.list:
      source: "deb https://packages.cloud.google.com/apt $RELEASE main"
      keyid: A362B822F6DEDC652817EA46B53DC80D13EDEF05
    docker.list:
      source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable"
      keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88

# Install System Packages
package_update: true
packages:
  - apt-utils
  - apt-transport-https
  - software-properties-common
  - python3.9
  - python3-pip
  - python3-dev
  - ca-certificates
  - curl
  - gnupg
  - gnupg-agent
  - lsb-release
  - gcsfuse
  - docker-ce
  - docker-ce-cli
  - pass
  - sysstat
  - nmon
  - vim
  - cron
  - ubuntu-drivers-common

# create the docker group
groups:
  - docker

# Add default auto created user to docker group
system_info:
  default_user:
    groups: [docker]

runcmd:
  - [ sh, -c, /usr/bin/pip3 install docker ]
  - [ sh, -c, "/usr/bin/curl -L 'https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64' -o /usr/bin/docker-compose" ]
  - [ sh, -c, chmod +x /usr/bin/docker-compose ]

 

 Receiving these errors:

 

2023-11-09 00:18:57,878 - debian.py[DEBUG]: Waiting for apt lock
2023-11-09 00:18:57,879 - debian.py[DEBUG]: apt lock available
2023-11-09 00:18:57,879 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsa
fe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2023-11-09 00:19:09,999 - util.py[DEBUG]: apt-update [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-
yes --quiet update] took 12.119 seconds
2023-11-09 00:19:10,000 - handlers.py[DEBUG]: finish: modules-config/config-apt-configure: FAIL: running config-apt-configure with frequency once-per-instance
2023-11-09 00:19:10,000 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/
config/cc_apt_configure.py'>) failed
2023-11-09 00:19:10,000 - util.py[DEBUG]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/co
nfig/cc_apt_configure.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/modules.py", line 246, in _run_modules
    ran, _r = cc.run(
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 67, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 196, in handle
    apply_apt(apt_cfg, cloud, target)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 244, in apply_apt
    add_apt_sources(
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 609, in add_apt_sources
    update_packages(cloud)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 522, in update_packages
    cloud.distro.update_package_sources()
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 293, in update_package_sources
    self._runner.run(
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 287, in package_command
    self._wait_for_apt_command(
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 223, in _wait_for_apt_command
    return util.log_time(
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2721, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update']
Exit code: 100
Reason: -
Stdout: -
Stderr: -



2023-11-09 00:19:10,874 - debian.py[DEBUG]: Waiting for apt lock
2023-11-09 00:19:10,874 - debian.py[DEBUG]: apt lock available
2023-11-09 00:19:10,874 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsa
fe-io', '--assume-yes', '--quiet', 'install', 'apt-utils', 'apt-transport-https', 'software-properties-common', 'python3.9', 'python3-pip', 'python3-dev', 'atop', 'c
a-certificates', 'curl', 'gnupg', 'gnupg-agent', 'lsb-release', 'gcsfuse', 'docker-ce', 'docker-ce-cli', 'pass', 'sysstat', 'nmon', 'vim', 'cron', 'ubuntu-drivers-co
mmon'] with allowed return codes [0] (shell=False, capture=False)
2023-11-09 00:19:11,193 - util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume
-yes --quiet install apt-utils apt-transport-https software-properties-common python3.9 python3-pip python3-dev atop ca-certificates curl gnupg gnupg-agent lsb-relea
se gcsfuse docker-ce docker-ce-cli pass sysstat nmon vim cron ubuntu-drivers-common] took 0.319 seconds
2023-11-09 00:19:11,194 - util.py[WARNING]: Failed to install packages: ['apt-utils', 'apt-transport-https', 'software-properties-common', 'python3.9', 'python3-pip'
, 'python3-dev', 'atop', 'ca-certificates', 'curl', 'gnupg', 'gnupg-agent', 'lsb-release', 'gcsfuse', 'docker-ce', 'docker-ce-cli', 'pass', 'sysstat', 'nmon', 'vim', 'cron', 'ubuntu-drivers-common']
2023-11-09 00:19:11,194 - util.py[DEBUG]: Failed to install packages: ['apt-utils', 'apt-transport-https', 'software-properties-common', 'python3.9', 'python3-pip',
'python3-dev', 'atop', 'ca-certificates', 'curl', 'gnupg', 'gnupg-agent', 'lsb-release', 'gcsfuse', 'docker-ce', 'docker-ce-cli', 'pass', 'sysstat', 'nmon', 'vim', '
cron', 'ubuntu-drivers-common']
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 116, in handle
    cloud.distro.install_packages(pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 138, in install_packages
    self.package_command("install", pkgs=pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 287, in package_command
    self._wait_for_apt_command(
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 223, in _wait_for_apt_command
    return util.log_time(
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2721, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'apt-utils', 'apt-transport-https', 'software-properties-common', 'python3.9', 'python3-pip', 'python3-dev', 'atop', 'ca-certificates', 'curl', 'gnupg', 'gnupg-agent', 'lsb-release', 'gcsfuse', 'docker-ce', 'docker-ce-cli', 'pass', 'sysstat', 'nmon', 'vim', 'cron', 'ubuntu-drivers-common']
Exit code: 100
Reason: -
Stdout: -
Stderr: -

2023-11-09 00:19:11,197 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2023-11-09 00:19:11,197 - handlers.py[DEBUG]: finish: modules-final/config-package-update-upgrade-install: FAIL: running config-package-update-upgrade-install with f
requency once-per-instance
2023-11-09 00:19:11,197 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/li
b/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
2023-11-09 00:19:11,197 - util.py[DEBUG]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/
python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/modules.py", line 246, in _run_modules
    ran, _r = cc.run(
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 67, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 185, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 142, in handle
    raise errors[-1]
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py", line 116, in handle
    cloud.distro.install_packages(pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 138, in install_packages
    self.package_command("install", pkgs=pkglist)
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 287, in package_command
    self._wait_for_apt_command(
  File "/usr/lib/python3/dist-packages/cloudinit/distros/debian.py", line 223, in _wait_for_apt_command
    return util.log_time(
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2721, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'ap
t-utils', 'apt-transport-https', 'software-properties-common', 'python3.9', 'python3-pip', 'python3-dev', 'atop', 'ca-certificates', 'curl', 'gnupg', 'gnupg-agent',
'lsb-release', 'gcsfuse', 'docker-ce', 'docker-ce-cli', 'pass', 'sysstat', 'nmon', 'vim', 'cron', 'ubuntu-drivers-common']
Exit code: 100
Reason: -
Stdout: -
Stderr: -

 

Any feedback is appreciated.  Not sure if I should open a case, but it seems that something changed on GCP's back-end somewhere related to cloud-init and/or how it processes the user-data metadata field.

Thanks,
Nick

 

1 2 806
Topic Labels
2 REPLIES 2

Posted on --/--/---- --:-- AM
NickK
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

A little more troubleshooting and it seems to only fail with both source lists enabled.  If I comment out either one with the associated packages, the rest of the config runs successfully.

Posted on --/--/---- --:-- AM
NickK
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

Finally just abandoned using the apt config and setting my repos up with bash under runcmd.  

Top Labels in this Space
Top Solution Authors
View all