GAR vulnerability reports include files that don't...

jth · 07-29-2024 12:02 PM

The subject says most of it. We have numerous instances of Docker containers that are reporting vulnerable packages are installed (for example setuptools < v70) that, when interacting with the image, do not have the specific vulnerable version installed. Further, the reported vulnerable file doesn't exist. Is there some way to dig deeper into what the GAR vuln scanning is looking at or has anyone else noticed incorrect issue reporting like this? Because at this point it seems to be fairly useless if we want to confirm addressable vulns have been remediated.

alexmoore

Hi,

Curious to dig deeper on this one, is there anything you can share that would reproduce what you are seeing?

All the best,

Alex

jth

Found some better info than my reply this morning.

We are running Docker from the puthon:3.9-slim base. This base has history where it had the identified vulnerable version of setuptools (and Pip as well) installed. https://hub.docker.com/layers/library/python/3.9-slim/images/sha256-6a168e1359fe0cb7746746fe418bab1f...

And even though our Dockerfile is updating the image immediately, the container is still getting flagged as containing these vulnerabilities.

# more Dockerfile
FROM  --platform=linux/amd64 python:3.9-slim AS builder-base

RUN apt update -y && apt upgrade -y

RUN mkdir app
WORKDIR /app/

...

So GAR's vulnerability scanning is flagging files that exist in the base image/layers embedded in the base image, but which are resolved by our steps in creating the container.

alexmoore

So this is very helpful, thank you. I tried it myself just now and in my case a simple apt update/upgrade isn't updating the python packages themselves, so for me setuptools was showing as v58.1.0 and then correctly being flagged by GAR.

Do you have a setup that is also updating the python packages too?

alexmoore

Just to follow my reply up 🙂 If I modify the Dockerfile to update the python packages I don't see the vulnerability in GAR, for example:

FROM  --platform=linux/amd64 python:3.9-slim AS builder-base

RUN apt update -y && apt upgrade -y && pip freeze --all | cut -d "=" -f 1 | xargs pip install -U --root-user-action=ignore

RUN mkdir app
WORKDIR /app/

jth

Ah I trimmed our Dockerfile too much -- here's the whole thing. Guessing the next lines are what are getting the right pip and setuptools in place.

# more Dockerfile
FROM  --platform=linux/amd64 python:3.9-slim AS builder-base

RUN apt update -y && apt upgrade -y

RUN mkdir app
WORKDIR /app/

RUN pip install -U pip setuptools && \
    pip install "poetry>=1.5.0" \
    && pip install keyring \
    && pip install keyrings-google-artifactregistry-auth

FROM builder-base AS builder-deps
COPY . .
RUN --mount=type=secret,id=adc,dst=./creds/key.json \
    export GOOGLE_APPLICATION_CREDENTIALS=./creds/key.json && \
    poetry config virtualenvs.create false && \
    poetry install


FROM builder-deps
RUN mkdir config

ENTRYPOINT ["poetry", "run", "python"]

When I run the container, I see the following:

root@1bbffdb58b3f:/app# ls /usr/local/lib/python3.9/site-packages/setuptools-58.1.0.dist-info/METADATA
/usr/bin/ls: cannot access '/usr/local/lib/python3.9/site-packages/setuptools-58.1.0.dist-info/METADATA': No such file or directory
root@1bbffdb58b3f:/app# ls /usr/local/lib/python3.9/site-packages/setuptools*
/usr/local/lib/python3.9/site-packages/setuptools:
 __init__.py	     _entry_points.py   _normalization.py   archive_util.py   cli-arm64.exe   config	     errors.py	    gui-32.exe	    installer.py   monkey.py	      sandbox.py	   version.py
 __pycache__	     _imp.py	        _path.py	    build_meta.py     cli.exe	      depends.py     extension.py   gui-64.exe	    launch.py	   msvc.py	     'script (dev).tmpl'   warnings.py
 _core_metadata.py   _importlib.py      _reqs.py	    cli-32.exe	      command	      discovery.py   extern	    gui-arm64.exe   logging.py	   namespaces.py      script.tmpl	   wheel.py
 _distutils	     _itertools.py      _vendor		    cli-64.exe	      compat	      dist.py	     glob.py	    gui.exe	    modified.py    package_index.py   unicode_utils.py	   windows_support.py

/usr/local/lib/python3.9/site-packages/setuptools-70.0.0.dist-info:
INSTALLER  LICENSE  METADATA  RECORD  REQUESTED  WHEEL	entry_points.txt  top_level.txt
root@1bbffdb58b3f:/app#

But GAR still flags the issue claiming it seems 58.1.

alexmoore

So for me, that exact Dockerfile does not raise a vulnerability for setuptools when I tried it just now. Most strange. And if you re-download that exact image hash that GAR is flagging and look inside it the version is as you expect?

One side question though, where does this container run? because in what you shared it looks like you are loading service account keys into the container. If this is running on Google Cloud itself (or even in some cases other places like Azure and AWS etc) you should be able to use workload identity federation:

https://cloud.google.com/iam/docs/workload-identity-federation

This is a better security practice if possible as it means you don't have to manage keys and key lifecycle.

Alex

jth

@alexmoore wrote:
So for me, that exact Dockerfile does not raise a vulnerability for setuptools when I tried it just now. Most strange. And if you re-download that exact image hash that GAR is flagging and look inside it the version is as you expect?
One side question though, where does this container run? because in what you shared it looks like you are loading service account keys into the container. If this is running on Google Cloud itself (or even in some cases other places like Azure and AWS etc) you should be able to use workload identity federation:
https://cloud.google.com/iam/docs/workload-identity-federation
This is a better security practice if possible as it means you don't have to manage keys and key lifecycle.
And if you re-download that exact image hash that GAR is flagging and look inside it the version is as you expect?

I'm not sure how to do this but I'll see if I can figure that out to check.

We load this in GCP. I know some of our teams are shifting to workload identity federation, I'll add this to the team's list (they're loving me right now...)

jth

Yes, pulling the specific exact image hash had identical results - shows setuptools-70 installed. Very curious you can't recreate.

jth

Not sure why the forum here marked your reply as a solution...

Anyway, I don't know how to reproduce this. Many of our Docker images similarly have this issue.

In one image, GAR vulnerability report shows setuptools-58.1 installed:

Neither GitHub or Snyk show this vulnerability for this image.

When running the Docker image the METADATA file (and setuptools-58.1.0.dist-info directory) do not exist:

Pip agrees:

Nothing my devs have been able to do or find shows this vulnerability existing in this image, let alone found a way to patch/update, since it doesn't exist...

GAR vulnerability reports include files that don't exist