Unexpected cache miss during docker build step in Cloud Build

[lovitt]

On January 8, the docker build step in my Cloud Build config stopped using the docker layer cache where it would previously.

An early step in my Dockerfile is:

RUN apt-get update -qq && \
apt-get install --no-install-recommends -y build-essential default-libmysqlclient-dev git libvips pkg-config

That step would previously be a cache hit during most builds. Now it misses every time. 

I am using gcr.io/kaniko-project/executor:latest with the "cache=true" argument to build my image. But I tried switching to the approach described here, using gcr.io/cloud-builders/docker with "--cache-from", and I get a cache miss on the same step.

I've tried re-running the same build repeatedly (using the same git revision) and get a cache miss at the same step every time.

When I build the docker image on my local machine, the cached layer is used as expected.

I was using Container Registry previously. Once I started seeing this problem, I migrated to Artifact Registry, and I see the same behavior.

I'm not sure how to proceed in debugging this issue. Any suggestions would be appreciated.