Problem with saving multiple files to cloud storage

[mdubrowski]

My problem appears only sometimes(what is really strange) while saving multiple files simultaneously to cloud storage. I am using [google-cloud/storage](https://www.npmjs.com/package/@google-cloud/storage) library version 6.9.5.
I was able to find out people having the same issue however none of their solutions worked for me(neither changing stream options nor applying timeout). Firstly I thought that bandwidth could be exceeded, however I am almost sure we are not saving over 50Gb of data(I tried to check it using cloud charts; however you can check bandwidth usage only if other services are using cloud storage - in my case i get "No data available").

For further explanation:
In my code there is a function called "uploadFileToGCS" which is invoked around 57 times per each process. Process includes saving 35jpg(20KB per one) images, 1 tiff image(500KB) and 7 json(150KB per one) file. Simoultaneously 30 processes are invoked.

Here is exactly what error says:

error - unhandledRejection: FetchError: request to https://storage.googleapis.com/upload/storage/v1/b/satellite-photos/o?uploadType=multipart&name=64d162a80103b55f87a6cdb4_64db59204f7cd1f34a65e5c7_2023_6_31_raw.tiff failed, reason: read ECONNRESET
    at ClientRequest.<anonymous> (D:\WORK\nirby-project\node_modules\next\dist\compiled\node-fetch\index.js:1:65756)
    at ClientRequest.emit (node:events:525:35)
    at TLSSocket.socketErrorListener (node:_http_client:496:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNRESET',
  code: 'ECONNRESET'
}

function saving 5 images(which is invoked 7 times in the same time):

saveLayers.ts:

import { saveSinglePhoto } from "~/helpers/sentinel/saveSinglePhoto";

interface INdviData {
    min: number;
    max: number;
    averageNDVI: number;
    buffer: Buffer;
}

const saveLayers = async (
    fileNameBase: string,
    trueColorData: Buffer,
    ndviData: INdviData,
    contrastNdviData: Buffer,
    sclData: Buffer,
    clmData: Buffer
) => {
    console.log(fileNameBase);

    const data = await Promise.all([
        saveSinglePhoto(trueColorData, fileNameBase, "TRUECOLOR"),
        saveSinglePhoto(ndviData.buffer, fileNameBase, "NDVI"),
        saveSinglePhoto(contrastNdviData, fileNameBase, "CONTRAST_NDVI"),
        saveSinglePhoto(sclData, fileNameBase, "SCL"),
        saveSinglePhoto(clmData, fileNameBase, "CLM"),
    ]);

    return data;
};

export default saveLayers;

saveSinglePhoto.ts

import { SatelliteData } from "~/interfaces/sentinel/SatelliteData";
import uploadFileToGCS from "../gcs/uploadFileToGCS";

const baseUrl = `https://storage.googleapis.com/${process.env.GCLOUD_STORAGE_BUCKET}/`;

export const saveSinglePhoto = async (
    buffer: Buffer,
    fileNameBase: string,
    layerId: string
): Promise<SatelliteData | null> => {
    const fileName = fileNameBase + layerId + ".jpg";

    await uploadFileToGCS(fileName, buffer, "image/jpeg");

    const satelliteData: SatelliteData = {
        layerId: layerId,
        fileUrl: baseUrl + fileName,
    };

    return satelliteData;
};

uploadFileToGCS.ts

import { Storage } from "@google-cloud/storage";

const gcsKey = JSON.parse(Buffer.from(process.env.GCLOUD_CRED_FILE, "base64").toString());

const storage = new Storage({
    credentials: {
        client_email: gcsKey.client_email,
        private_key: gcsKey.private_key,
    },
    projectId: process.env.GCLOUD_PROJECT_ID,
});

const uploadFileToGCS = (filename: string, data: any, contentType: string) => {
    return new Promise((resolve, reject) => {
        const file = storage.bucket(process.env.GCLOUD_STORAGE_BUCKET).file(filename);

        const stream = file.createWriteStream({
            metadata: {
                contentType,
            },
            resumable: false,
            validation: false,
            timeout: 86400,
        });

        stream.on("error", (err) => {
            reject(err);
        });

        stream.on("finish", () => {
            resolve("ok");
        });

        stream.end(data);
    });
};

export default uploadFileToGCS;

[Marramirez]

Hello @mdubrowski,

Welcome to the Google Cloud Community!

Take a look at this Stack Overflow post as you might have the same problem. 

As you have createWriteStream in your code, you don't seem to have the createReadStreams which results in you getting the error. Take a look at this blog to better Understand Streams in Node.js.

You can also check on the official documentation on how to Upload objects from a filesystem, choose client libraries and select Node.js.

If the above options don't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[mdubrowski]

Hello @Marramirez ,
Your solution didn't work for me, because my variable "data" already contained content of read file. The problem was about [concurrent requests limit](https://cloud.google.com/run/docs/about-concurrency). I was adived to use Cloud Tasks in order to prevent the error. It seems like a suitable solution for me.

Nevertheless - thanks for your help!