ECONNRESET error socket hang up when creating task...

Hamdi_Masalmeh · 07-30-2024 02:16 PM

Our server had a problem with Deadline exceeded which appeared frequently, I looked for a solution to solve the problem and found an issue from 2020 and the google team pointed out a solution to use:

const client = new CloudTasksClient({ fallback: true });

This solved the problem for me, however it created a new problem when trying to create so many tasks (some users data creates 14000 tasks). This problem didn't appear when the deadline exceeded was appearing. Like the cloud run server could work for a week before starting to throw deadline exceeded. Now the new problem which I am trying to understand is:

FetchError: request to https://cloudtasks.googleapis.com/v2/projects/dev/locations/europe-west3/queues/XAPI/tasks?$alt=json%3Benum-encoding=int failed, reason: socket hang up"
 at ClientRequest.<anonymous> (/usr/src/app/node_modules/node-fetch/lib/index.js:1501:11)
  at ClientRequest.emit (node:events:517:28)
at TLSSocket.socketOnEnd (node:_http_client:525:9)
 at TLSSocket.emit (node:events:529:35)
  at endReadableNT (node:internal/streams/readable:1400:12)
  at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {

  type: 'system',
 errno: 'ECONNRESET',
 code: 'ECONNRESET'

this is the time of the error: 2024-07-30 09:40:08.513 then the server starts to create tasks, this run created approx 4500 tasks and this is the time of the last task created: 2024-07-30 09:40:10.139

this is the code where I create the google tasks:

const { CloudTasksClient } = require("@google-cloud/tasks").v2;
const client = new CloudTasksClient({ fallback: true });

const createTask = async (url: string, payload: TaskPayload): Promise<void> => {
  const parent = client.queuePath(project, location, queue);
  const task: protos.google.cloud.tasks.v2.ITask = {
    httpRequest: {
      httpMethod: "POST",
      url,
      headers: { "Content-Type": "application/json" },
      body: Buffer.from(JSON.stringify(payload)),
    },
  };

  try {
    const [response] = await client.createTask({ parent, task });
    console.log(`Created task ${response.name}`);
  } catch (error) {
    console.error("Error creating task:", error);
    throw error;
  }
};

and this is where i use the function:

const BATCH_SIZE = 250;

export const processBatches = async (  tasks: Promise<void>[],
  batchSize: number = BATCH_SIZE) => {
  for (let i = 0; i < tasks.length; i += batchSize) {
    const batch = tasks.slice(i, i + batchSize);
    try {
      await Promise.all(batch);
    } catch (error) {
      console.error("Error processing batch:", error);
    }
  }
};

 try {
    const data = payload.data as unknown as Body[];
    const user = payload.user as User;
    const userRefId = user.reference_id;
    let tasks = [];

    for (const bodyData of data) {
      const startTime = bodyData.metadata.start_time;
      const endTime = bodyData.metadata.end_time;
      const collectionId = generateCollectionId(startTime, endTime);

      const collectionsData = generateBodyCollections(bodyData);

      for (const [collectionName, collectionData] of Object.entries(
        collectionsData
      )) {
        const filteredData = removeNullValues(collectionData);
        if (!areAllValuesNullOrEmpty(filteredData)) {
          tasks.push(
            createTask(uploadUrl, {
              userRefId,
              collectionName,
              docId: collectionId,
              data: filteredData,
            })
          );
        }
      }
    }

    await processBatches(tasks);
  } catch (error) {
    console.error("Error processing body data:", error); //this is where the error comes up.
  }

This is the function that handle the google tasks:

const functionConfig: HttpsOptions = {
  region: "europe-west3",
  memory: "4GiB",
  cpu: 2,
  timeoutSeconds: 500,
  concurrency: 80,
  minInstances: 0,
  maxInstances: 50,
};

export const uploadToFirestore = onRequest(
  functionConfig,
  async (request, response) => {
    const { userRefId, collectionName, docId, data } = request.body;
    try {
      const docRef = firestore
        .collection("patients")
        .doc(userRefId)
        .collection(collectionName)
        .doc(docId);
      await docRef.set(data, { merge: true });

      response.status(200).send("Data uploaded successfully");
    } catch (error) {
      console.error("Error uploading data to Firestore:", error);
      response.status(500).send("Internal Server Error");
    }
  }
);

this is the queue details: Location europe-west3 Rate limits Max rate 500/s Max concurrent 5000 Max burst size 100 Retry parameters Max attempts 100 Min interval 0.100s Max interval 3600s Max doublings 16 Max retry duration Unlimited

As i mentioned this error started to appear when i used fallback true, If i remember right when i looked at the tasks yesterday I saw an error with permission denied which didn't appear before when everything was working (sometimes if the deadline exceeded didn't appear).

Any insights or suggestions to resolve this ECONNRESET error would be greatly appreciated.

Feel free to adjust any specific details or code snippets as needed.

I tried to reduce the batch size to 250 and got this error:

FetchError: request to https://cloudtasks.googleapis.com/v2/projects/dev/locations/europe-west3/queues/XAPI/tasks?$alt=json%3Benum-encoding=int failed, reason: Client network socket disconnected before secure TLS connection was established"

Or it maybe because i changed the version of CloudTasksClient from "@google-cloud/tasks";@google-cloud/tasks.v2.

MaximGerb

If I remove

fallback: true

I got the following error: 4 DEADLINE_EXCEEDED: Deadline exceeded

https://github.com/googleapis/nodejs-tasks/issues/397#issuecomment-618580649

ECONNRESET error socket hang up when creating tasks using client.CloudTasksClient with fallback true