Storage Transfer Service - job duration is unexpectedly increasing over time

[Vasco]

I have set up a job in Storage Transfer Service that sincronizes the content of a folder in Azure Blob Storage to a folder in Google Cloud Storage, once per day. The duration of this job is increasing over time with no increase in data volume transferred per run. This is unexplained and a strangely shaped loss of performance, since the duration is increasing linearly every day after the previous.

Does anyone have a clue why this could be happening?

[KirTitievsky]

Any chance the # of files transferred is growing? How many files do you transfer every day?  How many are there in the buckets? Also, another guess: part of an STS job is to list all the files on both sides and compare them.  Since the # of files is growing every day, the duration of this phase, should, in principle, grow as well.  If there are very few new objects transferred, it's possible that the total time for the STS job is dominated by this "diff" step which would explain your observations. 

[Vasco]

Hello @KirTitievsky 
No, the # of files transferred is fairly constant (around 27k files), as well as the data skipped (~3 TB) and the data deleted from destination (~300 MB). The average speed estimation just keeps reducing linearly, and likewise the duration is growing linearly over time.
Can the "diff" step (list and compare) take longer if the # of files to screen is constant?

[KirTitievsky]

"Constant"  meaning you are not adding any files? Or adding and deleting the same number of files between every cycle?  If that's the case, my theory does not explain the changes you see.  Would you confirm this directly from Monitoring metrics?

[Vasco]

Yes, each cycle adds, deletes and skips about the same number of files.

[KirTitievsky]

Can you send me job and any monitoring data you use to observe this change in a private message.  I'd like to see if I can get the storage transfer service team to look at this.  No guarantees I get get a response since this is turning into a support case, effectively. If you have a support plan, please submit this as a support case. That's guaranteed to get you a response.