Filestore much slower than Persistent Disk for some operations, despite superior specs

[yaroslavvb]

Is there some kind of rate-limit that would make Filestore much slower than Persistent Disk for copying many small files?

I have filestore + persistent disk which are similar on paper (bandwidth + iops), but in practice can be 10x slower.

CFor instance, copying 30k files takes 10 seconds for Persistent Disk, but 100 seconds for Filestore, using gsutil -m rsync for both. My filestore IOPS and bandwidth are 1-2% of maximum during this time. 

On other hand, fio benchmarks make filestore + PD look similar, in accordance with advertised rates.

My guess is that there's some kind of undocumented rate-limit, perhaps around small-file creation. I can't get past 300 files/second, even going to HIGHSCALE tier with 60TB. Any ideas?

The practical impact is that operations like "pip install; conda install" take much longer on filestore than on persistent disk.

[Marramirez]

Hi @yaroslavvb,

Welcome to the Google Cloud Community!

In order to comply with NFS (Network File System) Protocol and to keep data safe, Filestore uses the sync export option.

Filestore instance waits for the data to be committed to storage before replying to requests from the Client VM. In an event that many files are involved in an operation, such as copying a large number of small files, the client will perform a long series of synchronous operations which increases the overall latency. As a result, performance is reduced.

You can read more about Workloads Involving High Volumes of Small Files.

You can also read Optimize and Test Instance Performance. It also contains the expected performance of the service tier you are using, as well Improving the Performance Across Google Cloud Resources.

If the above options don't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[yaroslavvb]

Thanks for the tips. I've contacted support and they were able to reproduce the issue on their own filestore, only getting about 170 files per second. I'm able to copy 270 files per second to HIGHSCALE-SSD 60TB filestore. Meanwhile, copying to basic EBS proceeds at the rate of 3000 files per second.

- where does this limit come from? I imagine there's a limit to number of "sync" operations that Filestore can achieve, how do I estimate this number?

- is there a way to configure Filestore to achieve similar performance to EBS?

[yaroslavvb]

By "EBS" I meant "persistent disk" (too much AWS back in the day for me :))