Is millions of files & many TB *mirrored* to local file systems within scope for Drive for Desktop?

[JRF]

Question:  Is having millions of files and terabytes *mirrored* to local file systems within the design parameters of the new "Drive for Desktop"?  

More philosophically:  will the concept of "backup" disappear when "Backup and Sync" becomes "Drive for Desktop"?

Context:  I use "Backup and Sync" as a core piece of infrastructure.  While we make extensive use of Google Docs, the majority of our files are PDFs and MS Office files.  Many of the MS Office files are point-in-time snapshots of Google Docs.  We use Google Drive as a central archive for all of these files.  I found that Google Shared Drives have file count limits (400,000) that are too small for these archives, so I share projects only as shared *folders* in my My Drive. 

My total file counts is over ten million and several TB.

I keep a small set of *current project folders* on my laptop.  When a project or time period ends, I stop syncing that folder, so that the files are no longer on my laptop and instead are *backed up* in Google Drive.

For redundancy, I have multiple geographic sites with workstations running Backup and Sync that download the entire My Drive to an external disk.  These sites also run Apple Time Machine backups to a second external disk.

I love Google Drive because it unifies three things: (a) central nexus for files created on my computers or in the cloud, (b) searchable index of those files, and (c) distribution point for getting files to multiple backup sites.

How should I interpret Google's move to "Drive for Desktop"?  Is my usage pattern part of your target vision for what you want to support?  Or is having millions of files and terabytes *mirrored* to local file systems not within the design parameters of the new "Drive for Desktop"?  

Here's another way to ask the same question:  which of these two choices would you recommend as more reliable?

(1) Use "Drive for Desktop" to mirror a My Drive with 10+ million files and several TB to offline storage on half a dozen different computers.

(2) Use cronjobs to run rsync to mirror between sites and have just one of those sites syncing with Google Drive so that files in Google Drive get into the backup sites and files added to the offline sites become searchable in Google Drive.  

(Anecdote: I believe strongly in using tools in the way that their designers/maintainers intend.  I once had a disastrous experience with the non-atomic file moves in garbage that is Dropbox because I didn't realize that they weren't intending to offer a real file system.)

[KAM]

@JRF I would personally say that Drive for Desktop, previously Drive File Stream, was designed to use Google's "magic" to cache and sync files locally using a minimal amount of space on the desktop while keeping the files in the cloud.

I've used DFS successfully with very low-end Chromebooks attached to instances approaching petabytes of data.

It's very smart and with a decent internet connection, you rarely know you are working on files that are remote.

Beyond that, I think if you are trying to use the tool to make a backup of your data at Google, you'll be better off looking into a tool dedicated to backups like Backupify.

If you just want to use stuff on Google drive AND backup some things on your laptop to the cloud, that's like more the intention for Drive for Desktop.

As a final note, Backup and Sync was considered a consumer product.  Drive for Desktop is an Enterprise product.  So I think you'll find it pretty amazing.  

Google Workspace + Drive for Desktop is an absolute file server killer for all but things like video editing and accounting software.  For accounting software, we deploy VDI networks for that these days.  Video editing, just better to have it local due to the size and data grinding.

HTH, KAM

[JRF]

@KAM 

Thank you for your reply!  I agree that tools like backupify are a good idea.

You mentioned "decent internet connection," and unfortunately that's not realistic.  Nobody has a consistently good internet connection when traveling, e.g. on an airplane.  I love gDocs "offline mode" and need the analog of that for all my flat files.

Further, only one of my permanent work sites has a really reliable internet connection.  My other work sites have frequent internet outages of a few minutes and occasionally for hours---it's the nature of working in the field.

For these, full mirroring to the local file system is crucial.  Have you seen the mirroring functionality in "Drive for Desktop" work well on these?

JRF

[KAM]

@JRF I have not had any work with the new Drive for Desktop to use the Backup & Sync features, sorry.  -KAM

[GerhardZelenka]

Hi @JRF, my concern around the number and size of files you mention is that whilst Google Drive uses Colossus in Google Servers, Googel Drive mounts onto your computer as a FAT32 system:

The limitiations of FAT32 may cause issues for the volume of files you're managing and thus may cuase reliability issues - you'd need to talk to someone who knows about Colossus and how it's mounted to get a definitive answer.

The backup functionality of Backup and Sync has now been intetegrated into Google Drive Desktop so that the new system has most features of both previous systems - the Upcoming changes to Google Drive sync clients document has a feature comparison table.

[JRF]

@GerhardZelenka thank you for your note!

What type of file system does Google Drive Desktop use on macOS? 

Most of my computers are Apple, and all of the ones relevant to this question are Apple running macOS.

[KAM]

@JRF & @GerhardZelenka very interesting point.  Google Drive is mounting and appear to a Windows System as FAT32 HOWEVER, I would not pay attention to how DFS aka Drive for Desktop is tricking the OS into thinking the cloud content is on the local machine while remaining in the cloud.  

However, I don't think he's going to be using that mode which on my machine is more the stream mode.  He's going to be using the mirror mode which I think is the same as what he was doing before.

JRF, take a look at "What it means to mirror or stream your files," on this KB: https://support.google.com/drive/answer/10838124 and see if that helps.

It's looking a lot like to me, the backup/sync is the old system so if it worked with the old system in your scenario, I would posit you are good to go in this scenario.  -KAM