Adding exemptions in firebase automatic indexing

[mudasir]

I'm storing data in firebase with this structure:

/aggregate_data/{user id}/{product id}/data

The aggregate_data collection is the root collection id and the rest subcollections, documents under it except data are dynamically generated. The data stored under data document is a map with structure

{
  'YYYYMMDD': {
     'field1': value1,
     'field2': value2,
     'field3': value3,
     'field4': value4,

  },
  .
  .
  .
  .
  .
  'YYYYMMDD': {
     'field1': value1,
     'field2': value2,
     'field3': value3,
     'field4': value4,

  },

}

The data inside this data document has now reached a point where it's not allowing me to add (update) more data in this map since it's being auto indexed by firebase and this error is produced as a result:

google.api_core.exceptions.InvalidArgument: 400 too many index entries for entity

I tried adding an exception in indexing to prevent indexing on this document so that I can add more data but not sure if I'm doing it the right way. So far I've tried the combinations shown below:

where sessions is one of the fields nested inside the map. Please note that the map keys YYYYMMDD are dynamically generated and contain date strings. What would be the right Collection ID & Field path combinations needed for me to stop the indexing so that I can add more data?

[jaydubu]

Hi @mudasir,

Welcome to Google Cloud Community!

I understand that you are hitting “too many index entries for entity” even with these setups: 

• Using sub collections to store your data: the recommended approach to structure data hierarchically, making it easier to access
• Added single-field index exemption: disabled for all collection and group scopes

I can see that you already found a workaround on the Stack Overflow question that you raised by adding exception to the entire Collection ID with the use of "*" wildcard. Likewise, you can refer to the listed recommendations below in case this occur again.

Recommendations: 

• If there are certain fields in these documents that can be deleted, or can be moved to other kinds or other documents, consider modifying the data structure so that each document has fewer and smaller fields (nested maps, and arrays are usually big contributors to large index fan outs)
• Remove certain indexes that you no longer need
  - Especially if there are some fields that don't need single-field indexing for (which is enabled by default), you can disable single-field indexing on those fields
• Remove those documents as last resort

Check out the best practices on how to manage your indexes for more information.

I hope the above information is helpful.

[smartplay121]

Adding exemptions in Firebase automatic indexing involves excluding specific data from being indexed to optimize search performance. Though unrelated, Smart Play (URL Removed by Staff) can help developers test and manage app performance efficiently.

[mudasir · Accepted solution]

Seemed to be any easy one since the subcollection in my case was a dynamic one I used the * wildcard as Collection ID and the field names field1, field2 etc. were known so I had to add them specifically to prevent them from being auto-indexed and this fixed the issue. One such example is given below for field name sessions.

Collection ID: * & Field Path: sessions
This fixed the issue!