Problem of Multiprocess in Component of Vertex AI

[guillaume0683]

Hello community,

I'm trying to parralelize my code in my component Vertex.
For this, i used some librairies like concurrent.futures(), multiprocessing or multiprocess.

But each time, i've got an error with pickle. For multiprocessing and concurrent.futures, something like this : AttributeError: Can't pickle local object 'preprocess_series.<locals>.preprocess_serie_by_serie'

And for multiprocess (which use Dill instead of Pickle) i've got TypeError: cannot pickle 'SSLContext' object

I give you a part of my code: 

import multiprocess as mp
def preprocess_serie_by_serie(args):
       .... #list of pandas dataframe operations
       return df
list_trainable_series = ... #a list of string
pool = mp.Pool(processes = (mp2.cpu_count() - 1))
lst_args = [(df1, serie_id, cursor) for serie_id in list_trainable_series]
results = pool.map(preprocess_serie_by_serie, lst_args)
pool.close()
pool.join()

results_df = pd.concat(results)

I test this code in local and it's working but i can not find a solution for Vertex AI.

Thanks for your help