Hello community,
I'm trying to parralelize my code in my component Vertex.
For this, i used some librairies like concurrent.futures(), multiprocessing or multiprocess.
But each time, i've got an error with pickle. For multiprocessing and concurrent.futures, something like this : AttributeError: Can't pickle local object 'preprocess_series.<locals>.preprocess_serie_by_serie'
And for multiprocess (which use Dill instead of Pickle) i've got TypeError: cannot pickle 'SSLContext' object
I give you a part of my code:
import multiprocess as mp
def preprocess_serie_by_serie(args):
.... #list of pandas dataframe operations
return df
list_trainable_series = ... #a list of string
pool = mp.Pool(processes = (mp2.cpu_count() - 1))
lst_args = [(df1, serie_id, cursor) for serie_id in list_trainable_series]
results = pool.map(preprocess_serie_by_serie, lst_args)
pool.close()
pool.join()
results_df = pd.concat(results)
I test this code in local and it's working but i can not find a solution for Vertex AI.
Thanks for your help
LinkedIn
Twitter
