The Cloud Run logs don't show any errors or anything that could indicate a problem. This is the code I'm calling from the Flask endpoint:
The run_spider is called by the Flask endpoint. It should wait the spider finish before returning the result, but it isn't. In other words, it should block on line 18 (result = queue.get(...)) for some seconds, before returning the result, but instead, the request is returned in less than 1 second, meaning that the spider process didn't run.
def run_spider():
def script(queue):
settings = get_project_settings()
process = CrawlerProcess(settings)
process.crawl("SpiderName")
process.start()
queue.put(None)
get_or_create_eventloop()
queue = Queue()
try:
main_process = Process(target=script, args=(queue,))
main_process.start()
main_process.join()
result = queue.get(block=True, timeout=None)
if result is not None:
raise result
return result
except Exception as e:
queue.put(e)
raise e
def get_or_create_eventloop():
try:
return asyncio.get_event_loop()
except RuntimeError as ex:
if "There is no current event loop in thread" in str(ex):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
return asyncio.get_event_loop()
if __name__ == '__main__':
app.run()
LinkedIn
Twitter
