Hi @MaxiChava,
To my understanding, you are wondering why the training progressed more than the budgeted 8 node hours, given that the "Enable early stopping" is activated hence cancelling the training because it has been running more than 18 hours and you are worried to be billed accordingly.
Let me itemize these components:
- Budget/ Node hours - is the unit of measure used to track Vertex AI resources usage. For example, running a prediction job that uses 1 machine for 1 hour, then you are charged for 1 node hour. Another example is, if an endpoint was deployed with 4 machines and if it is active for 24 hours, then you will be charged for 96 node hours. See Prediction and explanation document for detailed pricing information.
A node hour represents the time a virtual machine spends running your prediction job or waiting in an active state (an endpoint with one or more models deployed) to handle prediction or explanation requests.
- Enable early stopping - if enabled, Vertex AI will monitor the metric that you specified and stop training the model when it reaches the threshold or the number of steps that you specified. This can help you save time and money because you will not be paying for the used model-training resources beyond the specified metric threshold or steps.
- Billing - unfortunately you will still be charged because it is a user-initiated cancellation, see published Pricing for AutoML models document for your reference.
You pay only for compute hours used; if training fails for any reason other than a user-initiated cancellation, you are not billed for the time. You are charged for training time if you cancel the operation.
To sum it up, the node hours will not trigger the early stopping feature because it is not related and you will still be billed for the incurred hours since you initiated the cancellation.
Hope this helps.
LinkedIn
Twitter
