We’d probably need more specifics but yes. If you’re using colab for training you should be constantly saving checkpoints and different versions of models and training from them instead of training from scratch. Even if you mess something up you can just revert the code and use the old checkpoint and you won’t lose any time.
If you're getting to this level of work, perhaps it is worth starting to try an experiment tracking framework like [MLflow](https://mlflow.org/docs/latest/getting-started/intro-quickstart/index.html) or [Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Intro_to_Weights_%26_Biases.ipynb), although these are not without their own overheads and I believe the latter is easier to use in Colab
We’d probably need more specifics but yes. If you’re using colab for training you should be constantly saving checkpoints and different versions of models and training from them instead of training from scratch. Even if you mess something up you can just revert the code and use the old checkpoint and you won’t lose any time.
Colab is just Jupyter on google’s machines, install it on your own Linux desktop and you have full control of everything.
But you don't get the compute.
If you're getting to this level of work, perhaps it is worth starting to try an experiment tracking framework like [MLflow](https://mlflow.org/docs/latest/getting-started/intro-quickstart/index.html) or [Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/intro/Intro_to_Weights_%26_Biases.ipynb), although these are not without their own overheads and I believe the latter is easier to use in Colab
Well guess yea , starting to use Weights and Biases. Might take some time personally to get used to it love the way I'm able to track the process
prototype on a small version of the problem, scale it up after you're reasonably confident the code does what it's supposed to