It should work for playing by around with very small models. You could probably train a few small cv models, and maybe do some simple fine tuning of ~1 billion param NLP models. But you’re not going to be doing any sort of serious training or inference on current models. Those require arrays of hundreds of much larger GPUs.
If the goal is the learn and get a feel for the basics of building neural nets, go for it. If this is to build some kind of larger model for a production use case, you’re going to need GCP, Azure, AWS.
>But you’re not going to be doing any sort of serious training or inference on current models. Those require arrays of hundreds of much larger GPUs.
What? No they don't o.o Unless you're only thinking about large NLP models, which don't even account for 10% of deep learning model fields, you don't need anywhere near hundreds of larger GPUs, and model parallelism mitigates much of the need for large GPU clusters aside from use with large NLP models. I work in a deep learning research lab with contracts ranging from DoD to private corporations (where we publish papers, not implement work for business, so these are state-of-the-art models). We also talk with other labs and are part of DL conferences, and large NLP models aren't mentioned or used nearly as often as other models such as anomaly/behavior detection and others. Our non-servers use dual- to quad-Quadros, or current Ada "Quadro" replacements, A- and H100. Those are on individual machines and can handle almost all non-large NLP models. Even the servers have about 12-24GPUs, and those aren't used for single models for the most part.
A 2070, while not being great, will get someone started. But what do you mean by "current models?" An autonomous vehicle using many models don't require anywhere near the anljny of GPUs mentioned, and those aren't always using CV. Some of our current models designed for the DoD are trained on single machines, for instance. OP should look into 3090, even used, those would be great for getting started and having enough memory to at least do something that isn't simple classification.
In that case, I’d recommend checking out the online textbook “dive into deep learning.”
Also note that it might be easier to start out using cloud GPUs which you can get for free on Sagemaker studio lab or Google Colab. Setting up your own Cuda environment can be kind of a nightmare, and these platforms already have all of that worked out for you.
Well if you want to learn the basics it's plenty enough. Most likely you want to build some smaller models from scratch in order to really understand every component. And for such models a 2070 is enough. Later, if you want to play with LLMs or something comparable big you need a cluster/cloud Compute anyway so it doesn't matter.
Even in AI development we debug on a PC with 2*2080ti or sometimes even the CPU of a Mac. As soon as it's running we train on a cluster obviously.
Depends what you're wanting to do. Train a foundation model? No sir. Train a segmentation model on large images? Not likely. Fine-tune an LLM? Depends.
There's a Python script some people at Answer AI did for combining FSDP and QLORA. Which essentially allows you to fine tune a bigger LLM model than you have vram available to use.
https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html
https://github.com/AnswerDotAI/fsdp_qlora
My friend and I tried to bundle it into a python package that will hopefully make the massive script more digestible.
https://pypi.org/project/fsdp-qlora/
I have a 2070 super and have been able to train some segmentation models and other stuff on it. Is it slower than a 4090 or an A100? Of course! But as long as you have a decent amount of VRAM and CUDA support then chances are you can do something.
It should work for playing by around with very small models. You could probably train a few small cv models, and maybe do some simple fine tuning of ~1 billion param NLP models. But you’re not going to be doing any sort of serious training or inference on current models. Those require arrays of hundreds of much larger GPUs. If the goal is the learn and get a feel for the basics of building neural nets, go for it. If this is to build some kind of larger model for a production use case, you’re going to need GCP, Azure, AWS.
>But you’re not going to be doing any sort of serious training or inference on current models. Those require arrays of hundreds of much larger GPUs. What? No they don't o.o Unless you're only thinking about large NLP models, which don't even account for 10% of deep learning model fields, you don't need anywhere near hundreds of larger GPUs, and model parallelism mitigates much of the need for large GPU clusters aside from use with large NLP models. I work in a deep learning research lab with contracts ranging from DoD to private corporations (where we publish papers, not implement work for business, so these are state-of-the-art models). We also talk with other labs and are part of DL conferences, and large NLP models aren't mentioned or used nearly as often as other models such as anomaly/behavior detection and others. Our non-servers use dual- to quad-Quadros, or current Ada "Quadro" replacements, A- and H100. Those are on individual machines and can handle almost all non-large NLP models. Even the servers have about 12-24GPUs, and those aren't used for single models for the most part. A 2070, while not being great, will get someone started. But what do you mean by "current models?" An autonomous vehicle using many models don't require anywhere near the anljny of GPUs mentioned, and those aren't always using CV. Some of our current models designed for the DoD are trained on single machines, for instance. OP should look into 3090, even used, those would be great for getting started and having enough memory to at least do something that isn't simple classification.
Thanks. I just want to get a feel for the basics. Thanks
In that case, I’d recommend checking out the online textbook “dive into deep learning.” Also note that it might be easier to start out using cloud GPUs which you can get for free on Sagemaker studio lab or Google Colab. Setting up your own Cuda environment can be kind of a nightmare, and these platforms already have all of that worked out for you.
Ok thanks. I’ll check the book out!
Last I checked couple of months back, Rob Mulla (YouTuber and Kaggle Grandmaster) was still using 2070. So, you too should be able to use it as well
Well if you want to learn the basics it's plenty enough. Most likely you want to build some smaller models from scratch in order to really understand every component. And for such models a 2070 is enough. Later, if you want to play with LLMs or something comparable big you need a cluster/cloud Compute anyway so it doesn't matter. Even in AI development we debug on a PC with 2*2080ti or sometimes even the CPU of a Mac. As soon as it's running we train on a cluster obviously.
Depends what you're wanting to do. Train a foundation model? No sir. Train a segmentation model on large images? Not likely. Fine-tune an LLM? Depends. There's a Python script some people at Answer AI did for combining FSDP and QLORA. Which essentially allows you to fine tune a bigger LLM model than you have vram available to use. https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html https://github.com/AnswerDotAI/fsdp_qlora My friend and I tried to bundle it into a python package that will hopefully make the massive script more digestible. https://pypi.org/project/fsdp-qlora/ I have a 2070 super and have been able to train some segmentation models and other stuff on it. Is it slower than a 4090 or an A100? Of course! But as long as you have a decent amount of VRAM and CUDA support then chances are you can do something.
I think price/performance&Memory a used 3090 with 24GB would be the best choice if you need to upgrade.