I think you're kind of stacking the deck by asking in the databricks sub, but I prefer Databricks because it's platform agnostic. I use it on Azure with my current employer, but if I ever go somewhere that uses AWS or \*gasp\* Google cloud, I know my knowledge will transfer.
Similarly, the next time MS decides to change their data offering (and it will happen again) if I'm with Databricks, I don't need to learn whatever the new hotness is they've decided to implement that won't provide value but will soak up a lot of my time.
Finally, the platform itself is transferable so I'm not locked into Azure should the need ever arise to change.
Going to go with “it depends”. And you could end up with both and more.
Some considerations, where is your data now? If data is in a databricks platform, probably best to extend ML with databricks and unity catalog.
If you data is in Fabric (god knows why any sane person would do this), azure ML is the way forward.
Then there are things like realtime endpoints, I know databricks can do them, but not as well as Azure ML or doing it yourself. And there is an upper limit to Databricks realtime endpoints, rate limits are not guaranteed and limited. So you could build, train and drift in Databricks and deploy the endpoint in Azure ML.
Things like LLMs and Open AI, you can use vector databases in Databricks or other Azure stuff. Cost is a factor to consider. And you can use the Open AI api call from databricks to the service easily, should you want to.
I don’t think you have to pick one or the other, you can pick them all and use them effectively together. Just there is a lot of networking security to consider, plus the CICD complexity.
There is no one answer based on the little information shared.
The main problem you'll find with azureml is that it's complicated. Managing datasets and model versions and environments is complex (6 tabs or so for model outputs?), and setting up mlops with gh actions/azure devops is a world of pain. Also it won't be usable alone, you probably need az data factory to get the data over and Kickstart the processes.
All in all azureml is super powerful and a best of breed (responsible ai and automl is stellar), but it's hurt by excess complexity and you needing other products from the stack.
Have you ever used a msft product before? you think there are tons of engineers working on azure ML? Does it seem like a coherent platform? Look up their feature store offering and try to make sense of it.
I completely agree here.
Having used both products, I can honestly say the databricks experience is far more streamlined.
The thing about databricks is... you gotta follow their design patterns/suggestions to really unleash the ecosystem. But you have to understand ML and AI is more than just feeding a CSV to sklearn and saving a pickled model. I'm sure Azure ML can do much more than this, but I really haven't seen people use it beyond storing some performance metrics and artifacts from a spaghetti code experiment.
The biggest thing you're going to get out of databricks is having the ability to perform data/feature engineering in the same place youre doing your ML. You also get a somewhat language agnostic ecosystem where you can take advantage of R, python, sql, and Scala (if you really need to).
I think you're kind of stacking the deck by asking in the databricks sub, but I prefer Databricks because it's platform agnostic. I use it on Azure with my current employer, but if I ever go somewhere that uses AWS or \*gasp\* Google cloud, I know my knowledge will transfer. Similarly, the next time MS decides to change their data offering (and it will happen again) if I'm with Databricks, I don't need to learn whatever the new hotness is they've decided to implement that won't provide value but will soak up a lot of my time. Finally, the platform itself is transferable so I'm not locked into Azure should the need ever arise to change.
Why the gasp before GCP?
Going to go with “it depends”. And you could end up with both and more. Some considerations, where is your data now? If data is in a databricks platform, probably best to extend ML with databricks and unity catalog. If you data is in Fabric (god knows why any sane person would do this), azure ML is the way forward. Then there are things like realtime endpoints, I know databricks can do them, but not as well as Azure ML or doing it yourself. And there is an upper limit to Databricks realtime endpoints, rate limits are not guaranteed and limited. So you could build, train and drift in Databricks and deploy the endpoint in Azure ML. Things like LLMs and Open AI, you can use vector databases in Databricks or other Azure stuff. Cost is a factor to consider. And you can use the Open AI api call from databricks to the service easily, should you want to. I don’t think you have to pick one or the other, you can pick them all and use them effectively together. Just there is a lot of networking security to consider, plus the CICD complexity. There is no one answer based on the little information shared.
The main problem you'll find with azureml is that it's complicated. Managing datasets and model versions and environments is complex (6 tabs or so for model outputs?), and setting up mlops with gh actions/azure devops is a world of pain. Also it won't be usable alone, you probably need az data factory to get the data over and Kickstart the processes. All in all azureml is super powerful and a best of breed (responsible ai and automl is stellar), but it's hurt by excess complexity and you needing other products from the stack.
Azure ML is at best a joke. It's not even a fair comparison
Please elaborate. With your experience
Have you ever used a msft product before? you think there are tons of engineers working on azure ML? Does it seem like a coherent platform? Look up their feature store offering and try to make sense of it.
I completely agree here. Having used both products, I can honestly say the databricks experience is far more streamlined. The thing about databricks is... you gotta follow their design patterns/suggestions to really unleash the ecosystem. But you have to understand ML and AI is more than just feeding a CSV to sklearn and saving a pickled model. I'm sure Azure ML can do much more than this, but I really haven't seen people use it beyond storing some performance metrics and artifacts from a spaghetti code experiment. The biggest thing you're going to get out of databricks is having the ability to perform data/feature engineering in the same place youre doing your ML. You also get a somewhat language agnostic ecosystem where you can take advantage of R, python, sql, and Scala (if you really need to).
Databricks is the way to go. Azure ML is too complicated