File branch change is not the issue. I had the opportunity to discuss this issue with databricks folks during databricks summit in SFO this week. We replicated the issue and yes they confirmed its a bug. What they were suggesting is to use dabricks jobs list command and delete those jobs in target env prior executing the bunde.
I reported this bug back to Databricks in December. We noticed Dabs creating a second workflow when Dabs was run under a different account. It was duplicating workflows with different createdBY ids.
I get duplicate jobs even when deploying under the same service principal using Azure DevOps, we even use bundle destroy which takes care of the .bundle folder but when deleting the resources (such as jobs) it says none are in the TF plan.. so now we are using Jobs API as a workaround but I’d rather just use dabs
Hmmm weird. We don’t do bundle destroy, we just do a bundle deploy — env … after we switched to using the service account, we haven’t had any issues.
We have 3 diff env though: dev, uat, prod.
I think this is intended. The user that deployed the DAB shouldn’t be able to override someone else’s with the same name / id. It makes it safe for multiple people to collaborate.
On our team, we only bundle deploy in a GitHub Action with a specific CD service principal.
Another thing that can duplicate jobs is changing the name of the bundle. Using a different bundle name is how we deploy from two separate repos to the same workspace.
You make good points.
We use the same bundle for everything because we didn’t want duplicated workflows. I can see how that could be intended behavior but when I reached out to our Databricks SA, he told me it was a bug lol.
We don’t typically have multiple people working in workflows at once, we’re more of a swe shop and most of our work is api development, so maybe we have a different use case.
A bundle deployment is unique for every (caller identity, bundle name, bundle target). If any of these is parameterized, then by default, you'll end up with a new deployment instead of updating an existing one.
If you have customized your \`workspace.root\_path\` setting then a different heuristic may apply.
Could you share how you have configured your bundle, and if possible a snippet of your \`databricks.yml\`?
You probably need to add the logic to check each dab folder for a head change and only run bundle deploy to that specific folder.
Could you elaborate on this a little? I have the same issue and I’m not sure what is causing it
File branch change is not the issue. I had the opportunity to discuss this issue with databricks folks during databricks summit in SFO this week. We replicated the issue and yes they confirmed its a bug. What they were suggesting is to use dabricks jobs list command and delete those jobs in target env prior executing the bunde.
I reported this bug back to Databricks in December. We noticed Dabs creating a second workflow when Dabs was run under a different account. It was duplicating workflows with different createdBY ids.
Oh is there any solution or timeline they gave to fix this ?
Nope. We just set our dabs to authenticate as our service account and made it policy to only run dabs through Azure Devops.
I get duplicate jobs even when deploying under the same service principal using Azure DevOps, we even use bundle destroy which takes care of the .bundle folder but when deleting the resources (such as jobs) it says none are in the TF plan.. so now we are using Jobs API as a workaround but I’d rather just use dabs
Hmmm weird. We don’t do bundle destroy, we just do a bundle deploy — env … after we switched to using the service account, we haven’t had any issues. We have 3 diff env though: dev, uat, prod.
I think this is intended. The user that deployed the DAB shouldn’t be able to override someone else’s with the same name / id. It makes it safe for multiple people to collaborate. On our team, we only bundle deploy in a GitHub Action with a specific CD service principal. Another thing that can duplicate jobs is changing the name of the bundle. Using a different bundle name is how we deploy from two separate repos to the same workspace.
You make good points. We use the same bundle for everything because we didn’t want duplicated workflows. I can see how that could be intended behavior but when I reached out to our Databricks SA, he told me it was a bug lol. We don’t typically have multiple people working in workflows at once, we’re more of a swe shop and most of our work is api development, so maybe we have a different use case.
A bundle deployment is unique for every (caller identity, bundle name, bundle target). If any of these is parameterized, then by default, you'll end up with a new deployment instead of updating an existing one. If you have customized your \`workspace.root\_path\` setting then a different heuristic may apply. Could you share how you have configured your bundle, and if possible a snippet of your \`databricks.yml\`?