shell - Create a profile with databricks cli without passing arguments

Question

I am trying to use databricks-cli in a devops pipeline on azure. For that I need to create a profile, using:

databricks configure --profile Profile --token

The problem is that when I run that command, it asks me for host and token which breaks my pipeline waiting for user input. I would like to know if it is possible to do this without passing the arguments.

I already tried this:

echo "configuring databrick-cli authentication"

declare DATABRICKS_URL="https://westeurope.azuredatabricks.net"
declare DATABRICKS_ACCESS_TOKEN="authentication_token_generated_from_databricks_ux"

declare dbconfig=$(<~/.databrickscfg)
if [[ $dbconfig = *"host = "* && $dbconfig = *"token = "* ]]; then
  echo "file [~/.databrickscfg] is already configured"
else
  if [[ -z "$DATABRICKS_URL" || -z "$DATABRICKS_ACCESS_TOKEN" ]]; then
    echo "file [~/.databrickscfg] is not configured, but [DATABRICKS_URL],[DATABRICKS_ACCESS_TOKEN] env vars are not set"
  else
    echo "populating [~/.databrickscfg]"
    > ~/.databrickscfg
    echo "[DEFAULT]" >> ~/.databrickscfg
    echo "host = $DATABRICKS_URL" >> ~/.databrickscfg
    echo "token = $DATABRICKS_ACCESS_TOKEN" >> ~/.databrickscfg
    echo "" >> ~/.databrickscfg
  fi
fi

I am trying to put host and token in databrickscfg but it just hangs in there, I guess it is waiting for the user input again

What am I doing wrong?

question from:https://stackoverflow.com/questions/65889083/create-a-profile-with-databricks-cli-without-passing-arguments

Answer 1

You can achieve that by piping several values at the same time to configure command, like this is done for Databricks Connect here:

echo "$(databricks_host)
    $(databricks_token)" | databricks configure --token

But in reality, you can just use environment variables DATABRICKS_HOST and DATABRICKS_TOKEN as described in documentation - Databricks CLI will pickup them.

The only thing that you need to take into account - that sensitive data like Databricks token need to be stored securely in pipeline definition, and you'll need to access them special way - via env (full definition is here):

    - script: |
        echo "[{{cookiecutter.profile}}]" >> ~/.databrickscfg
        echo "host = $DATABRICKS_HOST" >> ~/.databrickscfg
        echo "token = $DATABRICKS_TOKEN" >> ~/.databrickscfg
      env:
        DATABRICKS_HOST: $(DATABRICKS_HOST)
        DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)

P.S. You can use cicd-templates project to generate a template for Azure DevOps (or for Github Actions) that works out of box with Databricks.