Azure 'azcopy sync' issue in syncing across containers using sas token for a multitenant app

Question

I am trying to sync data across Azure containers in different accounts using a Multitetant App and azcopy tool.

The syncing happens via "azcopy sync" and using separate SAS tokens for both source storage account and destination storage account.

I am generating short lived sas tokens using the Java SDK following the user delegation key method.

Here is the scenario:

Account1 (destination) has App1 registered. i.e. Account1 is home tenant for App1. Account1 has StorageAccount1 and Container1 configured App1 is given "Storage Blob Data Contributor" role on StorageAccount1

Account2 (source) has StorageAccount2 and Container2 configured. It is the data source for us. Here, App1 is added as a ServicePrincipal via:

az ad sp create --id client-id-of-App1-in-Account1

In Account2, for this SP, we also gave the Storage Blob Data Reader Role as:

az role assignment create 
   --assignee-object-id <object-id-for-this-sp> 
   --role 2a2b9908-6ea1-4ae2-8e65-a410df84e7d1 
   --scope /subscriptions/<subsid-account2>/resourceGroups/<resgrpname>/providers/Microsoft.Storage/storageAccounts/<storagename>

This completes the setup.

Now using Java SDK, I generated a user delegation key for both source and destination. The snippet looks something like below.

genSasToken(String storageAccountName, String containerName,
                             String tenantId,
                             String azureAppClientId,
                             String azureAppClientSecret,
                             boolean isDestinationAccount) {
    BlobContainerSasPermission blobContainerSasPermission =
        new BlobContainerSasPermission().setReadPermission(true).setListPermission(true);
    if (isDestinationAccount) {
      blobContainerSasPermission.setCreatePermission(true)
              .setAddPermission(true)
              .setWritePermission(true)
              .setExecutePermission(true);
    }
    BlobServiceSasSignatureValues builder =
        new BlobServiceSasSignatureValues(OffsetDateTime.now().plusHours(1), blobContainerSasPermission)
            .setProtocol(SasProtocol.HTTPS_ONLY);
    // Create a BlobServiceClient object which will be used to create a container client
    String endpoint = String.format(Locale.ROOT, "https://%s.blob.core.windows.net",
            storageAccountName);
    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
                                                        .clientId(azureAppClientId)
                                                        .clientSecret(azureAppClientSecret)
                                                        .tenantId(tenantId)
                                                        .build();

    BlobServiceClient blobServiceClient =
        new BlobServiceClientBuilder().endpoint(endpoint).credential(clientSecretCredential).buildClient();
    BlobContainerClient blobContainerClient =
        blobServiceClient.getBlobContainerClient(containerName);
    // Get a user delegation key for the Blob service that's valid for one hour.
    // You can use the key to generate any number of shared access signatures over the lifetime of the key.
    OffsetDateTime keyStart = OffsetDateTime.now();
    OffsetDateTime keyExpiry = OffsetDateTime.now().plusHours(1);
    UserDelegationKey userDelegationKey = blobServiceClient.getUserDelegationKey(keyStart, keyExpiry);

    String sas = blobContainerClient.generateUserDelegationSas(builder, userDelegationKey);
    return sas;
  }

Above method is called for both source and destination and gives us SAS tokens generated programmatically.

Interesting thing happening is this:

azcopy sync https://storageaccount2/container2/?sas-token-for2 https://storageaccount1/container1/?sas-token-for1

above sync errors out as

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /Users/runner/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=This request is not authorized to perform this operation.
RequestId:xxx
Time:2021-01-27T10:26:34.9282634Z, Details:
   Code: AuthorizationFailure
   GET https://storageaccount1.blob.core.windows.net/container1/?comp=properties&restype=account&se=2021-01-27t11%3A10%3A12z&sig=-REDACTED-&ske=2021-01-27t11%3A10%3A12z&skoid=xxx&sks=b&skt=2021-01-27t10%3A10%3A12z&sktid=xxx&skv=2020-02-10&sp=racwle&spr=https&sr=c&sv=2020-02-10&timeout=901
   User-Agent: [AzCopy/10.8.0 Azure-Storage/0.10 (go1.13; darwin)]
   X-Ms-Client-Request-Id: [xxx]
   X-Ms-Version: [2019-12-12]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [246]
   Content-Type: [application/xml]
   Date: [Wed, 27 Jan 2021 10:26:34 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [xxx]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [xxx]
   X-Ms-Version: [2019-12-12]

But, when I try to copy from source to localhost using same sas token 2, it works.

azcopy sync https://storageaccount2/container2/sas-token-for2 /tmp

and

when I try to copy a localhost folder to destination using same sas token it also works.

azcopy sync /tmp https://storageaccount1/container1/sas-token-for1

So the tokens work individually like above.

But azcopy sync https://storageaccount2/container2/sas-token-for2 https://storageaccount1/container1/sas-token-for1

Fails.

Any pointers what might be the issue here?

question from:https://stackoverflow.com/questions/65923143/azure-azcopy-sync-issue-in-syncing-across-containers-using-sas-token-for-a-mul

Answer 1

For syncing you don't need execution permission (which is still in preview in any case). Just remove .setExecutePermission(true) you should be good. In fact syncing should work with only read, write and list permission on destination.