3. Integrations

If you already have your data managed and organized on a cloud storage service, such as S3, you may want toutilize that with Dataloop, and not upload the binaries and create duplicates.

3.1. Amazon Web Services integration

Access & Permissions - Creating an integration with AWS requires allowing dataloop specific permissions for accessing the resource

To learn more about setting up integrations, please visit our Dataloop documentation

3.2. Create AWS Access-Key integration

import dtlpy as dl
if dl.token_expired():
    dl.login()
organization = dl.organizations.get(organization_name='my-org')
organization.integrations.create(name='S3integration', integrations_type=dl.ExternalStorage.S3,
                                 options={'key': "my_key", 'secret': "my_secret"})

3.3. Create AWS Cross Account integration

Follow these steps:

  1. Create an S3 bucket on your AWS account

  2. Create an IAM policy on your AWS account

  3. Create the integration and get an IAM user from DataloopTo learn more about setting up integrations, please visit our Dataloop documentation

To create the integration and get the IAM user follow this code snippet

import dtlpy as dl
if dl.token_expired():
    dl.login()
# Fetch your project id
project = dl.projects.get(project_id='<YOUR_PROJECT_ID>')
# Create a partial AWS Cross Account integration (before you add the IAM user to the Trust relationship of the role)
integration: dl.Integration = project.integrations.create(integrations_type=dl.IntegrationType.AWS_CROSS_ACCOUNT,
                                                          name='<YOUR_INTEGRATION_NAME>')
# value of the IAM user ARN
for metadata in integration.meatadata:
    if metadata['name'] == 'userArn':
        # value of the IAM user ARN
        iam_user_arn_value = metadata['value']
    elif metadata['name'] == 'status':
        # value of the integration - should be 'waiting-trust'
        integration_status = metadata['value']
# Go to your Role Trust relationship and add this IAM user ARN
print('The IAM User ARN is: ', iam_user_arn_value)
print('The Integration status is: ', integration_status)

  1. Create an IAM role that can access the bucket on your AWS account

  2. Add The IAM user to the trust relationship of the roleTo learn more about setting up integrations, please visit our Dataloop documentation

To update the integration and provide the IAM role ARN follow this code snippet

# After Adding the user ARN to your role trust relationship, continue and update the integration
integration.update(new_options={'roleArn': '<YOUR_IAMֹֹֹֹֹ_ROLE_ֹֹARN>'})
# checking the integration was created correctly
for metadata in integration.meatadata:
    if metadata['name'] == 'status':
        if metadata['value'] != 'trust-established':
            raise ValueError('ERROR: Integration was not setup correctly - please check the trust relationship in your IAM Role')