Repositories¶
Organizations¶
- class Organizations(client_api: ApiClient)[source]¶
Bases:
object
Organizations Repository
Read our documentation to learn more about Organizations in the Dataloop platform.
- add_member(email: str, role: MemberOrgRole = MemberOrgRole.MEMBER, organization_id: Optional[str] = None, organization_name: Optional[str] = None, organization: Optional[Organization] = None)[source]¶
Add members to your organization. Read about members and groups here.
Prerequisities: To add members to an organization, you must be an owner in that organization.
You must provide at least ONE of the following params: organization, organization_name, or organization_id.
- Parameters:
- Returns:
True if successful or error if unsuccessful
- Return type:
Example:
success = dl.organizations.add_member(email='user@domain.com', organization_id='organization_id', role=dl.MemberOrgRole.MEMBER)
- cache_action(organization_id: Optional[str] = None, organization_name: Optional[str] = None, organization: Optional[Organization] = None, mode=CacheAction.APPLY, pod_type=PodType.SMALL)[source]¶
Add or remove Cache for the org
Prerequisites: You must be an organization owner
You must provide at least ONE of the following params: organization, organization_name, or organization_id.
- Parameters:
- Returns:
True if success
- Return type:
Example:
success = dl.organizations.enable_cache(organization_id='organization_id', mode=dl.CacheAction.APPLY)
- delete_member(user_id: str, organization_id: Optional[str] = None, organization_name: Optional[str] = None, organization: Optional[Organization] = None, sure: bool = False, really: bool = False) bool [source]¶
Delete member from the Organization.
Prerequisites: Must be an organization owner to delete members.
You must provide at least ONE of the following params: organization_id, organization_name, organization.
- Parameters:
- Returns:
True if success and error if not
- Return type:
Example:
success = dl.organizations.delete_member(user_id='user_id', organization_id='organization_id', sure=True, really=True)
- get(organization_id: Optional[str] = None, organization_name: Optional[str] = None, fetch: Optional[bool] = None) Organization [source]¶
Get Organization object to be able to use it in your code.
Prerequisites: You must be a superuser to use this method.
You must provide at least ONE of the following params: organization_name or organization_id.
- Parameters:
- Returns:
Organization object
- Return type:
Example:
org = dl.organizations.get(organization_id='organization_id')
- list() List[Organization] [source]¶
Lists all the organizations in Dataloop.
Prerequisites: You must be a superuser to use this method.
- Returns:
List of Organization objects
- Return type:
Example:
organizations = dl.organizations.list()
- list_groups(organization: Optional[Organization] = None, organization_id: Optional[str] = None, organization_name: Optional[str] = None)[source]¶
List all organization groups (groups that were created within the organization).
Prerequisites: You must be an organization owner to use this method.
You must provide at least ONE of the following params: organization, organization_name, or organization_id.
- Parameters:
- Returns:
groups list
- Return type:
Example:
groups_list = dl.organizations.list_groups(organization_id='organization_id')
- list_integrations(organization: Optional[Organization] = None, organization_id: Optional[str] = None, organization_name: Optional[str] = None, only_available=False)[source]¶
List all organization integrations with external cloud storage.
Prerequisites: You must be an organization owner to use this method.
You must provide at least ONE of the following params: organization_id, organization_name, or organization.
- Parameters:
- Returns:
integrations list
- Return type:
Example:
list_integrations = dl.organizations.list_integrations(organization='organization-entity', only_available=True)
- list_members(organization: Optional[Organization] = None, organization_id: Optional[str] = None, organization_name: Optional[str] = None, role: Optional[MemberOrgRole] = None)[source]¶
List all organization members.
Prerequisites: You must be an organization owner to use this method.
You must provide at least ONE of the following params: organization_id, organization_name, or organization.
- Parameters:
- Returns:
projects list
- Return type:
Example:
list_members = dl.organizations.list_members(organization='organization-entity', role=dl.MemberOrgRole.MEMBER)
- update(plan: str, organization: Optional[Organization] = None, organization_id: Optional[str] = None, organization_name: Optional[str] = None) Organization [source]¶
Update an organization.
Prerequisites: You must be a superuser to update an organization.
You must provide at least ONE of the following params: organization, organization_name, or organization_id.
- Parameters:
- Returns:
organization object
- Return type:
Example:
org = dl.organizations.update(organization='organization-entity', plan=dl.OrganizationsPlans.FREEMIUM)
- update_member(email: str, role: MemberOrgRole = MemberOrgRole.MEMBER, organization_id: Optional[str] = None, organization_name: Optional[str] = None, organization: Optional[Organization] = None)[source]¶
Update member role.
Prerequisites: You must be an organization owner to update a member’s role.
You must provide at least ONE of the following params: organization, organization_name, or organization_id.
- Parameters:
- Returns:
json of the member fields
- Return type:
Example:
member_json = dl.organizations.update_member(email='user@domain.com', organization_id='organization_id', role=dl.MemberOrgRole.MEMBER)
Integrations¶
Integrations Repository
- class Integrations(client_api: ApiClient, org: Optional[Organization] = None, project: Optional[Project] = None)[source]¶
Bases:
object
Integrations Repository
The Integrations class allows you to manage data integrations from your external storage (e.g., S3, GCS, Azure) into your Dataloop’s Dataset storage, as well as sync data in your Dataloop’s Datasets with data in your external storage.
For more information on Organization Storage Integration see the Dataloop documentation and developers’ docs.
- create(integrations_type: IntegrationType, name: str, options: dict)[source]¶
Create an integration between an external storage and the organization.
Examples for options include: s3 - {key: “”, secret: “”}; gcs - {key: “”, secret: “”, content: “”}; azureblob - {key: “”, secret: “”, clientId: “”, tenantId: “”}; key_value - {key: “”, value: “”} aws-sts - {key: “”, secret: “”, roleArns: “”} aws-cross - {} gcp-cross - {}
Prerequisites: You must be an owner in the organization.
- Parameters:
integrations_type (IntegrationType) – integrations type dl.IntegrationType
name (str) – integrations name
options (dict) – dict of storage secrets
- Returns:
success
- Return type:
Example:
project.integrations.create(integrations_type=dl.IntegrationType.S3, name='S3ntegration', options={key: "Access key ID", secret: "Secret access key"})
- delete(integrations_id: str, sure: bool = False, really: bool = False) bool [source]¶
Delete integrations from the organization.
Prerequisites: You must be an organization owner to delete an integration.
- Parameters:
- Returns:
success
- Return type:
Example:
project.integrations.delete(integrations_id='integrations_id', sure=True, really=True)
- get(integrations_id: str)[source]¶
Get organization integrations. Use this method to access your integration and be able to use it in your code.
Prerequisites: You must be an owner in the organization.
- Parameters:
integrations_id (str) – integrations id
- Returns:
Integration object
- Return type:
Example:
project.integrations.get(integrations_id='integrations_id')
- list(only_available=False)[source]¶
List all the organization’s integrations with external storage.
Prerequisites: You must be an owner in the organization.
- Parameters:
only_available (bool) – if True list only the available integrations.
- Returns:
groups list
- Return type:
Example:
project.integrations.list(only_available=True)
- update(new_name: Optional[str] = None, integrations_id: Optional[str] = None, integration: Optional[Integration] = None, new_options: Optional[dict] = None)[source]¶
Update the integration’s name.
Prerequisites: You must be an owner in the organization.
- Parameters:
new_name (str) – new name
integrations_id (str) – integrations id
integration (Integration) – integration object
new_options (dict) – new value
- Returns:
Integration object
- Return type:
Examples for options include: s3 - {key: “”, secret: “”}; gcs - {key: “”, secret: “”, content: “”}; azureblob - {key: “”, secret: “”, clientId: “”, tenantId: “”}; key_value - {key: “”, value: “”} aws-sts - {key: “”, secret: “”, roleArns: “”} aws-cross - {roleArn: “”} gcp-cross - {“email: “”, “resourceName”: “”}
Example:
project.integrations.update(integrations_id='integrations_id', new_options={roleArn: ""})
Projects¶
- class Projects(client_api: ApiClient, org=None)[source]¶
Bases:
object
Projects Repository
The Projects class allows the user to manage projects and their properties.
For more information on Projects see the Dataloop documentation.
- add_member(email: str, project_id: str, role: MemberRole = MemberRole.DEVELOPER)[source]¶
Add a member to the project.
Prerequisites: You must be in the role of an owner to add a member to a project.
- Parameters:
- Returns:
dict that represent the user
- Return type:
Example:
user_json = dl.projects.add_member(project_id='project_id', email='user@dataloop.ai', role=dl.MemberRole.DEVELOPER)
- checkout(identifier: Optional[str] = None, project_name: Optional[str] = None, project_id: Optional[str] = None, project: Optional[Project] = None)[source]¶
Checkout (switch) to a project to work on.
Prerequisites: All users can open a project in the web.
You must provide at least ONE of the following params: project_id, project_name.
- Parameters:
identifier (str) – project name or partial id that you wish to switch
project_name (str) – The Name of the project
project_id (str) – The Id of the project
project (dtlpy.entities.project.Project) – project object
Example:
dl.projects.checkout(project_id='project_id')
- create(project_name: str, checkout: bool = False) Project [source]¶
Create a new project.
Prerequisites: Any user can create a project.
- Parameters:
- Returns:
Project object
- Return type:
Example:
project = dl.projects.create(project_name='project_name')
- delete(project_name: Optional[str] = None, project_id: Optional[str] = None, sure: bool = False, really: bool = False) bool [source]¶
Delete a project forever!
Prerequisites: You must be in the role of an owner to delete a project.
- Parameters:
- Returns:
True if success, error if not
- Return type:
Example:
is_deleted = dl.projects.delete(project_id='project_id', sure=True, really=True)
- get(project_name: Optional[str] = None, project_id: Optional[str] = None, checkout: bool = False, fetch: Optional[bool] = None, log_error=True) Project [source]¶
Get a Project object.
Prerequisites: You must be in the role of an owner to get a project object.
You must check out to a project or provide at least one of the following params: project_id, project_name
- Parameters:
- Returns:
Project object
- Return type:
Example:
project = dl.projects.get(project_id='project_id')
- list() List[Project] [source]¶
Get the user’s project list
Prerequisites: You must be a superuser to list all users’ projects.
- Returns:
List of Project objects
Example:
projects = dl.projects.list()
- list_members(project: Project, role: Optional[MemberRole] = None)[source]¶
Get a list of the project members.
Prerequisites: You must be in the role of an owner to list project members.
- Parameters:
project (dtlpy.entities.project.Project) – Project object
role – The required role for the user. Use the enum dl.MemberRole
- Returns:
list of the project members
- Return type:
Example:
users_jsons_list = dl.projects.list_members(project_id='project_id', role=dl.MemberRole.DEVELOPER)
- open_in_web(project_name: Optional[str] = None, project_id: Optional[str] = None, project: Optional[Project] = None)[source]¶
Open the project in our web platform.
Prerequisites: All users can open a project in the web.
- Parameters:
project_name (str) – The Name of the project
project_id (str) – The Id of the project
project (dtlpy.entities.project.Project) – project object
Example:
dl.projects.open_in_web(project_id='project_id')
- remove_member(email: str, project_id: str)[source]¶
Remove a member from the project.
Prerequisites: You must be in the role of an owner to delete a member from a project.
- Parameters:
- Returns:
dict that represents the user
- Return type:
Example:
user_json = dl.projects.remove_member(project_id='project_id', email='user@dataloop.ai')
- update(project: Project, system_metadata: bool = False) Project [source]¶
Update a project information (e.g., name, member roles, etc.).
Prerequisites: You must be in the role of an owner to add a member to a project.
- Parameters:
project (dtlpy.entities.project.Project) – project object
system_metadata (bool) – optional - True, if you want to change metadata system
- Returns:
Project object
- Return type:
Example:
project = dl.projects.delete(project='project_entity')
- update_member(email: str, project_id: str, role: MemberRole = MemberRole.DEVELOPER)[source]¶
Update member’s information/details in the project.
Prerequisites: You must be in the role of an owner to update a member.
- Parameters:
- Returns:
dict that represent the user
- Return type:
Example:
user_json = = dl.projects.update_member(project_id='project_id', email='user@dataloop.ai', role=dl.MemberRole.DEVELOPER)
Datasets¶
Datasets Repository
- class Datasets(client_api: ApiClient, project: Optional[Project] = None)[source]¶
Bases:
object
Datasets Repository
The Datasets class allows the user to manage datasets. Read more about datasets in our documentation and SDK documentation.
- checkout(identifier: Optional[str] = None, dataset_name: Optional[str] = None, dataset_id: Optional[str] = None, dataset: Optional[Dataset] = None)[source]¶
Checkout (switch) to a dataset to work on it.
Prerequisites: You must be an owner or developer to use this method.
You must provide at least ONE of the following params: dataset_id, dataset_name.
- Parameters:
identifier (str) – project name or partial id that you wish to switch
dataset_name (str) – The Name of the dataset
dataset_id (str) – The Id of the dataset
dataset (dtlpy.entities.dataset.Dataset) – dataset object
Example:
project.datasets.checkout(dataset_id='dataset_id')
- clone(dataset_id: str, clone_name: str, filters: Optional[Filters] = None, with_items_annotations: bool = True, with_metadata: bool = True, with_task_annotations_status: bool = True)[source]¶
Clone a dataset. Read more about cloning datatsets and items in our documentation and SDK documentation.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
dataset_id (str) – id of the dataset you wish to clone
clone_name (str) – new dataset name
filters (dtlpy.entities.filters.Filters) – Filters entity or a query dict
with_items_annotations (bool) – true to clone with items annotations
with_metadata (bool) – true to clone with metadata
with_task_annotations_status (bool) – true to clone with task annotations’ status
- Returns:
dataset object
- Return type:
Example:
dataset = project.datasets.clone(dataset_id='dataset_id', clone_name='dataset_clone_name', with_metadata=True, with_items_annotations=False, with_task_annotations_status=False)
- create(dataset_name: str, labels=None, attributes=None, ontology_ids=None, driver: Optional[Driver] = None, driver_id: Optional[str] = None, checkout: bool = False, expiration_options: Optional[ExpirationOptions] = None, index_driver: Optional[IndexDriver] = None, recipe_id: Optional[str] = None) Dataset [source]¶
Create a new dataset
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
dataset_name (str) – The Name of the dataset
labels (list) – dictionary of {tag: color} or list of label entities
attributes (list) – dataset’s ontology’s attributes
ontology_ids (list) – optional - dataset ontology
driver (dtlpy.entities.driver.Driver) – optional - storage driver Driver object or driver name
driver_id (str) – optional - driver id
checkout (bool) – set the dataset as a default dataset object (cookies)
expiration_options (ExpirationOptions) – dl.ExpirationOptions object that contain definitions for dataset like MaxItemDays
index_driver (str) – dl.IndexDriver, dataset driver version
recipe_id (str) – optional - recipe id
- Returns:
Dataset object
- Return type:
Example:
dataset = project.datasets.create(dataset_name='dataset_name', ontology_ids='ontology_ids')
- delete(dataset_name: Optional[str] = None, dataset_id: Optional[str] = None, sure: bool = False, really: bool = False)[source]¶
Delete a dataset forever!
Prerequisites: You must be an owner or developer to use this method.
Example:
is_deleted = project.datasets.delete(dataset_id='dataset_id', sure=True, really=True)
- directory_tree(dataset: Optional[Dataset] = None, dataset_name: Optional[str] = None, dataset_id: Optional[str] = None)[source]¶
Get dataset’s directory tree.
Prerequisites: You must be an owner or developer to use this method.
You must provide at least ONE of the following params: dataset, dataset_name, dataset_id.
- Parameters:
dataset (dtlpy.entities.dataset.Dataset) – dataset object
dataset_name (str) – The Name of the dataset
dataset_id (str) – The Id of the dataset
- Returns:
DirectoryTree
Example:
- static download_annotations(dataset: Dataset, local_path: Optional[str] = None, filters: Optional[Filters] = None, annotation_options: Optional[ViewAnnotationOptions] = None, annotation_filters: Optional[Filters] = None, overwrite: bool = False, thickness: int = 1, with_text: bool = False, remote_path: Optional[str] = None, include_annotations_in_output: bool = True, export_png_files: bool = False, filter_output_annotations: bool = False, alpha: Optional[float] = None, export_version=ExportVersion.V1) str [source]¶
Download dataset’s annotations by filters.
You may filter the dataset both for items and for annotations and download annotations.
Optional – download annotations as: mask, instance, image mask of the item.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
dataset (dtlpy.entities.dataset.Dataset) – dataset object
local_path (str) – local folder or filename to save to.
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
annotation_options (list) – type of download annotations: list(dl.ViewAnnotationOptions)
annotation_filters (dtlpy.entities.filters.Filters) – Filters entity to filter annotations for download
overwrite (bool) – optional - default = False to overwrite the existing files
thickness (int) – optional - line thickness, if -1 annotation will be filled, default =1
with_text (bool) – optional - add text to annotations, default = False
remote_path (str) – DEPRECATED and ignored
include_annotations_in_output (bool) – default - False , if export should contain annotations
export_png_files (bool) – default - if True, semantic annotations should be exported as png files
filter_output_annotations (bool) – default - False, given an export by filter - determine if to filter out annotations
alpha (float) – opacity value [0 1], default 1
export_version (str) – exported items will have original extension in filename, V1 - no original extension in filenames
- Returns:
local_path of the directory where all the downloaded item
- Return type:
Example:
file_path = project.datasets.download_annotations(dataset='dataset_entity', local_path='local_path', annotation_options=dl.ViewAnnotationOptions, overwrite=False, thickness=1, with_text=False, alpha=1 )
- get(dataset_name: Optional[str] = None, dataset_id: Optional[str] = None, checkout: bool = False, fetch: Optional[bool] = None) Dataset [source]¶
Get dataset by name or id.
Prerequisites: You must be an owner or developer to use this method.
You must provide at least ONE of the following params: dataset_id, dataset_name.
- Parameters:
- Returns:
Dataset object
- Return type:
Example:
dataset = project.datasets.get(dataset_id='dataset_id')
- list(name=None, creator=None) List[Dataset] [source]¶
List all datasets.
Prerequisites: You must be an owner or developer to use this method.
- Parameters:
- Returns:
List of datasets
- Return type:
Example:
datasets = project.datasets.list(name='name')
- merge(merge_name: str, dataset_ids: list, project_ids: str, with_items_annotations: bool = True, with_metadata: bool = True, with_task_annotations_status: bool = True, wait: bool = True)[source]¶
Merge a dataset. See our SDK docs for more information.
Prerequisites: You must be an owner or developer to use this method.
- Parameters:
merge_name (str) – new dataset name
dataset_ids (list) – list id’s of the datatsets you wish to merge
project_ids (str) – the project id that include the datasets
with_items_annotations (bool) – true to merge with items annotations
with_metadata (bool) – true to merge with metadata
with_task_annotations_status (bool) – true to merge with task annotations’ status
wait (bool) – wait for the command to finish
- Returns:
True if success
- Return type:
Example:
success = project.datasets.merge(dataset_ids=['dataset_id1','dataset_id2'], merge_name='dataset_merge_name', with_metadata=True, with_items_annotations=False, with_task_annotations_status=False)
- open_in_web(dataset_name: Optional[str] = None, dataset_id: Optional[str] = None, dataset: Optional[Dataset] = None)[source]¶
Open the dataset in web platform.
Prerequisites: You must be an owner or developer to use this method.
- Parameters:
dataset_name (str) – The Name of the dataset
dataset_id (str) – The Id of the dataset
dataset (dtlpy.entities.dataset.Dataset) – dataset object
Example:
project.datasets.open_in_web(dataset_id='dataset_id')
- set_readonly(state: bool, dataset: Dataset)[source]¶
Set dataset readonly mode.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
state (bool) – state to update readonly mode
dataset (dtlpy.entities.dataset.Dataset) – dataset object
Example:
project.datasets.set_readonly(dataset='dataset_entity', state=True)
- sync(dataset_id: str, wait: bool = True)[source]¶
Sync dataset with external storage.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
True if success
- Return type:
Example:
success = project.datasets.sync(dataset_id='dataset_id')
- update(dataset: Dataset, system_metadata: bool = False, patch: Optional[dict] = None) Dataset [source]¶
Update dataset field.
Prerequisites: You must be an owner or developer to use this method.
- Parameters:
dataset (dtlpy.entities.dataset.Dataset) – dataset object
system_metadata (bool) – True, if you want to change metadata system
patch (dict) – Specific patch request
- Returns:
Dataset object
- Return type:
Example:
dataset = project.datasets.update(dataset='dataset_entity')
- upload_annotations(dataset, local_path, filters: Optional[Filters] = None, clean=False, remote_root_path='/', export_version=ExportVersion.V1)[source]¶
Upload annotations to dataset.
Example for remote_root_path: If the item filepath is a/b/item and remote_root_path is /a the start folder will be b instead of a
Prerequisites: You must have a dataset with items that are related to the annotations. The relationship between the dataset and annotations is shown in the name. You must be in the role of an owner or developer.
- Parameters:
dataset (dtlpy.entities.dataset.Dataset) – dataset to upload to
local_path (str) – str - local folder where the annotations files are
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
clean (bool) – True to remove the old annotations
remote_root_path (str) – the remote root path to match remote and local items
export_version (str) – exported items will have original extension in filename, V1 - no original extension in filenames
Example:
project.datasets.upload_annotations(dataset='dataset_entity', local_path='local_path', clean=False, export_version=dl.ExportVersion.V1 )
Drivers¶
- class Drivers(client_api: ApiClient, project: Optional[Project] = None)[source]¶
Bases:
object
Drivers Repository
The Drivers class allows users to manage drivers that are used to connect with external storage. Read more about external storage in our documentation and developers’ docs.
- create(name: str, driver_type: ExternalStorage, integration_id: str, bucket_name: str, integration_type: IntegrationType, project_id: Optional[str] = None, allow_external_delete: bool = True, region: Optional[str] = None, storage_class: str = '', path: str = '')[source]¶
Create a storage driver.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
name (str) – the driver name
driver_type (ExternalStorage) – dl.ExternalStorage (Enum). For all options run: list(dl.ExsternalStorage)
integration_id (str) – the integration id
bucket_name (str) – the external bucket name
integration_type (IntegrationType) – dl.IntegrationType (Enum). For all options run: list(dl.IntegrationType)
project_id (str) – project id
allow_external_delete (bool) – true to allow deleting files from external storage when files are deleted in your Dataloop storage
region (str) – relevant only for s3 - the bucket region
storage_class (str) – relevant only for s3
path (str) – Optional. By default path is the root folder. Path is case sensitive integration
- Returns:
driver object
- Return type:
Example:
project.drivers.create(name='driver_name', driver_type=dl.ExternalStorage.S3, integration_id='integration_id', bucket_name='bucket_name', project_id='project_id', region='ey-west-1')
- delete(driver_name: Optional[str] = None, driver_id: Optional[str] = None, sure: bool = False, really: bool = False)[source]¶
Delete a driver forever!
Prerequisites: You must be an owner or developer to use this method.
Example:
project.drivers.delete(dataset_id='dataset_id', sure=True, really=True)
- get(driver_name: Optional[str] = None, driver_id: Optional[str] = None) Driver [source]¶
Get a Driver object to use in your code.
Prerequisites: You must be in the role of an owner or developer.
You must provide at least ONE of the following params: driver_name, driver_id.
- Parameters:
- Returns:
Driver object
- Return type:
Example:
project.drivers.get(driver_id='driver_id')
Items¶
- class Items(client_api: ApiClient, datasets: Optional[Datasets] = None, dataset: Optional[Dataset] = None, dataset_id=None, items_entity=None, project=None)[source]¶
Bases:
object
Items Repository
The Items class allows you to manage items in your datasets. For information on actions related to items see https://developers.dataloop.ai/tutorials/data_management/upload_and_manage_items/chapter/
- clone(item_id: str, dst_dataset_id: str, remote_filepath: Optional[str] = None, metadata: Optional[dict] = None, with_annotations: bool = True, with_metadata: bool = True, with_task_annotations_status: bool = False, allow_many: bool = False, wait: bool = True)[source]¶
Clone item. Read more about cloning datatsets and items in our documentation and SDK documentation.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
item_id (str) – item to clone
dst_dataset_id (str) – destination dataset id
remote_filepath (str) – complete filepath
metadata (dict) – new metadata to add
with_annotations (bool) – clone annotations
with_metadata (bool) – clone metadata
with_task_annotations_status (bool) – clone task annotations status
allow_many (bool) – bool if True, using multiple clones in single dataset is allowed, (default=False)
wait (bool) – wait for the command to finish
- Returns:
Item object
- Return type:
Example:
dataset.items.clone(item_id='item_id', dst_dataset_id='dist_dataset_id', with_metadata=True, with_task_annotations_status=False, with_annotations=False)
- delete(filename: Optional[str] = None, item_id: Optional[str] = None, filters: Optional[Filters] = None)[source]¶
Delete item from platform.
Prerequisites: You must be in the role of an owner or developer.
You must provide at least ONE of the following params: item id, filename, filters.
- Parameters:
filename (str) – optional - search item by remote path
item_id (str) – optional - search item by id
filters (dtlpy.entities.filters.Filters) – optional - delete items by filter
- Returns:
True if success
- Return type:
Example:
dataset.items.delete(item_id='item_id')
- download(filters: Optional[Filters] = None, items=None, local_path: Optional[str] = None, file_types: Optional[list] = None, save_locally: bool = True, to_array: bool = False, annotation_options: Optional[ViewAnnotationOptions] = None, annotation_filters: Optional[Filters] = None, overwrite: bool = False, to_items_folder: bool = True, thickness: int = 1, with_text: bool = False, without_relative_path=None, avoid_unnecessary_annotation_download: bool = False, include_annotations_in_output: bool = True, export_png_files: bool = False, filter_output_annotations: bool = False, alpha: float = 1, export_version=ExportVersion.V1)[source]¶
Download dataset items by filters.
Filters the dataset for items and saves them locally.
Optional – download annotation, mask, instance, and image mask of the item.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
items (List[dtlpy.entities.item.Item] or dtlpy.entities.item.Item) – download Item entity or item_id (or a list of item)
local_path (str) – local folder or filename to save to.
file_types (list) – a list of file type to download. e.g [‘video/webm’, ‘video/mp4’, ‘image/jpeg’, ‘image/png’]
save_locally (bool) – bool. save to disk or return a buffer
to_array (bool) – returns Ndarray when True and local_path = False
annotation_options (list) – download annotations options: list(dl.ViewAnnotationOptions)
annotation_filters (dtlpy.entities.filters.Filters) – Filters entity to filter annotations for download
overwrite (bool) – optional - default = False
to_items_folder (bool) – Create ‘items’ folder and download items to it
thickness (int) – optional - line thickness, if -1 annotation will be filled, default =1
with_text (bool) – optional - add text to annotations, default = False
without_relative_path (bool) – bool - download items without the relative path from platform
avoid_unnecessary_annotation_download (bool) – default - False
include_annotations_in_output (bool) – default - False , if export should contain annotations
export_png_files (bool) – default - if True, semantic annotations should be exported as png files
filter_output_annotations (bool) – default - False, given an export by filter - determine if to filter out annotations
alpha (float) – opacity value [0 1], default 1
export_version (str) – exported items will have original extension in filename, V1 - no original extension in filenames
- Returns:
generator of local_path per each downloaded item
- Return type:
generator or single item
Example:
dataset.items.download(local_path='local_path', annotation_options=dl.ViewAnnotationOptions, overwrite=False, thickness=1, with_text=False, alpha=1, save_locally=True )
- get(filepath: Optional[str] = None, item_id: Optional[str] = None, fetch: Optional[bool] = None, is_dir: bool = False) Item [source]¶
Get Item object
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
Item object
- Return type:
Example:
dataset.items.get(item_id='item_id')
- get_all_items(filters: Optional[Filters] = None) [<class 'dtlpy.entities.item.Item'>] [source]¶
Get all items in dataset.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filters (dtlpy.entities.filters.Filters) – dl.Filters entity to filters items
- Returns:
list of all items
- Return type:
Example:
dataset.items.get_all_items()
- list(filters: Optional[Filters] = None, page_offset: Optional[int] = None, page_size: Optional[int] = None) PagedEntities [source]¶
List items in a dataset.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
page_offset (int) – start page
page_size (int) – page size
- Returns:
Pages object
- Return type:
Example:
dataset.items.list(page_offset=0, page_size=100)
- make_dir(directory, dataset: Optional[Dataset] = None) Item [source]¶
Create a directory in a dataset.
Prerequisites: All users.
- Parameters:
directory (str) – name of directory
dataset (dtlpy.entities.dataset.Dataset) – dataset object
- Returns:
Item object
- Return type:
Example:
dataset.items.make_dir(directory='directory_name')
- move_items(destination: str, filters: Optional[Filters] = None, items=None, dataset: Optional[Dataset] = None) bool [source]¶
Move items to another directory. If directory does not exist we will create it
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
destination (str) – destination directory
filters (dtlpy.entities.filters.Filters) – optional - either this or items. Query of items to move
items – optional - either this or filters. A list of items to move
dataset (dtlpy.entities.dataset.Dataset) – dataset object
- Returns:
True if success
- Return type:
Example:
dataset.items.move_items(destination='directory_name')
- open_in_web(filepath=None, item_id=None, item=None)[source]¶
Open the item in web platform
Prerequisites: You must be in the role of an owner or developer or be an annotation manager/annotator with access to that item through task.
- Parameters:
filepath (str) – item file path
item_id (str) – item id
item (dtlpy.entities.item.Item) – item entity
Example:
dataset.items.open_in_web(item_id='item_id')
- set_items_entity(entity)[source]¶
Set the item entity type to Artifact, Item, or Codebase.
- Parameters:
entity (entities.Item, entities.Artifact, entities.Codebase) – entity type [entities.Item, entities.Artifact, entities.Codebase]
- update(item: Optional[Item] = None, filters: Optional[Filters] = None, update_values=None, system_update_values=None, system_metadata: bool = False)[source]¶
Update item metadata.
Prerequisites: You must be in the role of an owner or developer.
You must provide at least ONE of the following params: update_values, system_update_values.
- Parameters:
item (dtlpy.entities.item.Item) – Item object
filters (dtlpy.entities.filters.Filters) – optional update filtered items by given filter
update_values – optional field to be updated and new values
system_update_values – values in system metadata to be updated
system_metadata (bool) – True, if you want to update the metadata system
- Returns:
Item object
- Return type:
Example:
dataset.items.update(item='item_entity')
- update_status(status: ItemStatus, items=None, item_ids=None, filters=None, dataset=None, clear=False)[source]¶
Update item status in task
Prerequisites: You must be in the role of an owner or developer or annotation manager who has been assigned a task with the item.
You must provide at least ONE of the following params: items, item_ids, filters.
- Parameters:
status (str) – ItemStatus.COMPLETED, ItemStatus.APPROVED, ItemStatus.DISCARDED
items (list) – list of items
item_ids (list) – list of items id
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
dataset (dtlpy.entities.dataset.Dataset) – dataset object
clear (bool) – to delete status
Example:
dataset.items.update_status(item_ids='item_id', status=dl.ItemStatus.COMPLETED)
- upload(local_path: str, local_annotations_path: ~typing.Optional[str] = None, remote_path: str = '/', remote_name: ~typing.Optional[str] = None, file_types: ~typing.Optional[~dtlpy.repositories.items.Items.list] = None, overwrite: bool = False, item_metadata: ~typing.Optional[dict] = None, output_entity=<class 'dtlpy.entities.item.Item'>, no_output: bool = False, export_version: str = ExportVersion.V1, item_description: ~typing.Optional[str] = None)[source]¶
Upload local file to dataset. Local filesystem will remain unchanged. If “*” at the end of local_path (e.g. “/images/*”) items will be uploaded without the head directory.
Prerequisites: Any user can upload items.
- Parameters:
local_path (str) – list of local file, local folder, BufferIO, numpy.ndarray or url to upload
local_annotations_path (str) – path to dataloop format annotations json files.
remote_path (str) – remote path to save.
remote_name (str) – remote base name to save. when upload numpy.ndarray as local path, remote_name with .jpg or .png ext is mandatory
file_types (list) – list of file type to upload. e.g [‘.jpg’, ‘.png’]. default is all
item_metadata (dict) – metadata dict to upload to item or ExportMetadata option to export metadata from annotation file
overwrite (bool) – optional - default = False
output_entity – output type
no_output (bool) – do not return the items after upload
export_version (str) – exported items will have original extension in filename, V1 - no original extension in filenames
item_description (str) – add a string description to the uploaded item
- Returns:
Output (generator/single item)
- Return type:
generator or single item
Example:
dataset.items.upload(local_path='local_path', local_annotations_path='local_annotations_path', overwrite=True, item_metadata={'Hellow': 'Word'} )
Annotations¶
- class Annotations(client_api: ApiClient, item=None, dataset=None, dataset_id=None)[source]¶
Bases:
object
Annotations Repository
The Annotation class allows you to manage the annotations of data items. For information on annotations explore our documentation at: Classification SDK, Annotation Labels and Attributes, Show Video with Annotations.
- builder()[source]¶
Create Annotation collection.
- Prerequisites: You must have an item to be annotated. You must have the role of an owner or developer
or be assigned a task that includes that item as an annotation manager or annotator.
- Returns:
Annotation collection object
- Return type:
Example:
annotation_collection = item.annotations.builder()
- delete(annotation: Optional[Annotation] = None, annotation_id: Optional[str] = None, filters: Optional[Filters] = None) bool [source]¶
Remove an annotation from item.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
annotation (dtlpy.entities.annotation.Annotation) – Annotation object
annotation_id (str) – The id of the annotation
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
True/False
- Return type:
Example:
is_deleted = item.annotations.delete(annotation_id='annotation_id')
- download(filepath: str, annotation_format: ViewAnnotationOptions = ViewAnnotationOptions.JSON, img_filepath: Optional[str] = None, height: Optional[float] = None, width: Optional[float] = None, thickness: int = 1, with_text: bool = False, alpha: float = 1)[source]¶
Save annotation to file.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
filepath (str) – Target download directory
annotation_format (str) – the format that want to download ,options: list(dl.ViewAnnotationOptions)
img_filepath (str) – img file path - needed for img_mask
height (float) – optional - image height
width (float) – optional - image width
thickness (int) – optional - line thickness, default=1
with_text (bool) – optional - draw annotation with text, default = False
alpha (float) – opacity value [0 1], default 1
- Returns:
file path to where save the annotations
- Return type:
Example:
file_path = item.annotations.download( filepath='file_path', annotation_format=dl.ViewAnnotationOptions.MASK, img_filepath='img_filepath', height=100, width=100, thickness=1, with_text=False, alpha=1)
- get(annotation_id: str) Annotation [source]¶
Get a single annotation.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
annotation_id (str) – The id of the annotation
- Returns:
Annotation object or None
- Returns:
Annotation object or None
- Return type:
Example:
annotation = item.annotations.get(annotation_id='annotation_id')
- list(filters: Optional[Filters] = None, page_offset: Optional[int] = None, page_size: Optional[int] = None)[source]¶
List Annotations of a specific item. You must get the item first and then list the annotations with the desired filters.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
page_offset (int) – starting page
page_size (int) – size of page
- Returns:
Pages object
- Return type:
Example:
annotations = item.annotations.list(filters=dl.Filters( resource=dl.FiltersResource.ANNOTATION, field='type', values='box'), page_size=100, page_offset=0)
- show(image=None, thickness: int = 1, with_text: bool = False, height: Optional[float] = None, width: Optional[float] = None, annotation_format: ViewAnnotationOptions = ViewAnnotationOptions.MASK, alpha: float = 1)[source]¶
Show annotations. To use this method, you must get the item first and then show the annotations with the desired filters. The method returns an array showing all the annotations.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
image (ndarray) – empty or image to draw on
thickness (int) – optional - line thickness, default=1
with_text (bool) – add label to annotation
height (float) – item height
width (float) – item width
annotation_format (str) – the format that want to show ,options: list(dl.ViewAnnotationOptions)
alpha (float) – opacity value [0 1], default 1
- Returns:
ndarray of the annotations
- Return type:
ndarray
Example:
image = item.annotations.show(image='nd array', thickness=1, with_text=False, height=100, width=100, annotation_format=dl.ViewAnnotationOptions.MASK, alpha=1)
- update(annotations, system_metadata=False)[source]¶
Update an existing annotation. For example, you may change the annotation’s label and then use the update method.
- Prerequisites: You must have an item that has been annotated. You must have the role of an owner or
developer or be assigned a task that includes that item as an annotation manager or annotator.
- Parameters:
annotations (dtlpy.entities.annotation.Annotation) – Annotation object
system_metadata (bool) – bool - True, if you want to change metadata system
- Returns:
True if successful or error if unsuccessful
- Return type:
bool
Example:
annotations = item.annotations.update(annotation='annotation')
- update_status(annotation: Optional[Annotation] = None, annotation_id: Optional[str] = None, status: AnnotationStatus = AnnotationStatus.ISSUE) Annotation [source]¶
Set status on annotation.
Prerequisites: You must have an item that has been annotated. You must have the role of an owner or developer or be assigned a task that includes that item as an annotation manager.
- Parameters:
annotation (dtlpy.entities.annotation.Annotation) – Annotation object
annotation_id (str) – optional - annotation id to set status
status (str) – can be AnnotationStatus.ISSUE, APPROVED, REVIEW, CLEAR
- Returns:
Annotation object
- Return type:
Example:
annotation = item.annotations.update_status(annotation_id='annotation_id', status=dl.AnnotationStatus.ISSUE)
- upload(annotations) AnnotationCollection [source]¶
Upload a new annotation/annotations. You must first create the annotation using the annotation builder method.
Prerequisites: Any user can upload annotations.
- Parameters:
annotations (List[dtlpy.entities.annotation.Annotation] or dtlpy.entities.annotation.Annotation) – list or
single annotation of type Annotation :return: list of annotation objects :rtype: entities.AnnotationCollection
Example:
annotations = item.annotations.upload(annotations='builder')
Recipes¶
- class Recipes(client_api: ApiClient, dataset: Optional[Dataset] = None, project: Optional[Project] = None, project_id: Optional[str] = None)[source]¶
Bases:
object
Recipes Repository
The Recipes class allows you to manage recipes and their properties. For more information on Recipes, see our documentation and developers’ documentation.
- clone(recipe: Optional[Recipe] = None, recipe_id: Optional[str] = None, shallow: bool = False)[source]¶
Clone recipe.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
recipe (dtlpy.entities.recipe.Recipe) – Recipe object
recipe_id (str) – Recipe id
shallow (bool) – If True, link to existing ontology, clones all ontologies that are linked to the recipe as well
- Returns:
Cloned ontology object
- Return type:
Example:
dataset.recipes.clone(recipe_id='recipe_id')
- create(project_ids=None, ontology_ids=None, labels=None, recipe_name=None, attributes=None, annotation_instruction_file=None) Recipe [source]¶
Create a new Recipe. Note: If the param ontology_ids is None, an ontology will be created first.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
Recipe entity
- Return type:
Example:
dataset.recipes.create(recipe_name='My Recipe', labels=labels))
- delete(recipe_id: str, force: bool = False)[source]¶
Delete recipe from platform.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
True if success
- Return type:
Example:
dataset.recipes.delete(recipe_id='recipe_id')
- get(recipe_id: str) Recipe [source]¶
Get a Recipe object to use in your code.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
recipe_id (str) – recipe id
- Returns:
Recipe object
- Return type:
Example:
dataset.recipes.get(recipe_id='recipe_id')
- list(filters: Optional[Filters] = None) List[Recipe] [source]¶
List recipes for a dataset.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
list of all recipes
- Retype:
list
Example:
dataset.recipes.list()
- open_in_web(recipe: Optional[Recipe] = None, recipe_id: Optional[str] = None)[source]¶
Open the recipe in web platform.
Prerequisites: All users.
- Parameters:
recipe (dtlpy.entities.recipe.Recipe) – recipe entity
recipe_id (str) – recipe id
Example:
dataset.recipes.open_in_web(recipe_id='recipe_id')
- update(recipe: Recipe, system_metadata=False) Recipe [source]¶
Update recipe.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
recipe (dtlpy.entities.recipe.Recipe) – Recipe object
system_metadata (bool) – True, if you want to change metadata system
- Returns:
Recipe object
- Return type:
Example:
dataset.recipes.update(recipe='recipe_entity')
Ontologies¶
- class Ontologies(client_api: ApiClient, recipe: Optional[Recipe] = None, project: Optional[Project] = None, dataset: Optional[Dataset] = None)[source]¶
Bases:
object
Ontologies Repository
The Ontologies class allows users to manage ontologies and their properties. Read more about ontology in our developers docs.
- create(labels, title=None, project_ids=None, attributes=None) Ontology [source]¶
Create a new ontology.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
Ontology object
- Return type:
Example:
recipe.ontologies.create(labels='labels_entity', title='new_ontology', project_ids='project_ids')
- delete(ontology_id)[source]¶
Delete Ontology from the platform.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
ontology_id – ontology id
- Returns:
True if success
- Return type:
Example:
recipe.ontologies.delete(ontology_id='ontology_id')
- delete_attributes(ontology_id, keys: list)[source]¶
Delete a bulk of attributes
- Parameters:
- Returns:
True if success
- Return type:
Example:
ontology.delete_attributes(['1'])
- get(ontology_id: str) Ontology [source]¶
Get Ontology object to use in your code.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
ontology_id (str) – ontology id
- Returns:
Ontology object
- Return type:
Example:
recipe.ontologies.get(ontology_id='ontology_id')
- static labels_to_roots(labels)[source]¶
Converts labels dictionary to a list of platform representation of labels.
- Parameters:
labels (dict) – labels dict
- Returns:
platform representation of labels
- list(project_ids=None) List[Ontology] [source]¶
List ontologies for recipe
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
project_ids –
- Returns:
list of all the ontologies
Example:
recipe.ontologies.list(project_ids='project_ids')
- update(ontology: Ontology, system_metadata=False) Ontology [source]¶
Update the Ontology metadata.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
ontology (dtlpy.entities.ontology.Ontology) – Ontology object
system_metadata (bool) – bool - True, if you want to change metadata system
- Returns:
Ontology object
- Return type:
Example:
recipe.ontologies.delete(ontology='ontology_entity')
- update_attributes(ontology_id: str, title: str, key: str, attribute_type: AttributesTypes, scope: Optional[list] = None, optional: Optional[bool] = None, values: Optional[list] = None, attribute_range: Optional[AttributesRange] = None)[source]¶
ADD a new attribute or update if exist
- Parameters:
ontology_id (str) – ontology_id
title (str) – attribute title
key (str) – the key of the attribute must br unique
attribute_type (AttributesTypes) – dl.AttributesTypes your attribute type
scope (list) – list of the labels or * for all labels
optional (bool) – optional attribute
values (list) – list of the attribute values ( for checkbox and radio button)
attribute_range (dict or AttributesRange) – dl.AttributesRange object
- Returns:
true in success
- Return type:
Example:
ontology.update_attributes(key='1', title='checkbox', attribute_type=dl.AttributesTypes.CHECKBOX, values=[1,2,3])
Tasks¶
- class Tasks(client_api: ApiClient, project: Optional[Project] = None, dataset: Optional[Dataset] = None, project_id: Optional[str] = None)[source]¶
Bases:
object
Tasks Repository
The Tasks class allows the user to manage tasks and their properties. For more information, read in our developers’ documentation about Creating Tasks, and Redistributing and Reassigning Tasks.
- add_items(task: Optional[Task] = None, task_id=None, filters: Optional[Filters] = None, items=None, assignee_ids=None, query=None, workload=None, limit=None, wait=True) Task [source]¶
Add items to a Task.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
- Parameters:
task (dtlpy.entities.task.Task) – task object
task_id (str) – the Id of the task
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
items (list) – list of items (item Ids or objects) to add to the task
assignee_ids (list) – list to assignee who works in the task
query (dict) – query to filter the items for the task
workload (list) – list of WorkloadUnit objects. Customize distribution (percentage) between the task assignees. For example: [dl.WorkloadUnit(annotator@hi.com, 80), dl.WorkloadUnit(annotator2@hi.com, 20)]
limit (int) – the limit items that task can include
wait (bool) – wait until add items will to finish
- Returns:
task entity
- Return type:
Example:
dataset.tasks.add_items(task= 'task_entity', items = [items])
- create(task_name, due_date=None, assignee_ids=None, workload=None, dataset=None, task_owner=None, task_type='annotation', task_parent_id=None, project_id=None, recipe_id=None, assignments_ids=None, metadata=None, filters=None, items=None, query=None, available_actions=None, wait=True, check_if_exist: Filters = False, limit=None, batch_size=None, max_batch_workload=None, allowed_assignees=None, priority=TaskPriority.MEDIUM, consensus_task_type=None, consensus_percentage=None, consensus_assignees=None) Task [source]¶
Create a new Task (Annotation or QA).
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
- Parameters:
task_name (str) – the name of the task
due_date (float) – date by which the task should be finished; for example, due_date=datetime.datetime(day=1, month=1, year=2029).timestamp()
assignee_ids (list) – list the task assignees (contributors) that should be working on the task. Provide a list of users’ emails
workload (List[WorkloadUnit] List[WorkloadUnit]) – list of WorkloadUnit objects. Customize distribution (percentage) between the task assignees. For example: [dl.WorkloadUnit(annotator@hi.com, 80), dl.WorkloadUnit(annotator2@hi.com, 20)]
dataset (entities.Dataset) – dataset object, the dataset that refer to the task
task_owner (str) – task owner. Provide user email
task_type (str) – task type “annotation” or “qa”
task_parent_id (str) – optional if type is qa - parent annotation task id
project_id (str) – the Id of the project where task will be created
recipe_id (str) – recipe id for the task
assignments_ids (list) – assignments ids to the task
metadata (dict) – metadata for the task
filters (entities.Filters) – dl.Filters entity to filter items for the task
items (List[entities.Item]) – list of items (item Id or objects) to insert to the task
query (dict DQL) – filter items for the task
available_actions (list) – list of available actions (statuses) that will be available for the task items; The default statuses are: “completed” and “discard”
wait (bool) – wait until create task finish
check_if_exist (entities.Filters) – dl.Filters check if task exist according to filter
limit (int) – the limit items that the task can include
batch_size (int) – Pulling batch size (items), use with pulling allocation method. Restrictions - Min 3, max 100
max_batch_workload (int) – max_batch_workload: Max items in assignment, use with pulling allocation method. Restrictions - Min batchSize + 2, max batchSize * 2
allowed_assignees (list) – list the task assignees (contributors) that should be working on the task. Provide a list of users’ emails
priority (entities.TaskPriority) – priority of the task options in entities.TaskPriority
consensus_task_type (entities.ConsensusTaskType) – consensus_task_type of the task options in entities.ConsensusTaskType
consensus_percentage (int) – percentage of items to be copied to multiple annotators (consensus items)
consensus_assignees (int) – the number of different annotators per item (number of copies per item)
- Returns:
Task object
- Return type:
Example:
dataset.tasks.create(task= 'task_entity', due_date = datetime.datetime(day= 1, month= 1, year= 2029).timestamp(), assignee_ids =[ 'annotator1@dataloop.ai', 'annotator2@dataloop.ai'])
- create_qa_task(task: Task, assignee_ids, due_date=None, filters=None, items=None, query=None, workload=None, metadata=None, available_actions=None, wait=True, batch_size=None, max_batch_workload=None, allowed_assignees=None, priority=TaskPriority.MEDIUM) Task [source]¶
Create a new QA Task.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
- Parameters:
task (dtlpy.entities.task.Task) – the parent annotation task object
assignee_ids (list) – list the QA task assignees (contributors) that should be working on the task. Provide a list of users’ emails
due_date (float) – date by which the QA task should be finished; for example, due_date=datetime.datetime(day=1, month=1, year=2029).timestamp()
filters (entities.Filters) – dl.Filters entity to filter items for the task
items (List[entities.Item]) – list of items (item Id or objects) to insert to the task
query (dict DQL) – filter items for the task
workload (List[WorkloadUnit]) – list of WorkloadUnit objects. Customize distribution (percentage) between the task assignees. For example: [dl.WorkloadUnit(annotator@hi.com, 80), dl.WorkloadUnit(annotator2@hi.com, 20)]
metadata (dict) – metadata for the task
available_actions (list) – list of available actions (statuses) that will be available for the task items; The default statuses are: “approved” and “discard”
wait (bool) – wait until create task finish
batch_size (int) – Pulling batch size (items), use with pulling allocation method. Restrictions - Min 3, max 100
max_batch_workload (int) – Max items in assignment, use with pulling allocation method. Restrictions - Min batchSize + 2, max batchSize * 2
allowed_assignees (list) – list the task assignees (contributors) that should be working on the task. Provide a list of users’ emails
priority (entities.TaskPriority) – priority of the task options in entities.TaskPriority
- Returns:
task object
- Return type:
Example:
dataset.tasks.create_qa_task(task= 'task_entity', due_date = datetime.datetime(day= 1, month= 1, year= 2029).timestamp(), assignee_ids =[ 'annotator1@dataloop.ai', 'annotator2@dataloop.ai'])
- delete(task: Optional[Task] = None, task_name: Optional[str] = None, task_id: Optional[str] = None, wait: bool = True)[source]¶
Delete the Task.
Prerequisites: You must be in the role of an owner or developer or annotation manager who created that task.
- Parameters:
task (dtlpy.entities.task.Task) – the task object
task_name (str) – the name of the task
task_id (str) – the Id of the task
wait (bool) – wait until delete task finish
- Returns:
True is success
- Return type:
Example:
dataset.tasks.delete(task_id='task_id')
- get(task_name=None, task_id=None) Task [source]¶
Get a Task object to use in your code.
Prerequisites: You must be in the role of an owner or developer or annotation manager who has been assigned the task.
- Parameters:
- Returns:
task object
- Return type:
Example:
dataset.tasks.get(task_id='task_id')
- get_items(task_id: Optional[str] = None, task_name: Optional[str] = None, dataset: Optional[Dataset] = None, filters: Optional[Filters] = None, get_consensus_items: bool = False, task: Optional[Task] = None) PagedEntities [source]¶
Get the task items to use in your code.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
If a filters param is provided, you will receive a PagedEntity output of the task items. If no filter is provided, you will receive a list of the items.
- Parameters:
task_id (str) – the id of the task
task_name (str) – the name of the task
get_consensus_items (bool) – get the items from the consensus assignment
task (dtlpy.entities.Task) – task object
dataset (dtlpy.entities.dataset.Dataset) – dataset object that refer to the task
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
list of the items or PagedEntity output of items
- Return type:
Example:
dataset.tasks.get_items(task_id= 'task_id')
- list(project_ids=None, status=None, task_name=None, pages_size=None, page_offset=None, recipe=None, creator=None, assignments=None, min_date=None, max_date=None, filters: Optional[Filters] = None) Union[List[Task], PagedEntities] [source]¶
List all tasks.
Prerequisites: You must be in the role of an owner or developer or annotation manager who has been assigned the task.
- Parameters:
project_ids – search tasks by given list of project ids
status (str) – search tasks by a given task status
task_name (str) – search tasks by a given task name
pages_size (int) – pages size of the output generator
page_offset (int) – page offset of the output generator
recipe (dtlpy.entities.recipe.Recipe) – Search tasks that use a given recipe. Provide the required recipe object
creator (str) – search tasks created by a given creator (user email)
assignments (dtlpy.entities.assignment.Assignment recipe) – assignments object
min_date (double) – search all tasks created AFTER a given date, use a milliseconds format. For example: 1661780622008
max_date (double) – search all tasks created BEFORE a given date, use a milliseconds format. For example: 1661780622008
filters (dtlpy.entities.filters.Filters) – dl.Filters entity to filters tasks using DQL
- Returns:
List of Task objects
Example:
dataset.tasks.list(project_ids='project_ids',pages_size=100, page_offset=0)
- open_in_web(task_name: Optional[str] = None, task_id: Optional[str] = None, task: Optional[Task] = None)[source]¶
Open the task in the web platform.
Prerequisites: You must be in the role of an owner or developer or annotation manager who has been assigned the task.
- Parameters:
task_name (str) – the name of the task
task_id (str) – the Id of the task
task (dtlpy.entities.task.Task) – the task object
Example:
dataset.tasks.open_in_web(task_id='task_id')
- query(filters=None, project_ids=None)[source]¶
List all tasks by filter.
Prerequisites: You must be in the role of an owner or developer or annotation manager who has been assigned the task.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
project_ids (list) – list of project ids of the required tasks
- Returns:
Paged entity - task pages generator
- Return type:
Example:
dataset.tasks.query(project_ids='project_ids')
- remove_items(task: Optional[Task] = None, task_id=None, filters: Optional[Filters] = None, query=None, items=None, wait=True)[source]¶
remove items from Task.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
- Parameters:
task (dtlpy.entities.task.Task) – task object
task_id (str) – the Id of the task
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
query (dict) – query to filter the items use it
items (list) – list of items to add to the task
wait (bool) – wait until remove items finish
- Returns:
True if success and an error if failed
- Return type:
Examples:
dataset.tasks.remove_items(task= 'task_entity', items = [items])
- set_status(status: str, operation: str, task_id: str, item_ids: List[str])[source]¶
Update an item status within a task.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned to be owner of the annotation task.
- Parameters:
- Returns:
True if success
- Return type:
Example:
dataset.tasks.set_status(task_id= 'task_id', status='complete', operation='create')
- update(task: Optional[Task] = None, system_metadata=False) Task [source]¶
Update a Task.
Prerequisites: You must be in the role of an owner or developer or annotation manager who created that task.
- Parameters:
task (dtlpy.entities.task.Task) – the task object
system_metadata (bool) – True, if you want to change metadata system
- Returns:
Task object
- Return type:
Example:
dataset.tasks.update(task='task_entity')
Assignments¶
- class Assignments(client_api: ApiClient, project: Optional[Project] = None, task: Optional[Task] = None, dataset: Optional[Dataset] = None, project_id=None)[source]¶
Bases:
object
Assignments Repository
The Assignments class allows users to manage assignments and their properties. Read more about Task Assignment in our Developers documentation.
- create(assignee_id: str, task: Optional[Task] = None, filters: Optional[Filters] = None, items: Optional[list] = None) Assignment [source]¶
Create a new assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
assignee_id (str) – the email of the user that want to assign the assignment
task (dtlpy.entities.task.Task) – the task object that include the assignment
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
items (list) – list of items (item Id or objects) to insert to the assignment
- Returns:
Assignment object
- Return type:
dtlpy.entities.assignment.Assignment assignment
Example:
assignment = task.assignments.create(assignee_id='annotator1@dataloop.ai')
- get(assignment_name: Optional[str] = None, assignment_id: Optional[str] = None)[source]¶
Get Assignment object to use it in your code.
- Parameters:
- Returns:
Assignment object
- Return type:
Example:
assignment = task.assignments.get(assignment_id='assignment_id')
- get_items(assignment: Optional[Assignment] = None, assignment_id=None, assignment_name=None, dataset=None, filters=None) PagedEntities [source]¶
Get all the items in the assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
assignment (dtlpy.entities.assignment.Assignment) – assignment object
assignment_id – the Id of the assignment
assignment_name (str) – the name of the assignment
dataset (dtlpy.entities.dataset.Dataset) – dataset object, the dataset that refer to the assignment
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
pages of the items
- Return type:
Example:
items = task.assignments.get_items(assignment_id='assignment_id')
- list(project_ids: Optional[list] = None, status: Optional[str] = None, assignment_name: Optional[str] = None, assignee_id: Optional[str] = None, pages_size: Optional[int] = None, page_offset: Optional[int] = None, task_id: Optional[int] = None) List[Assignment] [source]¶
Get Assignment list to be able to use it in your code.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
project_ids (list) – search assignment by given list of project ids
status (str) – search assignment by a given task status
assignment_name (str) – search assignment by a given assignment name
assignee_id (str) – the user email that assignee the assignment to it
pages_size (int) – pages size of the output generator
page_offset (int) – page offset of the output generator
task_id (str) – search assignment by given task id
- Returns:
List of Assignment objects
- Return type:
miscellaneous.List[dtlpy.entities.assignment.Assignment]
Example:
assignments = task.assignments.list(status='complete', assignee_id='user@dataloop.ai', pages_size=100, page_offset=0)
- open_in_web(assignment_name: Optional[str] = None, assignment_id: Optional[str] = None, assignment: Optional[str] = None)[source]¶
Open the assignment in the platform.
Prerequisites: All users.
- Parameters:
assignment_name (str) – the name of the assignment
assignment_id (str) – the Id of the assignment
assignment (dtlpy.entities.assignment.Assignment) – assignment object
Example:
task.assignments.open_in_web(assignment_id='assignment_id')
- reassign(assignee_id: str, assignment: Optional[Assignment] = None, assignment_id: Optional[str] = None, task: Optional[Task] = None, task_id: Optional[str] = None, wait: bool = True)[source]¶
Reassign an assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
assignee_id (str) – the email of the user that want to assign the assignment
assignment (dtlpy.entities.assignment.Assignment) – assignment object
assignment_id – the Id of the assignment
task (dtlpy.entities.task.Task) – task object
task_id (str) – the Id of the task that include the assignment
wait (bool) – wait until reassign assignment finish
- Returns:
Assignment object
- Return type:
Example:
assignment = task.assignments.reassign(assignee_ids='annotator1@dataloop.ai')
- redistribute(workload: Workload, assignment: Optional[Assignment] = None, assignment_id: Optional[str] = None, task: Optional[Task] = None, task_id: Optional[str] = None, wait: bool = True)[source]¶
Redistribute an assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
Example:
- Parameters:
workload (dtlpy.entities.assignment.Workload) – list of WorkloadUnit objects. Customize distribution (percentage) between the task assignees. For example: [dl.WorkloadUnit(annotator@hi.com, 80), dl.WorkloadUnit(annotator2@hi.com, 20)]
assignment (dtlpy.entities.assignment.Assignment) – assignment object
assignment_id (str) – the Id of the assignment
task (dtlpy.entities.task.Task) – the task object that include the assignment
task_id (str) – the Id of the task that include the assignment
wait (bool) – wait until redistribute assignment finish
- Returns:
Assignment object
- Return type:
dtlpy.entities.assignment.Assignment assignment
assignment = task.assignments.redistribute(workload=dl.Workload([dl.WorkloadUnit(assignee_id="annotator1@dataloop.ai", load=50), dl.WorkloadUnit(assignee_id="annotator2@dataloop.ai", load=50)]))
- set_status(status: str, operation: str, item_id: str, assignment_id: str) bool [source]¶
Set item status within assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
- Returns:
True id success
- Return type:
Example:
success = task.assignments.set_status(assignment_id='assignment_id', status='complete', operation='created', item_id='item_id')
- update(assignment: Optional[Assignment] = None, system_metadata: bool = False) Assignment [source]¶
Update an assignment.
Prerequisites: You must be in the role of an owner, developer, or annotation manager who has been assigned as owner of the annotation task.
- Parameters:
assignment (dtlpy.entities.assignment.Assignment assignment) – assignment entity
system_metadata (bool) – True, if you want to change metadata system
- Returns:
Assignment object
- Return type:
dtlpy.entities.assignment.Assignment assignment
Example:
assignment = task.assignments.update(assignment='assignment_entity', system_metadata=False)
Packages¶
- class LocalServiceRunner(client_api: ApiClient, packages, cwd=None, multithreading=False, concurrency=10, package: Optional[Package] = None, module_name='default_module', function_name='run', class_name='ServiceRunner', entry_point='main.py', mock_file_path=None)[source]¶
Bases:
object
Service Runner Class
- class Packages(client_api: ApiClient, project: Optional[Project] = None)[source]¶
Bases:
object
Packages Repository
The Packages class allows users to manage packages (code used for running in Dataloop’s FaaS) and their properties. Read more about Packages.
- build(package: Package, module_name=None, init_inputs=None, local_path=None, from_local=None)[source]¶
Instantiate a module from the package code. Returns a loaded instance of the runner class
- Parameters:
package – Package entity
module_name – Name of the module to build the runner class
init_inputs (str) – dictionary of the class init variables (if exists). will be used to init the module class
local_path (str) – local path of the package (if from_local=False - codebase will be downloaded)
from_local (bool) – bool. if true - codebase will not be downloaded (only use local files)
- Returns:
dl.BaseServiceRunner
- build_requirements(filepath) list [source]¶
Build a requirement list (list of packages your code requires to run) from a file path. The file listing the requirements MUST BE a txt file.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filepath – path of the requirements file
- Returns:
a list of dl.PackageRequirement
- Return type:
- static build_trigger_dict(actions, name='default_module', filters=None, function='run', execution_mode: TriggerExecutionMode = 'Once', type_t: TriggerType = 'Event')[source]¶
Build a trigger dictionary to trigger FaaS. Read more about FaaS Triggers.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
actions – list of dl.TriggerAction
name (str) – trigger name
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
function (str) – function name
execution_mode (str) – execution mode dl.TriggerExecutionMode
type_t (str) – trigger type dl.TriggerType
- Returns:
trigger dict
- Return type:
Example:
project.packages.build_trigger_dict(actions=dl.TriggerAction.CREATED, function='run', execution_mode=dl.TriggerExecutionMode.ONCE)
- static check_cls_arguments(cls, missing, function_name, function_inputs)[source]¶
Check class arguments. This method checks that the package function is correct.
Prerequisites: You must be in the role of an owner or developer.
- checkout(package: Optional[Package] = None, package_id: Optional[str] = None, package_name: Optional[str] = None)[source]¶
Checkout (switch) to a package.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) – package entity
package_id (str) – package id
package_name (str) – package name
Example:
project.packages.checkout(package='package_entity')
- delete(package: Optional[Package] = None, package_name=None, package_id=None)[source]¶
Delete a Package object.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) – package entity
package_id (str) – package id
package_name (str) – package name
- Returns:
True if success
- Return type:
Example:
project.packages.delete(package_name='package_name')
- deploy(package_id: Optional[str] = None, package_name: Optional[str] = None, package: Optional[Package] = None, service_name: Optional[str] = None, project_id: Optional[str] = None, revision: Optional[str] = None, init_input: Optional[Union[List[FunctionIO], FunctionIO, dict]] = None, runtime: Optional[Union[KubernetesRuntime, dict]] = None, sdk_version: Optional[str] = None, agent_versions: Optional[dict] = None, bot: Optional[Union[Bot, str]] = None, pod_type: Optional[InstanceCatalog] = None, verify: bool = True, checkout: bool = False, module_name: Optional[str] = None, run_execution_as_process: Optional[bool] = None, execution_timeout: Optional[int] = None, drain_time: Optional[int] = None, on_reset: Optional[str] = None, max_attempts: Optional[int] = None, force: bool = False, secrets: Optional[list] = None, **kwargs) Service [source]¶
Deploy a package. A service is required to run the code in your package.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package_id (str) – package id
package_name (str) – package name
package (dtlpy.entities.package.Package) – package entity
service_name (str) – service name
project_id (str) – project id
revision (str) – package revision - default=latest
init_input – config to run at startup
runtime (dict) – runtime resources
sdk_version (str) –
optional - string - sdk version
agent_versions (dict) –
dictionary - - optional -versions of sdk, agent runner and agent proxy
bot (str) – bot email
pod_type (str) – pod type dl.InstanceCatalog
verify (bool) – verify the inputs
checkout (bool) – checkout
module_name (str) – module name
run_execution_as_process (bool) – run execution as process
execution_timeout (int) – execution timeout
drain_time (int) – drain time
on_reset (str) – on reset
max_attempts (int) – Maximum execution retries in-case of a service reset
force (bool) – optional - terminate old replicas immediately
secrets (list) – list of the integrations ids
- Returns:
Service object
- Return type:
Example:
project.packages.deploy(service_name=package_name, execution_timeout=3 * 60 * 60, module_name=module.name, runtime=dl.KubernetesRuntime( concurrency=10, pod_type=dl.InstanceCatalog.REGULAR_S, autoscaler=dl.KubernetesRabbitmqAutoscaler( min_replicas=1, max_replicas=20, queue_length=20 ) ) )
- deploy_from_file(project, json_filepath)[source]¶
Deploy package and service from a JSON file.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
project (dtlpy.entities.project.Project) – project entity
json_filepath (str) – path of the file to deploy
- Returns:
the package and the services
Example:
project.packages.deploy_from_file(project='project_entity', json_filepath='json_filepath')
- static generate(name=None, src_path: Optional[str] = None, service_name: Optional[str] = None, package_type='default_package_type')[source]¶
Generate a new package. Provide a file path to a JSON file with all the details of the package and service to generate the package.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
Example:
project.packages.generate(name='package_name', src_path='src_path')
- get(package_name: Optional[str] = None, package_id: Optional[str] = None, checkout: bool = False, fetch=None) Package [source]¶
Get Package object to use in your code.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
- Returns:
Package object
- Return type:
Example:
project.packages.get(package_id='package_id')
- list(filters: Optional[Filters] = None, project_id: Optional[str] = None) PagedEntities [source]¶
List project packages.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
project_id (str) – project id
- Returns:
Paged entity
- Return type:
Example:
project.packages.list()
- open_in_web(package: Optional[Package] = None, package_id: Optional[str] = None, package_name: Optional[str] = None)[source]¶
Open the package in the web platform.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) – package entity
package_id (str) – package id
package_name (str) – package name
Example:
project.packages.open_in_web(package_id='package_id')
- pull(package: Package, version=None, local_path=None, project_id=None)[source]¶
Pull (download) the package to a local path.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) – package entity
version (str) – the package version to pull
local_path – the path of where to save the package
project_id – the project id that include the package
- Returns:
local path where the package pull
- Return type:
Example:
project.packages.pull(package='package_entity', local_path='local_path')
- push(project: Optional[Project] = None, project_id: Optional[str] = None, package_name: Optional[str] = None, src_path: Optional[str] = None, codebase: Optional[Union[GitCodebase, ItemCodebase, FilesystemCodebase]] = None, modules: Optional[List[PackageModule]] = None, is_global: Optional[bool] = None, checkout: bool = False, revision_increment: Optional[str] = None, version: Optional[str] = None, ignore_sanity_check: bool = False, service_update: bool = False, service_config: Optional[dict] = None, slots: Optional[List[PackageSlot]] = None, requirements: Optional[List[PackageRequirement]] = None, package_type=None, metadata=None) Package [source]¶
Push your local package to the UI.
Prerequisites: You must be in the role of an owner or developer.
Project will be taken in the following hierarchy: project(input) -> project_id(input) -> self.project(context) -> checked out
- Parameters:
project (dtlpy.entities.project.Project) – optional - project entity to deploy to. default from context or checked-out
project_id (str) – optional - project id to deploy to. default from context or checked-out
package_name (str) – package name
src_path (str) – path to package codebase
codebase (dtlpy.entities.codebase.Codebase) – codebase object
modules (list) – list of modules PackageModules of the package
is_global (bool) – is package is global or local
checkout (bool) – checkout package to local dir
revision_increment (str) – optional - str - version bumping method - major/minor/patch - default = None
version (str) – semver version f the package
ignore_sanity_check (bool) – NOT RECOMMENDED - skip code sanity check before pushing
service_update (bool) – optional - bool - update the service
:param dict service_config : Service object as dict. Contains the spec of the default service to create. :param list slots: optional - list of slots PackageSlot of the package :param list requirements: requirements - list of package requirements :param str package_type: default ‘faas’, options: ‘app’, ‘ml :param dict metadata: dictionary of system and user metadata
- Returns:
Package object
- Return type:
Example:
project.packages.push(package_name='package_name', modules=[module], version='1.0.0', src_path=os.getcwd() )
- revisions(package: Optional[Package] = None, package_id: Optional[str] = None)[source]¶
Get the package revision history.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) – package entity
package_id (str) – package id
Example:
project.packages.revisions(package='package_entity')
- test_local_package(cwd: Optional[str] = None, concurrency: Optional[int] = None, package: Optional[Package] = None, module_name: str = 'default_module', function_name: str = 'run', class_name: str = 'ServiceRunner', entry_point: str = 'main.py', mock_file_path: Optional[str] = None)[source]¶
Test local package in local environment.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
cwd (str) – path to the file
concurrency (int) – the concurrency of the test
package (dtlpy.entities.package.Package) – entities.package
module_name (str) – module name
function_name (str) – function name
class_name (str) – class name
entry_point (str) – the file to run like main.py
mock_file_path (str) – the mock file that have the inputs
- Returns:
list created by the function that tested the output
- Return type:
Example:
project.packages.test_local_package(cwd='path_to_package', package='package_entity', function_name='run')
- update(package: Package, revision_increment: Optional[str] = None) Package [source]¶
Update Package changes to the platform.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
package (dtlpy.entities.package.Package) –
revision_increment – optional - str - version bumping method - major/minor/patch - default = None
- Returns:
Package object
- Return type:
Example:
project.packages.delete(package='package_entity')
Codebases¶
- class Codebases(client_api: ApiClient, project: Optional[Project] = None, dataset: Optional[Dataset] = None, project_id: Optional[str] = None)[source]¶
Bases:
object
Codebase Repository
The Codebases class allows the user to manage codebases and their properties. The codebase is the code the user uploads for the user’s packages to run. Read more about codebase in our FaaS (function as a service).
- clone_git(codebase: Codebase, local_path: str)[source]¶
Clone code base
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
codebase (dtlpy.entities.codebase.Codebase) – codebase object
local_path (str) – local path
- Returns:
path where the clone will be
- Return type:
str
Example:
package.codebases.clone_git(codebase='codebase_entity', local_path='local_path')
- get(codebase_name: Optional[str] = None, codebase_id: Optional[str] = None, version: Optional[str] = None)[source]¶
Get a Codebase object to use in your code.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
Example:
package.codebases.get(codebase_name='codebase_name')
- static get_current_version(all_versions_pages, zip_md)[source]¶
This method returns the current version of the codebase and other versions found.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
all_versions_pages (codebase) – codebase object
zip_md – zipped file of codebase
- Returns:
current version and all versions found of codebase
- Return type:
Example:
package.codebases.get_current_version(all_versions_pages='codebase_entity', zip_md='path')
- list() PagedEntities [source]¶
List all codebases.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
Example:
package.codebases.list()
- Returns:
Paged entity
- Return type:
- list_versions(codebase_name: str) PagedEntities [source]¶
List all codebase versions.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
Example:
package.codebases.list_versions(codebase_name='codebase_name')
- pack(directory: str, name: Optional[str] = None, extension: str = 'zip', description: str = '', ignore_directories: Optional[List[str]] = None)[source]¶
Zip a local code directory and post to codebases.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
- Returns:
Codebase object
- Return type:
dtlpy.entities.codebase.Codebase
Example:
package.codebases.pack(directory='path_dir', name='codebase_name')
- pull_git(codebase, local_path)[source]¶
Pull (download) a codebase.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
codebase (dtlpy.entities.codebase.Codebase) – codebase object
local_path (str) – local path
- Returns:
path where the Pull will be
- Return type:
Example:
package.codebases.pull_git(codebase='codebase_entity', local_path='local_path')
- unpack(codebase: Optional[Codebase] = None, codebase_name: Optional[str] = None, codebase_id: Optional[str] = None, local_path: Optional[str] = None, version: Optional[str] = None)[source]¶
Unpack codebase locally. Download source code and unzip.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
- Returns:
String (dirpath)
- Return type:
Example:
package.codebases.unpack(codebase='codebase_entity', local_path='local_path')
Services¶
- class ServiceLog(_json: dict, service: Service, services: Services, start=None, follow=None, execution_id=None, function_name=None, replica_id=None, system=False, model_id=None, model_operation=None, project_id=None)[source]¶
Bases:
object
Service Log
- class Services(client_api: ApiClient, project: Optional[Project] = None, package: Optional[Package] = None, project_id=None)[source]¶
Bases:
object
Services Repository
The Services class allows the user to manage services and their properties. Services are created from the packages users create. See our documentation for more information about services.
- activate_slots(service: Service, project_id: Optional[str] = None, task_id: Optional[str] = None, dataset_id: Optional[str] = None, org_id: Optional[str] = None, user_email: Optional[str] = None, slots: Optional[List[PackageSlot]] = None, role=None, prevent_override: bool = True, visible: bool = True, icon: str = 'fas fa-magic', **kwargs)[source]¶
Activate service slots (creates buttons in the UI that activate services).
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service (dtlpy.entities.service.Service) – service entity
project_id (str) – project id
task_id (str) – task id
dataset_id (str) – dataset id
org_id (str) – org id
user_email (str) – user email
slots (list) – list of entities.PackageSlot
role (str) – user role MemberOrgRole.ADMIN, MemberOrgRole.owner, MemberOrgRole.MEMBER, MemberOrgRole.WORKER
prevent_override (bool) – True to prevent override
visible (bool) – visible
icon (str) – icon
kwargs – all additional arguments
- Returns:
list of user setting for activated slots
- Return type:
Example:
setting = package.services.activate_slots(service='service_entity', project_id='project_id', slots=List[entities.PackageSlot], icon='fas fa-magic')
- checkout(service: Optional[Service] = None, service_name: Optional[str] = None, service_id: Optional[str] = None)[source]¶
Checkout (switch) to a service.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service (dtlpy.entities.service.Service) – Service entity
service_name (str) – service name
service_id (str) – service id
Example:
package.services.checkout(service_id='service_id')
- delete(service_name: Optional[str] = None, service_id: Optional[str] = None)[source]¶
Delete Service object
Prerequisites: You must be in the role of an owner or developer. You must have a package.
You must provide at least ONE of the following params: service_id, service_name.
Example:
is_deleted = package.services.delete(service_id='service_id')
- deploy(service_name: Optional[str] = None, package: Optional[Package] = None, bot: Optional[Union[Bot, str]] = None, revision: Optional[str] = None, init_input: Optional[Union[List[FunctionIO], FunctionIO, dict]] = None, runtime: Optional[Union[KubernetesRuntime, dict]] = None, pod_type: Optional[InstanceCatalog] = None, sdk_version: Optional[str] = None, agent_versions: Optional[dict] = None, verify: bool = True, checkout: bool = False, module_name: Optional[str] = None, project_id: Optional[str] = None, driver_id: Optional[str] = None, func: Optional[Callable] = None, run_execution_as_process: Optional[bool] = None, execution_timeout: Optional[int] = None, drain_time: Optional[int] = None, max_attempts: Optional[int] = None, on_reset: Optional[str] = None, force: bool = False, secrets: Optional[list] = None, **kwargs) Service [source]¶
Deploy service.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service_name (str) – name
package (dtlpy.entities.package.Package) – package entity
bot (str) – bot email
revision (str) – package revision of version
init_input – config to run at startup
runtime (dict) – runtime resources
pod_type (str) – pod type dl.InstanceCatalog
sdk_version (str) –
optional - string - sdk version
agent_versions (str) –
dictionary - - optional -versions of sdk
verify (bool) – if true, verify the inputs
checkout (bool) – if true, checkout (switch) to service
module_name (str) – module name
project_id (str) – project id
driver_id (str) – driver id
func (Callable) – function to deploy
run_execution_as_process (bool) – if true, run execution as process
execution_timeout (int) – execution timeout in seconds
drain_time (int) – drain time in seconds
max_attempts (int) – maximum execution retries in-case of a service reset
on_reset (str) – what happens on reset
force (bool) – optional - if true, terminate old replicas immediately
secrets (list) – list of the integrations ids
kwargs – list of additional arguments
- Returns:
Service object
- Return type:
Example:
service = package.services.deploy(service_name=package_name, execution_timeout=3 * 60 * 60, module_name=module.name, runtime=dl.KubernetesRuntime( concurrency=10, pod_type=dl.InstanceCatalog.REGULAR_S, autoscaler=dl.KubernetesRabbitmqAutoscaler( min_replicas=1, max_replicas=20, queue_length=20 ) ) )
- deploy_from_local_folder(cwd=None, service_file=None, bot=None, checkout=False, force=False) Service [source]¶
Deploy from local folder in local environment.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
- Returns:
Service object
- Return type:
Example:
service = package.services.deploy_from_local_folder(cwd='file_path', service_file='service_file')
- execute(service: Optional[Service] = None, service_id: Optional[str] = None, service_name: Optional[str] = None, sync: bool = False, function_name: Optional[str] = None, stream_logs: bool = False, execution_input=None, resource=None, item_id=None, dataset_id=None, annotation_id=None, project_id=None) Execution [source]¶
Execute a function on an existing service.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service (dtlpy.entities.service.Service) – service entity
service_id (str) – service id
service_name (str) – service name
sync (bool) – wait for function to end
function_name (str) – function name to run
stream_logs (bool) – prints logs of the new execution. only works with sync=True
execution_input – input dictionary or list of FunctionIO entities
resource (str) – dl.PackageInputType - input type.
item_id (str) – str - optional - input to function
dataset_id (str) – str - optional - input to function
annotation_id (str) – str - optional - input to function
project_id (str) – str - resource’s project
- Returns:
entities.Execution
- Return type:
Example:
execution = package.services.execute(service='service_entity', function_name='run', item_id='item_id', project_id='project_id')
- get(service_name=None, service_id=None, checkout=False, fetch=None) Service [source]¶
Get service to use in your code.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
- Returns:
Service object
- Return type:
Example:
service = package.services.get(service_id='service_id')
- list(filters: Optional[Filters] = None) PagedEntities [source]¶
List all services (services can be listed for a package or for a project).
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
Paged entity
- Return type:
Example:
services = package.services.list()
- log(service, size=100, checkpoint=None, start=None, end=None, follow=False, text=None, execution_id=None, function_name=None, replica_id=None, system=False, view=True, until_completed=True, log_level='DEBUG', model_id: Optional[str] = None, model_operation: Optional[str] = None, project_id: Optional[str] = None)[source]¶
Get service logs.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service (dtlpy.entities.service.Service) – service object
size (int) – size
checkpoint (dict) – the information from the lst point checked in the service
start (str) – iso format time
end (str) – iso format time
follow (bool) – if true, keep stream future logs
text (str) – text
execution_id (str) – execution id
function_name (str) – function name
replica_id (str) – replica id
system (bool) – system
view (bool) – if true, print out all the logs
until_completed (bool) – wait until completed
log_level (str) – the log level to display dl.LoggingLevel
model_id (str) – model id
model_operation (str) – model operation action
project_id (str) – project id
- Returns:
ServiceLog entity
- Return type:
Example:
service_logs = package.services.log(service='service_entity')
- name_validation(name: str)[source]¶
Validation service name.
Prerequisites: You must be in the role of an owner or developer.
- Parameters:
name (str) – service name
Example:
package.services.name_validation(name='name')
- open_in_web(service: Optional[Service] = None, service_id: Optional[str] = None, service_name: Optional[str] = None)[source]¶
Open the service in web platform
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service_name (str) – service name
service_id (str) – service id
service (dtlpy.entities.service.Service) – service entity
Example:
package.services.open_in_web(service_id='service_id')
- pause(service_name: Optional[str] = None, service_id: Optional[str] = None, force: bool = False)[source]¶
Pause service.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
You must provide at least ONE of the following params: service_id, service_name
- Parameters:
- Returns:
True if success
- Return type:
Example:
success = package.services.pause(service_id='service_id')
- resume(service_name: Optional[str] = None, service_id: Optional[str] = None, force: bool = False)[source]¶
Resume service.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
You must provide at least ONE of the following params: service_id, service_name.
- Parameters:
- Returns:
json of the service
- Return type:
Example:
service_json = package.services.resume(service_id='service_id')
- revisions(service: Optional[Service] = None, service_id: Optional[str] = None)[source]¶
Get service revisions history.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
You must provide at leats ONE of the following params: service, service_id
- Parameters:
service (dtlpy.entities.service.Service) – Service entity
service_id (str) – service id
Example:
service_revision = package.services.revisions(service_id='service_id')
- status(service_name=None, service_id=None)[source]¶
Get service status.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
You must provide at least ONE of the following params: service_id, service_name
- Parameters:
- Returns:
status json
- Return type:
Example:
status_json = package.services.status(service_id='service_id')
- update(service: Service, force: bool = False) Service [source]¶
Update service changes to platform.
Prerequisites: You must be in the role of an owner or developer. You must have a package.
- Parameters:
service (dtlpy.entities.service.Service) – Service entity
force (bool) – optional - terminate old replicas immediately
- Returns:
Service entity
- Return type:
Example:
service = package.services.update(service='service_entity')
Bots¶
- class Bots(client_api: ApiClient, project: Project)[source]¶
Bases:
object
Bots Repository
The Bots class allows the user to manage bots and their properties.
- create(name: str, return_credentials: bool = False)[source]¶
Create a new Bot.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
Bot object
- Return type:
Example:
service.bots.delete(name='bot', return_credentials=False)
- delete(bot_id: Optional[str] = None, bot_email: Optional[str] = None)[source]¶
Delete a Bot.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
You must provide at least ONE of the following params: bot_id, bot_email
- Parameters:
- Returns:
True if successful
- Return type:
Example:
service.bots.delete(bot_id='bot_id')
- get(bot_email: Optional[str] = None, bot_id: Optional[str] = None, bot_name: Optional[str] = None)[source]¶
Get a Bot object.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
Bot object
- Return type:
Example:
service.bots.get(bot_id='bot_id')
Triggers¶
- class Triggers(client_api: ApiClient, project: Optional[Project] = None, service: Optional[Service] = None, project_id: Optional[str] = None, pipeline: Optional[Pipeline] = None)[source]¶
Bases:
object
Triggers Repository
The Triggers class allows users to manage triggers and their properties. Triggers activate services. See our documentation for more information on triggers.
- create(service_id: Optional[str] = None, trigger_type: TriggerType = TriggerType.EVENT, name: Optional[str] = None, webhook_id=None, function_name='run', project_id=None, active=True, filters=None, resource: TriggerResource = TriggerResource.ITEM, actions: Optional[TriggerAction] = None, execution_mode: TriggerExecutionMode = TriggerExecutionMode.ONCE, start_at=None, end_at=None, inputs=None, cron=None, pipeline_id=None, pipeline=None, pipeline_node_id=None, root_node_namespace=None, **kwargs) BaseTrigger [source]¶
Create a Trigger. Can create two types: a cron trigger or an event trigger. Inputs are different for each type
Prerequisites: You must be in the role of an owner or developer. You must have a service.
Inputs for all types:
- Parameters:
service_id (str) – Id of services to be triggered
trigger_type (str) – can be cron or event. use enum dl.TriggerType for the full list
name (str) – name of the trigger
webhook_id (str) – id for webhook to be called
function_name (str) – the function name to be called when triggered (must be defined in the package)
project_id (str) – project id where trigger will work
active (bool) – optional - True/False, default = True, if true trigger is active
Inputs for event trigger: :param dtlpy.entities.filters.Filters filters: optional - Item/Annotation metadata filters, default = none :param str resource: optional - Dataset/Item/Annotation/ItemStatus, default = Item :param str actions: optional - Created/Updated/Deleted, default = create :param str execution_mode: how many times trigger should be activated; default is “Once”. enum dl.TriggerExecutionMode
Inputs for cron trigger: :param start_at: iso format date string to start activating the cron trigger :param end_at: iso format date string to end the cron activation :param inputs: dictionary “name”:”val” of inputs to the function :param str cron: cron spec specifying when it should run. more information: https://en.wikipedia.org/wiki/Cron :param str pipeline_id: Id of pipeline to be triggered :param pipeline: pipeline entity to be triggered :param str pipeline_node_id: Id of pipeline root node to be triggered :param root_node_namespace: namespace of pipeline root node to be triggered
- Returns:
Trigger entity
- Return type:
Example:
service.triggers.create(name='triggername', execution_mode=dl.TriggerExecutionMode.ONCE, resource='Item', actions='Created', function_name='run', filters={'$and': [{'hidden': False}, {'type': 'file'}]} )
- delete(trigger_id=None, trigger_name=None)[source]¶
Delete Trigger object
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
True is successful error if not
- Return type:
Example:
service.triggers.delete(trigger_id='trigger_id')
- get(trigger_id=None, trigger_name=None) BaseTrigger [source]¶
Get Trigger object
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
Trigger entity
- Return type:
Example:
service.triggers.get(trigger_id='trigger_id')
- list(filters: Optional[Filters] = None) PagedEntities [source]¶
List triggers of a project, package, or service.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
filters (dtlpy.entities.filters.Filters) – Filters entity or a dictionary containing filters parameters
- Returns:
Paged entity
- Return type:
Example:
service.triggers.list()
- name_validation(name: str)[source]¶
This method validates the trigger name. If name is not valid, this method will return an error. Otherwise, it will not return anything.
- Parameters:
name (str) – trigger name
- resource_information(resource, resource_type, action='Created')[source]¶
Returns which function should run on an item (based on global triggers).
Prerequisites: You must be a superuser to run this method.
- Parameters:
resource – ‘Item’ / ‘Dataset’ / etc
resource_type – dictionary of the resource object
action – ‘Created’ / ‘Updated’ / etc.
Example:
service.triggers.resource_information(resource='Item', resource_type=item_object, action='Created')
- update(trigger: BaseTrigger) BaseTrigger [source]¶
Update trigger
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
trigger (dtlpy.entities.trigger.Trigger) – Trigger entity
- Returns:
Trigger entity
- Return type:
Example:
service.triggers.update(trigger='trigger_entity')
Executions¶
- class Executions(client_api: ApiClient, service: Optional[Service] = None, project: Optional[Project] = None)[source]¶
Bases:
object
Service Executions Repository
The Executions class allows the users to manage executions (executions of services) and their properties. See our documentation for more information about executions.
- create(service_id: Optional[str] = None, execution_input: Optional[list] = None, function_name: Optional[str] = None, resource: Optional[PackageInputType] = None, item_id: Optional[str] = None, dataset_id: Optional[str] = None, annotation_id: Optional[str] = None, project_id: Optional[str] = None, sync: bool = False, stream_logs: bool = False, return_output: bool = False, return_curl_only: bool = False, timeout: Optional[int] = None) Execution [source]¶
Execute a function on an existing service
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
service_id (str) – service id to execute on
execution_input (List[FunctionIO] or dict) – input dictionary or list of FunctionIO entities
function_name (str) – function name to run
resource (str) – input type.
item_id (str) – optional - item id as input to function
dataset_id (str) – optional - dataset id as input to function
annotation_id (str) – optional - annotation id as input to function
project_id (str) – resource’s project
sync (bool) – if true, wait for function to end
stream_logs (bool) – prints logs of the new execution. only works with sync=True
return_output (bool) – if True and sync is True - will return the output directly
return_curl_only (bool) – return the cURL of the creation WITHOUT actually do it
timeout (int) – int, seconds to wait until TimeoutError is raised. if <=0 - wait until done - by default wait take the service timeout
- Returns:
execution object
- Return type:
Example:
service.executions.create(function_name='function_name', item_id='item_id', project_id='project_id')
- create_batch(service_id: str, filters, function_name: Optional[str] = None, execution_inputs: Optional[list] = None, wait=True)[source]¶
Execute a function on an existing service
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
execution object
- Return type:
Example:
command = service.executions.create_batch( execution_inputs=dl.FunctionIO(type=dl.PackageInputType.STRING, value='test', name='string'), filters=dl.Filters(field='dir', values='/test'), function_name='run')
- get(execution_id: Optional[str] = None, sync: bool = False) Execution [source]¶
Get Service execution object
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
Service execution object
- Return type:
Example:
service.executions.get(execution_id='execution_id')
- increment(execution: Execution)[source]¶
Increment the number of attempts that an execution is allowed to attempt to run a service that is not responding.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
execution (dtlpy.entities.execution.Execution) –
- Returns:
int
- Return type:
Example:
service.executions.increment(execution='execution_entity')
- list(filters: Optional[Filters] = None) PagedEntities [source]¶
List service executions
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
filters (dtlpy.entities.filters.Filters) – dl.Filters entity to filters items
- Returns:
Paged entity
- Return type:
Example:
service.executions.list()
- logs(execution_id: str, follow: bool = True, until_completed: bool = True)[source]¶
executions logs
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
executions logs
Example:
service.executions.logs(execution_id='execution_id')
- progress_update(execution_id: str, status: Optional[ExecutionStatus] = None, percent_complete: Optional[int] = None, message: Optional[str] = None, output: Optional[str] = None, service_version: Optional[str] = None)[source]¶
Update Execution Progress.
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
- Returns:
Service execution object
- Return type:
Example:
service.executions.progress_update(execution_id='execution_id', status='complete', percent_complete=100)
- rerun(execution: Execution, sync: bool = False)[source]¶
Rerun execution
Prerequisites: You must be in the role of an owner or developer. You must have a service.
- Parameters:
execution (dtlpy.entities.execution.Execution) –
sync (bool) – wait for the execution to finish
- Returns:
Execution object
- Return type:
Example:
service.executions.rerun(execution='execution_entity')
- terminate(execution: