Tutorials

Getting Started With Dataloop SDK

Getting started with Dataloop SDK

Getting Started With Dataloop SDK

• 1. Dataloop SDK Overview
  • 1.1. About this guide
  • 1.2. Installing Prerequisite Software
    • 1.2.1. Python
      • 1.2.1.1. To download Python:
    • 1.2.2. Dataloop SDK Package
    • 1.2.3. pip
      • 1.2.3.1. To verify if pip exists on your system:
      • 1.2.3.2. To bootstrap pip:
    • 1.2.4. DTLPY Package
      • 1.2.4.1. To install the Dataloop SDK Package:
  • 1.3. SDK Login
  • 1.4. Machine-to-Machine Login
  • 1.5. Datasets
  • 1.6. Creating a New Dataset
    • 1.6.1. To create a new project:
    • 1.6.2. To select the new project:
    • 1.6.3. To select the new project using a project_id:
    • 1.6.4. To create a new dataset:
  • 1.7. Uploading items
    • 1.7.1. To select the dataset:
    • 1.7.2. To upload an item to a dataset:
    • 1.7.3. Exercise 1
  • 1.8. Getting Items
    • 1.8.1. Getting a Single Item
      • 1.8.1.1. To get a single item:
    • 1.8.2. Exercise 2
    • 1.8.3. Getting All Items
      • 1.8.3.1. To get all items:
  • 1.9. Annotating Items
    • 1.9.1. Classification
      • 1.9.1.1. To Add a Classification Label to a Dataset Recipe:
  • 1.10. Point Markers
    • 1.10.1. To Add/Upload a Point Marker Label to a Dataset Recipe:
    • 1.10.2. Exercise 3
  • 1.11. Working with Filters
    • 1.11.1. Creating Filters
      • 1.11.1.1. To Create a Filter Query:
      • 1.11.1.2. To Add a Filter Parameter:
      • 1.11.1.3. To Apply the Filter Query:
  • 1.12. Using Filters to Replace Data
    • 1.12.1. To Create the Replacement Filter Query:
    • 1.12.2. Exercise 4
  • 1.13. Working with Item Metadata
    • 1.13.1. Adding a New User Metadata Field to an Item
      • 1.13.1.1. To Import the datetime Module:
      • 1.13.1.2. To Assign the Current Date to a New Metadata Field:
      • 1.13.1.3. To Create Metadata fields for Multiple Items using Filters:
    • 1.13.2. Exercise 5
  • 1.14. Creating Tasks
    • 1.14.1. To Create a Task
    • 1.14.2. Exercise 6
  • 1.15. Logging out

Data Management Tutorial

Tutorials for data management

Data Management Tutorial

• 1. Dataset Binding with AWS
  • 1.1. Create the Lambda
  • 1.2. Add a Layer to the Lambda
  • 1.3. Create the Bucket Events
• 2. Dataset Manual item upstream sync
• 3. Integrations
  • 3.1. Amazon Web Services integration
  • 3.2. Create AWS Access-Key integration
  • 3.3. Create AWS Cross Account integration
• 4. Storage Drivers
  • 4.1. External Storage Driver
  • 4.2. Create Drivers in the Platform (browser)
• 5. Dataset Binding with Azure
  • 5.1. Create the Blob Function
  • 5.2. Deploy your function
• 6. Integrations
  • 6.1. Microsoft Azure integration
  • 6.2. Create integration With Microsoft Azure
• 7. Storage Drivers
  • 7.1. External Storage Driver
  • 7.2. Create Drivers in the Platform (browser)
• 8. Dataset Binding with Google Cloud Storage
  • 8.1. Create the Cloud Function
  • 8.2. You’re good to go!
    • 8.2.1. For pictures examples:
• 9. Integrations
  • 9.1. GCP integration
  • 9.2. Create integration With Google Cloud Provider
• 10. Storage Drivers
  • 10.1. External Storage Driver
  • 10.2. Create Drivers in the Platform (browser)
• 11. Manage Datasets
  • 11.1. Create Dataset
  • 11.2. Create Dataset With Cloud Storage Driver
  • 11.3. Retrieve Datasets
  • 11.4. Create Directory
  • 11.5. Deep Copy a Folder to Another Dataset
• 12. Data Versioning
  • 12.1. Clone Datasets
  • 12.2. Merge Datasets
• 13. Upload & Manage Data & Metadata
  • 13.1. Upload Specific Files
  • 13.2. Upload All Files in a Folder
  • 13.3. Upload Items From URL Links
  • 13.4. Upload Items with Metadata
• 14. Upload & Manage Annotations
  • 14.1. Upload User Metadata
  • 14.2. Convert Annotations To COCO Format
  • 14.3. Upload Entire Directory and their Corresponding Dataloop JSON Annotations
  • 14.4. Upload Annotations To Video Item
  • 14.5. Upload Annotations In VTT Format
  • 14.6. Upload Audio Annotation to an Audio File
  • 14.7. Set Attributes On Annotations
  • 14.8. Free Text Attribute
  • 14.9. Range Attributes (Slider in UI)
  • 14.10. CheckBox Attribute (Multiple choice)
  • 14.11. Radio Button Attribute (Single Choice)
  • 14.12. Yes/No Attribute
  • 14.13. Show Annotations Over Image
  • 14.14. Download Data, Annotations & Metadata
  • 14.15. Download Items and Annotations
  • 14.16. Filter by Item and/or Annotation
  • 14.17. Filter by Annotations
  • 14.18. Download Annotations in COCO/YOLO/VOC Format
• 15. Advance SDK Filters
  • 15.1. Filter Operators
    • 15.1.1. Equal
    • 15.1.2. Not Equal
    • 15.1.3. Greater Than
    • 15.1.4. Less Than
    • 15.1.5. In a List
    • 15.1.6. Exist
  • 15.2. SDK defaults
    • 15.2.1. Hidden Items and Directories
  • 15.3. Delete a Filter
  • 15.4. Full Examples
    • 15.4.1. How to filter items that were created between specific dates?
    • 15.4.2. How to filter items that don’t have a specific label?
• 16. The Dataloop Query Language - DQL
  • 16.1. Filters
    • 16.1.1. Filters - Field & Value
      • 16.1.1.1. Field
      • 16.1.1.2. Value
    • 16.1.2. Sort - Field & Value
      • 16.1.2.1. Field
      • 16.1.2.2. Value
  • 16.2. Filter Annotations
  • 16.3. Filter Annotations by the Annotations’ Item
  • 16.4. Filters Method - “Or” and “And”
    • 16.4.1. And
    • 16.4.2. Or
  • 16.5. Delete Filtered Items
  • 16.6. Annotation Filtering Fields
    • 16.6.1. More Filter Options
  • 16.7. Full Examples
    • 16.7.1. How to filter annotations by their label?
  • 16.8. Advanced Filtering Operators
• 17. The Dataloop Query Language - DQL
  • 17.1. Filters
    • 17.1.1. Filters - Field & Value
      • 17.1.1.1. Field
      • 17.1.1.2. Value
    • 17.1.2. Sort - Field & Value
      • 17.1.2.1. Field
      • 17.1.2.2. Value
  • 17.2. Filter Items
  • 17.3. Filter Items by the Items’ Annotations
  • 17.4. Filters Method - “Or” and “And”
    • 17.4.1. And
    • 17.4.2. Or
  • 17.5. Update User Metadata of Filtered Items
  • 17.6. Delete Filtered Items
  • 17.7. Item Filtering Fields
    • 17.7.1. More Filter Options
  • 17.8. Full Examples
    • 17.8.1. How to filter items by their annotations label?
    • 17.8.2. How to filter items by completed and approved status?
    • 17.8.3. How to filter items by completed status (with items who are approved as well)?
    • 17.8.4. How to filter items by only completed status?
    • 17.8.5. How to filter unassigned items?
    • 17.8.6. How to filter items by a specific folder?
    • 17.8.7. Get all items named foo.bar
    • 17.8.8. Sort files of size 0-5 MB by name, in ascending order
    • 17.8.9. Sort with multiple fields: Sort Items by labels ascending and createdAt descending
  • 17.9. Advanced Filtering Operators
  • 17.10. Response to DQL Query
  • 17.11. Using Custom DQL Filter
• 18. Pagination
  • 18.1. Pages
  • 18.2. Iterator of Items
  • 18.3. Set page_size
• 19. Working with Metadata
  • 19.1. User Metadata
    • 19.1.1. Metadata Data Types
    • 19.1.2. String
    • 19.1.3. Number
    • 19.1.4. Boolean
    • 19.1.5. Null – add metadata with no information
    • 19.1.6. List
    • 19.1.7. Add new metadata to a list without losing existing data
    • 19.1.8. Add metadata to an item’s user metadata
    • 19.1.9. Modify an existing user metadata field
    • 19.1.10. Add metadata to annotations’ user metadata
    • 19.1.11. Filter items by user metadata
      • 19.1.11.1. 1. Get your dataset
      • 19.1.11.2. 2. Add metadata to an item
      • 19.1.11.3. 3. Create a filter
      • 19.1.11.4. 4. Filter by your written key
      • 19.1.11.5. 5. Get filtered items

FaaS Tutorial

Tutorials for FaaS

FaaS Tutorial

• 1. Concept
  • 1.1. Use Cases
• 2. Introduction
• 3. Basic Use Case: Single Function
  • 3.1. Create and Deploy a Sample Function
  • 3.2. Execute the function
• 4. Multiple Functions and Modules
  • 4.1. Multiple Functions
    • 4.1.1. Create and Deploy a Package of Several Functions
    • 4.1.2. Write your code
    • 4.1.3. Define the module
    • 4.1.4. Push the package
    • 4.1.5. Deploy a service
    • 4.1.6. Trigger the service
    • 4.1.7. Execute the function
    • 4.1.8. Review the function’s logs
    • 4.1.9. Pause the service:
  • 4.2. Multiple Modules
• 5. Advance Configurations
  • 5.1. Service Runtime
  • 5.2. Autoscaler
  • 5.3. Using Secrets in a Function
• 6. UI Toolbars
  • 6.1. Overview
  • 6.2. Define a UI slot in the platform
  • 6.3. Example
  • 6.4. Update the Package and Service with the Slot
  • 6.5. Activate the UI slot
• 7. Executions Control
  • 7.1. Execution Termination
  • 7.2. Execution Timeout
• 8. FaaS Docker Image
  • 8.1. Our Docker Image
  • 8.2. Public Docker Images
  • 8.3. Build Your Own Docker Image
  • 8.4. Using Private Docker Registry
• 9. Example: Model Annotations Service
  • 9.1. Service Code
  • 9.2. Define the module
  • 9.3. Model and weights files
  • 9.4. Push the Package
  • 9.5. Deploy The Service
  • 9.6. Trigger the Service
  • 9.7. Uploading Model Weights as Artifacts

Task Workflows

Tutorials for workforce management

Task Workflows

• 1. Creating Tasks
  • 1.1. Create A Task (Annotation task or QA task) Using Filter
  • 1.2. List of Items
  • 1.3. Entire Dataset
• 2. Create a Pulling Task
• 3. Add items to an existing task
  • 3.1. By Filters
  • 3.2. Single or a List of Items
  • 3.3. Creating Consensus Tasks
  • 3.4. Creating Pipeline Tasks Node
• 4. Managing Tasks & Assignments
  • 4.1. Get Task
  • 4.2. Get Assignments
    • 4.2.1. Get Assignment Items
• 5. Redistribute and Reassign Assignments
  • 5.1. Redistribute
    • 5.1.1. Reassign
    • 5.1.2. Delete Task and Assignments
      • 5.1.2.1. Delete Task
      • 5.1.2.2. Delete Assignment

Image Annotations

Tutorials for creating all types of image annotations

Image Annotations

• 1. Setup
  • 1.1. Initiation
  • 1.2. Optional Plotting
• 2. Classification Point and Pose
  • 2.1. Classification
  • 2.2. Classify Multiple Items
  • 2.3. Create a Point Annotation
  • 2.4. Pose Annotation
• 3. Bounding Box and Cuboid
  • 3.1. Create Box Annotation
  • 3.2. Create a Rotated Bounding Box Annotation
  • 3.3. Convert Semantic Segmentation to Bounding Box
  • 3.4. Create Cuboid (3D Box) Annotation
• 4. Polygon and Polyline
  • 4.1. Create Single Polygon/Polyline Annotation
  • 4.2. Create Multiple Polygons from Mask
  • 4.3. Convert Mask Annotations to Polygon
  • 4.4. Convert Polygon Annotation to Mask
• 5. Ellipse and Item Description
  • 5.1. Create Ellipse Annotation
  • 5.2. Item Description
• 6. Segmentation
  • 6.1. Init Segmentation
  • 6.2. Create a Semantic Segmentation Annotation
  • 6.3. Convert Mask to Polygon
  • 6.4. Convert Polygon to Mask
  • 6.5. Create Semantic Segmentation from Image Mask and Upload
• 7. Advance Tutorials
  • 7.1. Copy Annotations Between Items
  • 7.2. Show Images & Annotations
  • 7.3. Show Annotations from JSON file (Dataloop format)
  • 7.4. Count total number of annotations
  • 7.5. Parenting Annotations
  • 7.6. Change Annotations’ Label

Video Annotations

Tutorials for annotating videos

Video Annotations

• 1. Setup
• 2. Create A Single annotation
• 3. Adding Multiple Annotations Using Annotation Builder
• 4. Read Frames of an Annotation
• 5. Create Frame Snapshots from Video
• 6. Play An Item In Video Player
• 7. Show Annotations in a Specified Frame

Recipe and Ontology

Tutorials for managing ontologies, labels, and recipes

Recipe and Ontology

• 1. Recipe and Ontology Concepts
• 2. Preparing - Entities setup
• 3. Create an Ontology
• 4. Labels
• 5. Attributes
• 6. Add attributes to the ontology
• 7. Read Ontology Attributes
• 8. Getting all labels is (including children):
• 9. Working With Recipes
• 10. Cloning Recipes
• 11. View Dataset Labels
• 12. Add Labels by Dataset
• 13. Add Labels Using Label Object
• 14. Add a Label and Sub-Labels
• 15. Add Hierarchy Labels with Nested
• 16. Delete Labels by Dataset
• 17. Update Label Features

Model Management

Tutorials for creating and managing ML model

Model Management

• 1. Model Management
  • 1.1. Quick overview
  • 1.2. Introduction
    • 1.2.1. Package and Model Entities
      • 1.2.1.1. Package
      • 1.2.1.2. Model
    • 1.2.2. Additional Package components
      • 1.2.2.1. Artifacts and Codebase
      • 1.2.2.2. The Model adapter
    • 1.2.3. Model comparison
      • 1.2.3.1. Offline vs online mode
• 2. Offline model training, logging metrics in Dataloop
  • 2.1. Create Dataloop entities
• 3. Tutorial: using models from the AI library
  • 3.1. Clone and deploy a model
  • 3.2. Train on a custom dataset
    • 3.2.1. Dataset subsets
    • 3.2.2. Deploying the model
  • 3.3. Predict on a single item
• 4. Training a classification model with ResNet
  • 4.1. Create the Package and pretrained Model in your project
    • 4.1.1. Run a pretrained model
    • 4.1.2. Predict on an item
  • 4.2. Train on new dataset
  • 4.3. Start the training
  • 4.4. Save the Model
  • 4.5. Predict on our newly trained model
• 5. Create your own model
  • 5.1. Inference from a pre-trained model
    • 5.1.1. Create a model adapter
    • 5.1.2. Push the package
    • 5.1.3. Upload artifacts and create the model
  • 5.2. Checking that your model works
    • 5.2.1. Via the UI
    • 5.2.2. Via the SDK
• 6. Dataloop dataloader
  • 6.1. Object detection examples
  • 6.2. Segmentation examples
  • 6.3. Setting a label map
  • 6.4. Batch size and batch_size and collate_fn

Applications

Developing and working with the Dataloop Applications

Applications

• 1. What are Dataloop Applications?
  • 1.1. Start an App!
  • 1.2. Apps
  • 1.3. DPK - Dataloop Package
    • 1.3.1. Panels and Toolbars
    • 1.3.2. Modules and Functions
    • 1.3.3. Services
    • 1.3.4. All the Others
  • 1.4. App Manifest - dataloop.json
  • 1.5. Store
  • 1.6. Typical Workflow
• 2. Create and test a Dataloop app locally.
  • 2.1. How to create and test Dataloop Applications?
    • 2.1.1. Tl;dr
    • 2.1.2. Too Short; Want More?
    • 2.1.3. Prerequisites:
    • 2.1.4. Create a React Application:
    • 2.1.5. Create a host:
    • 2.1.6. Generate SSL certificate
      • 2.1.6.1. CREATE-REACT-APP:
      • 2.1.6.2. APACHE SETUP:
    • 2.1.7. Test the app locally
    • 2.1.8. Using Typescript :
    • 2.1.9. FAQ section:
• 3. What are Dataloop Applications?
• 4. Debug Applications (Development stage)
  • 4.1. Basic requirements
  • 4.2. How to add a debug app in the Dataloop Platform
  • 4.3. Tests
  • 4.4. Item Viewer Example
• 5. Dataloop JSON Examples
  • 5.1. Floating Window
  • 5.2. Item Viewer
  • 5.3. Pipeline Node
  • 5.4. Toolbars
• 6. UI Toolbars
  • 6.1. When are we going to need toolbars?
  • 6.2. The current available toolbar locations in the platform
  • 6.3. Toolbar Structure in a DPK/App entity
  • 6.4. Toolbar Examples in a DPK/App entity