A new cookbook about alignment evaluation
This commit is contained in:
Tomasz Wrona committed
1 parent
6ba9e70b69
commit
f53e0d8aa5
2 files changed
+623
No files matched your search
@@ -0,0 +1,622 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"gpuType": "T4"
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"accelerator": "GPU"
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Evaluating Alignment of Text-to-image Diffusion Models\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"[](https://colab.research.google.com/github/roboflow/supervision/blob/develop/docs/notebooks/evaluating-alignment-of-text-to-image-diffusion-models.ipynb)\n",
|
||||
"\n",
|
||||
"Click the `Open in Colab` button to run the cookbook on Google Colab."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "pviK1VFFNp8h"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Introduction\n",
|
||||
"\n",
|
||||
"It is a common scenario to evaluate text-to-image models for its alignment to the prompt. One way to test it is to use a set of prompts, consisting of number of objects and their basic physical properties (e.g. color), to generate images and manually evaluate the results. This process can be greatly improved using object detection models."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FtZXoFjPt569"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Before you start\n",
|
||||
"\n",
|
||||
"Let's make sure that we have access to GPU. We can use `nvidia-smi` command to do that. In case of any problems navigate to `Edit` -> `Notebook settings` -> `Hardware accelerator`, set it to `GPU`, and then click `Save`."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "1yOkAwH-UYPM"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "MVfLwWd8D1ne",
|
||||
"outputId": "5269d336-94e8-4a70-b5ec-9d30ee3fc57d"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Thu Feb 15 12:41:37 2024 \n",
|
||||
"+---------------------------------------------------------------------------------------+\n",
|
||||
"| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |\n",
|
||||
"|-----------------------------------------+----------------------+----------------------+\n",
|
||||
"| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
|
||||
"| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n",
|
||||
"| | | MIG M. |\n",
|
||||
"|=========================================+======================+======================|\n",
|
||||
"| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n",
|
||||
"| N/A 69C P0 28W / 70W | 0MiB / 15360MiB | 0% Default |\n",
|
||||
"| | | N/A |\n",
|
||||
"+-----------------------------------------+----------------------+----------------------+\n",
|
||||
" \n",
|
||||
"+---------------------------------------------------------------------------------------+\n",
|
||||
"| Processes: |\n",
|
||||
"| GPU GI CI PID Type Process name GPU Memory |\n",
|
||||
"| ID ID Usage |\n",
|
||||
"|=======================================================================================|\n",
|
||||
"| No running processes found |\n",
|
||||
"+---------------------------------------------------------------------------------------+\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!nvidia-smi"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Install required packages¶\n",
|
||||
"\n",
|
||||
"In this cookbook, we'll leverage the following Python packages:\n",
|
||||
"* `diffusers` for image generation pipelines,\n",
|
||||
"* `ultralytics` for running object detection,\n",
|
||||
"* and `supervision` for visualizing detections."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "xg5JFGa4UjZq"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"!pip install -q torch diffusers accelerate ultralytics dill git+https://github.com/openai/CLIP.git supervision==0.18.0"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "ri8GPZ3SD7LT",
|
||||
"outputId": "e5799a1d-aba3-496f-f30a-2cc627288ea2"
|
||||
},
|
||||
"execution_count": 2,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
" Preparing metadata (setup.py) ... \u001B[?25l\u001B[?25hdone\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Imports"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "U0qBpU5YU3P9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import itertools\n",
|
||||
"\n",
|
||||
"import cv2\n",
|
||||
"from diffusers import StableDiffusionXLPipeline\n",
|
||||
"import numpy as np\n",
|
||||
"from PIL import Image\n",
|
||||
"import supervision as sv\n",
|
||||
"import torch\n",
|
||||
"from ultralytics import YOLO\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "sBKUQMdwJTv1"
|
||||
},
|
||||
"execution_count": 3,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Generating an image\n",
|
||||
"\n",
|
||||
"We'll use [SDXL](https://huggingface.co/docs/diffusers/using-diffusers/sdxl) model to generate our image. Let's initialize our pipeline first:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ZzEhndAyU_EF"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"pipeline = StableDiffusionXLPipeline.from_pretrained(\n",
|
||||
" \"stabilityai/stable-diffusion-xl-base-1.0\",\n",
|
||||
" torch_dtype=torch.float16,\n",
|
||||
" variant=\"fp16\",\n",
|
||||
" use_safetensors=True,\n",
|
||||
").to(\"cuda\")"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/",
|
||||
"height": 49,
|
||||
"referenced_widgets": [
|
||||
"7c3c01ceea8a43e3b567318f52a07aaf",
|
||||
"1f46daee02914c0ca99461709e4b6df5",
|
||||
"545343e7f952440380c0f4d55c68d269",
|
||||
"a1bac32e59f747e88ec3f4617ff2364b",
|
||||
"7b66e7324c3c4378bc3b1f811d8a34ee",
|
||||
"26e1f67a586f44198cadc423c26fc0e0",
|
||||
"d5f5be390f364ca6bec373f320990e5f",
|
||||
"949a6331579c43d18a3176ad3bd16ffa",
|
||||
"e3724b0575da46a492f466475f34e873",
|
||||
"91fd00244246412883f0be99c3946b73",
|
||||
"4ec1a82ed16e454dbcd8b175be7a9340"
|
||||
]
|
||||
},
|
||||
"id": "qCwxwGcuEHJ-",
|
||||
"outputId": "585e93e0-1c91-4af3-caf1-641fdd085ed6"
|
||||
},
|
||||
"execution_count": 4,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]"
|
||||
],
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"version_major": 2,
|
||||
"version_minor": 0,
|
||||
"model_id": "7c3c01ceea8a43e3b567318f52a07aaf"
|
||||
}
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"In this example, we'll focus on generating an image of a black cat playing with a blue ball next to a parked white car. We don't care about the aesthetic aspect of the image."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XrQ8ose0Vm-T"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"image = pipeline(\n",
|
||||
" prompt=\"a black cat playing with a blue ball next to a parked white car, wide angle, photorealistic\",\n",
|
||||
" negative_prompt=\"low quality, blurred, text, illustration\",\n",
|
||||
" generator=torch.manual_seed(9213799),\n",
|
||||
" num_inference_steps=50,\n",
|
||||
" width=1024,\n",
|
||||
" height=768,\n",
|
||||
").images[0]\n",
|
||||
"image"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/",
|
||||
"height": 817,
|
||||
"referenced_widgets": [
|
||||
"f90eda761c76428d8e70b3c4964146fd",
|
||||
"ff14ced44585465c93176a7fae9cd743",
|
||||
"b4a1f88509324fc2a4d9a0dba5bffc87",
|
||||
"752619d45e5547b39351fef9f70cbbd4",
|
||||
"737dc8acba5a4a1cbe1ce610b1d2808e",
|
||||
"ce25b27561634668b22d456bcabe7364",
|
||||
"22336fd9896b468a831ee105a86199e1",
|
||||
"f5d1ff459ebe487e82a58d2d4da07e0f",
|
||||
"f6388be9dd9a4874afb1dc7fd3023ffa",
|
||||
"eadd966752464146ad7ad8396dba7fbf",
|
||||
"60a4c01a6e8843fb8fe1b42a2ef7bf0d"
|
||||
]
|
||||
},
|
||||
"id": "TaWaIpjQERtf",
|
||||
"outputId": "88c8b8d6-1058-4977-f750-716f3b50e76a"
|
||||
},
|
||||
"execution_count": 5,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
" 0%| | 0/50 [00:00<?, ?it/s]"
|
||||
],
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"version_major": 2,
|
||||
"version_minor": 0,
|
||||
"model_id": "f90eda761c76428d8e70b3c4964146fd"
|
||||
}
|
||||
},
|
||||
"metadata": {}
|
||||
},
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<PIL.Image.Image image mode=RGB size=1024x768>"
|
||||
],
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABAAAAAMACAIAAAA12IJaAAEAAElEQVR4nHz97ZbkypEkCIp+GOARecmqrp595n2P+bVP1lNn92x3kTczwgFTVZkfaoB7XvaZIJmMD3c4YGYwiIqKisr/+//8/xgrQv9Mgw8bA/MpX//c5OkaJqlarKKKmbk5RZiUnKqC/aNs5/hIGcQ+aztznzKwPUxVikpslrvjv/+x/3Ca688T/zX117OqLEqTWgRESAgoFAAQAiAJQlVAiICEqLAoAoGISlUJQEBE0N/09yxVLVJESEIgBEQAyHrV+j8SqsL1SvRXf5ZK/x4goGBR1hHW79f5sM/47ev6PUCBEAQhIvztNRTtSwbX9+scZB3iPtN1TMjv//apooeOfS2iazj6aHi7oipKD6SApED4fnxArutlQRSv0bhGmOwh7NO/ZkTWGNY9VvKar/dzvufunsf+9GvuCMrv49kX97re+/CAgHy73j4+7uPfk/h+XWveec3julK55wLX5wiu2VmTICRVtY8j6/yr16GpgFQFCLM1QqoiQFWtFQj0Iq+i6DW411j11ayTErDYax5CQV8hVAVF0XXxIlABiwKKqgBArXkSsEgBihDJKhbVtIpRFYyZZ2UgpxtMfEBReqYceWb+RP3T+c34/vX1D41jZGRqjQ8Zw1CM73n84/z6n8fxDFYm6njO57Mis2ob7oKKzMqI+vXrCN3+/u//bgJkfv35j+M4fHN1U1WpcjdWHsc5g99nzZK/fTz+9rFLnIKSTb+POM6pyOFKlA0HJGaKEMiImM8QtbENAY4zI7ENI+s8JlREJSPnJKQ/sCpTRKpSTVmoLHdVM4Gai4hkFEFWDZeqAsTciqiCAD60inFGVanChzHLh2fRVMW0MiNLBe46Z2SUmLnpcBeomkREzGnu5xH7PnwYUCQzYtt3UY3IiOp7bAwfriB9jDFcVSr5/X3Y0O+fh6iObYxhc8bzeaqImqr0akXEVFV3V8F5HL9+HeoWQYqKwE0FNLcq5gwb+vjYTHXOIuDmJAhGpKo89mEmmQmWu5M4z+ihczd1JfHzH19kPT4eH58fonKeh6nM49g2+/yxu/s85/N5mI7t48Pcnt9n5KyqnGGmP358EnKcc56VBYiN3Z9f3zkPMwD4eGy+jY/PxxCZ53GesX98PD4e5/f8/LEfx/H166kmm49t358z//nn158/v845//bH5zacxRmZVFEjmVFxnvOc5ioimdn3/nBXFxIsRjKTWSUibq6KyIwZouLu7gaR88wxjEUzrbVFIDMFrCzfXFVUNTLyTDVV0+3jAYAREVNNFTojfXjvdDHzeD5N4cO24WZWVRFVEDUxkZ5lES1WzlA1ilSEqWyPvZgZwUhTMaGZiThEznOKKqtU6MPGYyMlMp/PE1UqIoCaJaWqMhKC4cbKyAmiiCrZHruAUlVVvnlGZRHFQpl7RjJLVLbdUaWQ7WMnKIAAGdUjLGpQK2LOmRHuVlljM5DnGZkppqbqpu6qOpJQVVWpDAFVpIpZOc8QU98GiiIiKqpkBAgdFlFV3IeP4VATVWaNzQVCFCszozLUnJSkVPHzY1OROSMjzR0ABSRNUZmVMVw/Ph5j+yAEAhNhhm8bBPP7uQ3zzcbuMSerhgGFmGHb2PaRGXFOENvYfRs2tswkMmKaiKqCdLdtG9vYKMyq8/klVZ+fH/v+qTbO4yzG2Eaep5qSU+G6fT4+Ps3s++fP8/ilKiKEKMV830lUVR7P4V5FERmbQaBqoKr/8P1BSuXz/PrH8XxCsioBVtFcK2GuJipqKrp/fKhqVZ7Pw/dNfZi6b3p8fcV5qqm5FZNVJq62kaZmQA53EVMfY/+EyHweggIri2o2Rk+KjO0jE1FSZJxPYVWeEIHCVIolkG37EB1mH6KGYkYRpSpkkFVV5gPU8djmmaYoVj88oVoZpkKSBRERg4oBBh2iYISqZYLUTKoaSRQj8tev7+fzAPh4PLbHw32IyAInKiQVSiwoJzeUesdWQhAFsAr90gtt9sN6YRrV/kP/cEE4kguLFdl/JQhQVciClEpAmDVVUjHVRVSLFecUMfMtiSxWFgok6AKSSYWUlGemQzlz2wb3B3wvJIZnQarIBA6AYx/mppsUUDOAVHc1lMtkqhilBIRQFWxspYooSQFlPoGHDrMBCEvIqoU0ZaG8HlSCC8mJQABWCXQhvx6djgzqets16GtkAIjUGrQ+bMPshfzWDFwg8xUnXAiwv+01RwKglAAXRgd7AQguUPmGooHG0NWnweuF7A+9ATQuML4QbR9trZz1DF+n0n+5z24d8YKQ4AXkRfF2dPkNN6+jAlyv76t+jTvfjiPrZLnipUbAb7HTFQlc/8oVg11jLvfpyfXxr4X+lxjmmheRa7jvy7hiGPblcF3vOl8BmFc80PHDPW7XJKLvuiu8XC8Er5PCen3hdT0gOl5a1yP9uZUpouvIRbziQF7jg0qaSl/sWot8W5Vva7jXp1yj8Ar0rrhonSTqimkp0pHkNWqEvoU1K5ToaRUtsl+kouu+FxHTnFKiVOEsilAoQ0Vd1ES14stUn1+nMrJyPr+zptlDVHR4zefx/OY85xEiNcbAmUfOWVGRNiyZGcXMSlQVlYVZNc3sPM4kt48BVaiCMmeySk2SMNPPT/ftse+PoconwWnuThYSxWCA5GRFQWR/DDMDmEaRdQ+DMAWkMpIoEpINV0qUmZIRLNpQcm0zqhBhzAmompuaDSuwUoCSa3moiCgyq5JZBYUKRDoIlMyMyDJ1OASqooKGIBCyEiY5E0ijVVURCqoBAhGKSFWZa7EQHVWyilWlKkAJZX/otm2qMs+5bT4jyJ7hAkxkPZJUzV3V9TxPTvqwfR+Zxan7x3acCYG7VlZkCEihDxexfpJlJcCMcldTjQJAVRXt2LUy0oeLCgRqQlYWju9zHgnAhlXleZw6LKsgUJf9Y2z7bmpA/vnzPBEyXFTdNSksFkqBiBDRiuydlcg5mZkXGuP22H78+DF8sKb54/n8Z2Q8v89hXtBj4vsZY1cQ6iYq6mKuf3t8jmE54zyDQJWYEyKVyevOJFhZEHk8xr5vIhKRJSUVGQHAhotUZjXQFCHAzOynQVWRxaoKjjFEVVUjZlUZKxN9EDWpqmLh+VRVoKqKhG46djNTAQj1YdtuzOy9IjOLLJSIZpS6VRWI5Mwsc4WQLAqTnBmiIiIlPCM/P7bHx6eK/vp6zjl9eGXBSEhVNUToh6gaWH3DoE91DBfh8zvnzDFsuG/bEBuVlXGooLIyM6LU1N0hWByKAhBVFzBmVNFMts1lG6oSZ/RjvVj9iKyqJk0qqx+dAkQES9VGzXP72H0b80yyMrKDH0LEFBfLRnKYgxRRNVSDwaqCZpWrATQTc53HqaYQYZWaiso2tiTmmSSTdZ6REVaNdrQq1YSQCsJUzNRgY8wZc05T2lAbVnGqSZEzMmYQdHdVbo/t48dHAud3RKVAQqoiPx7S2+zz+3CXjx8f53P6kIjyUVWMzMxSKXUbm0FkYNi2H8/n1/M5xnjsLqoRkVn74+G7nweez9OH+S5ZIaE+tu3xOBHH12k+xjCYgvj+Otx9s/nY/6aq5zM4bJ56xhQVAjAkSRV3U9OcbIKPBQA2VETMHBBW03AmqgTMfOaM4mM3lqibip7HaS6bCaAsrRQzhYKVPoa6ZlRmxvNbYL7tm++nFDNCkiAEM5JFH54FV1U3sy3PGG6iyMyoqqKPDZCizKMunCBgqamYBfoWBgRQYTEyzcVcCVQCpIpFUESrChQmKu+Ht7KfykUx7aOzGuP1RnAxiIvLvLEMF2UMkP1Ng8l6vWu98g1XXhCuX3NzuY0E7qhAFaKoouii/ghApMAqQrSqoYlRSFRGEOrbxmRV6CBTHPOMgpq7RiFFQjcROk6r76qYJmEuhjKmnFRRBWyYsJTTyiHMSEiVIFQhlgwmCmoCCqLqz/MM6A+Wqj42nbQ8mJkQvwD5Is1lZQEW1DP1F9K9Qq8LGd44CbhA2HqhvBDti8i/8N6LJb4wJa556EN31uGiuG+kJr+Dy85XXIz0zTpfqOwVZFyM/v2pd4jYV3GhN1wZgYsXfh3kjhrXZd3Y7wKtfZyb93/Dwdcnc62uVxhxn88FOl8B1fto9bAXe3uXZqAbAV/5B3l95oV2uaZEcIUavGbiBfPv318/XoHSypxcMPqKi3DfZnKP33Wev5+yCEG9QpG+NOENuV+gnFwgf0U/uJJG6266Qpv1ceuzeX1EozyhuPaJL6D/FomwyAXKZd2a145Q1yos8GIW1nDwWjl37PTKEkg/alErJOgn9xVgANB1AgQ6P1MkZzKCScRzSoVQ/DFKVUsEVFApeUx+Pyt/fY6aw86vX6IqedSzKs7MmTNYKWZjbCrx/csAE0VkKnQ+Z2aa6Lb5vm+b2Ri6jTFUt4ccx1EQVTNVkHE+tRpe2+bbeDy2fVMwy87nmfM0ETclRIpjc3P9/vXdrNS2be7DbH5/PZPFrKwiBFMhYu6d+hARM61qCgYQZC5CofnyKooWyYgEYWa9FESUxaxANLlFknNGrzF3d+/kDMlOsFRl2nAzVRFyioiqVaFhY2RWVBUzQwRjjG0bPkyEERIz5hlXNkwAdMYpk2Z6nmEjlRqZRZ7HLFLBc0ZkiKpvljOrgvCYFZEE5kyxBGVsD/Oc8RWRkpFJEYoq2SGrRYbcdwI5z2lmmaWqAuSMEFSVmi7mdbgAlTnnzEwKmawoGV6s43uSuW3+8di3x+Pzb38IOf/5hMp5Tj2OGWlm5zkzEmBmnRHDLTIqpdYEpAhsOET33d3t8fkw1Xnmr59fFIrI/tg+P3+4b3/87d9+/u3nzz//K3N+Pb9FB4jPz4eKiMhMZiUpVajq9Eg2N9/UnKo1r29m275nVrHw9YzIKpJ1nKUi6ubDq+rKppJAnCcJd3d3UY3MiMgIEWn0DFEzhUCqKisjqRSFqJoKK5quMndLine truncated
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 5
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Not bad! The results seem to be well-aligned with the prompt.\n",
|
||||
"\n",
|
||||
"## Detecting objects\n",
|
||||
"\n",
|
||||
"Now, let's see how can we detect the objects automatically. For this, we'll use [YOLO-World](https://docs.ultralytics.com/models/yolo-world/) model from `ultralytics` library."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wKuyU-GDXPhr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"model = YOLO(\"yolov8m-world.pt\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "R9pERjDkGIlN"
|
||||
},
|
||||
"execution_count": 6,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"YOLO-World model allows us to [define our own set of labels](https://docs.ultralytics.com/models/yolo-world/#set-prompts). Let's create it by combining lists of pre-defined colors and objects."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "VOxtES_qaoWe"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"colors = [\"green\", \"yellow\", \"black\", \"blue\", \"red\", \"white\", \"orange\"]\n",
|
||||
"objects = [\"car\", \"cat\", \"ball\", \"dog\", \"tree\", \"house\", \"person\"]\n",
|
||||
"pairs = [f\"{color} {obj}\" for color, obj in itertools.product(colors, objects)]\n",
|
||||
"print(\"Number of labels:\", len(pairs))"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "SkY3TI4JHmkj",
|
||||
"outputId": "ef7943b1-f5ab-4422-a5ff-db2ac87df88f"
|
||||
},
|
||||
"execution_count": 7,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Number of labels: 49\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's feed these labels into our model:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "by_NYnqkoET-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"model.set_classes(pairs)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "oinXZU3YWZqw"
|
||||
},
|
||||
"execution_count": 8,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Time to detect some objects!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "RTHHpRsFoJsk"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"results = model(image, verbose=False)[0]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "zDPd-LOqGLHJ"
|
||||
},
|
||||
"execution_count": 9,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We'll convert the results to the [sv.Detections](https://supervision.roboflow.com/latest/detection/core/#supervision.detection.core.Detections) format to enable features like filtering or annotations."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "TGCRxUpeu8Vi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"detections = sv.Detections.from_ultralytics(results)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "GWbmFKscvUhU"
|
||||
},
|
||||
"execution_count": 10,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Speaking of which: we only care about strong detections, so we filter out ones that are below 0.5 confidence."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "AscEY8SioQQ7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"valid_detections = detections[detections.confidence >= 0.5]"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "f2fvqc8lWxGT"
|
||||
},
|
||||
"execution_count": 11,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"A quick peek on the detected labels and their score:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "klstslXPop-9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"labels = [\n",
|
||||
" f\"{results.names[class_id]} {confidence:0.2f}\"\n",
|
||||
" for class_id, confidence\n",
|
||||
" in zip(valid_detections.class_id, valid_detections.confidence)\n",
|
||||
"]\n",
|
||||
"labels"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "0jbyOEGdGOhd",
|
||||
"outputId": "2b0ba2d0-c481-4824-b9fa-7682a44bbec4"
|
||||
},
|
||||
"execution_count": 12,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['blue ball 0.91', 'black cat 0.70', 'white car 0.64']"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 12
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Visualizing results\n",
|
||||
"\n",
|
||||
"Now, let's use the power of `supervision` to visualize them. Our output image is in [Pillow](https://pillow.readthedocs.io/en/stable/index.html) format, but annotators expect the image to be a BGR `np.ndarray`. Here's a simple helper function that's going to help us with the conversion:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "GOccbqMDo2Ec"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def pil_to_bgr_array(image: Image.Image) -> np.array:\n",
|
||||
" rgb_array = np.array(image)\n",
|
||||
" bgr_array = cv2.cvtColor(rgb_array, cv2.COLOR_RGB2BGR)\n",
|
||||
" return bgr_array"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6FxIgA2SJPAS"
|
||||
},
|
||||
"execution_count": 13,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Time to define how we want our detections to be visualized. A combination of [sv.BoundingBoxAnnotator](https://supervision.roboflow.com/latest/annotators/#supervision.annotators.core.BoundingBoxAnnotator) and [sv.LabelAnnotator](https://supervision.roboflow.com/latest/annotators/#supervision.annotators.core.LabelAnnotator) should be perfect."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "K32SxeSrpq15"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"bounding_box_annotator = sv.BoundingBoxAnnotator(thickness=2)\n",
|
||||
"label_annotator = sv.LabelAnnotator(text_thickness=1, text_scale=0.5)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "X9HMMnTppp-c"
|
||||
},
|
||||
"execution_count": 14,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Finally, annotating our image is as simple as calling `annotate` methods from our annotators:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "u3y59TgWp4Fx"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"annotated_image = pil_to_bgr_array(image)\n",
|
||||
"annotated_image = bounding_box_annotator.annotate(annotated_image, valid_detections)\n",
|
||||
"annotated_image = label_annotator.annotate(annotated_image, valid_detections, labels)\n",
|
||||
"sv.plot_image(annotated_image, (12, 12))"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/",
|
||||
"height": 734
|
||||
},
|
||||
"id": "QmMzgD8DGU8t",
|
||||
"outputId": "19de76dd-c1f4-412a-8aac-92373dbfec65"
|
||||
},
|
||||
"execution_count": 15,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "display_data",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<Figure size 1200x1200 with 1 Axes>"
|
||||
],
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7YAAALNCAYAAAAbaP0uAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9W5Ikya4kCDIgImrmkXkfXauZz1nKbGbWMV+ztf6Yqa57TmaGu6mKCDAfDIiquXtExq3qpqFL5HooT7ibq+lDnmCAwRB3d3wdX8fX8XV8HV/H1/F1fB1fx9fxdXwdX8d/0UP///0AX8fX8XV8HV/H1/F1fB1fx9fxdXwdX8fX8b9yfAHbr+Pr+Dq+jq/j6/g6vo6v4+v4Or6Or+O/9PEFbL+Or+Pr+Dq+jq/j6/g6vo6v4+v4Or6O/9LHF7D9Or6Or+Pr+Dq+jq/j6/g6vo6v4+v4Ov5LH1/A9uv4Or6Or+Pr+Dq+jq/j6/g6vo6v4+v4L318Aduv4+v4Or6Or+Pr+Dq+jq/j6/g6vo6v47/08QVsv46v4+v4Or6Or+Pr+Dq+jq/j6/g6vo7/0scXsP06vo6v4+v4Or6Or+Pr+Dq+jq/j6/g6/ksfX8D26/g6vo6v4+v4Or6Or+Pr+Dq+jq/j6/gvfdRfPfH/+f/6f6O54BiCcfsG377BfQJv/0AZf6DMB9Q6BDtEgHbf0LYNUgvmnMAYqKoo7Q6rL+hocN1guONhdxy4w/QGqQ0KhUzHTQw3FfzrS8W/3RWlKv7RHf94ON46ML1iusAgAARw8D8A8PhBAInfBXr+zpPOFxQB4HAAIgKB8OenVnCe9u7T59/ef5g/8Gry9Pvn3xURvL/5p/eI93p+B57r7vwhPsPZHHCR5++9u75f2kWeGwx52Wyvd0/+w2c9X8c/fGt1lfzwLX9yXJ4j2u16mbyXfHhnWU/yw7v+zfPknT977p9+8/lxzy566sv4Jz7yaze4PJ3z4Z4So2yN/fyjnM/qMR4lh8nZjiqAPv0NKBqjVy7P6Hxmj3s6zvFh7nHN55GFNQN83TPPkHju83ecz4Dwwq12OPsu56S5w93gEPjl2SaAx+jo1uHzgB8PbEVRS8NWGqZXPKZjf/wTYv+B8fo/IPaGx/4H/O0vFJso7TfIy++AOPr3PzD3P3B8/w+4HEBtOB4d3//8J8bjANyhRSDTYXPAJ5/l7ThgpeFffv9XiDv21zeYdTZ4UagLxn6wrRU4+gSg0LZhu71gqwXYd/g8oFvBo3f044DbhPuEqsCnw+HYbg21Kvb9wPEY7FdVTAPmMEgR2DSMMTicHJjTMIYDonB32JiQwkFalP0Ld6gKzByqBdtWUVThAMwMAoObwdxRamW/GPvLAbgZAIOIQAWxzjrmNJRaoOX0tbo55pxwNwCCVisEyv4WwZwDogJ3R6sNrfFvZjPmEq9l5pjTYWaotaAUgYrit9++4XbfAHcce8dxHHg8DpgD29bQWsUYE8fRoaqotaJUxbSJ47GjtoLb/Qabhsdjx+gT+zFyVMZzO0pV1FYBc4gKauVzzWHQUlBrgRkwxkQtiu1WISIYR+deujWMaewP8BpjTvRjQgCUWlBLQW0NBoeIQeH49m3Dt9++oYji8fgL//jnX9Cy4eXlN6gqeu/ovWOMjqKC++0OQHAcA3MCBoGLYPSBcexom0JV8G//9jt+++031FJg88Dj7YExDa3dsbWGdrvh9fWB73/+hdIEWy2o24bHMfDnX2+AA7UV+JjY9x7rhkJLtEMfGKOvvWeOCYjg5WXDtjUAgjknxhh4PDpsOseOCswsxoZBFChaoi9iYXPjmJlAa22NozEGiirn7WXFMuOCVmuNsTbhxucXFZRSzl1PAJsTbrbWKHP+x3ngaLVAnJvUtAkAKKUAEs8OoDbeC3NiRt/8/u0btu2GfT/w/a9XPos5SnW0VlHaBodjPzrmMJS1OSlceC4A1KqYc2J/0E6rtWLbGrQ22DTMfsDdYjxyHtfKd7XJ+ahFcGsbVASArbbatgLRClXBOLjmQEuM1QGAa0dRBczQ5+Saocq5WQraVlFKwXa/8RmOHbMPaFGIKKYZprEPi2q8U+GaMCdU+US9TwCOrTXU2lAq+6m2itE7tHCNm/3gtWqFaMWYtvaU/a1j9I7aGuqtohSuNQLH6B2lKL693HHbbihtQ+8dNgdqVdxe7rCwe0UFUgSzDzgcL7eCAqC2hnbbsPeB/e0VNidq21C04f7bC2CGt9c39GPH7d6gqmg19q3bBggw5sTYH6gF+Nd//Vfcbi9wF0AFfRz4/s8/UWvF/VYBVUBu+Pbtd6gC3//8Bx6vHEv1VuAQ1HpD224QFezf/8LsjrpVjncRHI+OUgpuL7/jX/7tv0HgeLz+ide//om3xxsMcxkdHsbNtjWOlSlobUOpDRCHW4eWitK47ogY+uMBj3GiynnuBtzuL3AoSmkQGOaYqNsdt9tv0Lqh7wfEJ1wm5jTc7jeoKsewOUQ4R2q7YfQDNjp6fywbcM4JEUFrN9R6R7t9Q603WDfaYOJcJ2YHhHupu0CkhK3CcQEHtFS4CuYYy6ZbNr0BpVaU7YV7bB8QUdgUmKfdwr1v9IHXtweO44CWgvv9htZuHKs56d7ZdRfY8/T397a6u3Nt47djjYoLBP6BAKJ62mNP1ll+3xfsEj4454gAqg5Rg/sA1OHeoegQDfvDHDYNbgKtFS4FYxpmH7Bp0Fohqph9olSHTeD/9n//f+BXjl8GthgDwwCtFVUHHB1aHHJXYC+wAfiYUHWUKihu0DkAM1QBShUIDOodxSoHypgQcVQRmBRMFJgJDAUKLlCHGf55dHQovt34ottN0d0x+wSkEsxA4OLRf6cBHlAVKmUZxQv0LfTAnnkPuD47/B1w4j3Oy3zyjU//IiIx6K+DTp5x6OVefgEE7y704cdcUD5/uHdo2Z9//tAG7x9dLuD2CnqfMOMnH3567XdA+Ol53gGiH3VOPo9c+vLz2/z8+Mwp8IO+e99rnx15ThruTw8n/um5nz/XJ+fJz5/r6pJZQ8Hf3VeezzudOrn+OsSBkiA5x+ZT/3v8T9b3r76lZ6eLP7WbrL87PO7hl1X5OjLOd3LAhUbfep24uxPGWE6ceD5zx5iOORzTHOPRIT6h4ihtwwyHh8JQHBj7gD12YLzhZVOM7Yb+eAV8AMeD17RBZx0cqhW1Nfg0qBaIFtgcGMNhfWL0AYVg2yq22wapDbUpqhYQpwpGgMSiCrhjHA/AHCoKrRXbdsPtvqEWxcTA/kqDTAHUooAKRCpaK+hHxzQa6Vu7oZQG1QOPtx3TDGNMzOlQL8ugm8bNSFVRisMMa3PycAg6JIxJbkii7Pkx5zlfHDS4ITCbQAA3GsoWw8FRSkFrZY2WdEC4GaAKVYWIwDBhJtAAJ1oK4ATB7mFkq2DbCHhofDlGFwK2OS6bMd8PAOZ0oDiO3iEl2nxOHJ39Kqo4DgI+LQWlKtwM7my7PgbbrAOiNEy22x2iHUefMTY4rqTE6DWEoTXgblChETPHiGsDIgl4B+egGwoEYwyOj63BzTBGh00DhMaFT0BV0EfHnBOlKr59u2G73/Htt29wM7w+6A/rowOPB0opCxzmnOxzosQzch+OeSZ0UogAt1tDKYrt1qACPKajz2iDreL3f/kXbNsd//7v/w3fX7/jn//4HziOB+Z+wFxw2xqdGqroDjiOGG8WDogYi+FccWO/iSpKKaiV883McRwH5jR0sE2Pw6Cq0KootcDNw6ilLWF9wIwguNUGF2AMgmg3GuJNS9wrnOAWjpo5Ia4BZkEDbmI5cNw41kWVzjV3qAhqPLtEu4oAYo5pBpsOc4MD2FpDaRVzcPwUFKgDqhXbreG3f/lX/PbtN7y+foc5gWLdFA6OndE5ZhT8P4t5LprOIonxz3FXCgForWm4WhijdH45jEN4hgFrvoD31m54+XZH1YLedxwHjX0RpfNJBFMn32VMTGNbiCos1gKJ5ymF69AY87oV8XcQcIw5UFH5XmbhyBMC4VpRt8r5NEfsj4JSJma0N100CnPH/nhgmqFYQa2K0mrsbXTabKVhjknnmBimGyocsw8AXH9UCNAI7BW9D/RwEiroWBMV1CKAGHo3+EGHFBw4jomXlxtau6PWBtEKMcP+eKAfHVYceOUetu8PgpBpUCj3BAX2tzc6gty5B8BwHAPbxr4ttQAO3O53wJ3tpRXtdkNrBfv+oGNrDvg0GAratqE1OjpGP+hoVofbRO+G9IDOMWCzQ2QS7MNoBSj3ZNq2PLdoASzsBGc/p9O7tcIxaYYx2V9wCxvEOQ7D4OZa3Nj38HDKhCOkNtQi6PsbxhjYWuG8dcCGYZihKNDaBjFgKw2HjTVPATpuOeDpFMScmH5AwDHqAGb6cIwOdNUKVTrUcs0uReP97cny4z4KSFGUIqitQFHQRdCPEQ60WPN87YzLyfJsDp8BuGWJncbfu79f7aF1+gcs8Rz3STCu67f3+Jgwy9eeftqDiV24r4sZuOycBpq7wbj4QwKUiwPTJmY/563bhE3ORR2AoOBXj18GtlZumOIoAjQMYHwHl8QJEwG0wrWg3AqkxCvagIBAt5WEqgL3jgZOGIOjqsKV3r6BgglgQnA4vfp9OEyBWQUtjNpSHHUCMItNkM3rEh3qXPTZQWxMRAMiz7kCAHHoMvVlGf7PXoo8zh6WMFDOj9/DHVn/JsjhaedofQYKPwJo7wHtxz/lIFv3k5haT/gy3mkBwfdg5zQEr9fPiOFzBPcTgCWf//IURT1xxyff+exy5z0/xnuvE/jpkd/1nF9+WVDtw7f8/cM+f+X5MvnR1Sv31Hbnz9f/f/cYn9zrXGVygXo61f0J3D6zCJ7bZ31T/Gyn+OoCmrHo+oeulAVcz8js5QLrlqd39nzl5+dxdzqecALl9feLt+jkNPinTiRGk6NvNZ+Lyyk3AX83BU+obTbRzVBUYVoxzAFMqA8oBorLine truncated
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Testing it automatically\n",
|
||||
"\n",
|
||||
"We can also test if all requested objects are in the generated image by comparing a set of ground-truth labels with predicted ones:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "kl5kO59CquMq"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"gt = {\"black cat\", \"blue ball\", \"white car\"}\n",
|
||||
"pred = {results.names[class_id] for class_id in valid_detections.class_id}\n",
|
||||
"\n",
|
||||
"pred.issubset(gt)"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "laJS1E76qynK",
|
||||
"outputId": "b5b8566e-d638-4999-8e4d-830241d2064c"
|
||||
},
|
||||
"execution_count": 16,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "execute_result",
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"True"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"execution_count": 16
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Using [sv.Detections](https://supervision.roboflow.com/latest/detection/core/#supervision.detection.core.Detections) makes it super easy to do."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "8L4lXpvFvxZR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"In this tutorial you learned how to detect and visualize objects for a simple image generation evaluation study.\n",
|
||||
"\n",
|
||||
"Having a pipeline capable of evaluating a single image, the natural next step should be to run it on a set of pre-defined scenarios and calculate metrics."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "j2GgYtNGqYmY"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Vendored
+1
@@ -16,6 +16,7 @@
|
||||
<p class="card repo-card" data-url="/develop/notebooks/download-supervision-assets" data-name="Downloading Supervision Assets" data-labels="ASSETS" data-version="v0.18.0" data-author="nickherrig"></p>
|
||||
<p class="card repo-card" data-url="/develop/notebooks/annotate-video-with-detections" data-name="Annotate Video with Detections" data-labels="INFERENCE,YOLOV8" data-version="v0.18.0" data-author="nickherrig"></p>
|
||||
<p class="card repo-card" data-url="/develop/notebooks/object-tracking" data-name="Object Tracking" data-labels="TRACKING, ANNOTATOR" data-version="v0.18.0" data-author="nickherrig"></p>
|
||||
<p class="card repo-card" data-url="/develop/notebooks/evaluating-alignment-of-text-to-image-diffusion-models" data-name="Evaluating Alignment of Text-to-image Diffusion Models" data-labels="ANNOTATORS,YOLO WORLD" data-version="v0.18.0" data-author="iamhatesz"></p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
Reference in new issue
Block a user