{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Explore Golden Volumes and Select a Recipe\n", "\n", "In this guide, we will walk through the process of exploring Golden Recipe volumes using the GRDB API and selecting a suitable recipe ([Golden Recipe](https://docs.latentai.io/leip/design/latest/content/explanation/golden_recipes/)) from the Golden DataFrame.\n", "\n", "## Understanding Golden Recipe Volumes\n", "Golden Recipe volumes are collections of recipes that have demonstrated strong performance on specific datasets. Latent AI continuously works to discover better recipes for increasingly diverse datasets, ensuring that you have access to high-performing models.\n", "\n", "To begin, let's list the available Golden Recipe volumes:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Recipe Volumes:\n", " - chemistrylab \n", " - bdd100k \n", " - insects \n", " - wheat \n", " - carsimple \n", " - xval_det " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import leip_recipe_designer as rd\n", "volumes = rd.GoldenVolumes()\n", "volumes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Note:\n", "The code snippet above retrieves Golden Recipe volumes for the **Detector GRDB**. If you're working on **classifiers**, you can adjust the `task` parameter to specify classification tasks. For example:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Recipe Volumes:\n", " - cifar100 \n", " - fgvc_aircraft \n", " - bdd100k_phase \n", " - stanford_cars \n", " - eurosat \n", " - gtsrb \n", " - xval_cls " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Listing Golden Volumes for the classifier GRDB\n", "classifier_volumes = rd.GoldenVolumes(task=\"vision.classification.2d\")\n", "classifier_volumes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each name in the list corresponds to a Golden Recipe Volume. These collections contain recipes that have excelled on particular datasets. By exploring the datasets used to train each volume, you can identify the one that closely matches your own data.\n", "\n", "Volumes named **`xval_...`** are cross-validated across diverse datasets and are the recommended starting points.\n", "\n", "- **`xval_cls`**: A cross-validated volume containing recipes optimized for **image classification** tasks across multiple datasets.\n", "- **`xval_det`**: A cross-validated volume containing recipes optimized for **object detection** tasks across multiple datasets.\n", "\n", "If any of the volumes seem to have data that is very similar to yours, feel free to start with that one. Keep in mind that data similarity is not a well-defined concept, so choose one volume to begin with and iterate as necessary. For more information, consult the guide on [GRDB Model Training and Optimization](https://docs.latentai.io/leip/design/latest/notebooks/HowToGuides/GRDB_Model_Training_and_Optimization/).\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Skipped downloading goldenrecipedb with name \"carsimple\" and variant \"Carsimplev0.3\" (0), as it already exists.\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "91ac2babea304201a527e0cb8da88d01", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Tab(children=(HTML(value='\\n\\n
Description 'Simple and small dataset …" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "volumes[\"carsimple\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can click around on the widgets to explore the data, including the training and validation samples, as well as other metadata such as class distribution and resolution details." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exploring the Golden DataFrame\n", "\n", "Golden Recipe Volumes are structured as [Pandas DataFrames](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), allowing you to query and sort them based on your specific performance criteria. For example, you can retrieve the DataFrame for a specific volume like this:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtimestampneptune_idhardwarequantizationcalib_methodper_channelheightwidthmodel_family...od_task_metricod_inf_rate_msod_inf_energy_mJod_size_MBod_gpu_mem_MBod_cpu_mem_MBod_unified_mem_MBod_total_mem_MBod_avg_cpu_freq_Hzsppr
1189841189842024-10-15 09:38:45.048419GRDBCAR-102cuda:A4500float32nonenone736.0416.0Yolo v5...0.4289541.401320117.98551216.544411243.000000697.101562NaN940.101562NaNeJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS...
1189851189852024-10-15 09:38:45.048419GRDBCAR-102cuda:A4500float16entropyFalse736.0416.0Yolo v5...0.4323551.12605985.76365713.229023218.770279364.319977NaN583.090256NaNeJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU...
1189861189862024-10-15 09:38:45.048419GRDBCAR-102cuda:A4500int8entropyFalse736.0416.0Yolo v5...0.1622371.00904065.30338311.787714222.000000381.723969NaN603.723969NaNeJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU...
1189871189872024-10-15 09:38:45.048419GRDBCAR-102cuda:AGX-Xavierfloat32nonenone736.0416.0Yolo v5...0.4289547.1776970.00000021.236946NaNNaN1324.9708251324.970825NaNeJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS...
1189881189882024-10-15 09:38:45.048419GRDBCAR-102cuda:AGX-Xavierfloat16entropyFalse736.0416.0Yolo v5...0.4323554.6591350.00000014.740621NaNNaN1326.9239501326.923950NaNeJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU...
..................................................................
1230241230242024-10-15 09:38:45.048419GRDBCAR-99rpi4_8gbint8percentileFalse512.0512.0EfficientDet...0.0357334296.565918NaN69.839480NaN324.141998NaN324.1419981757.000000eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9...
1230251230252024-10-15 09:38:45.048419GRDBCAR-99i9-10920Xint8percentileTrue512.0512.0EfficientDet...0.029460134.523331NaN40.135960NaN266.424469NaN266.4244691621.237564eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9...
1230261230262024-10-15 09:38:45.048419GRDBCAR-99rpi4_8gbint8percentileTrue512.0512.0EfficientDet...0.0294604543.513184NaN71.299512NaN326.242004NaN326.2420041757.000000eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9...
1230271230272024-10-15 09:38:45.048419GRDBCAR-99i9-10920XuintpercentileFalse512.0512.0EfficientDet...0.030770142.222061NaN42.585248NaN234.094345NaN234.0943451480.753062eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9...
1230281230282024-10-15 09:38:45.048419GRDBCAR-99rpi4_8gbuintpercentileFalse512.0512.0EfficientDet...0.0307704499.782715NaN71.369056NaN324.790039NaN324.7900391757.000000eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9...
\n", "

4045 rows × 26 columns

\n", "
" ], "text/plain": [ " id timestamp neptune_id hardware \\\n", "118984 118984 2024-10-15 09:38:45.048419 GRDBCAR-102 cuda:A4500 \n", "118985 118985 2024-10-15 09:38:45.048419 GRDBCAR-102 cuda:A4500 \n", "118986 118986 2024-10-15 09:38:45.048419 GRDBCAR-102 cuda:A4500 \n", "118987 118987 2024-10-15 09:38:45.048419 GRDBCAR-102 cuda:AGX-Xavier \n", "118988 118988 2024-10-15 09:38:45.048419 GRDBCAR-102 cuda:AGX-Xavier \n", "... ... ... ... ... \n", "123024 123024 2024-10-15 09:38:45.048419 GRDBCAR-99 rpi4_8gb \n", "123025 123025 2024-10-15 09:38:45.048419 GRDBCAR-99 i9-10920X \n", "123026 123026 2024-10-15 09:38:45.048419 GRDBCAR-99 rpi4_8gb \n", "123027 123027 2024-10-15 09:38:45.048419 GRDBCAR-99 i9-10920X \n", "123028 123028 2024-10-15 09:38:45.048419 GRDBCAR-99 rpi4_8gb \n", "\n", " quantization calib_method per_channel height width model_family \\\n", "118984 float32 none none 736.0 416.0 Yolo v5 \n", "118985 float16 entropy False 736.0 416.0 Yolo v5 \n", "118986 int8 entropy False 736.0 416.0 Yolo v5 \n", "118987 float32 none none 736.0 416.0 Yolo v5 \n", "118988 float16 entropy False 736.0 416.0 Yolo v5 \n", "... ... ... ... ... ... ... \n", "123024 int8 percentile False 512.0 512.0 EfficientDet \n", "123025 int8 percentile True 512.0 512.0 EfficientDet \n", "123026 int8 percentile True 512.0 512.0 EfficientDet \n", "123027 uint percentile False 512.0 512.0 EfficientDet \n", "123028 uint percentile False 512.0 512.0 EfficientDet \n", "\n", " ... od_task_metric od_inf_rate_ms od_inf_energy_mJ od_size_MB \\\n", "118984 ... 0.428954 1.401320 117.985512 16.544411 \n", "118985 ... 0.432355 1.126059 85.763657 13.229023 \n", "118986 ... 0.162237 1.009040 65.303383 11.787714 \n", "118987 ... 0.428954 7.177697 0.000000 21.236946 \n", "118988 ... 0.432355 4.659135 0.000000 14.740621 \n", "... ... ... ... ... ... \n", "123024 ... 0.035733 4296.565918 NaN 69.839480 \n", "123025 ... 0.029460 134.523331 NaN 40.135960 \n", "123026 ... 0.029460 4543.513184 NaN 71.299512 \n", "123027 ... 0.030770 142.222061 NaN 42.585248 \n", "123028 ... 0.030770 4499.782715 NaN 71.369056 \n", "\n", " od_gpu_mem_MB od_cpu_mem_MB od_unified_mem_MB od_total_mem_MB \\\n", "118984 243.000000 697.101562 NaN 940.101562 \n", "118985 218.770279 364.319977 NaN 583.090256 \n", "118986 222.000000 381.723969 NaN 603.723969 \n", "118987 NaN NaN 1324.970825 1324.970825 \n", "118988 NaN NaN 1326.923950 1326.923950 \n", "... ... ... ... ... \n", "123024 NaN 324.141998 NaN 324.141998 \n", "123025 NaN 266.424469 NaN 266.424469 \n", "123026 NaN 326.242004 NaN 326.242004 \n", "123027 NaN 234.094345 NaN 234.094345 \n", "123028 NaN 324.790039 NaN 324.790039 \n", "\n", " od_avg_cpu_freq_Hz sppr \n", "118984 NaN eJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS... \n", "118985 NaN eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... \n", "118986 NaN eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... \n", "118987 NaN eJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS... \n", "118988 NaN eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... \n", "... ... ... \n", "123024 1757.000000 eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... \n", "123025 1621.237564 eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... \n", "123026 1757.000000 eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... \n", "123027 1480.753062 eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... \n", "123028 1757.000000 eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... \n", "\n", "[4045 rows x 26 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = volumes[\"carsimple\"].get_golden_df()\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get a detailed overview of the metrics and configuration parameters stored in a Golden Recipe Volume, use the `describe_table()` method. This will display information like model architecture, hardware deployment metrics, memory usage, and inference performance:\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "------------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n", "id Index ID for the recipe wrt central db\n", "timestamp Time the recipe was added to the DB.\n", "neptune_id Neptune AI Id of the model trained.\n", "hardware Deployment hardware on which the metrics were calculated.\n", "quantization Quantization method used for On Device metrics.\n", "calib_method Quantization calibration method used for On Device metrics.\n", "per_channel If per-channel quantization was used for On Device metrics.\n", "height Height of the input to the model\n", "width Width of the input to the model\n", "model_family Architecture of the model\n", "backbone Backbone used to extract the features.\n", "optimizer Optimization algorithm used to adjust weights during training\n", "scheduler Learning rate scheduler used to adjust the optimizer learning rate during training\n", "if_task_metric Relative accuracy using the coco average on the validation set, computed Pytorch (not on edge hardware).\n", "if_inf_size Size of the Pytorch model in bytes in (not on edge hardware).\n", "if_inf_macs Number of Multiply-Accumulate Operations during in framework inference (Not on edge hardware)\n", "od_task_metric On Device Mean Average Precision using the coco average of the model on the validation set.\n", "od_inf_rate_ms On Device Average inference rate in ms of the core artifact, excluding pre- and post processing. Note- post-processing for detector models includes nms and can consume substantial time.\n", "od_inf_energy_mJ On Device Energy per inference in mJ measured (Not avaliable for all devices).\n", "od_size_MB On Device size in MB of the compiled model artifact (modelLibrary.so) saved to disk.\n", "od_gpu_mem_MB On Device GPU Memory usage in MB (Not avaliable for all devices).\n", "od_cpu_mem_MB On Device CPU memory usage in MB, i.e. system memory.\n", "od_unified_mem_MB On Device GPU and CPU memory useage in MB for devices with shared CPU/GPU memory (eg. Jetsons). (Not avaliable for all devices)\n", "od_total_mem_MB On Device combined memory usage in MB CPU + GPU. For CPU, only system memory. For GPU, it is GPU memory + System memory. For devices with combined GPU and CPU (eg. Jetsons), it is combined memory.\n", "od_avg_cpu_freq_Hz On Device CPU clock rate in Hz.\n", "sppr Serialized Packaged Portable Recipes (SPPR) ready to train on selected data\n", "------------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n" ] } ], "source": [ "volumes[\"carsimple\"].describe_table()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Filtering for Suitable Recipes\n", "\n", "In our scenario, we aim to identify candidate recipes that offer a balance between accuracy and fast inference. To achieve this, we will analyze the multiply-accumulate (MAC) operations during inference." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Calculate the 75th Percentile of the Task Metric:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "threshold = df[\"if_task_metric\"].quantile(0.75)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Include only those recipes where the task accuracy metric meets or exceeds the calculated threshold and sort the recipes by inference MACs in ascending order:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idtimestampneptune_idhardwarequantizationcalib_methodper_channelheightwidthmodel_family...od_task_metricod_inf_rate_msod_inf_energy_mJod_size_MBod_gpu_mem_MBod_cpu_mem_MBod_unified_mem_MBod_total_mem_MBod_avg_cpu_freq_Hzsppr
1228341228342024-10-15 09:38:45.048419GRDBCAR-9cuda:AGX-Orinint8entropyFalse320.0256.0NanoDet...0.6477672.24372532.9573869.641616NaNNaN2400.0004882400.0004881581.438461eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e...
1228261228262024-10-15 09:38:45.048419GRDBCAR-9cuda:A4500float32nonenone320.0256.0NanoDet...0.6411981.54556996.8210078.764759373.0702.741089NaN1075.741089NaNeJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8...
1228271228272024-10-15 09:38:45.048419GRDBCAR-9cuda:A4500float16entropyFalse320.0256.0NanoDet...0.6403560.76079764.6802677.014277224.0446.127960NaN670.127960NaNeJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e...
1228281228282024-10-15 09:38:45.048419GRDBCAR-9cuda:A4500int8entropyFalse320.0256.0NanoDet...0.6477670.87593954.0213707.835824213.0368.084015NaN581.084015NaNeJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e...
1228291228292024-10-15 09:38:45.048419GRDBCAR-9cuda:AGX-Xavierfloat32nonenone320.0256.0NanoDet...0.6411985.0115650.0000009.152363NaNNaN1944.8311771944.831177NaNeJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8...
\n", "

5 rows × 26 columns

\n", "
" ], "text/plain": [ " id timestamp neptune_id hardware \\\n", "122834 122834 2024-10-15 09:38:45.048419 GRDBCAR-9 cuda:AGX-Orin \n", "122826 122826 2024-10-15 09:38:45.048419 GRDBCAR-9 cuda:A4500 \n", "122827 122827 2024-10-15 09:38:45.048419 GRDBCAR-9 cuda:A4500 \n", "122828 122828 2024-10-15 09:38:45.048419 GRDBCAR-9 cuda:A4500 \n", "122829 122829 2024-10-15 09:38:45.048419 GRDBCAR-9 cuda:AGX-Xavier \n", "\n", " quantization calib_method per_channel height width model_family ... \\\n", "122834 int8 entropy False 320.0 256.0 NanoDet ... \n", "122826 float32 none none 320.0 256.0 NanoDet ... \n", "122827 float16 entropy False 320.0 256.0 NanoDet ... \n", "122828 int8 entropy False 320.0 256.0 NanoDet ... \n", "122829 float32 none none 320.0 256.0 NanoDet ... \n", "\n", " od_task_metric od_inf_rate_ms od_inf_energy_mJ od_size_MB \\\n", "122834 0.647767 2.243725 32.957386 9.641616 \n", "122826 0.641198 1.545569 96.821007 8.764759 \n", "122827 0.640356 0.760797 64.680267 7.014277 \n", "122828 0.647767 0.875939 54.021370 7.835824 \n", "122829 0.641198 5.011565 0.000000 9.152363 \n", "\n", " od_gpu_mem_MB od_cpu_mem_MB od_unified_mem_MB od_total_mem_MB \\\n", "122834 NaN NaN 2400.000488 2400.000488 \n", "122826 373.0 702.741089 NaN 1075.741089 \n", "122827 224.0 446.127960 NaN 670.127960 \n", "122828 213.0 368.084015 NaN 581.084015 \n", "122829 NaN NaN 1944.831177 1944.831177 \n", "\n", " od_avg_cpu_freq_Hz sppr \n", "122834 1581.438461 eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... \n", "122826 NaN eJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8... \n", "122827 NaN eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... \n", "122828 NaN eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... \n", "122829 NaN eJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8... \n", "\n", "[5 rows x 26 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filtered_df = df[df[\"if_task_metric\"] >= threshold]\n", "sorted_df = filtered_df.sort_values(\"if_inf_macs\")\n", "sorted_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selecting a Recipe\n", "\n", "After filtering and sorting the DataFrame to identify a suitable recipe, the next step is to retrieve the [Serialized Packaged Portable Recipe (SPPR)](https://docs.latentai.io/leip/design/latest/content/reference/recipe_creators/#leip_recipe_designer.create.from_sppr) for your chosen model configuration. For instance, from the table above, let’s select the recipe with ID `122834` tailored for `cuda:AGX-Orin` (Jetson AGX Orin), which employs `int8` quantization and `entropy` calibration." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-10-29 11:53:04,808 | WARNING | pantry.build-119 | You requested to build a Pantry, but haven't specified the desired execution contexts. Therefore, will use the installed ones ['leip_af', 'leip_forge', 'leip_stub_gen']\n" ] } ], "source": [ "from pathlib import Path\n", "\n", "workspace = Path('./workspace')\n", "pantry = rd.Pantry.build(workspace / \"./my_combined_pantry/\", force_rebuild=False)\n", "\n", "# Select the recipe with the specified ID\n", "recipe_id = 122834\n", "selected_recipe = sorted_df[sorted_df['id'] == recipe_id].iloc[0]\n", "\n", "sppr = selected_recipe['sppr']\n", "recipe = rd.create.from_sppr(sppr, pantry=pantry, allow_upgrade=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By following these steps, you can effectively explore Golden Volumes, filter for recipes that meet your performance needs, and select a recipe that best fits your dataset. Now, you can replace the data ingredient with your dataset and begin training as shown in the [Getting Started tutorial](https://docs.latentai.io/leip/design/latest/notebooks/GettingStartedwithLEIPDesign/). \n", "\n", "Happy experimenting with your new model!" ] } ], "metadata": { "kernelspec": { "display_name": "main", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 2 }