# ClearML Integration
<img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_dark.png#gh-light-mode-only" alt="Clear|ML"><img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_light.png#gh-dark-mode-only" alt="Clear|ML">
## About ClearML
[ClearML](https://cutt.ly/yolov5-tutorial-clearml) is an [open-source](https://github.com/allegroai/clearml) toolbox designed to save you time â±ï¸.
ð¨ Track every YOLOv5 training run in the <b>experiment manager</b>
ð§ Version and easily access your custom training data with the integrated ClearML <b>Data Versioning Tool</b>
ð¦ <b>Remotely train and monitor</b> your YOLOv5 training runs using ClearML Agent
ð¬ Get the very best mAP using ClearML <b>Hyperparameter Optimization</b>
ð Turn your newly trained <b>YOLOv5 model into an API</b> with just a few commands using ClearML Serving
<br />
And so much more. It's up to you how many of these tools you want to use, you can stick to the experiment manager, or chain them all together into an impressive pipeline!
<br />
<br />

<br />
<br />
## 𦾠Setting Things Up
To keep track of your experiments and/or data, ClearML needs to communicate to a server. You have 2 options to get one:
Either sign up for free to the [ClearML Hosted Service](https://cutt.ly/yolov5-tutorial-clearml) or you can set up your own server, see [here](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server). Even the server is open-source, so even if you're dealing with sensitive data, you should be good to go!
1. Install the `clearml` python package:
```bash
pip install clearml
```
1. Connect the ClearML SDK to the server by [creating credentials](https://app.clear.ml/settings/workspace-configuration) (go right top to Settings -> Workspace -> Create new credentials), then execute the command below and follow the instructions:
```bash
clearml-init
```
That's it! You're done ð
<br />
## ð Training YOLOv5 With ClearML
To enable ClearML experiment tracking, simply install the ClearML pip package.
```bash
pip install clearml>=1.2.0
```
This will enable integration with the YOLOv5 training script. Every training run from now on, will be captured and stored by the ClearML experiment manager.
If you want to change the `project_name` or `task_name`, use the `--project` and `--name` arguments of the `train.py` script, by default the project will be called `YOLOv5` and the task `Training`.
PLEASE NOTE: ClearML uses `/` as a delimter for subprojects, so be careful when using `/` in your project name!
```bash
python train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
```
or with custom project and task name:
```bash
python train.py --project my_project --name my_training --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
```
This will capture:
- Source code + uncommitted changes
- Installed packages
- (Hyper)parameters
- Model files (use `--save-period n` to save a checkpoint every n epochs)
- Console output
- Scalars (mAP_0.5, mAP_0.5:0.95, precision, recall, losses, learning rates, ...)
- General info such as machine details, runtime, creation date etc.
- All produced plots such as label correlogram and confusion matrix
- Images with bounding boxes per epoch
- Mosaic per epoch
- Validation images per epoch
- ...
That's a lot right? ð¤¯
Now, we can visualize all of this information in the ClearML UI to get an overview of our training progress. Add custom columns to the table view (such as e.g. mAP_0.5) so you can easily sort on the best performing model. Or select multiple experiments and directly compare them!
There even more we can do with all of this information, like hyperparameter optimization and remote execution, so keep reading if you want to see how that works!
<br />
## ð Dataset Version Management
Versioning your data separately from your code is generally a good idea and makes it easy to aqcuire the latest version too. This repository supports supplying a dataset version ID and it will make sure to get the data if it's not there yet. Next to that, this workflow also saves the used dataset ID as part of the task parameters, so you will always know for sure which data was used in which experiment!

### Prepare Your Dataset
The YOLOv5 repository supports a number of different datasets by using yaml files containing their information. By default datasets are downloaded to the `../datasets` folder in relation to the repository root folder. So if you downloaded the `coco128` dataset using the link in the yaml or with the scripts provided by yolov5, you get this folder structure:
```
..
|_ yolov5
|_ datasets
|_ coco128
|_ images
|_ labels
|_ LICENSE
|_ README.txt
```
But this can be any dataset you wish. Feel free to use your own, as long as you keep to this folder structure.
Next, â ï¸**copy the corresponding yaml file to the root of the dataset folder**â ï¸. This yaml files contains the information ClearML will need to properly use the dataset. You can make this yourself too, of course, just follow the structure of the example yamls.
Basically we need the following keys: `path`, `train`, `test`, `val`, `nc`, `names`.
```
..
|_ yolov5
|_ datasets
|_ coco128
|_ images
|_ labels
|_ coco128.yaml # <---- HERE!
|_ LICENSE
|_ README.txt
```
### Upload Your Dataset
To get this dataset into ClearML as a versionned dataset, go to the dataset root folder and run the following command:
```bash
cd coco128
clearml-data sync --project YOLOv5 --name coco128 --folder .
```
The command `clearml-data sync` is actually a shorthand command. You could also run these commands one after the other:
```bash
# Optionally add --parent <parent_dataset_id> if you want to base
# this version on another dataset version, so no duplicate files are uploaded!
clearml-data create --name coco128 --project YOLOv5
clearml-data add --files .
clearml-data close
```
### Run Training Using A ClearML Dataset
Now that you have a ClearML dataset, you can very simply use it to train custom YOLOv5 ð models!
```bash
python train.py --img 640 --batch 16 --epochs 3 --data clearml://<your_dataset_id> --weights yolov5s.pt --cache
```
<br />
## ð Hyperparameter Optimization
Now that we have our experiments and data versioned, it's time to take a look at what we can build on top!
Using the code information, installed packages and environment details, the experiment itself is now **completely reproducible**. In fact, ClearML allows you to clone an experiment and even change its parameters. We can then just rerun it with these new parameters automatically, this is basically what HPO does!
To **run hyperparameter optimization locally**, we've included a pre-made script for you. Just make sure a training task has been run at least once, so it is in the ClearML experiment manager, we will essentially clone it and change its hyperparameters.
You'll need to fill in the ID of this `template task` in the script found at `utils/loggers/clearml/hpo.py` and then just run it :) You can change `task.execute_locally()` to `task.execute()` to put it in a ClearML queue and have a remote agent work on it instead.
```bash
# To use optuna, install it first, otherwise you can change the optimizer to just be RandomSearch
pip install optuna
python utils/loggers/clearml/hpo.py
```

weixin58692541
- 粉丝: 4129
- 资源: 7560
最新资源
- sqlserver jtds 驱动
- 微电网逆变器控制策略与并网仿真研究:下垂控制仿真模型及其参考资料综述,微电网逆变器控制策略与并网仿真研究:下垂控制仿真模型及其应用探讨,微电网逆变器控制、并网仿真、下垂控制等仿真模型以及一些参考资料
- DeepSeek清华大学全套学习资料1-5
- 【毕业设计】Python的Django-html深度学习的安全帽佩戴检测系统源码(完整前后端+mysql+说明文档+LW+PPT).zip
- 【毕业设计】Python的Django-html深度学习的聊天机器人设计源码(完整前后端+mysql+说明文档+LW+PPT).zip
- 【毕业设计】Python的Django-html搜索的目标站点内容监测系统源码(完整前后端+mysql+说明文档+LW+PPT).zip
- 蒙特卡洛思想下的充电汽车负荷曲线生成研究报告:探索1万台汽车的充电负荷曲线模拟,基于蒙特卡洛思想的充电汽车充电负荷曲线生成研究报告-涵盖负荷曲线的精细分析与实际应用解析,基于蒙特卡洛思想生成1000
- DeepSeek 保姆级新手教程
- MATLAB下电转气协同与碳捕集垃圾焚烧虚拟电厂优化调度复现程序及仿真结果展示,基于MATLAB和CPLEX的碳捕集与垃圾焚烧虚拟电厂电转气协同优化调度算法研究,MATLAB代码:计及电转气协同的含碳
- 基于直齿行星齿轮传动系统的ode45求解方法及自由度收敛研究,基于直齿行星齿轮传动系统的ODE45求解及其自由度全收敛分析研究,直齿行星齿轮传动系统ode45求解,自由度全部收敛 本人主攻齿轮动力学
- 洪泽湖流域1951-2008年面雨量变化特征与趋势分析及其防洪减灾应用
- 基于TMS320F2812的PMSM有传感器矢量控制双闭环程序系统,TMS320F2812 DSP矢量控制实现PMSM传感器转速与电流双闭环控制程序,矢量控制dsp2812 主控为TMS320F281
- blender插件,mmd-tools-v2.10.3
- 黑龙江省近45年积温变化
- SVPWM过调制控制技术:深入解析与实际应用,基于SVPWM技术的过调制控制方法的研究与应用,SVPWM过调制控制方法 ,SVPWM; 过调制; 控制方法; 数字信号处理,SVPWM过调制控制策略
- 基于PI控制器与滑膜控制器的三相永磁同步电机无速度传感器控制策略及其SVPWM-MRAS-PI SMC算法研究与应用,基于PI控制器与滑膜控制器的三相永磁同步电机无速度传感器控制策略研究:SVPWM
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈


