# ClearML Integration
<img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_dark.png#gh-light-mode-only" alt="Clear|ML"><img align="center" src="https://github.com/thepycoder/clearml_screenshots/raw/main/logos_light.png#gh-dark-mode-only" alt="Clear|ML">
## About ClearML
[ClearML](https://cutt.ly/yolov5-tutorial-clearml) is an [open-source](https://github.com/allegroai/clearml) toolbox designed to save you time â±ï¸.
ð¨ Track every YOLOv5 training run in the <b>experiment manager</b>
ð§ Version and easily access your custom training data with the integrated ClearML <b>Data Versioning Tool</b>
ð¦ <b>Remotely train and monitor</b> your YOLOv5 training runs using ClearML Agent
ð¬ Get the very best mAP using ClearML <b>Hyperparameter Optimization</b>
ð Turn your newly trained <b>YOLOv5 model into an API</b> with just a few commands using ClearML Serving
<br />
And so much more. It's up to you how many of these tools you want to use, you can stick to the experiment manager, or chain them all together into an impressive pipeline!
<br />
<br />
![ClearML scalars dashboard](https://github.com/thepycoder/clearml_screenshots/raw/main/experiment_manager_with_compare.gif)
<br />
<br />
## 𦾠Setting Things Up
To keep track of your experiments and/or data, ClearML needs to communicate to a server. You have 2 options to get one:
Either sign up for free to the [ClearML Hosted Service](https://cutt.ly/yolov5-tutorial-clearml) or you can set up your own server, see [here](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server). Even the server is open-source, so even if you're dealing with sensitive data, you should be good to go!
1. Install the `clearml` python package:
```bash
pip install clearml
```
1. Connect the ClearML SDK to the server by [creating credentials](https://app.clear.ml/settings/workspace-configuration) (go right top to Settings -> Workspace -> Create new credentials), then execute the command below and follow the instructions:
```bash
clearml-init
```
That's it! You're done ð
<br />
## ð Training YOLOv5 With ClearML
To enable ClearML experiment tracking, simply install the ClearML pip package.
```bash
pip install clearml>=1.2.0
```
This will enable integration with the YOLOv5 training script. Every training run from now on, will be captured and stored by the ClearML experiment manager.
If you want to change the `project_name` or `task_name`, use the `--project` and `--name` arguments of the `train.py` script, by default the project will be called `YOLOv5` and the task `Training`.
PLEASE NOTE: ClearML uses `/` as a delimiter for subprojects, so be careful when using `/` in your project name!
```bash
python train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
```
or with custom project and task name:
```bash
python train.py --project my_project --name my_training --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache
```
This will capture:
- Source code + uncommitted changes
- Installed packages
- (Hyper)parameters
- Model files (use `--save-period n` to save a checkpoint every n epochs)
- Console output
- Scalars (mAP_0.5, mAP_0.5:0.95, precision, recall, losses, learning rates, ...)
- General info such as machine details, runtime, creation date etc.
- All produced plots such as label correlogram and confusion matrix
- Images with bounding boxes per epoch
- Mosaic per epoch
- Validation images per epoch
- ...
That's a lot right? ð¤¯
Now, we can visualize all of this information in the ClearML UI to get an overview of our training progress. Add custom columns to the table view (such as e.g. mAP_0.5) so you can easily sort on the best performing model. Or select multiple experiments and directly compare them!
There even more we can do with all of this information, like hyperparameter optimization and remote execution, so keep reading if you want to see how that works!
<br />
## ð Dataset Version Management
Versioning your data separately from your code is generally a good idea and makes it easy to acquire the latest version too. This repository supports supplying a dataset version ID, and it will make sure to get the data if it's not there yet. Next to that, this workflow also saves the used dataset ID as part of the task parameters, so you will always know for sure which data was used in which experiment!
![ClearML Dataset Interface](https://github.com/thepycoder/clearml_screenshots/raw/main/clearml_data.gif)
### Prepare Your Dataset
The YOLOv5 repository supports a number of different datasets by using yaml files containing their information. By default datasets are downloaded to the `../datasets` folder in relation to the repository root folder. So if you downloaded the `coco128` dataset using the link in the yaml or with the scripts provided by yolov5, you get this folder structure:
```
..
|_ yolov5
|_ datasets
|_ coco128
|_ images
|_ labels
|_ LICENSE
|_ README.txt
```
But this can be any dataset you wish. Feel free to use your own, as long as you keep to this folder structure.
Next, â ï¸**copy the corresponding yaml file to the root of the dataset folder**â ï¸. This yaml files contains the information ClearML will need to properly use the dataset. You can make this yourself too, of course, just follow the structure of the example yamls.
Basically we need the following keys: `path`, `train`, `test`, `val`, `nc`, `names`.
```
..
|_ yolov5
|_ datasets
|_ coco128
|_ images
|_ labels
|_ coco128.yaml # <---- HERE!
|_ LICENSE
|_ README.txt
```
### Upload Your Dataset
To get this dataset into ClearML as a versioned dataset, go to the dataset root folder and run the following command:
```bash
cd coco128
clearml-data sync --project YOLOv5 --name coco128 --folder .
```
The command `clearml-data sync` is actually a shorthand command. You could also run these commands one after the other:
```bash
# Optionally add --parent <parent_dataset_id> if you want to base
# this version on another dataset version, so no duplicate files are uploaded!
clearml-data create --name coco128 --project YOLOv5
clearml-data add --files .
clearml-data close
```
### Run Training Using A ClearML Dataset
Now that you have a ClearML dataset, you can very simply use it to train custom YOLOv5 ð models!
```bash
python train.py --img 640 --batch 16 --epochs 3 --data clearml://<your_dataset_id> --weights yolov5s.pt --cache
```
<br />
## ð Hyperparameter Optimization
Now that we have our experiments and data versioned, it's time to take a look at what we can build on top!
Using the code information, installed packages and environment details, the experiment itself is now **completely reproducible**. In fact, ClearML allows you to clone an experiment and even change its parameters. We can then just rerun it with these new parameters automatically, this is basically what HPO does!
To **run hyperparameter optimization locally**, we've included a pre-made script for you. Just make sure a training task has been run at least once, so it is in the ClearML experiment manager, we will essentially clone it and change its hyperparameters.
You'll need to fill in the ID of this `template task` in the script found at `utils/loggers/clearml/hpo.py` and then just run it :) You can change `task.execute_locally()` to `task.execute()` to put it in a ClearML queue and have a remote agent work on it instead.
```bash
# To use optuna, install it first, otherwise you can change the optimizer to just be RandomSearch
pip install optuna
python utils/loggers/clearml/hpo.py
```
![HPO](https://github.com/thepycoder/clearml_screenshots/raw/main/
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
我们对yolov5源码中的common.py、yolo.py、loss.py、general.py 等进行了修改,更换了主干网络、改进了损失函数。 系统的前后端代码则体现在sever.py、detect_web.py和head-detect-web 文件夹中,主要基于Flask实现。 基于改进后的YOLOv5目标检测模型,实现人群密度检测系统。 改进的YOLOv5模型主要是用FasterNet主干网络更换了原YOLOv5的主干网络, 并应用Soft-NMS、最优运输分配(OTA)等方法改进了损失函数。
资源推荐
资源详情
资源评论
收起资源包目录
基于改进后的YOLOv5目标检测模型实现人群密度检测系统源码+模型+详细说明.zip (252个子文件)
CITATION.cff 406B
setup.cfg 2KB
.eslintrc.cjs 2KB
semantic.min-CANXK2OJ.css 654KB
app.css 21KB
results.csv 15KB
Dockerfile 3KB
Dockerfile 846B
Dockerfile-arm64 2KB
Dockerfile-cpu 2KB
.dockerignore 4KB
icons-7JOAMH2Y.eot 104KB
brand-icons-UXBLK6EO.eot 96KB
outline-icons-HR6CQBVC.eot 30KB
.gitignore 65B
show_web.html 417B
favicon.ico 17KB
tutorial.ipynb 103KB
tutorial.ipynb 54KB
tutorial.ipynb 43KB
video.JPG 5.38MB
image.JPG 2.38MB
train_batch0.jpg 604KB
train_batch1.jpg 592KB
train_batch2.jpg 555KB
val_batch2_pred.jpg 494KB
val_batch2_labels.jpg 491KB
bus.jpg 476KB
val_batch1_pred.jpg 443KB
val_batch1_labels.jpg 435KB
val_batch0_pred.jpg 407KB
val_batch0_labels.jpg 393KB
labels_correlogram.jpg 382KB
labels.jpg 201KB
zidane.jpg 165KB
flask.jpg 97KB
test.jpg 44KB
chunk-GIAAE3CH.js 925KB
chunk-BFLSOAUD.js 336KB
chunk-CXYJAP3H.js 297KB
chunk-BOXFZXVX.js 78KB
jsx-runtime-7NXSP56X.js 36KB
chunk-XU7DNSPJ.js 36KB
index.js 26KB
chunk-UWV35TSL.js 18KB
imageinput-SRLDF4YB.js 11KB
root-TIVC3ZIS.js 8KB
_index-P32U4Q7J.js 4KB
videoinput-WLBTJPVK.js 2KB
esm-NQK47IKO.js 2KB
chunk-XWXMIAY6.js 2KB
manifest-209E707D.js 2KB
chunk-PNG5AS42.js 2KB
history-VYYQU67J.js 2KB
heatmap-W4OL5MGH.js 2KB
entry.client-LD3RDIMD.js 1KB
chunk-ZWGWGGVF.js 1KB
__remix_entry_dev-UBDFYCZB.js 659B
client-DFZR44C7.js 298B
jsx-dev-runtime-RDH4Y5YT.js 281B
react-dom-BEFB7ICU.js 263B
remix_hmr-KULZA7QD.js 259B
remix.config.js 247B
runtime-GC7QIU56.js 213B
react-3OYUNTOK.js 207B
package-lock.json 426KB
metafile.js.json 385KB
metafile.server.json 6KB
metafile.css.json 3KB
package.json 1KB
tsconfig.json 566B
.keep 0B
chunk-GIAAE3CH.js.map 1.42MB
chunk-CXYJAP3H.js.map 667KB
chunk-BFLSOAUD.js.map 648KB
chunk-BOXFZXVX.js.map 123KB
jsx-runtime-7NXSP56X.js.map 59KB
chunk-XU7DNSPJ.js.map 58KB
index.js.map 30KB
chunk-UWV35TSL.js.map 30KB
imageinput-SRLDF4YB.js.map 10KB
root-TIVC3ZIS.js.map 5KB
chunk-XWXMIAY6.js.map 3KB
_index-P32U4Q7J.js.map 3KB
videoinput-WLBTJPVK.js.map 2KB
history-VYYQU67J.js.map 2KB
heatmap-W4OL5MGH.js.map 2KB
entry.client-LD3RDIMD.js.map 1KB
chunk-ZWGWGGVF.js.map 1KB
__remix_entry_dev-UBDFYCZB.js.map 659B
react-3OYUNTOK.js.map 100B
react-dom-BEFB7ICU.js.map 100B
chunk-PNG5AS42.js.map 100B
esm-NQK47IKO.js.map 100B
jsx-dev-runtime-RDH4Y5YT.js.map 100B
runtime-GC7QIU56.js.map 100B
remix_hmr-KULZA7QD.js.map 100B
client-DFZR44C7.js.map 100B
README.md 11KB
README.md 11KB
共 252 条
- 1
- 2
- 3
资源评论
FL1768317420
- 粉丝: 4438
- 资源: 4766
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功