SAM-Med 2D视觉大模型复现、训练自定义数据集：脊椎分割

# SAM-Med2D \[[Paper](https://arxiv.org/abs/2308.16184)] [](https://openxlab.org.cn/apps/detail/GMAI/SAM-Med2D) </a> <a src="https://img.shields.io/badge/Data-SAMed2D_20M-blue?logo=red" href="https://openxlab.org.cn/datasets/GMAI/SA-Med2D-20M"> <img src="https://img.shields.io/badge/Data-SAMed2D_20M-blue?logo=red"> <a src="https://img.shields.io/badge/cs.CV-2308.16184-b31b1b?logo=arxiv&logoColor=red" href="https://arxiv.org/abs/2308.16184"> <img src="https://img.shields.io/badge/cs.CV-2308.16184-b31b1b?logo=arxiv&logoColor=red"> <a src="https://img.shields.io/badge/WeChat-Group-green?logo=wechat" href="https://github.com/OpenGVLab/SAM-Med2D/blob/main/assets/SAM-Med2D_wechat_group.jpeg"> <img src="https://img.shields.io/badge/WeChat-Group-green?logo=wechat"> </a> <a target="_blank" href="https://colab.research.google.com/github/OpenGVLab/SAM-Med2D/blob/main/predictor_example.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a> [](https://github.com/OpenGVLab/SAM-Med2D)🔥🔥🔥 <!-- ## Description --> ## 🌤️ Highlights - 🏆 Collected and curated the largest medical image segmentation dataset (4.6M images and 19.7M masks) to date for training models. - 🏆 The most comprehensive fine-tuning based on Segment Anything Model (SAM). - 🏆 Comprehensive evaluation of SAM-Med2D on large-scale datasets. ## 🔥 Updates - (2023.12.05) We open the download of the dataset on the [Hugging Face](https://huggingface.co/datasets/OpenGVLab/SA-Med2D-20M) platform - (2023.11.23) We have released the [SA-Med2D-20M](https://openxlab.org.cn/datasets/GMAI/SA-Med2D-20M) dataset - (2023.11.21) We have released article introducing the [SA-Med2D-20M](https://arxiv.org/abs/2311.11969) dataset - (2023.10.24) We now released [SAM-Med3D](https://github.com/uni-medical/SAM-Med3D), which focus on segmentation of 3D medical imaging - (2023.09.14) Train code release - (2023.09.02) Test code release - (2023.08.31) Pre-trained model release - (2023.08.31) Paper release - (2023.08.26) Online Demo release ## 👉 Dataset SAM-Med2D is trained and tested on a dataset that includes **4.6M images** and **19.7M masks**. This dataset covers 10 medical data modalities, 4 anatomical structures + lesions, and 31 major human organs. To our knowledge, this is currently the largest and most diverse medical image segmentation dataset in terms of quantity and coverage of categories. <p align="center"><img width="800" alt="image" src="https://github.com/OpenGVLab/SAM-Med2D/blob/main/assets/dataset.png"></p> ## 👉 Framework The pipeline of SAM-Med2D. We freeze the image encoder and incorporate learnable adapter layers in each Transformer block to acquire domain-specific knowledge in the medical field. We fine-tune the prompt encoder using point, Bbox, and mask information, while updating the parameters of the mask decoder through interactive training. <p align="center"><img width="800" alt="image" src="https://github.com/OpenGVLab/SAM-Med2D/blob/main/assets/framwork.png"></p> ## 👉 Results <table> <caption align="center">Quantitative comparison of different methods on the test set: </caption> <thead> <tr> <th>Model</th> <th>Resolution</th> <th>Bbox (%)</th> <th>1 pt (%)</th> <th>3 pts (%)</th> <th>5 pts (%)</th> <th>FPS</th> <th>Checkpoint</th> </tr> </thead> <tbody> <tr> <td align="center">SAM</td> <td align="center">$256\times256$</td> <td align="center">61.63</td> <td align="center">18.94</td> <td align="center">28.28</td> <td align="center">37.47</td> <td align="center">51</td> <td align="center"><a href="https://drive.google.com/file/d/1_U26MIJhWnWVwmI5JkGg2cd2J6MvkqU-/view?usp=drive_link">Offical</a></td> </tr> <tr> <td align="center">SAM</td> <td align="center">$1024\times1024$</td> <td align="center">74.49</td> <td align="center">36.88</td> <td align="center">42.00</td> <td align="center">47.57</td> <td align="center">8</td> <td align="center"><a href="https://drive.google.com/file/d/1_U26MIJhWnWVwmI5JkGg2cd2J6MvkqU-/view?usp=drive_link">Offical</a></td> </tr> <tr> <td align="center">FT-SAM</td> <td align="center">$256\times256$</td> <td align="center">73.56</td> <td align="center">60.11</td> <td align="center">70.95</td> <td align="center">75.51</td> <td align="center">51</td> <td align="center"><a href="https://drive.google.com/file/d/1J4qQt9MZZYdv1eoxMTJ4FL8Fz65iUFM8/view?usp=drive_link">FT-SAM</a></td> </tr> <tr> <td align="center">SAM-Med2D</td> <td align="center">$256\times256$</td> <td align="center">79.30</td> <td align="center">70.01</td> <td align="center">76.35</td> <td align="center">78.68</td> <td align="center">35</td> <td align="center"><a href="https://drive.google.com/file/d/1ARiB5RkSsWmAB_8mqWnwDF8ZKTtFwsjl/view?usp=drive_link">SAM-Med2D</a></td> </tr> </tbody> </table> 百度云链接: https://pan.baidu.com/s/1HWo_s8O7r4iQI6irMYU8vQ?pwd=dk5x 提取码: dk5x <table> <caption align="center">Generalization validation on 9 MICCAI2023 datasets, where "*" denotes that we drop adapter layer of SAM-Med2D in test phase: </caption> <thead> <tr> <th rowspan="2">Datasets</th> <th colspan="3">Bbox prompt (%)</th> <th colspan="3">1 point prompt (%)</th> </tr> <tr> <th>SAM</th> <th>SAM-Med2D*</th> <th>SAM-Med2D</th> <th>SAM</th> <th>SAM-Med2D*</th> <th>SAM-Med2D</th> </tr> </thead> <tbody> <tr> <td align="center"><a href="https://www.synapse.org/#!Synapse:syn51236108/wiki/621615">CrossMoDA23</a></td> <td align="center">78.12</td> <td align="center">86.26</td> <td align="center">88.42</td> <td align="center">33.84</td> <td align="center">65.85</td> <td align="center">85.26</td> </tr> <tr> <td align="center"><a href="https://kits-challenge.org/kits23/">KiTS23</a></td> <td align="center">81.52</td> <td align="center">86.14</td> <td align="center">89.89</td> <td align="center">31.36</td> <td align="center">56.67</td> <td align="center">83.71</td> </tr> <tr> <td align="center"><a href="https://codalab.lisn.upsaclay.fr/competitions/12239#learn_the_details">FLARE23</a></td> <td align="center">73.20</td> <td align="center">77.18</td> <td align="center">85.09</td> <td align="center">19.87</td> <td align="center">32.01</td> <td align="center">77.17</td> </tr> <tr> <td align="center"><a href="https://atlas-challenge.u-bourgogne.fr/">ATLAS2023</a></td> <td align="center">76.98</td> <td align="center">79.09</td> <td align="center">82.59</td> <td align="center">29.07</td> <td align="center">45.25</td> <td align="center">64.76</td> </tr> <tr> <td align="center"><a href="https://multicenteraorta.grand-challenge.org/">SEG2023</a></td> <td align="center">64.82</td> <td align="center">81.85</td> <td align="center">85.09</td> <td align="center">21.15</td> <td align="center">34.71</td> <td align="center">72.08</td> </tr> <tr> <td align="center"><a href="https://lnq2023.grand-challenge.org/lnq2023/">LNQ2023</a></td> <td align="center">53.02</td> <td align="center">57.37</td> <td align="center">58.01</td> <td align="center">7.05</td> <td align="center">7.21</td> <td align="center">37.64</td> </tr> <tr> <td align="center"><a href="https://codalab.lisn.upsaclay.fr/competitions/9