# Cloud Utilities for Deep Learning ⛅️
A super lightweight cloud management tool designed with deep learning applications in mind.
**Built with the belief that managing cloud resources should be as easy as:**
```python
import cloud
cloud.connect()
train_my_network()
cloud.down()
```
We welcome all contributions, suggestions, and use-cases. Reach out to us over GitHub or at team@for.ai with ideas!
## Contents
- [Quickstart](#quickstart)
- [Install](#install)
- [Config](#config)
- [Usage](#usage)
- [Documentation](#documentation)
- [Amazon EC2](#amazon-ec2)
- [Azure](#azure)
- [Google Cloud](#google-cloud)
## Quickstart
### Install:
Sort of stable:
```python
sudo pip install dl-cloud
```
Bleeding edge:
```python
git clone git@github.com:for-ai/cloud.git
sudo pip install -e cloud
```
### Config:
See `configs/cloud.toml-*` for instructions on how to authenticate for each provider (Google Cloud, AWS EC2, and Azure).
Place your completed configuration file (named `cloud.toml`) in either root `/` or `$HOME`. Otherwise, provide a full path to the file in `$CLOUD_CFG`.
### Usage:
#### GPU
```python
import cloud
cloud.connect()
# gpu instances have a dedicated GPU so we don't need to worry
# about preemption or acquiring/releasing accelerators online.
while True:
# train your model or w/e
cloud.down() # stop the instance (does not delete instance)
```
#### TPU (Only on GCP)
```python
import cloud
cloud.connect()
tpu = cloud.instance.tpu.get(preemptible=True) # acquire an accelerator
while True:
if not tpu.usable:
tpu.delete(background=True) # release the accelerator in the background
tpu = cloud.instance.tpu.get(preemptible=True) # acquire a new accelerator
else:
# train your model or w/e
cloud.down() # release all resources, then stop the instance (does not delete instance)
```
---
# Documentation
### cloud.connect()
Takes/Creates a `cloud.Instance` object and sets `cloud.instance` to it.
| **returns** | **desc.** |
| :------- | :------- |
| cloud_env | a cloud.Instance. |
### cloud.down()
Calls `cloud.instance.down()`.
### cloud.delete(confirm=True)
Calls `cloud.instance.delete(confirm)`.
### cloud.Resource
Takes/Creates a `cloud.Instance` object and sets `cloud.instance` to it.
| properties | desc. |
| :------- | :------- |
| `name` | str, name of the instance |
| `usable ` | bool, whether this resource is usable |
| **methods** | **desc.** |
| `up(background=False)` | start an existing stopped resource |
| `down(background=False)` | stop the resource. Note: this should not necessarily delete this resource |
| `delete(background=False)` | delete this resource |
### cloud.Instance(Resource)
An object representing a cloud instance with a set of Resources that can be allocated/deallocated.
| properties | desc. |
| :------- | :------- |
| `resource_managers` | list of ResourceManagers |
| **methods** | **desc.** |
| `down(background=False, delete_resources=True)` | stop this instance and optionally delete all managed resources |
| `delete(background=False, confirm=True)` | delete this instance with optional user confirmation |
### cloud.ResourceManager
Class for managing the creation and maintanence of `cloud.Resources`.
| properties | desc. |
| :------- | :------- |
| `instance ` | `cloud.Instance` instance owning this resource manager |
| `resource_cls ` | `cloud.Resource` type, the class of the resource to be managed |
| `resources ` | list of `cloud.Resource`s, managed resources |
| **methods** | **desc.** |
| `__init__(instance, resource_cls)` | `instance`: the `cloud.Instance` object operating this ResourceManager |
| | `resource_cls `: the `cloud.Resource` class this object manages |
| `add(*args, **kwargs)` | add an existing resource to this manager |
| `remove(*args, **kwargs)` | remove an existing resource from this manager |
## Amazon EC2
### cloud.AWSInstance(Instance)
A `cloud.Instance` object for AWS EC2 instances.
## Azure
### cloud.AzureInstance(Instance)
A `cloud.Instance` object for Microsoft Azure instances.
## Google Cloud
Our GCPInstance requires that your instances have `gcloud` installed and properly authenticated so that `gcloud alpha compute tpus create test_name` runs without issue.
### cloud.GCPInstance(Instance)
A `cloud.Instance` object for Google Cloud instances.
| properties | desc. |
| :------- | :------- |
| `tpu ` | `cloud.TPUManager`, a resource manager for this instance's TPUs |
| `resource_managers ` | list of owned `cloud.ResourceManager`s |
| **methods** | **desc.** |
| `__init__(collect_existing_tpus=True, **kwargs)` | `collect_existing_tpus `: bool, whether to add existing TPUs to this manager |
| | `**kwargs `: passed to `cloud.Instance`'s initializer |
### cloud.TPU(Resource)
Resource class for TPU accelerators.
| properties | desc. |
| :------- | :------- |
| `ip` | str, IP address of the TPU |
| `preemptible` | bool, whether this TPU is preemptible or not |
| `details` | dict {str: str}, properties of this TPU |
| **methods** | **desc.** |
| `up(background=False)` | start this TPU |
| `down(background=False)` | stop this TPU |
| `delete(background=False)` | delete this TPU |
### cloud.TPUManager(ResourceManager)
ResourceManager class for TPU accelerators.
| properties | desc. |
| :------- | :------- |
| `names` | list of str, names of the managed TPUs |
| `ips` | list of str, ips of the managed TPUs |
| **methods** | **desc.** |
| `__init__(instance, collect_existing=True)` | `instance`: the `cloud.GCPInstance` object operating this TPUManager |
| | `collect_existing`: bool, whether to add existing TPUs to this manager |
| `clean(background=True)` | delete all managed TPUs with unhealthy states |
| `get(preemptible=True)` | get an available TPU, or create one using `up()` if none exist |
| `up(preemptible=True, background=False)` | allocate and manage a new instance of `resource_cls ` |
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
共18个文件
py:10个
txt:4个
pkg-info:2个
资源分类:Python库 所属语言:Python 资源全名:dl-cloud-0.1.3.tar.gz 资源来源:官方 安装方法:https://lanzao.blog.csdn.net/article/details/101784059
资源推荐
资源详情
资源评论
收起资源包目录
dl-cloud-0.1.3.tar.gz (18个子文件)
dl-cloud-0.1.3
PKG-INFO 8KB
setup.cfg 38B
dl_cloud.egg-info
PKG-INFO 8KB
requires.txt 38B
SOURCES.txt 350B
top_level.txt 6B
dependency_links.txt 1B
setup.py 977B
README.md 6KB
cloud
cloud.py 579B
__init__.py 1001B
envs
utils.py 3KB
__init__.py 0B
env.py 3KB
registry.py 216B
aws.py 968B
gcp.py 7KB
azure.py 949B
共 18 条
- 1
资源评论
挣扎的蓝藻
- 粉丝: 13w+
- 资源: 15万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功