61 DAC Proceedings.zip

DeepRIoT: Continuous Integration and Deployment Of

Robotic-IoT Applications

Meixun Qu

∗,†

, Jie He

∗,†

, Zlatan Tucaković

†

, Ezio Bartocci

†

Technische Universität Wien

†

, AIT Austrian Institute of Technology

‡

ABSTRACT

We present DeepRIoT, a continuous integration and continuous de-

ployment (CI/CD) based architecture that accelerates the learning

nally, with low cost, succeeded in deploying our controller in a

real F1tenth car that is able to reach the goal and avoid collision

from a virtual car through mixed reality.

KEYWORDS

Deep Reinforcement Learning, Sim2Real, DevOps, CI/CD

1 INTRODUCTION

Deep Reinforcement Learning [

1

] (RL) has been gaining momen-

tum in the control tasks of Robotic-IoT systems [

2

–

has achieved state-of-the-art in some simulation benchmarks, there

are still many factors that hinder its application in real Robotic-IoT

and deployment of RL in real Robotic-IoT systems. In this paper,

we ll in this blank by proposing DeepRIoT, a practical framework

monitoring of the real system during its operation (Ops). In our con-

RL policy in the simulation environment. Using DevOps machinery

for prot or commercial advantage and that copies bear this notice and the full citation

on the rst page. Copyrights for third-party components of this work must be honored.

For all other uses, contact the owner/author(s).

© 2024 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-0601-1/24/06.

https://doi.org/10.1145/3649329.3658250

Continuous Deployment [

8

] (CD). To accelerate RL in the simulation

avoiding collisions with static obstacles and other vehicles.

DeepRIoT Architecture. We sketch the architecture of DeepRIoT,

depicted in Fig. 1. The architecture consists of Pipelines (A-E).

Pipeline A. Here we specify requirements

for the given robotic

tasks. These requirements are given in the form of formal speci-

cations using Signal Temporal Logic (STL) [

use runtime monitors to measure the degree to which the observed

robotic behaviors satisfy or violate the specications (see Pipelines

C-E). Second, these same specications are used to engineer the

Gaussian noises to the LiDAR model to make it more realistic. When

we teach the car to avoid collision to obstacles, we only consider

laser scans whose lengths are within a specied range.

Pipeline C. This is responsible for training the RL policy. It takes

the inputs processed by the previous two pipelines, and starts from

picking a simple kinematics model (KiModel) to simulate the dy-

namics of the car. Every step of training yields a four-tuple called

experience that includes current state, current action, next state and

associated reward. We classify each piece of experience into dierent

types, and store them in dierent replay buers according to their

types. In each step, the trainer will proportionally sample batched

the success rate after each training stage, which is dened as the

percentage of satisfying

among episodes. Finally, we export the

satisfactory policy (Base Policy) to a corresponding repository. The

success of training in Pipeline C, however, is only based on a simple

plant model, very dierent from the actual dynamics of the car.

Pipeline D. This focuses on adapting the trained policy to richer

dynamics. We rst pick a dynamic model (DyModel) from the model

repository, and test the performance of the so-far learned policy

we directly go to the Sim2Real pipeline. Otherwise, we improve

the new version of the KiModel in the model repository, and go

back to Pipeline C to train a second time our policy on the updated

KiModel. Afterwards, we test the updated policy (Improved Policy)

to the DyModel without directly being trained on it. This further

indicates that, without training the policy on the real car, which

is extremely expensive, we can continue to update the policy by

training it on a DyModel whose traces are as similar as the real car.

Pipeline E. Here we rst evaluate the Improved Policy on the real car.

If requirements in

are satised by the observed behaviors, this

policy we can obtain through our framework. Finally, we evaluate

the Final Policy in a mixed-reality environment where a real car

and a simulated car should cross each other without collision, and

both reach their goal positions. The workow dened by the above

architecture and its pipelines is fully automated using the CI/CD

integration, which is described in more details in Section 6.

cost CI/CD framework for training and deployment of RL policies in

2 RELATED WORK

to continuous control of robotics tasks, as in [

generally requires millions of training episodes to be eective. Gen-

erating so many episodes in real-world Robotic-IoTs is practically

infeasible. Furthermore, RL may be unsafe and costly, consider-

ing the potential damage and harm it may cause by applying a

trial-and-error method directly on Robotic-IoTs.

5

14

17

4

17

algorithm with episodes generated in a simulation environment,

a digital twin reproducing the physical behavior of the real sys-

both reachability and safety properties. To this end, we rst specify

the reachability property in Signal Temporal Logic (STL) [10]: