A selfcalibrated photogeometric depth camera

In this paper, we present a practical 3D imaging system by combining the nearlight photometric stereo and the specklebased stereo matching method. The system is compact in structure and suitable for multialbedo targets. The parameters (including position and intensity) of the light sources can be
A SelfCalibrated PhotoGeometric Depth Camera (1)The near point light source model is adopted for making a compact sySten he parameters of the point light sources can be calibrated automati (3) The system can be adapted to the mult ialbedo objects (4)The PS method and the stereo matching method are combined reconstruct surfaces with rich details and high accuracv According to the guideline above. we design a 3D imaging system combin ing the PS method and the traditional stereo vision, called the photogeometric depth camera. The systcm consists of two CMOs cameras, a ncar infrared speckle projector, four near infrared LED and a synchronous circuit, as shown in Fig. 1 and Fig. 5. The experimental hardware cost is about $400 The main contributions of this paper are 1)A highprecision 3D imaging system is introduced, which combines the photometric stereo and the binocular stereo. The system is low cost, compact and can bc adapted to multialbcdo targets in the gencral indoor environment (2)A point, light The tradi tional point light source calibration method usually requires specific calibration spheres). In the distant lighting Illodel 7 to estinate the initial surface albedo Imap. Thel, with the estimated albedo map and the normal vector field fixed the parameters of the near lighting model are optimized. Next, with the optimized lighting model wc use the ncarlight photometric stereo(NLPS) mcthod to rccomputc tho albedo map and use the method 9 to compute higher quality depth map. We repeat the above two steps iteratively until convergence or the iteration times reaching a predefined maximum number The reInlaiIlder of this paper is organized as follows: The details of the pro posed system are presented in Section 2, and experimental results are provided in Section 3. Finally, Section 4 concludes the paper 2 PhotoGeometric Depth Camera 2.1 Hardware Our photogeometric depth camera consists of two CMOS cameras. a Kinect typc ncar infrared (NIR) spccklc projector, four NIR LEDs with a wavelength of 830 nm, and a microcontrollcrbascd circuit, as shown in Fig. 1. Thc cameras have a frame rate of 60 fps and a resolution of 1280x 960. The cameras are connected to the PC via Giga bit Ethernet interfaces. The narrowband filters are mounted on the 4 mm focal length lenses to filter out ambient light. The two caMeras and the four LEDs are mounted on a rigid structure to keep the stable relative positional relationship. The cameras, the LEDs, and the projector are synchronized by the trigger signal of the microcontrollerbased circuit, which is controlled by the PC via USB 2.0 interface. The NIR projector can emit a large number of random speckles to enhance the surface texture 4 Liang Xie et al ( Fig. 1 The proposed photogeametric depth camera D Speckle projector OFF LEDILEDA OFF LEDIED4 OF E Speckle projector OFF tor o Deckle projector OFI LEDILED OFF LEDI CN utens OFF LED2 ON, otcS OFF LED? ON, utleIs OFF LED4 oN, theiS OFF Each catera s,lap Eacli canela suay cue fi a e Each caela silap uue fiat Eh camela silap Jue iane Ead call.eia siap oe fiane Fig. 2 The timing diagram of a reconstruction cycle As illustratcd in Fig. 2, in a reconstruction cyclc, the projector is lit firstly and the cameras are triggered simultaneously to capture a pair of images of the speckles. The stereo image pair is used to generate the initial dept h map. Then the four LEDs are lit one by one, and the cameras are triggered to capture iNages under the illumination of each LED. Ii these inages, the images of the left camera are used for photometric computation 2.2 Nearlight photometric stereo We use the near point light source assumption in our image formation model [11]. In the jth image, the light vector lii from the surface point xi to the ght sourcc s, is written as With the near light source assumption, intensity observation O: is computed with accounting the inverse squarc law as E A SelfCalibrated PhotoGeometric Depth Camera 5 < surface Light Fig,3 Nearlight photometric stereo model Where, E; is the light source intensity at a unit distance, Pi is surface lbedo, and ni is the surface normal vector(Fig. 3) Once we know the light source parameters, we can estimate the normal vector I: anld the albedo Pi according to Ey.(2) froI at least three observa tiOns In order to balance the efficiency and the quality, we use four point light sources in our setup 2.3 Reconstruction pipeline We use the stereo camera calibration method in [17 to calibrate our stere o cameras. The light source will be sclfcalibrated using the mcthod to bo discussed in section 2 Given the calibrated cameras and lightings, the 3D reconstruction pipeline is illustrated in Fig. 4. Firstly, the stereo Latching Inlethiod is applied to the speckle image pair to generate the initial depth map Then, the photometric computation is applied to the four images under the illumination of the four LEDs rcspcctivcly to gcncratc the surfacc normal vector ficld. Finally, thc ob tained initial depth map and the normal vector field are integrated to generate the higher quality depth map 2.4 Initial dcpth generation The semiglobal matching(SGM) method[18] performs an energy minimiza tion using dynamic programming on multiple lD paths. The energy function consists of three terIns: d data terIll for photoconsistency, a sInloothness terI for slanted surfaces that change the disparity slightly(parameter P1), and a smoothness term for depth discontinuities(parameter P2). Due to that SGM has a good balance in efficiency and accuracy we use this method to estimate the initial depth map. 6 Liang Xie et al Compute the initial depth map using stereo match lhod Speckle image pair Optimized depth map Initial depth man Four images under illumination of leds Normal vector field he normal field using Ps method Fig. 4 The pipeline of the reconstructi 2.5 Point light source selfcalibration This scction proposcd a new calibration mcthod for point light sourcc includin the geometric parameters and the light intensity. With the selfcalibration method, our system does not rely on the fixed calibration such as mirror spheres[14[15[16, which makes the system more flexible and practical To estimate Ei and si, we use the system described in Section l to capture five image pairs of the target according to the timing diagram shown in Fi 2. Our light source calibration firstly makes a distant lighting assumption and estimate a rough albedo map. Then, an iterative manner optimization is applied to estimate the parameters of the near light sources. The calibration algorithm is summarized as follows Algorithm 1: Point light, source calibration (1)Initialization Rough depth map generation. Each stereo pair is rectified to obtain a rowaligned epipolar geometry. The stereo Latching Inethod described iI Section 2.4 o the speckle image pair to generate the initial depth Do of the target. A bilateral filter [19 glied to the to obtain a discontinuity preserved depth map with reduccd noisc Do )No (Do(u)Do(ql)Do(q)( 3) ∈b(u) Where, No=exp(t20), nb(u) denotes the neighborhood of the pixel u,and Wp is a normalizing constant A SelfCalibrated PhotoGeometric Depth Camera Initial position estimation. According to the mounting position of the LEDs relative to the reference camera we use the left calera as the reference camera), we can estimate an initial value Si, o for Sj Initial albedo estimation. Wc follow the automatic calibration mcthod 7 with distant, lighting assumption to estimate the initial al bedo map p We first robust y estimate a rank3 approximation of the observed brightness matrix using an iterative reweighting method, and then factorize this rank reduced brightness matrix into the corresponding lighting, albedo and surface IIOrInlal coInponents Initial intcnsity estimation. With the depth map dn and the camera parameters, the point cloud xi of the target surface can be generated. Fur thermore, the surface normal vectors ni can be estimated with the point cloud 20. Up to nOw, using Fq.(2), we can estimate the initial value of F, with the linear least square method O E i=i pi; Where, N is the number of the surface point (2) Position and intensity refinement q(2)is a typical nonlinear least squares problem. With the estimated initial values, we use the LevenbergMarquardt(LM)algorithm 21 to op timizc Ei and s, with the albedo map and the normal ficld fixcd. The cost function is defined as: cost(E, s) 4N ∑∑(n1EA Where,E={E},S={s},j=1,2.3,4. (3 Updating albedo map, surface points and normal With the optimized Fi and s,, we use the near lighting model(Eq(2)) to recompute the albedo map and the normal field. By combining the roug depth map and the normal field with the method in 9: the higher quality depth map can be acquired. Note that, the normal field used for the followin optimization is obtained from the optimized depth map, rather than the PS mcthod (4)Iterative optimization Jump to Step(2) until convergence or the iteration times reaching the predefined IllaxiInuIl nunber pth normal fusion To estimate the optimal depth by combining the normal vector field by the PS method and the rough depth map by the stereo matching, we can form Liang Xie et al Fig. 5 Prototype of the proposed photogeometric depth camera a linear system of equations as 9 to refine the quality of the reconstructed surtace AI 入D D (6) Where d is the refined depth Imap, V is a laplacian operator, I is all identity matrix and A is a weighting parameter controlling the contribution of sparse, it can be efficiently solved usIp ox ny. y ie& for each normalneN* pth constraint aN is the stacks of onea While it forms a la Lations. because the left matrix ng existing sparse linear solvers(e CHOLMOD 22) 3 Experimental results Fig 5 is the prototype of the proposed photogeometric depth camera. The baseline length of the stereo system is 176.92 mm. The dimension of the depth camera is 260m.. 76mm x 150mm 3. 1 Qualitativc cvaluation Wc firstly usc thrcc targets including a malc, a fcmalc and a shoc to cvaluatc our depth cameras. The gray images of the three targets are shown in Fig. 7 Fig. 6 shows the convergence curve of the iterative optimization process for the target in Fig 4. The Yaxis is the root Ileal square error defined ill Eq.(4). After 10 iterations, the error converges We compare our method with the distant lighting model [7. Fig. 7(a) shows the estimated albedo maps, and Fig. 7(b)shows the estimated normal vector fields From Fig. 7(a) we can know that the albedo of the eyebrows of A SelfCalibrated PhotoGeometric Depth Camera 8 teration times Fig. 6 Convergence curve of the iterative optinization of the light source paraineters for the dataset shown in Fig. 4. The error is defined in Equation (5) the two persons is relatively low and the albedo of the words on clothes of the Inale is relatively high. Our results correctly reflect these facts. However, the method in 7 cannot show these. The albedo bias of 7 is also severe for the shoc. Furthermore, the estimated surfacc normal vectors in facc regions of tho method in 7 are severely biased. These results show that the autocalibration method in Subsection 2.5 improves the quality of the estimated albedos and normaIs great ly Fig.8 shows the reconstruction results of three targets. Fig 8(a) shows the results of the Imale shown ill Fig. 7(a), Fig 8(b) shows the reconstructiOn result of a female, and Fig. 8(c)shows the results of a shoe. The left is the result using the speckle images, and the right is the result after combining the initial depth map by the stereo matching and the normal vector field by the PS mcthod. Thesc rcsults show that the reconstruction quality can bc improved remarkably by combining the nearlight ps method, in which the para meters of the point light sources are calibrated automatically using our calibration method described in Section 2.5 We also conpare our depth camera witl Kinect, a popular depth camera Fig. 9(a) shows the reconstruction results of a 30 cm tall David sculpture, and Fig 9(b) shows the results of our depth camera. The volume voxels resolution of KincctFusion is sct to 512. Wc can scc that thc rcsult of our dopth camera has higher quality in reconstruction details 3.2 Quantitativc cvaluation Fig. 10 shows the quantitative evaluation results, where the result of a com mercial phaseshifting system with nominal accuracy of 0.025 mm is treated 10 Liang Xie et al RYI∥ RYT∥ Fig. 7 Comparisons of the estimated albedo naps and the normal vector fields. (a) The gray mages of the three targets and the results of the estimated albedo maps.(b)Results of the estimated normal vector fields. In (a), the first column shows the gray images, the second column shows the results of [7, and the last column shows our results. In (b), the left shows the results of 7), and the right shows ours
 4KB
TH06温湿度检测DEMO程序
20190319本文件为模拟IIC驱动TH06的demo程序，The TH06 I2C Humidity and Temperature Sensor is a monolithic CMOS IC integrat
 1.93MB
论文研究台风“米雷”对掠海飞行的影响分析.pdf
20190920论文研究台风“米雷”对掠海飞行的影响分析.pdf, 以CCMP （CrossCalibrated， MultiPlatform）风场驱动海浪模式SWAN （Simulating Waves N
 3.71MB
STIM210_300.zip
20200209STIM210_300 陀螺仪 o Miniature package o Low noise o Low bias instability o Excellent performance in vi
 15.73MB
pix4d指导书
20140624本文件为pix4的指导书，欢迎参考，互相学习。
 2.94MB
Online Learning and Sequential Anomaly Detection in Trajectories
20190306Detection of anomalous trajectories is an important problem in the surveillance domain. Various algo
 50.19MB
计算机视觉（英文版）
20180720CONTENTS I IMAGEFORMATION 1 1 RADIOMETRY — MEASURING LIGHT 3 1.1 Light in Space 3 1.1.1 Foreshorteni
 1.86MB
雷达技术知识
20140506关于雷达方面的知识！ EFFECTIVENESS OF EXTRACTING WATER SURFACE SLOPES FROM LIDAR DATA WITHIN THE ACTIVE CHANNE
 820KB
大气水文耦合模型
20150811水文模型 全英的 Abstract: Much attention has recently been focused on the effects that climate variability
 831KB
Growing Like China
20100911This paper constructs a growth model that is consistent with salient features of the recent Chinese
 2.24MB
V400培训教程.pdf
20200326RIEGL地面激光扫描仪数据获取手册，riscan pro，应用于vz400 Introduction into RiSCAN PRO RiSCAN PRO is the companion soft
 6.30MB
ARTA Audio Measurement And Analysis
20120526解压密码 123456 因注册鸡会引起误报, 只共享了注册玛, 够用了. The ARTA software consists of following programs: ARTA  progra
 274KB
Android代码Ruler
20190806Privacy Friendly Ruler Privacy Friendly Ruler turns the phone screen into a ruler for quick measurme
 156.61MB
百度apollo3.5深度学习介绍_v4.4.pdf
20191106Apollo(阿波罗)是一个开放的、完整的、安全的自动驾驶平台，以灵活和高性能的技术架构，为全自动驾驶提供支持。 Apollo 3.5 is loaded with new modules and f
 357KB
日本人白平衡AWB算法
20181202这是日本著名相机公司的白平衡算法awb，全是English,希望在专业人士有所帮助，看得懂绝对有用，没用说明你们根本看不懂。
 268KB
磁力计校准
20170924Introduction The magnetoresistive sensors within the nIMU and µIMU units have been calibrated under
 20.20MB
navicat（内含激活码）
20180511navicat支持mysql的可视化操作，内涵激活码，不用再忍受弹框的痛苦。
 9.25MB
HTML期末大作业
20121124这是我自己做的HTML期末大作业，花了很多时间，稍加修改就可以作为自己的作业了，而且也可以作为学习参考

下载
TH06温湿度检测DEMO程序
TH06温湿度检测DEMO程序

下载
论文研究台风“米雷”对掠海飞行的影响分析.pdf
论文研究台风“米雷”对掠海飞行的影响分析.pdf

下载
STIM210_300.zip
STIM210_300.zip

下载
pix4d指导书
pix4d指导书

下载
Online Learning and Sequential Anomaly Detection in Trajectories
Online Learning and Sequential Anomaly Detection in Trajectories

下载
计算机视觉（英文版）
计算机视觉（英文版）

下载
雷达技术知识
雷达技术知识

下载
大气水文耦合模型
大气水文耦合模型

下载
Growing Like China
Growing Like China

下载
V400培训教程.pdf
V400培训教程.pdf

下载
ARTA Audio Measurement And Analysis
ARTA Audio Measurement And Analysis

下载
Android代码Ruler
Android代码Ruler

下载
百度apollo3.5深度学习介绍_v4.4.pdf
百度apollo3.5深度学习介绍_v4.4.pdf

下载
日本人白平衡AWB算法
日本人白平衡AWB算法

下载
磁力计校准
磁力计校准

博客
MySQL 8.0.19安装教程(windows 64位)
MySQL 8.0.19安装教程(windows 64位)

学院
Python+OpenCV计算机视觉
Python+OpenCV计算机视觉

学院
Vue.js 2.0之全家桶系列视频课程
Vue.js 2.0之全家桶系列视频课程

下载
navicat（内含激活码）
navicat（内含激活码）

下载
HTML期末大作业
HTML期末大作业

学院
150讲轻松搞定Python网络爬虫
150讲轻松搞定Python网络爬虫

学院
三个项目玩转深度学习（附1G源码）
三个项目玩转深度学习（附1G源码）

下载
基于STM32的电子时钟设计
基于STM32的电子时钟设计

下载
学生成绩管理系统（PHP + MYSQL）
学生成绩管理系统（PHP + MYSQL）

博客
面试了一个 31 岁程序员，让我有所触动，30岁以上的程序员该何去何从？
面试了一个 31 岁程序员，让我有所触动，30岁以上的程序员该何去何从？

学院
程序员的兼职技能课
程序员的兼职技能课

学院
实用主义学Python（小白也容易上手的Python实用案例）
实用主义学Python（小白也容易上手的Python实用案例）

学院
Java8零基础入门视频教程
Java8零基础入门视频教程

学院
Python数据挖掘简易入门
Python数据挖掘简易入门

学院
零基础学C#编程—C#从小白到大咖
零基础学C#编程—C#从小白到大咖