• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

PINTO0309/MobileNet-SSD-RealSense: [High Performance / MAX 30 FPS] RaspberryPi3( ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

PINTO0309/MobileNet-SSD-RealSense

开源软件地址(OpenSource Url):

https://github.com/PINTO0309/MobileNet-SSD-RealSense

开源编程语言(OpenSource Language):

Python 100.0%

开源软件介绍(OpenSource Introduction):

MobileNet-SSD-RealSense

RaspberryPi3(Raspbian Stretch) or Ubuntu16.04/UbuntuMate + Neural Compute Stick(NCS/NCS2) + RealSense D435(or USB Camera or PiCamera) + MobileNet-SSD(MobileNetSSD)

【Notice】December 19, 2018 OpenVINO has supported RaspberryPi + NCS2 !!
https://software.intel.com/en-us/articles/OpenVINO-RelNotes#inpage-nav-2-2

【Dec 31, 2018】 USB Camera + MultiStick + MultiProcess mode correspondence with NCS2 is completed.
【Jan 04, 2019】 Tune performance four times. MultiStickSSDwithRealSense_OpenVINO_NCS2.py. Core i7 -> NCS2 x1, 48 FPS
【Nov 12, 2019】 Compatible with OpenVINO 2019 R3 + RaspberryPi3/4 + Raspbian Buster.


Measure the distance to the object with RealSense D435 while performing object detection by MobileNet-SSD(MobileNetSSD) with RaspberryPi 3 boosted with Intel Movidius Neural Compute Stick.
"USB Camera mode / PiCamera mode" can not measure the distance, but it operates at high speed.
And, This is support for MultiGraph and FaceDetection, MultiProcessing, Background transparentation.
And, This is support for simple clustering function. (To prevent thermal runaway)

My blog

【Japanese Article1】
RaspberryPi3 (Raspbian Stretch) + Intel Movidius Neural Compute Stick(NCS) + RealSenseD435 + MobileNet-SSD(MobileNetSSD) で高速に物体検出しつつ悟空やモニタまでの距離を測る

【Japanese / English Article2】
Intel also praised me again ヽ(゚∀゚)ノ Yeah MobileNet-SSD(MobileNetSSD) object detection and RealSense distance measurement (640x480) with RaspberryPi3 At least 25FPS playback frame rate + 12FPS prediction rate

【Japanese / English Article3】
Detection rate approx. 30FPS RaspberryPi3 Model B(plus none) is slightly later than TX2, acquires object detection rate of MobilenetSSD and corresponds to MultiModel VOC+WIDER FACE

【Japanese Article4】
RaspberryPi3で複数のMovidius Neural Compute Stick をシームレスにクラスタ切り替えして高速推論性能を維持しつつ熱暴走(内部温度70℃前後)を回避する

【Japanese Article5】
Caffeで超軽量な "Semantic Segmentation" のモデルを生成する Sparse-Quantized CNN 512x1024_10MB_軽量モデル_その1

【Japanese / English Article6】
Boost RaspberryPi3 with Neural Compute Stick 2 (1 x NCS2) and feel the explosion performance of MobileNet-SSD (If it is Core i7, 21 FPS)

【Japanese / English Article7】
[24 FPS] Boost RaspberryPi3 with four Neural Compute Stick 2 (NCS2) MobileNet-SSD / YoloV3 [48 FPS for Core i7]

【Japanese / English Article8】
[24 FPS, 48 FPS] RaspberryPi3 + Neural Compute Stick 2, The day when the true power of one NCS2 was drawn out and "Goku" became a true "super saiya-jin"


Table of contents

1. Summary
 1.1 Verification environment NCSDK (1)
 1.2 Result of detection rate NCSDK (1)
 1.3 Verification environment NCSDK (2)
 1.4 Result of detection rate NCSDK (2)
2. Performance comparison as a mobile application (Based on sensory comparison)
3. Change history
4. Motion image
 4-1. NCSDK ver
  4-1-1. RealSense Mode about 6.5 FPS (Synchronous screen drawing)
  4-1-2. RealSense Mode about 25.0 FPS (Asynchronous screen drawing)
  4-1-3. USB Camera Mode MultiStick x4 Boosted 16.0 FPS+ (Asynchronous screen drawing)
  4-1-4. RealSense Mode SingleStick about 5.0 FPS(Transparent background / Asynchronous screen drawing
  4-1-5. USB Camera Mode MultiStick x3 Boosted (Asynchronous screen drawing / MultiGraph
  4-1-6. Simple clustering function (MultiStick / MultiCluster / Cluster switch cycle / Cluster switch temperature)
 4-2. OpenVINO ver
  4-2-1. USB Camera Mode NCS2 x 1 Stick + RaspberryPi3(Synchronous screen drawing)
  4-2-2. USB Camera Mode NCS2 x 1 Stick + Core i7(Synchronous screen drawing)
  4-2-3. USB Camera Mode NCS2 x 1 Stick + Core i7(Asynchronous screen drawing)
  4-2-4. USB Camera Mode NCS2 x 1 Stick + RaspberryPi3(Asynchronous screen drawing)
  4-2-5. USB Camera Mode NCS2 x 1 Stick + LattePanda Alpha(Asynchronous screen drawing)48 FPS
  4-2-6. PiCamera Mode NCS2 x 1 Stick + RaspberryPi3(Asynchronous screen drawing)
  4-2-7. USB Camera Mode NCS2 x 1 Stick + RaspberryPi4(Asynchronous screen drawing)40 FPS
5. Motion diagram of MultiStick
6. Environment
7. Firmware update with Windows 10 PC
8. Work with RaspberryPi3 (or PC + Ubuntu16.04 / RaspberryPi + Ubuntu Mate)
 8-1. NCSDK ver (Not compatible with NCS2)
 8-2. OpenVINO ver (Corresponds to NCS2)
9. Execute the program
10. 【Reference】 MobileNetv2 Model (Caffe) Great Thanks!!
11. Conversion method from Caffe model to NCS model (NCSDK)
12. Conversion method from Caffe model to NCS model (OpenVINO)
13. Construction of learning environment and simple test for model (Ubuntu16.04 x86_64 PC + GPU NVIDIA Geforce)
14. Reference articles, thanks

Summary

Performance measurement result each number of sticks. (It is Detection rate. It is not a Playback rate.)
The best performance can be obtained with QVGA + 5 Sticks.
However, It is important to use a good quality USB camera.

Verification environment (1)

No. Item Contents
1 Video device USB Camera (No RealSense D435) ELP-USB8MP02G-L75 $70
2 Auxiliary equipment (Required) self-powered USB2.0 HUB
3 Input resolution 640x480
4 Output resolution 640x480
5 Execution parameters $ python3 MultiStickSSDwithRealSense.py -mod 1 -wd 640 -ht 480

Result of detection rate (1)

No. Stick count FPS Youtube Movie Note
1 1 Stick 6 FPS https://youtu.be/lNbhutT8hkA base line
2 2 Sticks 12 FPS https://youtu.be/zuJOhKWoLwc 6 FPS increase
3 3 Sticks 16.5 FPS https://youtu.be/8UDFIJ1Z4v8 4.5 FPS increase
4 4 Sticks 16.5 FPS https://youtu.be/_2xIZ-IZwZc No improvement

Verification environment (2)

No. Item Contents
1 Video device USB Camera (No RealSense D435) PlayStationEye $5
2 Auxiliary equipment (Required) self-powered USB2.0 HUB
3 Input resolution 320x240
4 Output resolution 320x240
5 Execution parameters $ python3 MultiStickSSDwithRealSense.py -mod 1 -wd 320 -ht 240

Result of detection rate (2)

No. Stick count FPS Youtube Movie Note
1 4 Sticks   25 FPS https://youtu.be/v-Cei1TW88c
2 5 Sticks 30 FPS https://youtu.be/CL6PTNgWibI best performance

Performance comparison as a mobile application (Based on sensory comparison)

◯=HIGH, △=MEDIUM, ×=LOW

No. Model Speed Accuracy Adaptive distance
1 SSD × ALL
2 MobileNet-SSD Short distance
3 YoloV3 × ALL
4 tiny-YoloV3 × Long distance

Change history

Change history
[July 14, 2018] Corresponds to NCSDK v2.05.00.02
[July 17, 2018] Corresponds to OpenCV 3.4.2
[July 21, 2018] Support for multiprocessing [MultiStickSSDwithRealSense.py]
[July 23, 2018] Support for USB Camera Mode [MultiStickSSDwithRealSense.py]
[July 29, 2018] Added steps to build learning environment
[Aug 3, 2018] Background Multi-transparent mode implementation [MultiStickSSDwithRealSense.py]
[Aug 11, 2018] CUDA9.0 + cuDNN7.2 compatible with environment construction procedure
[Aug 14, 2018] Reference of MobileNetv2 Model added to README and added Facedetection Model
[Aug 15, 2018] Bug Fixed. `MultiStickSSDwithRealSense.py` depth_scale be undefined. Pull Requests merged. Thank you Drunkar!!
[Aug 19, 2018] 【Experimental】 Update Facedetection model [DeepFace] (graph.facedetectXX)
[Aug 22, 2018] Separate environment construction procedure of "Raspbian Stretch" and "Ubuntu16.04"
[Aug 22, 2018] 【Experimental】 FaceDetection model replaced [resnet] (graph.facedetection)
[Aug 23, 2018] Added steps to build NCSDKv2
[Aug 25, 2018] Added "Detection FPS View" [MultiStickSSDwithRealSense.py]
[Sep 01, 2018] FaceDetection model replaced [Mobilenet] (graph.fullfacedetection / graph.shortfacedetection)
[Sep 01, 2018] Added support for MultiGraph and FaceDetection mode [MultiStickSSDwithRealSense.py]
[Sep 04, 2018] Performance measurement result with 5 sticks is posted
[Sep 08, 2018] To prevent thermal runaway, simple clustering function of stick was implemented.
[Sep 16, 2018] 【Experimental】 Added Semantic Segmentation model [Tensorflow-UNet] (semanticsegmentation_frozen_person.pb)
[Sep 20, 2018] 【Experimental】 Updated Semantic Segmentation model [Tensorflow-UNet]
[Oct 07, 2018] 【Experimental】 Added Semantic Segmentation model [caffe-jacinto] (cityscapes5_jsegnet21v2_iter_60000.caffemodel)
[Oct 10, 2018] Corresponds to NCSDK 2.08.01
[Oct 12, 2018] 【Experimental】 Added Semantic Segmentation model [Tensorflow-ENet] (semanticsegmentation_enet.pb) https://github.com/PINTO0309/TensorFlow-ENet.git
[Dec 22, 2018] Only "USB Camera + single thread mode" correspondence with NCS 2 is completed
[Dec 31, 2018] "USB Camera + MultiStick + MultiProcess mode" correspondence with NCS2 is completed
[Jan 04, 2019] Tune performance four times. MultiStickSSDwithRealSense_OpenVINO_NCS2.py
[Feb 01, 2019] Pull request merged. Fix Typo. Thanks, nguyen-alexa!!
[Feb 09, 2019] Corresponds to PiCamera.
[Feb 10, 2019] Added support for SingleStickSSDwithRealSense_OpenVINO_NCS2.py
[Feb 10, 2019] Firmware v5.9.13 -> v5.10.6, RealSenseSDK v2.13.0 -> v2.16.5
[May 01, 2019] Corresponds to OpenVINO 2019 R1.0.1
[Nov 12, 2019] Corresponds to OpenVINO 2019 R3.0


Motion image

RealSense Mode about 6.5 FPS (Detection + Synchronous screen drawing / SingleStickSSDwithRealSense.py)

【YouTube Movie】 https://youtu.be/77cV9fyqJ1w

03 04

RealSense Mode about 25.0 FPS (Asynchronous screen drawing / MultiStickSSDwithRealSense.py)

However, the prediction rate is fairly low.(about 6.5 FPS)
【YouTube Movie】 https://youtu.be/tAf1u9DKkh4

09

USB Camera Mode MultiStick x4 Boosted 16.0 FPS+ (Asynchronous screen drawing / MultiStickSSDwithRealSense.py)

【YouTube Movie】 https://youtu.be/GedDpAc0JyQ

10 11

RealSense Mode SingleStick about 5.0 FPS(Transparent background in real time / Asynchronous screen drawing / MultiStickSSDwithRealSense.py)

【YouTube Movie】 https://youtu.be/ApyX-mN_dYA

12

USB Camera Mode MultiStick x3 Boosted (Asynchronous screen drawing / MultiGraph(SSD+FaceDetection) / FaceDetection / MultiStickSSDwithRealSense.py)

【YouTube Movie】 https://youtu.be/fQZpuD8mWok

13

Simple clustering function (MultiStick / MultiCluster / Cluster switch cycle / Cluster switch temperature)

14
[Execution log]
15

USB Camera Mode NCS2 SingleStick + RaspberryPi3(Synchronous screen drawing / SingleStickSSDwithUSBCamera_OpenVINO_NCS2.py)

【YouTube Movie】 https://youtu.be/GJNkX-ZBuC8

16

USB Camera Mode NCS2 SingleStick + Core i7(Synchronous screen drawing / SingleStickSSDwithUSBCamera_OpenVINO_NCS2.py)

【YouTube Movie】 https://youtu.be/1ogge90EuqI

17

USB Camera Mode NCS2 x 1 Stick + Core i7(Asynchronous screen drawing / MultiStickSSDwithRealSense_OpenVINO_NCS2.py)

【YouTube Movie】 https://youtu.be/Nx_rVDgT8uY

$ python3 MultiStickSSDwithRealSense_OpenVINO_NCS2.py -mod 1 -numncs 1

23

USB Camera Mode NCS2 x 1 Stick + RaspberryPi3(Asynchronous screen drawing / MultiStickSSDwithRealSense_OpenVINO_NCS2.py)

【YouTube Movie】 https://youtu.be/Xj2rw_5GwlI

$ python3 MultiStickSSDwithRealSense_OpenVINO_NCS2.py -mod 1 -numncs 1

24

USB Camera Mode NCS2 x 1 Stick + LattePanda Alpha(Asynchronous screen drawing / MultiStickSSDwithRealSense_OpenVINO_NCS2.py)[48 FPS]

https://twitter.com/PINTO03091/status/1081575747314057219

PiCamera Mode NCS2 x 1 Stick + RaspberryPi3(Asynchronous screen drawing / MultiStickSSDwithPiCamera_OpenVINO_NCS2.py)

$ python3 MultiStickSSDwithPiCamera_OpenVINO_NCS2.py

25

USB Camera Mode NCS2 x 1 Stick + RaspberryPi4(Asynchronous screen drawing / MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py)

$ python3 MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py

26


Motion diagram of MultiStick

20

Environment

1.RaspberryPi3 + Raspbian Stretch (USB2.0 Port) or RaspberryPi3 + Ubuntu Mate or PC + Ubuntu16.04
2.Intel RealSense D435 (Firmware Ver 5.10.6) or USB Camera or PiCamera Official stable version firmware
3.Intel Neural Compute Stick v1/v2 x1piece or more
4-1.OpenCV 3.4.2 (NCSDK)
4-2.OpenCV 4.1.1-openvino (OpenVINO)
5.VFPV3 or TBB (Intel Threading Building Blocks)
6.Numpy
7.Python3.5
8.NCSDK v2.08.01 (It does not work with NCSDK v1. v1 version is here)
9. OpenVINO 2019 R2.0.1
10.RealSenseSDK v2.16.5 (The latest version is unstable) Official stable version SDK
11.HDMI Display

Firmware update with Windows 10 PC

1.ZIP 2 types (1) Firmware update tool for Windows 10 (2) The latest firmware bin file Download and decompress
2.Copy Signed_Image_UVC_5_10_6_0.bin to the same folder as intel-realsense-dfu.exe
3.Connect RealSense D435 to USB port
4.Wait for completion of installation of device driver
5.Execute intel-realsense-dfu.exe
6.「1」 Type and press Enter and follow the instructions on the screen to update
01
7.Firmware version check 「2」
02

Work with RaspberryPi3 (or PC + Ubuntu16.04 / RaspberryPi + Ubuntu Mate)

1.NCSDK ver (Not compatible with NCS2)

Use of Virtualbox is not strongly recommended
[Note] Japanese Article
https://qiita.com/akitooo/items/6aee8c68cefd46d2a5dc
https://qiita.com/kikuchi_kentaro/items/280ac68ad24759b4c091

[Post of Official Forum]
https://ncsforum.movidius.com/discussion/950/problems-with-python-multiprocessing-using-sdk-2-0-0-4
https://ncsforum.movidius.com/discussion/comment/3921
https://ncsforum.movidius.com/discussion/comment/4316/#Comment_4316

1.Execute the following

$ sudo apt update;sudo apt upgrade
$ sudo reboot

2.Extend the SWAP area (RaspberryPi+Raspbian Stretch / RaspberryPi+Ubuntu Mate Only)

$ sudo nano /etc/dphys-swapfile
CONF_SWAPSIZE=2048

$ sudo /etc/init.d/dphys-swapfile restart;swapon -s

3.Install NSCDK

$ sudo apt install python-pip python3-pip
$ sudo pip3 install --upgrade pip
$ sudo pip2 install --upgrade pip

$ cd ~/ncsdk
$ make uninstall
$ cd ~;rm -r -f ncsdk
#=====================================================================================================
# [Oct 10, 2018] NCSDK 2.08.01 , Tensorflow 1.9.0
$ git clone -b ncsdk2 http://github.com/Movidius/ncsdk
#=====================================================================================================
$ cd ncsdk
$ nano ncsdk.conf

#MAKE_NJOBS=1
↓
MAKE_NJOBS=1

$ sudo apt install cython
$ sudo -H pip3 install cython
$ sudo -H pip3 install numpy
$ sudo -H pip3 install pillow
$ make install

$ cd ~
$ wget https://github.com/google/protobuf/releases/download/v3.5.1/protobuf-all-3.5.1.tar.gz
$ tar -zxvf protobuf-all-3.5.1.tar.gz
$ cd protobuf-3.5.1
$ ./configure
$ sudo make -j1
$ sudo make install
$ cd python
$ export LD_LIBRARY_PATH=../src/.libs
$ python3 setup.py build --cpp_implementation 
$ python3 setup.py test --cpp_implementation
$ sudo python3 setup.py install --cpp_implementation
$ sudo ldconfig
$ protoc --version

# Before executing "make examples", insert Neural Compute Stick into the USB port of the device.
$ cd ~/ncsdk
$ make examples -j1

【Reference】https://github.com/movidius/ncsdk

4.Update udev rule

$ sudo apt install -y git libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
$ sudo apt install -y libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev

$ cd /etc/udev/rules.d/
$ sudo wget https://raw.githubusercontent.com/IntelRealSense/librealsense/master/config/99-realsense-libusb.rules
$ sudo udevadm control --reload-rules && udevadm trigger

5.Upgrade to "cmake 3.11.4"

$ cd ~
$ wget https://cmake.org/files/v3.11/cmake-3.11.4.tar.gz
$ tar -zxvf cmake-3.11.4.tar.gz;rm cmake-3.11.4.tar.gz
$ cd cmake-3.11.4
$ ./configure --prefix=/home/pi/cmake-3.11.4
$ make -j1
$ sudo make install
$ export PATH=/home/pi/cmake-3.11.4/bin:$PATH
$ source ~/.bashrc
$ cmake --version
cmake version 3.11.4

6.Register LD_LIBRARY_PATH

$ nano ~/.bashrc
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

$ source ~/.bashrc

7.Install TBB (Intel Threading Building Blocks)

$ cd ~
$ wget https://github.com/PINTO0309/TBBonARMv7/raw/master/libtbb-dev_2018U2_armhf.deb
$ sudo dpkg -i ~/libtbb-dev_2018U2_armhf.deb
$ sudo ldconfig

8.Uninstall old OpenCV (RaspberryPi Only)
[Very Important] The highest performance can not be obtained unless VFPV3 is enabled.

$ cd ~/opencv-3.x.x/build
$ sudo make uninstall
$ cd ~
$ rm -r -f opencv-3.x.x
$ rm -r -f opencv_contrib-3.x.x

9.Build install "OpenCV 3.4.2" or Install by deb package.
[Very Important] The highest performance can not be obtained unless VFPV3 is enabled.

9.1 Build Install (RaspberryPi Only)

$ sudo apt update && sudo apt upgrade
$ sudo apt install build-essential cmake pkg-config libjpeg-dev libtiff5-dev \
libjasper-dev libavcodec-dev libavformat-dev libswscale-dev \
libv4l-dev libxvidcore-dev libx264-dev libgtk2.0-dev libgtk-3-dev \
libcanberra-gtk* libatlas-base-dev gfortran python2.7-dev python3-dev

$ cd ~
$ wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.4.2.zip
$ unzip opencv.zip;rm opencv.zip
$ wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.4.2.zip
$ unzip opencv_contrib.zip;rm opencv_contrib.zip
$ cd ~/opencv-3.4.2/;mkdir build;cd build
$ cmake -D CMAKE_CXX_FLAGS="-DTBB_USE_GCC_BUILTINS=1 -D__TBB_64BIT_ATOMICS=0" \
        -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D INSTALL_PYTHON_EXAMPLES=OFF \
        -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.4.2/modules \
        -D BUILD_EXAMPLES=OFF \
        -D PYTHON_DEFAULT_EXECUTABLE=$(which python3) \
        -D INSTALL_PYTHON_EXAMPLES=OFF \
        -D BUILD_opencv_python2=ON \
        -D BUILD_opencv_python3=ON \
        -D WITH_OPENCL=OFF \
        -D WITH_OPENGL=ON \
        -D WITH_TBB=ON \
        -D BUILD_TBB=OFF \
        -D WITH_CUDA=OFF \
        -D ENABLE_NEON:BOOL=ON \
        -D ENABLE_VFPV3=ON \
        -D WITH_QT=OFF \
        -D BUILD_TESTS=OFF ..
$ make -j1
$ sudo make install
$ sudo ldconfig

9.2 Install by deb package (RaspberryPi Only) [I already activated VFPV3 and built it]

$ cd ~
$ sudo apt autoremove libopencv3
$ wget https://github.com/PINTO0309/OpenCVonARMv7/raw/master/libopencv3_3.4.2-20180709.1_armhf.deb
$ sudo apt install -y ./libopencv3_3.4.2-20180709.1_armhf.deb
$ sudo ldconfig

10.Install Intel® RealSense™ SDK 2.0


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
hahwul/MobileHackersWeapons: Mobile Hacker's Weapons / A collection of cool ...发布时间:2022-08-30
下一篇:
Alfresco/alfresco-android-app发布时间:2022-08-30
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap