Running Dreamview (Apollo 3.0) for pre-Haswell processors

Problem overview

The Apollo docker includes a version of PCL which was built with the FMA instruction set (fused multiply-add). For any Intel processor older than Haswell (and perhaps even some Broadwell processors), this causes an illegal instruction and prevents Dreaview from starting. In previous version of Apollo, this was difficult to problem solve since no error details were provided – in fact, Dreamview would be reported to be running but localhost:8888 could not be accessed. Since 3.0, however, Dreamview reports the following:
dreamview: ERROR (spawn error)
Note: this can probably be caused by many other issues. However, if you are running Haswell or earlier, the solution provided here is likely to work. You can use gdb to troubleshoot spawn errors.

The solution

You have to build PCL within the docker to overcome this problem.

cd /apollo
git clone https://github.com/PointCloudLibrary/pcl.git
git checkout -b pcl-custom

Now you need to add the following to /apollo/pcl/CMakeLists.txt:
– Below the line set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "possible configurations" FORCE), add the following:

if (CMAKE_VERSION VERSION_LESS "3.1")
    set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
    message("Build with c++11 support")
else ()
    set (CMAKE_CXX_STANDARD 11)
endif ()

Build PCL with the default options.

cd /apollo/pcl
mkdir build
cd build
cmake ..
make

Replace the current PCL libraries with the new ones you just made (you can backup first).

sudo mkdir -p /usr/local/lib/pcl.backup
sudo mv /usr/local/lib/libpcl* /usr/local/lib/pcl.backup
sudo cp -a lib/* /usr/local/lib/
sudo ldconfig

Now you have to re-build Apollo.

cd /apollo
./apollo.sh clean
./apollo.sh build_gpu

And hopfully you can now start and access Dreamview with the usual ./scripts/bootstrap.sh.

Apollo messages

This post provides an overview of the messages used by Apollo, as captured from the demo rosbag for testing the perception module.

To do:

  • Investigate protobuf message definitions for variable types
  • Investiage contents of standard ROS type messages

Standard message types

sensor_msgs/Image
sensor_msgs/PointCloud2
tf2_msgs/TFMessage

These messages use the standard ROS message format for their respective types.

Custom message types

Apollo uses a custom version of ROS which replaces the msg message description language with a Real Time Publish Subscribe and Google protobuf messaging protocol for publishing and receiving the following messages.

pb_msgs/Chassis
pb_msgs/ContiRadar
pb_msgs/EpochObservation
pb_msgs/GnssBestPose
pb_msgs/GnssEphemeris
pb_msgs/GnssStatus
pb_msgs/Gps
pb_msgs/Imu
pb_msgs/InsStat
pb_msgs/LocalizationEstimate

Messages using these types cannot be interpreted using rosbag tools as only the header is compatible. However, their content can be dumped as text using read_messages().

Message topics

The demo rosbag contains the following topics:

topics: /apollo/canbus/chassis 5851 msgs : pb_msgs/Chassis
/apollo/localization/pose 5856 msgs : pb_msgs/LocalizationEstimate
/apollo/sensor/camera/traffic/image_long 471 msgs : sensor_msgs/Image
/apollo/sensor/camera/traffic/image_short 469 msgs : sensor_msgs/Image
/apollo/sensor/conti_radar 789 msgs : pb_msgs/ContiRadar
/apollo/sensor/gnss/best_pose 59 msgs : pb_msgs/GnssBestPose
/apollo/sensor/gnss/corrected_imu 5838 msgs : pb_msgs/Imu
/apollo/sensor/gnss/gnss_status 59 msgs : pb_msgs/GnssStatus
/apollo/sensor/gnss/imu 11630 msgs : pb_msgs/Imu
/apollo/sensor/gnss/ins_stat 118 msgs : pb_msgs/InsStat
/apollo/sensor/gnss/odometry 5848 msgs : pb_msgs/Gps
/apollo/sensor/gnss/rtk_eph 49 msgs : pb_msgs/GnssEphemeris
/apollo/sensor/gnss/rtk_obs 352 msgs : pb_msgs/EpochObservation
/apollo/sensor/velodyne64/compensator/PointCloud2 587 msgs : sensor_msgs/PointCloud2
/tf 11740 msgs : tf2_msgs/TFMessage
/tf_static 1 msg : tf2_msgs/TFMessage

Example messages

All custom messages were read using the following script, within the Apollo docker:

import rosbag
import std_msgs

from std_msgs.msg import String

bag = rosbag.Bag('../apollo_data/2018-01-03-19-37-16.bag', 'r')

read_topic = '/apollo/canbus/chassis'

counter = 0

for topic, msg, t in bag.read_messages():
    if topic == read_topic:
        out_file = 'messages/' + str(counter) + '.txt'
        f = file(out_file, 'w')
        f.write(str(msg))
        f.close()
        counter = counter + 1

Note that this will not work outside of the Apollo docker – the files will be empty.

pb_msgs/Chassis

Of custom protobuf type pb_msgs/Chassis.

engine_started: true
engine_rpm: 0.0
speed_mps: 0.261111110449
odometer_m: 0.0
fuel_range_m: 0
throttle_percentage: 14.9950408936
brake_percentage: 20.5966281891
steering_percentage: 4.36170196533
steering_torque_nm: -0.75
parking_brake: false
driving_mode: COMPLETE_AUTO_DRIVE
error_code: NO_ERROR
gear_location: GEAR_DRIVE
header {
    timestamp_sec: 1513807824.58
    module_name: "chassis"
    sequence_num: 96620
}
signal {
    turn_signal: TURN_NONE
    horn: false
}
chassis_gps {
    latitude: 37.416912
    longitude: -122.016053333
    gps_valid: true
    year: 17
    month: 12
    day: 20
    hours: 22
    minutes: 10
    seconds: 23
    compass_direction: 270.0
    pdop: 0.8
    is_gps_fault: false
    is_inferred: false
    altitude: -42.5
    heading: 285.75
    hdop: 0.4
    vdop: 0.6
    quality: FIX_3D
    num_satellites: 20
    gps_speed: 0.89408
}

/apollo/localization/pose

Of custom protobuf type pb_msgs/LocalizationEstimate.

header {
timestamp_sec: 1513807826.05
module_name: "localization"
sequence_num: 96460
}
pose {
    position {
    x: 587068.814494
    y: 4141577.34872
    z: -31.0619329279
    }
    orientation {
        qx: 0.0354450020653
        qy: 0.0137914670665
        qz: -0.608585615062
        qw: -0.792576177036
    }
    linear_velocity {
        x: -1.08479686092
        y: 0.30964034124
        z: -0.00187507107555
    }
    linear_acceleration {
        x: -2.10530393576
        y: 0.553837321635
        z: 0.170289232445
    }
    angular_velocity {
        x: 0.00630864843147
        y: 0.0111583669994
        z: 0.0146261464379
    }
    heading: 2.88124066533
    linear_acceleration_vrf {
        x: -0.0137881469155
        y: 2.15869303259
        z: 0.328470914294
    }
    angular_velocity_vrf {
        x: 0.0120972352232
        y: -0.00428235807315
        z: 0.0146133729179
    }
    euler_angles {
        x: 0.0213395674976
        y: 0.0730372234802
        z: -3.40194464185
    }
}
measurement_time: 1513807826.04

/sensor/camera/traffic/image_long

This is a standard sensor_msgs/Image message.

/apollo/sensor/camera/traffic/image_short

This is a standard sensor_msgs/Image message.

/apollo/sensor/conti_radar

Of custom protobuf type pb_msgs/ContiRadar.

header {
timestamp_sec: 1513807824.52
module_name: "conti_radar"
sequence_num: 12971
}
contiobs {
header {
timestamp_sec: 1513807824.52
module_name: "conti_radar"
sequence_num: 12971
}
clusterortrack: false
obstacle_id: 0
longitude_dist: 107.6
lateral_dist: -17.2
longitude_vel: -0.25
lateral_vel: -0.75
rcs: 7.5
dynprop: 1
longitude_dist_rms: 0.371
lateral_dist_rms: 0.616
longitude_vel_rms: 0.371
lateral_vel_rms: 0.616
probexist: 1.0
meas_state: 2
longitude_accel: 0.23
lateral_accel: 0.0
oritation_angle: 0.0
longitude_accel_rms: 0.794
lateral_accel_rms: 0.005
oritation_angle_rms: 1.909
length: 2.8
width: 2.4
obstacle_class: 1
}`
...[99 more `contiobs`]...
`object_list_status {
nof_objects: 100
meas_counter: 8464
interface_version: 0
}`

<h3>/apollo/sensor/gnss/best_pose</h3>
Of custom protobuf type `pb_msgs/GnssBestPose`.

`header {
timestamp_sec: 1513807825.02
}
measurement_time: 1197843043.0
sol_status: SOL_COMPUTED
sol_type: NARROW_INT
latitude: 37.4169108497
longitude: -122.016059063
height_msl: 2.09512365051
undulation: -32.0999984741
datum_id: WGS84
latitude_std_dev: 0.0114300707355
longitude_std_dev: 0.00970683153719
height_std_dev: 0.0248824004084
base_station_id: "0"
differential_age: 2.0
solution_age: 0.0
num_sats_tracked: 13
num_sats_in_solution: 13
num_sats_l1: 13
num_sats_multi: 11
extended_solution_status: 33
galileo_beidou_used_mask: 0
gps_glonass_used_mask: 51

/apollo/sensor/gnss/corrected_imu

Of custom protobuf type pb_msgs/Imu.

This appears to be empty, with just a timestamp header.

/apollo/sensor/gnss/gnss_status

Of custom protobuf type pb_msgs/GnssStatus.

header {
timestamp_sec: 1513807826.02
}
solution_completed: true
solution_status: 0
position_type: 50
num_sats: 14

/apollo/sensor/gnss/imu

Of custom protobuf type pb_msgs/Imu.

header {
timestamp_sec: 1513807824.58
}
measurement_time: 1197843042.56
measurement_span: 0.00499999988824
linear_acceleration {
x: -0.172816216946
y: -0.864528119564
z: 9.75685194135
}
angular_velocity {
x: -0.000550057197804
y: 0.00203638196634
z: 0.00155888550527
}

/apollo/sensor/gnss/ins_stat

Of custom protobuf type pb_msgs/InsStat.

header {
timestamp_sec: 1513807826.0
}
ins_status: 3
pos_type: 56

/apollo/sensor/gnss/odometry

Of custom protobuf type pb_msgs/Gps.

header {
timestamp_sec: 1513807824.58
}
localization {
position {
x: 587069.287353
y: 4141577.22403
z: -31.0546750054
}
orientation {
qx: 0.0399296504032
qy: 0.0164343412444
qz: -0.606054063971
qw: -0.79425059458
}
linear_velocity {
x: -0.231635539107
y: 0.0795322332148
z: 0.00147877897123
}
}

/apollo/sensor/gnss/rtk_eph

Of custom protobuf type pb_msgs/GnssEphemeris.

gnss_type: GPS_SYS
keppler_orbit {
gnss_type: GPS_SYS
sat_prn: 23
gnss_time_type: GPS_TIME
week_num: 1980
af0: -0.000219020526856
af1: 0.0
af2: 0.0
iode: 43.0
deltan: 5.23450375238e-09
m0: -1.97123659751
e: 0.0118623136077
roota: 5153.61836624
toe: 345600.0
toc: 345600.0
cic: 1.30385160446e-07
crc: 266.5
cis: -8.56816768646e-08
crs: -9.28125
cuc: -7.13393092155e-07
cus: 5.16884028912e-06
omega0: 2.20992318653
omega: -2.40973031377
i0: 0.943533454733
omegadot: -8.16355433052e-09
idot: -2.10723063176e-11
accuracy: 2
health: 0
tgd: -2.04890966415e-08
iodc: 43.0
}
glonass_orbit {
gnss_type: GLO_SYS
slot_prn: 10
gnss_time_type: GLO_TIME
toe: 339318.0
frequency_no: -7
week_num: 1980
week_second_s: 339318.0
tk: 3990.0
clock_offset: 1.41123309731e-05
clock_drift: 9.09494701773e-13
health: 0
position_x: -16965363.2812
position_y: -15829665.5273
position_z: -10698784.1797
velocity_x: -994.89402771
velocity_y: -1087.91637421
velocity_z: 3184.79061127
accelerate_x: -2.79396772385e-06
accelerate_y: -3.72529029846e-06
accelerate_z: -1.86264514923e-06
infor_age: 0.0
}

/apollo/sensor/gnss/rtk_obs

Of custom protobuf type pb_msgs/EpochObservation.

receiver_id: 0
gnss_time_type: GPS_TIME
gnss_week: 1980
gnss_second_s: 339042.6
sat_obs_num: 13
sat_obs {
sat_prn: 31
sat_sys: GPS_SYS
band_obs_num: 2
band_obs {
band_id: GPS_L1
frequency_value: 0.0
pseudo_type: CORSE_CODE
pseudo_range: 22104170.8773
carrier_phase: 116158209.251
loss_lock_index: 173
doppler: -2880.74853516
snr: 173.0
}
band_obs {
band_id: GPS_L2
frequency_value: 0.0
pseudo_type: CORSE_CODE
pseudo_range: 22104174.5591
carrier_phase: 90512897.445
loss_lock_index: 140
doppler: -2244.73510742
snr: 140.0
}
}

…[13 more sat_obs]…

/apollo/sensor/velodyne64/compensator/PointCloud2

This is a standard sensor_msgs/PointCloud2 message.

/tf

This is a standard tf2_msgs/TFMessage message. The messages alternate between frame_id: world to child_frame_id: localization transform messages and frame_id: world to child_frame_id: novatel transform messages.

/tf_static

This is a standard tf2_msgs/TFMessage message, for the frame_id: novatel to velodyne64 transform.

Running Apollo 2.0 – GPU

This is a brief guide to getting Apollo 2.0 up and running. It is based on the Apollo README with additional setup for the Perception modules.

Prerequisites

  • Ubuntu 16.04 (also works on 14.04).
  • Nvidia GPU. Install the drivers as described here. You don’t need CUDA installed (it’s included in the Apollo docker). On 16.04 you will need a new-ish version – the below is tested using 390.25. The Apollo recommended 275.39 will not work on 16.04,┬ábut will work on 14.04. However, as this requires a newer GCC version that breaks the build system, it is much easier to go straight to the 390.25 driver.

Download code and Docker image

  1. Get the code:
    git clone https://github.com/ApolloAuto/apollo.git
  2. If you don’t have Docker already:
    ./apollo/docker/scripts/install_docker.sh
  3. Then log out and log back in again.
  4. Pull the docker image. The dev_start.sh script downloads the docker image (or updates it if already downloaded) and starts the container.
    cd apollo/
    ./docker/scripts/dev_start.sh

Install Nvidia graphics drivers in the Docker image

  1. Check which driver you are using (in host) with nvidia-smi.
  2. First off we need to enter the container with root priveledges so we can install the matching graphics drivers.
    docker exec -it apollo_dev /bin/bash
    wget http://us.download.nvidia.com/XFree86/Linux-x86_64/***.**/NVIDIA-Linux-x86_64-***.**.run

    where ***.** is the driver version running on your host system.
    Note: Disregard the Apollo instructions to upgrade to GCC 4.9. Not only is it not necessary with newer versions of the Nvidia drivers, but it will make the build fail. Stick with the GCC version of 4.8.4 which comes in the Docker image.
  3. Now install the drivers:
    chmod +x NVIDIA-Linux-x86_64-***.**.run
    ./NVIDIA-Linux-x86_64-***.**.run -a --skip-module-unload --no-kernel-module --no-opengl-files

    Hit ‘enter’ to go with the default choices where prompted. Once done, check that the driver is working with nvidia-smi.
  4. To create a new image with your changes, check what the container ID of your image is (on the host):
    docker ps -l
  5. Use the resulting container ID with the following command to create a new image (on the host):
    docker commit CONTAINER_ID apolloauto/apollo:NEW_DOCKER_IMAGE_TAG
    where CONTAINER_ID is the container ID you found before, and NEW_DOCKER_IMAGE_TAG is the name you choose for your Apollo GPU image.

Build Apollo in your new Docker image

  1. To get into your new docker image, use the following:
    ./docker/scripts/dev_start.sh -l -t NEW_DOCKER_IMAGE_TAG
    ./docker/scripts/dev_into.sh
  2. Now you should be able to build the GPU version of Apollo:
    ./apollo.sh clean
    ./apollo.sh build_gpu

Run Apollo!

  1. From within the docker image, start Apollo:
    scripts/bootstrap.sh
  2. Check that Dreamview is running at http://localhost:8888.
  3. Set up in Dreamview by selecting the setup mode, vehicle, and map in the top right. For the sample data rosbag, select “Standard”, “Mkz8” and “Sunnyvale Big Loop”.
  4. Start the rosbag in the docker container with rosbag play path/to/rosbag.bag.
  5. Once you see the vehicle moving in Dreamview, pause the rosbag with the space bar.
  6. Wait a few seconds for the perception, prediction and traffic light modules to load.
  7. Resume playing the rosbag with the spacebar.

Once the rosbag playing is complete, to play it again you have to first shutdown with scripts/bootstrap.sh stop and then repeat the above from step 1 (otherwise the time discrepancy stops the modules from working).

Building Apollo outside of Docker

WARNING – this will probably mess up your system.
At some point I had an unrelated crash so I restarted my computer and now it is stuck in a boot loop. So don’t attempt this until I have updated with a safe version!

I will probably try this again in a VM, assuming graphics drivers will work. Apollo is extremely fussy about library versions so be warned!

Current status: Bricked my OS, but at last attempt built until hitting some ROS dependency issues, and some linker problems probably caused by Apollo wanting some archaic glibc version..

You must not have Boost or PCL installed before attempting this, as Apollo requires specific versions. You’ll also need to make sure you have the correct version of GCC (4.8) and probably glibc (TODO).

sudo apt-get install gcc-4.8 g++-4.8 g++-4.8-multilib gcc-4.8-multilib
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 50
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50

Now you need to build Boost.

wget http://downloads.sourceforge.net/project/boost/boost/1.54.0/boost_1_54_0.tar.gz
tar -zxvf boost_1_54_0.tar.gz
cd boost_1_54_0

You will also need to modify the Boost source code to get the threading library to compile with the old GCC version – see here:
Then you can build boost as normal.

./bootstrap.sh --prefix=/usr/local
sudo ./b2 --with=all -j 4

Now get some dependencies:

sudo apt-get install -y build-essential apt-utils curl debconf-utils doxygen lcov libcurl4-openssl-dev libfreetype6-dev lsof python-pip python-matplotlib python-scipy python-software-properties realpath software-properties-common unzip vim nano wget zip cppcheck libgtest-dev git bc apt-transport-https shellcheck cmake gdb psmisc python-empy librosconsole0d librosconsole-dev libtf-conversions0d
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get install -y oracle-java8-installer

Install Bazel (again, very fussy with which version):

wget https://github.com/bazelbuild/bazel/releases/download/0.5.3/bazel_0.5.3-linux-x86_64.deb
dpkg -i bazel_0.5.3-linux-x86_64.deb
sudo apt-get install -f

At this point I did some cleanup.

sudo apt-get clean autoclean
sudo apt-get autoremove -y
sudo rm -fr /var/lib/apt/lists/*

You have to build a specific version of protobuf. Get yourself a coffee ready:

wget https://github.com/google/protobuf/releases/download/v3.3.0/protobuf-cpp-3.3.0.tar.gz
tar xzf protobuf-cpp-3.3.0.tar.gz
cd protobuf-3.3.0/
./configure --prefix=/usr
make
sudo make install
sudo chmod 755 /usr/bin/protoc

Moving on to install node JS (again, a specific version required here):

wget https://github.com/tj/n/archive/v2.1.0.tar.gz
tar xzf v2.1.0.tar.gz
cd n-2.1.0/
sudo make install
sudo n 8.0.0

Make up an empty file called py27_requirements.txt and insert the following:

glog
protobuf == 3.1
python-gflags
flask
flask-socketio
gevent
requests >= 2.18
simplejson
PyYAML
catkin_pkg
rospkg

Now install the python corresponding bits and pieces for Python:

pip install -r py27_requirements.txt

I think the following might have to be done earlier, and might also get the wrong version of ROS given current build errors. Investigation needed (TODO):

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install -y yarn

sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116

And some more pesky libraries:

sudo apt-get install -y libbz2-dev libconsole-bridge-dev liblog4cxx10-dev libeigen3-dev liblz4-dev libpoco-dev libproj-dev libtinyxml-dev libyaml-cpp-dev sip-dev uuid-dev zlib1g-dev

sudo apt-get install -y libatlas-base-dev libflann-dev libhdf5-serial-dev libicu-dev libleveldb-dev liblmdb-dev libopencv-dev libopenni-dev libqhull-dev libsnappy-dev libvtk5-dev libvtk5-qt4-dev mpi-default-dev

sudo apt-get update && apt-get install -y --force-yes libglfw3 libglfw3-dev freeglut3-dev

You have to build glew to get the right version:

wget https://github.com/nigels-com/glew/releases/download/glew-2.0.0/glew-2.0.0.zip
unzip glew-2.0.0.zip
cd glew-2.0.0/
make
sudo make install

Same kind of thing for PCL:

wget https://github.com/PointCloudLibrary/pcl/archive/pcl-1.7.2.tar.gz
tar xzf pcl-1.7.2.tar.gz
cd pcl-pcl-1.7.2 && mkdir build && cd build
cmake ..
make
sudo make install

And the big bad CUDA (you need CUDA 8.0):

sudo apt-get install -y libgflags-dev libgoogle-glog-dev
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
sudo dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64-deb
sudo apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub
sudo apt-get update
sudo apt-get install -y cuda-8-0

Next up is Caffe:

wget https://github.com/BVLC/caffe/archive/rc5.zip
unzip rc5.zip
cd caffe-rcg/
cp Makefile.config.example Makefile.config

In the file Makefile.config you have to uncomment the line WITH_PYTHON_LAYER := 1 before continuing.

sudo ln -s libhdf5_serial.so.10.1.0 libhdf5.so
sudo ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so
cd python
for req in $(cat requirements.txt); do pip install --user $req; done
cd..
make all
make test
make runtest
make distribute

Now you can get to the actual Apollo part.

wget https://github.com/ApolloAuto/apollo/archive/master.zip
unzip master.zip
cd apollo-master
./apollo.sh build-gpu #will fail
source bazel-apollo/external/ros/setup.bash
./apollo.sh build-gpu

You might need to grab the docker image before doing the Apollo build – I will test if it works without. I think this is because it grabs a special, Apollo-specific version of ROS. You can see why they recommend you keep within the Docker!