This page will contain materials and progress reports related to my diploma thesis. The task is to enable robot navigation on a checkerboard floor of the Eurobot contest and to detect specific colored objects (skittles) with an omnidirectional visual sensor.

The thesis was defeneded and marked as excellent by the oponnent and the entire commitee.

The master thesis is finished and published online! You can either read the individual chapters or download the entire thesis.

Started creating a summary of all the known Vision-Based Localization - Relevant Papers. Each paper has own wikipage with abstract, short annotation and linked citations.

Implemented a method of position tracking using the preprocessed boundary data for motion estimation and MCL for verification and correction of the current pose estimation. More can be found in the Tracking the Absolute Position of a Mobile Robot Using Vision-Based Monte Carlo Localization for the DARH2005.

Here you can find some experiments:

• Ride along the long field side (1.7MB)
• Ride along the long field side (with MCL correction items, 5.1MB)
• Ride, turn and slide at the start corner (2.2MB)
• A long ride half-way around the field (3.4MB)
• An example of relocalization after prediction failure (3.2MB)
• An example of relocalization after prediction failure (with MCL correction items, 11.8MB)

Definitevely sick and tired of the camera’s unpredictible white balancing, I decided to try prepending color normalization in front of the filter chain.

The first filter now processes the captured frame to reach uniform distribution of all 3 channels. RGB values are recomputed so that each channel has mean (average value) 65 and standard deviation (square root of divergence) equal to 50. With these values, the RGB cube optimally fills the YCrCb color space:

Thanks to the normalization, all the tracked colors are always present in the thresholded image (even in the camera startup phase ). This enables to track the White playing table borders (which used to be very fugitive).

Due to different montage of the camera, whole playing field is visible all the time and even the Checkerboard Pattern in Eurobot 2005 boundaries are easier to track:

The current algorithm detects the orientation and shift of the Checkerboard Pattern in Eurobot 2005 boundaries performing the following filter chain:

The input image is thresholded, browN-beigE boundary is detected

Then it is transformed to ground plane where it gets summed in 128 different directions

and the peaks in the sum image denote orientation and shift of straight lines in the transformed image:

The vision testing application has been rewritten to support capturing into several so called filter chains, each of which consists of several image filters that perform some operation on the image going from the camera through the chain. The image below shows output of the new BoundaryFilter, which extracts browN / beigE boundary information from the thresholded image.

The next step is to sum these boundaries in different directions to get the most probable orientation and shift of the Checkerboard Pattern in Eurobot 2005 pattern.

Finished the definitive montage of the new camera to the Beacon Support with Omnivisual Sensor.

The montage allows fine-tuning of the camera position and orientation resulting in an evidently more precise shape of the projected surface.

Applied the pixelwise transformation on the camera image to get a sparse matrix of transformed camera pixels. It is optimistic, that the computed transformation seems to be correct On the other hand, proper camera montage is necessary to see the Checkerboard Pattern in Eurobot 2005 pattern as actual squares.

Also tried to make a pretty dense image using a naive interpolation algorithm (sparse pixels are interpolated on horizontal lines only). It looks bad because in the top and bottom parts, the pixels are very sparse… It would be necessary to find the nearest pixels in the euclidean metric.

Future work:

• Definitive montage of the new camera to the correct position on the Beacon Support with Omnivisual Sensor.
• Better dense transformation for visualization.

Performed a few computations to get the formulas necessary to describe the reflection of a ray corresponding to an image pixel.

I finally assumed that the center of camera projection (first principial point) is located in the second focus F (not contained in the mirror) of the mirror surface. This results in that the prolongations of all the reflected rays intersect in the fist focal point E.

The intersection of a ray passing from the camera chip towards the mirror and the mirror surface lays at the coordinates [t, F(t)] where F is the mirror surface function and t can be computed as:

where

and

Or alternatively (when fixing )

Todays meeting brought some specification to the corner detection algorithm (checkerboard processing).

• Using browN/beigE boundary pixels to locate edges and corners.
• Transformation of such pixels into plane using a predicted camera location.
• If location is well predicted, boundary pixels make rows and columns → SUM them.
• Fix rotation error either using:
  1. repetition of the same transform using different alpha,
  2. estimated alpha from the location of static beacons.

Yet another possible approach:

• Incorporate segmentated image into MCL treating the segmented image as a kind of “virtual floor color detectors”.

More notes:

1. You almost never get a whole square segment (either cut or melt with neighbor).
2. Use information of the white border - to stop processing squares?
3. Add another segmentation color (split White to “pure White”/graY).

Today I just created pieces of code to better understand the yesterday’s work

The code performs all the 6 thresholding operations at once (as described earlier) and shows the image in the six representing colors. The fact that the different cones in YUV space overlap is not a big problem because all the 6 classes are maintained, i.e. when searching for browN and beigE, it doesn’t matter that some of the beigE is classified as White too.

From the right image it is obvious how the YUV space is oriented in the RGB one. The bigger window contains a UV plane for a fixed Y while the upper one contains result of the segmenation process applied on the UV plane.

The colors will definitely be played with but I think it can be dismissed until the final cameras arrive because I fear that the colors will be a bit different.

Todays testing was based on the CMVision idea:

1. The image is transformed to the YUV (Brightness, Chromacity) color space where object color can be described as a cone with a wide span in the brightness dimension.
2. The cone is described as an intersection of three subspaces described by three binary tables.
3. If 32 such binary tables are stored in different bits of one int table, one can track at most 32 different colors at a time using just 2 binary and operators on every pixel.

The test code (color_segmentation.c) allows to calibrate the area in the YUV space interactively. Left mouse button adds the selected pixel to the set while right mouse button removes all the three associated coordinates. The visually selected region is then manually modified to form a cone. The selected areas will be described as (y, u, v, y_span, u_span, v_span) which allows to automatically adjust the span tolerances in reaction to ambient conditions.

Results:

• Blue, Red, beigE and browN – perfect sharp recognition (although beigE and browN result in Red when blurred together)
• Green and White – poor recognition (Green is very dark - dark Blue and blacK are often misclassified as Green; the chromacity of White stripe is different for Green and Red skittles and the resulting cone is very big)

Algorithm:

1. Mask out the camera, robot and other static stuff.
2. Color segmentation – all colors at once.
3. Detect and process floor segments (Blue, beigE, browN).
4. Crop the skittle segments far beyond table.
5. Process individual skittle segments (Red, Green) and find an adjacent stripe (White) if any.

Future work:

• Extract color information to enable auto-calibration.
• Color thresholding to segments (maybe using CMVision).

Today I tested the OpenCV cvPyrSegmentation function. This function joins color thresholding with descending an image pyramid to create initially interconnected segments. Then it creates larger connected components from the initial ones using a second threshold.

Features:

• Suppors both 1-channel and 3-channel images (involves the color perception weights into the distance function).
• Returns a list of all the interconnected components – FPS dramatically decreases for a small threshold and thus large amount of components.
• Resulting segment color as well as the size of the interconnected component varies a lot from frame to frame – segment color classification will be difficult.

Requires:

• ROI mask to speed up the process throwing away uninteresting pixels.
• Good illumination, fine-tuned thresholds (maybe individual for different tasks – skittles/table)
• Merging (or at least preserving) information accross frames.
• Better color classification for joining segments.

I tried to improve the results using cvCvtColor. The idea is that if the game elements are strongly differentiated in color, I can throw away the brightness and saturation information and process only the hue channel with much higher frame-rate.

In fact, the speed up is the only positive of this approach. Frame rate climbed to 4fps (from around 2.5fps for a 3-channel source) which still is not really impressive anyway.

Although the red skittles are pretty distinctive, the bad news is that the green skittle color (dark green-yellow) is very close to the floor color (beigE and brown become yellow and orange-yellow respectively). It is almost impossible to get any results from adjusting the thresholds.

Future work:

• Detailed study of OpenCV, LTI-lib and CMVision.
• Segment color classification.
• Try joining CMVision with OpenCV.
• Auto-callibrated ROI mask.

Meeting with Zbynek Winkler.
Created the outline of the localization alghoritm:

1. Apply a ROI mask to the omnidirectional image.
2. Perform the color-based segmentation:
   • Pyramid segmentation,
   • CMVision.
3. Recalculate the segment coordinates using a predefined lookup table.
4. The centroids of floor segments now make a rectangular grid …
5. … match the grid with a known map (using a predicted viewpoint).
6. Update the lookup table to minimize the grid-matching error.
7. Update the current viewpoint.

Based on the e-mail discussion transcripted below the skittle detection algorithm evolved to the following steps:

1. Perform color-based segmentation
2. A skittles consists of 2 or 3 individual segments:
   • Red/green body and white stripe (standing upside-down or laying with screws towards the camera)
   • Red/green head and body separated by a white stripe (standing upright or laying with head towards the camera)
3. Identify the corresponding segments (need a heuristic to join neighboring segments - consider direction of principial body axis)
4. Decide if the skittle is standing (consider direction from the camera - both panoramic and omnidirectional):
   • 2 segments - white stripe first, then the body (upside down)
   • 3 segments - larger (rectangular) body first, then white stripe, finally smaller (elliptical) head (on screws)

• 05-02-03 omni:Virtualni CNYcka (Jan Benda)
  Idea of straightforward cooperation of MCL with omnivision
• 04-12-01 omni:Re, Detekce kuzelek (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=40934&schema_id=1&q=flusser&s=1002829098|[cache]]] Outline of solution for upside-down skittles
• 04-11-29 Fwd: Detekce kuzelek (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=42037&schema_id=1&q=flusser&s=2361943936|[cache]]] Draft of skittle detection using segment moments
• 04-11-26 [No Subject] (Jan Flusser)
  [[http://127.0.0.1:4664/cache?event_id=40937&schema_id=1&q=flusser&s=1306126485|[cache]]] Outline of elipsoid detection (principial axis & moments)
• 04-11-16 Kamery (Jan Benda)
  Reasoning about possible camera systems
• 04-11-14 WiFi kamerka (Jan Benda)
  [[http://127.0.0.1:4664/getthread?doc_id=41137&flags=8&q=kamerka&s=3194096057|[cache]]] Stereo vision from playing field corners?
• 04-11-10 Cc: RE: Prevodovka (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=42245&schema_id=1&q=kamerka&s=1586290799|[cache]]] Omnidirectional vision vs. horizontal scanning for skittles
• 04-11-10 Kamerky na majacich (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=42241&schema_id=1&q=flusser&s=887280946|[cache]]] Idea of stereo from beacons in the corners of the table
• 04-11-10 Re: Vsesmerove videni (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=40961&schema_id=1&q=flusser&s=3849928923|[cache]]] Omnivision notes from ing. Pajdla
• 04-11-07 Eurobot - obrazky hriste (Jan Benda)
  Test images of a catadioptric sensor with a spherical mirror
• 04-11-05 Omnivision? (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=42281&schema_id=1&q=kamerka&s=1988944312|[cache]]] Second meeting - omnivision idea appeared
• 04-10-22 Re: Roboticka diplomka (Zbynek Winkler)
  [[http://127.0.0.1:4664/cache?event_id=42319&schema_id=1&q=kamerka&s=1049091706|[cache]]] References to similar works
• 04-10-21 Roboticka diplomka (Jan Benda)
  [[http://127.0.0.1:4664/cache?event_id=42321&schema_id=1&q=kamerka&s=354288122|[cache]]] First meeting - idea of rangefinder-enhanced stereo

• OpenCV (home) — Camera calibration, capture, color conversions etc. (C, Windows / Linux).
• CMVision (home) — Realtime color segmentation (C++, Linux).
• LTI-lib (home) — Corner detection etc. (C++, Windows / Linux).

• J. Sladek - Diploma Thesis
• Software for robotics