Please note that the /examples directory provides useful examples that are a good starting point to understand how to write an application using TLIB.
tlib-2.1-bbcp.zip pre-compiled for Borland C++ Builder 6.0
The current version is tlib-2.1. Take a look at the release notes to find out about features. Feel free to contact seb@tuyphon.com for contributions and/or requests.
After downloading the distribution from the download page, uncompress it to the location of your choice, et voil ! Under Windows, you may be prompted for missing third-party DLLs, depending on what is already installed on your system (see third-party run-time libraries requirements). The missing DLLs files must be placed in the same folder as your TLIB application, or in the system32 folder of your Windows installation (recommended).
Third-party run-time libraries requirements
- (optionally) SVS 4.1 stereo processing library from VidereDesign (website)
- naturally, any DLL/modules related to your particular hardware
Special instructions for Visual Studio Express 2005
TLIB is an image processing library written specifically for real-time object tracking. It is meant for people with basic C/C++ programming and image processing skills. TLIB aims at fulfilling several goals:
- Efficiency
While real-time computer vision-based tracking usually requires highly optimized code, writing optimized code is not a trivial task and can be quite time consuming. TLIB's intention is to provide a wrap-up to optimized low-level functions so that the programmer does not have to worry about it.
- Ease-of-use
Other libraries exist that offer a set of optimized image processing functions, in particular the Intel OpenCV library. However, and even though they are very powerful tools, these libraries make it hard to rapidly develop a simple code for people with basic programming skills. TLIB is designed to offer a high-level interface that makes abstraction of the low-level optimized code, making it very well suited for rapid image processing application development and student training.
- Portability
TLIB is written in ANSI C/C++, mostly independant of OS specific libraries. Only the image acquisition and display classes are OS dependant. Thus, TLIB can be ported to any OS with little effort. Current distribution supports MS Windows and Linux.
- Educational
TLIB offers high-level and overloaded routines, along with simple and straightforward data structures that make the code look compact and clean. It simplifies the dataflow design and the understanding of the software for the programmer, and is a good introduction to more complex image processing software.
- Use of multiple sources
TLIB is designed to work with various image acquisition hardware and image formats, in particular with color and stereoscopic images. The TLIB image formats support several standard and provide a wrapper to a stereo algorithm. It is also possible to design your own image format from multiple sources, along with algorithms dedicated to your custom format.
Thus, TLIB has been designed to make it easy for most programmers to include efficient computer vision code into their applications rapidly, in particular for applications that require vision-based detection and tracking.
This section covers some of the features and formats that TLIB supports.
TLIB uses a simple pixel array type that can be used for virtually any image format. Pixels are held in a unidimensional array, namely the pixel
member or the tlPixelArray class. Depending on the image format, an image will have a number of channels. Pixels from various channels are transparently recorded in the pixel
array. For instance, in the case of the RGB file format, pixel
will contain an array of pixels in the following sequence: RGBRGBRGBRGBRGB...
The pixel
array starts at the upper left corner of the image and save pixels line after line. The tl_format section describes the different image formats and color spaces that TLIB can handle.
The TLIB library is a set of optimized ANSI C functions that are wrapped up in a set of higher-level C++ objects. The benefits of this dual-level architecture are a significant reduction of the amount and of the redundancy of code written by the programmer.
This section describes the different high-level objects that TLIB is made of.
- tlPoint objects
Simple point coordinates are stored in tlPoint objects. tlPoint contains both 2D image coordonates, a disparity value, and 3D world coordonates. The tlPoint object also provides convenience routines to update and copy point coordinates. Details about the tlPoint object structure can be found in tlPoint.
- tlChain objects
A tlChain object is an arbitrary collection of tlPoints. tlChain objects are useful for storing extracted points, but also for continuous edges extraction and other related operations such as angle detections. Details about the tlChain object structure can be found in tlChain.
- tlRect objects
One of the most commonly used objects in TLIB is the tlRect object. The tlRect object contains simple rectangular data (position, size) and routines to duplicate, move and resize it. It is used mostly to define a ROI (Region Of Interest, i.e. to limit the image processing to a given portion of an image), leading to shorter execution times. Details about the tlRect object structure can be found in tlRect.
- tlColor objects
Color values of different color formats (described here) are stored in tlColor objects. They are used as arguments to color filtering and color extraction routines. Details about the tlColor object structure can be found in tlColor.
- tlConst objects
Tracking routines often relies on some "magic" constants that vary over time according to external conditions, or that simply cannot be determined anatically. The tlConst object is a Kalman-filter based estimator of constants that can be dynamically updated. Details about the tlConst object structure can be found in tlConst.
- tlPixelArray objects
The base class for every object that contains a pixel array. tlPixelArray contains a pixel array with its dimension, format and properties. It also encapsulates all the pixel operator methods that can be applied to it. tlImage, toObject and tlMask inherit from tlPixelArray. Details about the tlPixelArray object structure can be found in tlPixelArray.
- tlImage objects
Perhaps the most commonly used object in TLIB is the tlImage object. It contains the image data (from its tlPixelArray parent), as well as useful routines such as drawing functions, file read/write, etc. Details about the tlImage object structure can be found in tlImage.
- tlObject objects
Almost equally important in TLIB is the tlObject object. A tlObject is a set of data (pixel array, dimensions, velocity, ...) that describes an object in an image, in a given image format. It contains the method necessary to retrieve the object position in subsequent images, as well as the routines that update that object properties from an image. Details about the tlObject object structure can be found in tlObject.
- tlMask objects
In a similar way, the tlMask object is a very simple structure that is used extensively in TLIB routines. It is a basic binary mask with position and size information that can be used to apply image processing routines to specific pixels only. A tlMask object is also used to define which pixels are part of an object in a given area. Mask objects can be combined, and support binary image processing routines such as morphological operators. Details about the tlMask object structure can be found in tlMask.
- tlBlobs objects
A basic task in image processing, known as "connected compounds" extraction, consists in managing and sorting blobs defined by a series of filters. The tlBlobs object is a collection of tlMask objects that does just that. Given a binary image, it segments, extracts and labels a set of tlMask, that can then be sorted with respect to different criterions. This is extremely useful for object extraction. Details about the tlBlobs object structure can be found in tlBlobs.
- tlHist objects
tlHist provides a histogram class compatible with TLIB images and objects. It offers simple functions such as ranking, normalization and scaling, building from images, comparison and data manipulation. Details about the tlHist object structure can be found in tlHist.
- tlHist2D objects
tlHist2D provides a 2D histogram class compatible with TLIB images and objects. It offers simple functions such as ranking, normalization and scaling, building from images, comparison and data manipulation. Details about the tlHist2D object structure can be found in tlHist2D.
- tlSpiral objects
A tlSpiral object provides a discrete spiral coordinates computation respectively to a given starting point. This allows a routine to address image pixels in a spiral sequence, which can be extremely useful for some correlation implementations or simply to find the closest filtered neighbor of a filtered pixel. Details about the tlSpiral object structure can be found in tlSpiral.
- tlHough objects
The tlHough class offers a quick an efficient way to extract geometric features from tlImage objects. Currently, only straight lines are supported. Details about the tlHough object structure can be found in tlHough.
The following objects are wrappers for hardware/platform dependant components such as image digitizers and displays. They are provided as a convenience for commonly used hardware and operating systems, and can be used as templates for the programmer to write his/her own hardware drivers. The list below gives the currently available implementations available in TLIB.
- tlDigitizer objects
Simple image acquisition should be performed through tlDigitizer objects. Details about the tlDigitizer object structure can be found in tlDigitgizer. Under Windows, TLIB currently supports the DirectShow generic interface grabber that can be used with most firewire and USB webcams. Dedicated implementations of tlDigitizer are also available for the ImageNation PXC200 and the Data Translation DT3130 framegrabbers. Under Linux, TLIB supports Video4Linux2 compatible devices.
- tlDisplay objects
A tlDisplay object offers a simple image rendering window, along with platform-independant mouse and keyboard events processing with respect to the windows. Details about the tlDisplay object structure can be found in tlDisplay. Under Windows, TLIB currently uses DirectX to display images. Under Linux, TLIB supports an XWindows implementation of tlDisplay.
- tlCamera objects
The tlCamera base class offers an encapsulation for image sources that are in a known location in the workspace, and possibly return calibrated (undistorted) images. An important child class is the tlCameraStereo base class, which provides a generic interface to stereo engines. Details about the tlCamera object structure can be found in tlCamera.
- tlCameraStereo objects
The tlCameraStereo base class offers an encapsulation for stereo image sources and stereo processing engines. Stereo cameras are defined here as "special" images that can acquire not only images but also disparity maps, registered to the source image. Details about the tlCameraStereo object structure can be found in tlCameraStereo.
C functions are the core of TLIB. They implement optimized operators on specific data types. Programmers should try to use objects structure, mostly because the objects method performs all the necessary data consistency checks and conversion before calling the low-level C function. However, experienced programmers that are familiar with TLIB data types and architecture may find it useful to work with C function to enhance performance or implement new object operators.
All low-level C functions have a tl_ prefix. They are grouped by categories, as described below.
- Timer functions
TLIB offers platform-independant timing routines with a worst resolution of approximately 1 ms. On Windows machine, TLIB can usually use the resolution of the processor hardware clock. Details about the tlTimer object structure can be found in tlTimer.h.
- Channel operators
The list of channel operators can be found in tlChannel.h.
- Convolution operators
The list of convolution operators can be found in tlConvolution.h.
- Edges operators
The list of edges operators can be found in tlEdges.h.
- Object extraction operators
The list of object extraction operators can be found in tlExtract.h.
- File operators
The list of file operators can be found in tlFile.h.
- Filter operators
The list of filter operators can be found in tlFilter.h.
- Find operators
The list of find operators can be found in tlFind.h.
- Format conversion operators
The list of format conversion operators can be found in tlFormatConversion.h.
- Memory operators
The list of memory operators can be found in tlMemory.h.
- Morphological operators
The list of morphological operators can be found in tlMorph.h.
- Pixel operators
The list of pixel operators can be found in tlPixel.h.
Here are the most important data types used in TLIB.
- tlPixel type
The tlPixel type is the typewith which all pixel arrays are built, whatever the image format is. This uniformity is motivated by optimization requirements. The programmer is encouraged to use the tlPixel type explicitely to guarantee compatibility with future releases, should the tlPixel typedef change in the future. Details on the tlPixel format can be found in tlVision.h.
A word is required here on code optimization. There are two kinds of code optimization techniques, described below.
- Hardware-independant optimization
Hardware-independant optimization include all optimization that do not take into account specific hardware properties such as the physical memory cache management algorithm and the memory alignment. It consists mostly of coding guidelines that help the compiler optimization routines to do a better job.
- Hardware-dependant optimization
Hardware-dependant optimization usually produces better results in terms of code execution. However, the code becomes highly machine specific and might run much slower (or not run at all) on another computer or another operating system.
For portability reasons, TLIB relies only on hardware-independant code optimization.
Below is a list of simple guidelines for programmers who use TLIB and who wish to make their code more efficient. Please note that, generally, optimization leads to less readable code that is more difficult to debug (quite often, optimizing code introduces bugs). The recommended approach is to write "standard" code to test algorithms before applying optimization rules.
Also, please note that these techniques are here to help the compiler do a better job only. Results will vary depending on the compiler, the optimization flags and the hardware.
- Loop unrolling
The easiest technique to optimize code consists in explicitely coding loops instead of using for or while statements.
Example
instead of int i;
for (i=0; i<4; i++) {
printf ("hello world\n");
}
prefer printf ("hello world\n");
printf ("hello world\n");
printf ("hello world\n");
printf ("hello world\n");
- Function overhead
Because of the way a compiler works, every time a function is called, some time is lost. This time can be signiticant if the function is called numerous time (in a loop, for instance). Also, the overhead of a C++ method is higher than a C function.
Example
instead of int i;
int val;
for (i=0; i<100; i++) {
for (j=0; j<100; j++) {
val = image->getPixel(i,j);
}
}
prefer int i;
int val;
int width = image->width;
for (j=0; j<100; j++) {
for (i=0; i<100; i++) {
val = image->pixel[j*width+i];
}
}
- Invariants
When the compiler comes accross an indirect reference, such as an array element or an object field, it will explicitely compute the memory address of the reference even if it has not changed. This can have an impact on the termination condition of loops.
Example
instead of int i,j;
int val;
int index = 0;
for (j=0; j<(image->height); j++) {
for (i=0; i<(image->width); i++) {
val = image->pixel[index++];
}
}
prefer int i,j;
int val;
int index = 0;
int height = image->height;
int width = image->width;
tlPixel *src = image->pixel;
for (j=0; j<height; j++) {
for (i=0; i<width; i++) {
val = src[index++];
}
}
- Function arguments
Everytime a program calls a function, the function arguments are copied in the system memory. Obviously, copying a structure containing 150 elements will take more resources than a single byte. Therefore, when working with structures, programmer should try and use pointers.
- Function inlining
For the same reason, namely to avoid the overhead of function calls, it is possible to tell the compiler to execute a function "in place" by using the keyword inlin when declaring the function. Inlining can be tricky and sometimes impair performances. It is best used with small functions that get called repeatedly from one single place in the code (singletons).
- Keywords
Some compilers (including Visual C++ compiler) will use the register keyword in front of a variable declaration to try and force the variable in a processor register. The effect will depend on the hardware (in particular on the size of the register on the machine processor). It has close to no effects on Pentium class processors.
Example
int j;
register int height = image->height;
register int width = image->width;
for (j=0; j<height; j++) {
do_something(j, width);
}
TLIB is, by definition, an ever-evolving library. As such, it is also likely to contain bugs and to display some design flaws. Questions and comments can be addressed to seb@tuyphon.com (please mention TLIB in the subject line).
- How to contribute
Anybody who uses TLIB is welcome to contribute, either by providing code that is compatible with the TLIB architecture and is relevant to real-time tracking applications, or by sharing cool demo source code that makes use of TLIB.
TLIB documentation - generated on 7 May 2008
Please address any questions to seb@tuyphon.com
(C) 2001-2008 - S. Grange
(C) 2001-2007 - VRAI Group, EPFL
All Rights Reserved.