Open Extrapolation-API for tracking and object detection modules

With the new Extrapolation-API it is now possible to provide object-specific tracking or detection for use with Label Editor extrapolation feature. The extrapolation feature is used for automatically adding already defined geometrical objects into the next frame on frame change. Up to now only a simple copy or linear extrapolation was available. Now any algorithm can be used to provide the new object marker with high accuracy. And as the interface is open, customers can use their own proprietary algorithms.

Philosys adapts external software to Extrapolation-API on customer request.

Standard Tracker for Extrapolation

Under separate license Philosys makes available a tracker, that can be used when no own algorithm can be used. It works fine on most closed objects like rectangle and closed polygons. Dependent on the required accuracy and tracked object, no or minimal corrections are necessary. It can be optimized by a few parametes defined in the Structure-XML file.

On customer request Philosys also develops more sophisticated tracker moduls tailored to specific object types.

Combination of Extrapolation and Interpolation

This new feature speeds up annotation of objects with heavy movements considerably. With regular extrapolation you have to move from one image to the next, and the selected object is automatically placed in the next image. You have go through every single frame.

With interpolation, available since Version 2, the object has to be marked at first appearence and at last appearance. The range in between is interpolated. Then you have to iterate through the range and correct the interpolated object until the required accuracy has been achieved. For object movements that are mostly linear the number of required keyframes is low. This has the additional advantage that much less data has to be managed. But in Version 2 still all frames are stored for compatability reason.

With Version 3 now both annotation methods can be combined. The annotator now marks the object when it first appears, just like with interpolation. But in the next step, instead of looking for the frame where the object disappears, he steps by using a hotkey a specified number of frames forward with extrapolation. At the new position there will be automatically a keyframe added with the selected object delivered by the extrapolation algorithm. With good tracking most of the time no or only small corrections are necessary. But as specified number of frames are skipped, it is possible to step much faster through the scene. If the object disappears during a step, this step can be undone and the last frame of the interpolation region can be set.

Of course, finally there should be a control pass through the scene to check for interpolation failures, and to correct them as necessary. The number of frames to step can be configured. With little high frequency movement, it is even possible to jump 10 frames. This can be optimized during the first steps.

So a lot of manual stepping can be saved during annotation. With a step width of 3 the number of steps is cut to 1/3.

Extrapolation for multiple objects concurrently

When marking multiple objects in a scene with extrapolation, normally the scene is scanned for every object separately. This means that for 10 objects at least 10 passes covering the lifetime of the corresponding objects have to be made. This feature now allows to extrapolate multiple objects concurrently. The objects to extrapolate have to be marked for AutoAdd and then with a hotkey, the currently selected and all marked objects are extrapolated. This is very useful with the combination of extrapolation and interpolation, described above and the use of a tracker.

But one have to very careful not to be overwhelmed by to many objects. The experience up to now shows, that very high concentration is necessary to follow multiple objects concurrently without missing any necessary correction. The annotator has to find his sweet spot and especially in this case it is very important to do a careful control pass at the end.

Interpolation of data without geometrical objects

Up to now, interpolation of attribute values only worked when the timedependent subtree also contained a geometrical object. Therefor subtrees without gemetrical object had to be represendet by data in every frame, even there was no change in value.

This restriction could be overcome by defining a dummy geometrical object, that is just used to set a start frame in the first frame and a end frame in the last frame. But this is not very convenient.

Now such a subtree can be marked for interpolation and on creation of a new Label XML a start and end frame for interpolation is created automatically, and both are protect from deletion. Everything else ist just like with regular interpolation. A change of an attribute causes an automatic creation of a keyframe to hold the new value.

This new feature accelerates the handling of such kind of date considerably. And together with the "StoreMode" feature, described next, storing and loading of data is much faster, because of much less data to handle.

Store only keyframes in label data

As with very complex and long scenes a lot of data has to be stored and loaded from the Label XML file, it is now possible to store keyframes only. The users of the annotation files have then to do the interpolation by themself.

The advantages are faster store/load and much smaller annotation files.

Definition of colors, tool tips

Now it is possible to define colors for every enum value used for the InspectorView timeline. A corresponding geometrical object can also show the color of a specific attribute value.

For every attribute it is possible do define a explanatory tooltip shown when moving the mouse on top of the attribute.

Both make the Label Editor more intuitive and easier to use.

Miscellaneous Changes

Beside the addition of above new features, a lot of smaller changes have been made:

• Performance improvements ( InspectorView, Label XML handling, extrapolation, ...).
• Object IDs in front of path names in InspectorView.
• Scene can be defined with any name.
• Toolbar elements are now movable.
• Messages for errors in structure definition.
• Messages for erros on loading DAT files.

Pending Changes

The following additional features are planned for the final release:

• Handling of multiplexed data streams.
• Support of splines with polygons,

If you have special requirements, please contact us immediately.

The final release of Version 3 is planned for Q4/2013.

The Philosys Label Editor is used during development and test of diverse Advanced Driver Assistance Systems (ADAS) for ground-truth-data collection. Various objects are marked and tagged with detailed traits. All kinds of vehicles, lane boundaries, traffic signs as well as pedestrians and wildlife are registered and verified by the assistance system for use.

The ADTF based Philosys Label Editor is available as of the beginning of 2011. Its many features facilitate the annotation of video scenes. This results in a significant cost reduction for the annotation process.