HandTracker.java - sample program (java)

Source file: Click the following link to view the source code file:

• HandTracker.java\HandTracker.java

This section describes an OpenNI sample program for tracking a hand. This sample is encapsulated in the org.OpenNI.Samples.HandTracker.jar (java archive).

The documentation describes the sample program's code from the top of the program file(s) to bottom.

Every OpenNI feature is described the first time it appears in this sample program. Further appearances of the same feature are not described again.

HandTracker Class Title

The whole program is defined inside the HandTracker Class.

Handler for 'Gesture Recognized' Event

The 'Gesture Recognized' event signals that the GestureGenerator node has recognized the named gesture in the scene.

The following code block defines the event handler for the 'Gesture Recognized' event.

            class MyGestureRecognized implements IObserver<GestureRecognizedEventArgs>
            {
                ...
            }

The following is the main code in the body of the event handler.

The event handler calls the StartTracking() method of the HandsGenerator node. This method causes the HandsGenerator node to start tracking at the specific position where the application expects a hand, which the application supplies as a parameter of the org.OpenNI.Point3D type. The parameter is getEndPosition(), which is the position of the hand at the end of the gesture.

                handsGen.StartTracking(args.getEndPosition());
                gestureGen.removeGesture("Click");

The getEndPosition() method gets the pEndPosition parameter from, the event handler, which is the position of the hand at the end of the gesture.

Now that the HandsGenerator is tracking a hand, the GestureGenerator does not have to look for the gesture anymore. So it does this by calling the RemoveGesture() method to disable the GestureGenerator from looking for the named gesture in the FOV.

                gestureGen.removeGesture("Click");

Handler for 'Hand Create' Event

Some implementations may want to use the 'Hand Create' event to signal that the HandsGenerator node has recognized and started tracking a new hand in response to the application calling the xn::() method. This event returns the ID of the new hand, the time of its recognition, and its position on recognition in the current frame.

The following code block defines the event handler for the 'Hand Create' event.

            class MyHandCreateEvent implements IObserver<ActiveHandEventArgs>
            {
                ...
            }

The following is the main code in the body of the event handler.

The event handler calls a new list of org.OpenNI.Point3D type structs. The getPosition() method returns the position at which the hand was created.

                ArrayList<Point3D> newList = new ArrayList<Point3D>();
                newList.add(args.getPosition());

Now that the HandsGenerator is tracking the hand, it adds it to a history list created by this application. The history list is described later in declaration block.

                history.put(new Integer(args.getId()), newList);

Handler for 'Hand Update' Event

The 'Hand Update' event signals that the HandsGenerator node signals that a currently tracked hand was recognized at a specific position in the new frame. OpenNI continues to send this event at each further frame that the hand is still present. This event returns the ID of the hand, which is the same ID returned by the HandCreate event, the hand's new position, and the time of the update.

The following code block defines the event handler for the 'Hand Update' event.

            class MyHandUpdateEvent implements IObserver<ActiveHandEventArgs>
            {
                ...
            }

The following is the main code in the body of the event handler. The assignment in the first statement adds the new position of the hand to its own history buffer. The 'history' is a hash from an integer (the hand's unique, persistent ID) and a list of its last locations.

When adding a new location of the hand, we want to add them to the correct list.

            ArrayList<Point3D> historyList = history.get(args.getId());
            
            historyList.add(args.getPosition());
            
            while (historyList.size() > historySize)
            {
                historyList.remove(0);
            }

The above while loop provides a maximum history size, giving a 'moving tail' effect. This means that if there are too many points in the buffer, the oldest one is removed.

Handler for 'Hand Destroy' Event

The 'Hand Destroy' event signals that an existing hand has disappeared from the frame for any reason. This event returns the user ID – still the same user ID as before – and the time that it disappeared.

The following code block defines the event handler for the 'Hand Destroy' event.

            MyHandDestroyEvent implements IObserver<InactiveHandEventArgs>
            {
                ...
            }

The following code is the main code in the body of the event handler.

Now that the hand has been destroyed, in this example, the application wants to remove the user's ID from the history list. So the program gets the user's ID from the handler argument args.getId.

The code block below first removes the specific user's history buffer by calling history.remove(), and then we check if the general history is left empty, which means all hands were removed. The code then calls history.isEmpty() to check if the history listof hands has now become empty as a result of this hand having been destroyed. If it is empty, the program will then want to again start looking for the "Click" gesture. Thus, it then calls the AddGesture() method.

            history.remove(args.getId());
            if (history.isEmpty())
            {
                try
                {
                    gestureGen.addGesture("Click");
                } catch (StatusException e)
                {
                    e.printStackTrace();
                }
            }

Main Program Declaration Block: OpenNI Nodes

The declaration block immediately above the main program (try) declares the OpenNI objects required for building the OpenNI production graph. The production graph is the main object model in OpenNI.

            private static final long serialVersionUID = 1L;
            private OutArg<ScriptNode> scriptNode;
            private Context context;
            private DepthGenerator depthGen;
            private GestureGenerator gestureGen;
            private HandsGenerator handsGen;

Each of these concepts is described separately in the following paragraphs.

The production graph is a network of software objects - called production nodes - that can identify blobs as hands or human users. In this sample program the production graph identifies blobs as human users, and tracks them as they move.

the ScriptNode object loads an XML script from a file or string, and then runs the XML script to build a production graph.

a Context object is a workspace in which the application builds an OpenNI production graph.

a DepthGenerator node generates a depth map. Each map pixel value represents a distance from the sensor.

A GestureGenerator node recognizes certain hand gestures and raise corresponding events accordingly. A gesture is a specific hand movement. The GestureGenerator node scans the FOV to detect gestures and generates the gesture data.

A HandsGenerator node generates tracking data about a single hand or multiple hands with persistent IDs.

Main Program Declaration Block: Application Histogram

The following declaration block defines data structures for the application histogram. These are not OpenNI-specific declarations.

            private HashMap<Integer, ArrayList<Point3D>> history;
            private byte[] imgbytes;
            private float histogram[];

            private BufferedImage bimg;
            int width, height;

history is a history list of org.OpenNI.Point3D objects, which are the 3D (x,y,z) co-ordinates of each depth pixel in the depth map generated by a DepthGenerator node.

histogram is an array of floats to build a histogram of distribution of depth values. It will be used to represent different depth values with different colors..

The following declaration for SAMPLE_XML_FILE is the path to an OpenNI XML script file for inputting and building a stored production graph. private final String SAMPLE_XML_FILE = "../../../../Data/SamplesConfig.xml";

HandTracker() method: Setting up the Production Graph

The public HandTracker() routine sets up the Production Graph.

The following code initialization initializes the production graph from an OpenNI XML script file The call to Context.createFromXmlFile() combines the effects of two other initialization methods – xn::Context::Init() and then xn::Context::RunXmlScriptFromFile() – to initialize the context object and then create a production graph.

            scriptNode = new OutArg<ScriptNode>();
            context = Context.createFromXmlFile(SAMPLE_XML_FILE, scriptNode);

The following statements create and set up a GestureGenerator node. The addGesture() method activates the GestureGenerator node to start looking for the named gesture ("Click") in the FOV by adding the gesture's name to the node's list of gestures it actively scans for in the FOV.

            gestureGen = GestureGenerator.create(context);
            gestureGen.addGesture("Click"); {Not exactly the same as in the API}

The following statement sets up an event handler for the 'Gesture Recognized' event. The handler is the MyGestureRecognized() function declared elsewhere in the program sample.

            gestureGen.getGestureRecognizedEvent().addObserver(new MyGestureRecognized() ); 

The following code block sets up event handlers for a number of hand recognition events. For details about these events, see Hand Event Sequence Diagram, and 'Hand Create', 'Hand Update', and 'Hand Destroy'.

            handsGen = HandsGenerator.create(context);
            handsGen.getHandCreateEvent().addObserver(new MyHandCreateEvent());
            handsGen.getHandUpdateEvent().addObserver(new MyHandUpdateEvent());
            handsGen.getHandDestroyEvent().addObserver(new MyHandDestroyEvent());

The following statement creates a DepthGenerator.

            depthGen = DepthGenerator.create(context);

In the following statement, the latest data generated is copied to an easy-to-access <i>frame object</i>. In OpenNI terminology: the node has data that is designated as 'metadata to be placed in the node's metadata object'. The node's getMetaData() method gets this data and copies it to a metadata object, depthMD. The data includes all configuration information associated with the data itself. This metadata object is then termed the <i>frame object</i>.
For more explanation on this, see Frame Objects and Metadata Objects, Frame Objects, and Frame Data (Data Frame).

            DepthMetaData depthMD = depthGen.getMetaData();

The following statement ensures all created generator nodes are generating data.

            context.startGeneratingAll();

The following sets up the histogram (see Main Program Declaration Block: Application Histogram). See getFullXRes and getFullYRes get the number of rows and columns in the full frame (that is, the entire field-of-view), ignoring cropping. This enables the program set up the right size buffer.

            history = new HashMap<Integer, ArrayList<Point3D>>(); 

            histogram = new float[10000];
            width = depthMD.getFullXRes();
            height = depthMD.getFullYRes();

            imgbytes = new byte[width*height];

            DataBufferByte dataBuffer = new DataBufferByte(imgbytes, width*height);
            Raster raster = Raster.createPackedRaster(dataBuffer, width, height, 8, null);
            bimg = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);
            bimg.setData(raster);

CalcHist() - Using the Depth Values to Build an Accumulative Histogram

There are no OpenNI-specific declarations in this routine.

updateDepth() method: Updating the Depth Map

The following statement sets up the frame object. It is described in the HandTracker() method above. For more explanation on this, see Frame Objects and Metadata Objects, Frame Objects, and Frame Data (Data Frame).

            DepthMetaData depthMD = depthGen.getMetaData();

The following method call waits for any node to have generated new data. This method then 'updates' each and every node in the entire production graph. For an overview to reading data and the WaitXUpdateAll methods, see 'Wait and Update' Methods.

The waitAnyUpdateAll() method in the following statement waits for any node to have generated a new data frame. The method then makes the data frames of all nodes in the entire production graph available for getting. The application can then get the data (for example, using a metadata GetData() method). This method has a timeout.

            context.waitAnyUpdateAll();

The following code block creates a convenient buffer for the depth map and then calls the calcHist() method to calculate the histogram. There are no OpenNI-specific operations in this code block.

            ShortBuffer depth = depthMD.getData().createShortBuffer();
            calcHist(depth);
            depth.rewind();     

This code block copies the depth into a local buffer, so that it can be drawn on the screen.

            while(depth.remaining() > 0)
            {
                int pos = depth.position();
                short pixel = depth.get();
                imgbytes[pos] = (byte)histogram[pixel];
            }       

getPreferredSize() method

There are no OpenNI-specific operations in this routine.

paint() method

This function draws the current location of each known hand, with a tail consisting of its previous points. It uses the history list populated in the HandGenerator callbacks.