Hit Testing

See Also: Class BaseObject, Class GraphicsWindow, Class Mesh, Class ViewExp, Class Interface, Class HitRecord, Class HitRegion, Class HitLog.

Hit testing is used throughout 3ds max as the user selects items in the scene using the mouse. Hit testing is the process of determining if a given mouse point intersects an item (node, modifier gizmo/center, or controller gizmo). A plug-in developer's responsibility regarding hit testing depends upon the type of plug-in and how much functionality is supplied by the base class the plug-in is derived from. For example, plug-ins that are sub-classed from SimpleObject, SimpleMod or StdControl are not required to implement methods for hit testing as it is all handled internally by the base class. Other plug-ins that don't subclass from these classes (for example the boolean object or the mapping modifier) must perform their own hit testing.

There are two types of hit testing:

Node level. This is determining if a given node in the scene has been hit.

Sub-object level. This can be for objects, modifier or controllers. This is for determining if a sub-component of an item has been hit. Object space modifiers can have a visual representation in the scene that the user can manipulate. For example, many of the 3ds max object space modifiers have a gizmo and center mark. These are the modifier's sub-object levels. In the case of an edit modifier this can be the sub-object parts of the object itself (like vertices, edges, or face for the Edit Mesh modifier). An object may hit test sub-object parts as well. For example, a compound object like the boolean object may hit test the operands of the boolean operation. The loft model compound object may hit test parts of the loft model such as the path or shapes.

There is also a distinction to be made between 'simple' hit testing and 'smart' hit testing. Simple hit testing involves finding any hit on an item. As soon as a single hit is found the process of searching for hits can stop. Smart hit testing involves finding the part of an item that was the closest to the pick point. This involves hit testing everything and then searching through all the hits to find the closest one.

If the plug-in procedural object, modifier or controller must perform its own hit testing, methods of the GraphicsWindow class make this fairly simple. There is a special rendering mode that items may be drawn or rendered in that performs the hit testing. In this mode (called GW_PICK), the item is not actually drawn, but is instead tested for intersection with the specified hit region. So if an item needs to be hit tested, a developer must simply draw the item using a rendering level GW_PICK and then check the result. The following sample code demonstrates controller gizmo hit testing. This method hit tests a series of 'footstep' shaped polylines that serves as the controller's gizmo.

int FootStepControl::HitTest(TimeValue t, INode* inode,

 int type, int crossing, int flags, IPoint2 *p, ViewExp *vpt) {

 int savedLimits, res = 0;

 GraphicsWindow *gw = vpt->getGW();

 Matrix3 ntm = inode->GetNodeTM(t);

 HitRegion hr;

 MakeHitRegion(hr,type,crossing,4,p);

 gw->setHitRegion(&hr);

 gw->setRndLimits(((savedLimits =

  gw->getRndLimits()) | GW_PICK) & ~GW_ILLUM);

 gw->clearHitCode();

 BOOL abortOnHit = flags&SUBHIT_ABORTONHIT?TRUE:FALSE;

 BOOL selOnly = flags&SUBHIT_SELONLY?TRUE:FALSE;

 BOOL unselOnly = flags&SUBHIT_UNSELONLY?TRUE:FALSE;

 for (int i=0; i<NUM_FOOTSTEPS; i++) {

  if (selOnly && !sel[i]) continue;

  if (unselOnly && sel[i]) continue;

  gw->setTransform(fs[i]*ntm);

  gw->polyline(FOOT_POINTS,footPts,NULL,NULL,TRUE,NULL);

  if (gw->checkHitCode()) {

   res = TRUE;

   vpt->CtrlLogHit(inode,gw->getHitDistance(),i,0);

   if (abortOnHit) {

    break;

    }

   gw->clearHitCode();

   }

  }

 gw->setRndLimits(savedLimits);

 return res;

 }

Note that the example above hit tests polylines. If you are hit testing polygons they must be drawn as 3 sided entities for 3ds max to properly hit test them. Polygons drawn with more than 3 sides will not hit test properly. For more information on hit testing using the methods of Graphics Window see the Advanced Topics section on The Interactive Renderer : GraphicsWindow.

Setting hitCode and hitDistance in R4.0

setHitCode() and setHitDistance() are new methods that make it possible to work with GraphicsWindow hit-testing in otherwise impossible situations. Why are they necessary? An example from MAX’s CORE.DLL is shown below.

Sample Code:

The patch object contains bezier spline-based edges which can consist of up to 102 vertices. Since the GraphicsWindow::polyline function can only plot lines with up to 32 vertices, it is impossible to plot these in a single call to the polyline function. Multiple calls to the polyline call do not return a proper hitcode when using a "window"-type hit region. By using the new setHitCode method, code can properly handle this situation. The code below shows the function in use from the PatchMesh::renderEdge method:

int steps = GetMeshSteps();

int segNum = steps+2;

float fsegNum = (float) (segNum-1);

// If steps are too high for GraphicsWindow's buffer,

// we must draw it manually

 if((steps + 2) > GW_MAX_VERTS) {

Point3 line[2];

Point3 prev,current(.0f,.0f,.0f);

BOOL hitAll = TRUE;

BOOL hitAny = FALSE;

DWORD hitDist = 0xffffffff;

  for(int terp = 0; terp < segNum; terp++) {

  prev = current;

   current = work.InterpCurve3D((float)terp / fsegNum);

   if (terp != 0) {

    line[0] = prev;

    line[1] = current;

    gw->clearHitCode();

    gw->polyline(2, line, NULL, NULL, 0, NULL);

    if(gw->checkHitCode()) {

hitAny = TRUE;

     if(gw->getHitDistance() < hitDist)

      hitDist = gw->getHitDistance();

}

else hitAll = FALSE;

   } 

  }

  if(hr && !hr->crossing && hr->type != POINT_RGN)

  gw->setHitCode(hitAll);

  else

   gw->setHitCode(hitAny);

  gw->setHitDistance(hitDist);

 }

else {

 for(int terp = 0; terp < segNum; terp++)

   fixedBuf[terp] = work.InterpCurve3D((float)terp / fsegNum);

   gw->polyline(steps+2, fixedBuf, NULL, NULL, 0, NULL);

 }

Note that the gw->polyline call is preceded by a call to clearHitCode, and followed by code which checks the hit code, maintaining "hitAny" and "hitAll" flags. When all the segments are drawn, the gw->setHitCode call is made, setting the hit code depending on the hit region type. When the code which called this function checks the GraphicsWindow’s hit code, it will contain the proper value. This code also keeps track of the closest hit distance and places that into the GraphicsWindow when all line segments are drawn.

Sub-Object Hit Testing

Sub-object hit testing is similar to object level hit testing except that the object is not simply determining if it was hit or not; the sub-object element that was hit needs to be determined as well.

What exactly a sub-object element is depends on the modifier. It may be a modifier's gizmo or part of its gizmo. For example, when an FFD modifier is in vertex sub-selection mode, the sub-object elements that are being hit tested are the control points of the lattice. Edit modifiers usually hit test components of the object in the pipeline such as vertices or faces.

When a modifier's HitTest() method is called, it traverses the sub-elements of the current sub-object selection level and checks each one to see if it has been hit. If so, it registers a hit record with the active viewport. A hit record contains the following information:

A pointer to the node that was hit. Modifiers may be instanced across multiple nodes, so the instance that was hit must be identified.

A pointer to the ModContext.

The 'distance' of the hit. To classify as a hit, the sub-object component must be within some threshold distance of the mouse. This distance is recorded in the hit record so that of all the hits below the threshold, the one that is the closest can be identified. What the distance actually represents depends on the rendering level. For wireframe modes, it refers to the distance in the screen XY plane from the mouse to the sub-object component. In a shaded mode, it refers to the Z depth of the sub-object component. In both cases, smaller distances indicate that the sub-object element is 'closer' to the mouse cursor.

A general unsigned long value. Most modifiers will just need this to identify the sub-object element. The Edit Mesh modifier uses the value to store the index of the vertex or face that was hit.

In case the 4 bytes above isn't enough space to identify the sub-object element, a pointer to a HitData class is included. To use this, a developer would define a class derived from this class that would contain the necessary data. The HitData class has one member function, a virtual destructor, so the derived class can be properly deleted when the HitRecord instance is deleted.

When HitTest() is called on an edit modifier, the edit modifier needs to hit test elements of the object that is flowing through the pipeline. This presents a problem since a modifier usually only has a pointer to this object when its ModifyObject() method is called. In order to be able to hit test the object, a modifier must keep the object cached. Caching the object the modifier is modifying is generally useful as well. For example, the Edit Mesh modifier keeps a cache of the mesh it is editing. When the Edit Mesh modifier is evaluated, if it has a cache it can simply return the cache. This way, as the user applies incremental edits to the mesh, all of the previous edits don't need to be reapplied. Instead, when an edit is made it is applied to the cache and stored away in some form in case the modifier needs to be reapplied.

Again, modifiers can be instanced, so storing an instance-specific cache in the modifier might not be the best idea. It is the ModContext, specifically in the localData field, where instance-specific modifier data belongs. This is where edit modifiers can put a cache of the object that it modifies. NOTE: typically this cache is only needed while the object is being edited. A hit test will never be called on a modifier unless it is being edited. In terms of evaluation, the pipeline already places caches where appropriate so there is normally no need for the modifier to maintain a cache.

Below is a summary of the hit test related methods from Class GraphicsWindow.

virtual void setHitRegion(HitRegion *rgn) = 0;

Sets the hit region used for hit testing.

virtual void clearHitCode() = 0;

This methods clears the hit code. Call this method before performing a hit test.

virtual BOOL checkHitCode() = 0;

Returns TRUE if a hit was made; otherwise FALSE.

virtual DWORD getHitDistance() = 0;

If checkHitCode() returns TRUE you may call this method to return the hit distance. In wireframe mode this is the distance to the line. In shaded mode, this is the z distance. This allows you to perform 'smart' hit testing by choosing the item with the smallest hit distance.

The system maintains a list of nodes for node level hit testing, a HitLog for modifier sub-object hits, and a CtrlHitLog for controller sub-object hits. The methods of Class ViewExp are used to work with these results. After hit testing has been performed these methods are available to examine the results.

For node level hit-testing

virtual void ClearHitList()=0;

Clears the list of hits.

virtual INode *GetClosestHit()=0;

Returns the INode pointer of the node that was the closest all those hit.

virtual int HitCount()=0;

Returns the number of hits recorded.

For modifier sub-object level hit-testing

virtual void LogHit(INode *nr, ModContext *mc, DWORD dist,

 ulong info, HitData *hitdat = NULL)=0;

This method records a sub-object level hit record with the system using the specified parameters. This is frequently called when a plug-in needs to select a node from the scene.

virtual HitLog& GetSubObjHitList()=0;

Returns the sub-object hit list.

virtual void ClearSubObjHitList()=0;

Clears the sub-object hit list.

virtual int NumSubObjHits()=0;

Returns the number of sub-object hits.

For controller gizmo hit testing

virtual void CtrlLogHit(INode *nr,DWORD dist,ulong info,

 DWORD infoExtra)=0;

This method records a sub-object level hit record with the system using the specified parameters.

virtual CtrlHitLog& GetCtrlHitList()=0;

Returns the sub-object hit list.

virtual void ClearCtrlHitList()=0;

Clears the sub-object hit list.

If a plug-in wants to perform hit testing of other nodes in the scene it may use the following methods of Class Interface.

virtual INode *PickNode(HWND hWnd, IPoint2 pt)=0;

This method hit tests the screen position for nodes and returns a INode pointer if one is hit, NULL otherwise.

virtual int SubObHitTest(TimeValue t, int type, int crossing,

 int flags, IPoint2 *p, ViewExp *vpt)=0;

This method performs a sub-object hit test. You may access the number of hits using: vpt->NumSubObjHits(). To return a list of the hits use vpt->GetCtrlHitList().

The following code demonstrates the use of this method:

BOOL GenControlSelectionProcessor::HitTest(

  ViewExp *vpt, IPoint2 *p, int type, int flags )

 {

 vpt->ClearCtrlHitList();

 ip->SubObHitTest(ip->GetTime(),type,ip->GetCrossing(),flags,p,vpt);

 if (vpt->GetCtrlHitList().First()) {

  return TRUE;

 } else {

  return FALSE;

  }

 }

Plug-ins that are not subclassed from SimpleObject SimpleMod or StdControl need to implement one of the methods shown below. These allows the system to hit test the plug-in item.

From Class BaseObject. This is the procedural object version:

virtual int HitTest(TimeValue t, INode* inode, int type,

 int crossing, int flags, IPoint2 *p, ViewExp *vpt);

The modifier version of this method has an extra argument (ModContext *mc).

virtual int HitTest(TimeValue t, INode* inode, int type,

 int crossing, int flags, IPoint2 *p, ViewExp *vpt, ModContext* mc)

From Class Control.

virtual int HitTest(TimeValue t, INode* inode, int type,

  int crossing, int flags, IPoint2 *p, ViewExp *vpt)

Finally, the following function is commonly used to initialize the HitRegion data structure.

void MakeHitRegion(HitRegion& hr, int type,

 int crossing, int epsi, IPoint2 *p);